Getting started

LLMGrade is an independent prototype exploring app-specific evals for LLM applications. The current wedge: catching a hallucinated policy change in a RAG support bot before merge.

Right now, on a single bundled refund-policy example, you can:

• Run an interactive, deterministic eval in the browser — edit the inputs and re-score.
• See the rubric and a baseline-vs-candidate score on every run.
• Watch a critical-failure rule hard-block an answer that invents policy.
• Reproduce the same result offline with a local CLI (init / run / diff / html).
• Generate a self-contained local HTML report from the CLI.

Scoring is deterministic and tuned to this refund-policy example. It is not a general semantic judge yet — see What's planned.

The local CLI is a prototype that runs inside the cloned repo — it isn't published to npm. Clone the repo and run npm install first; then npx llmgrade resolves the local CLI. It runs the bundled fixture — no network and no API key — and reproduces the same regression you see in the web demo.

Start with npx llmgrade init — run, diff, and html need the files it scaffolds.

❯ npx llmgrade init
  Scaffolds llmgrade/ eval files, rubric, and the example
❯ npx llmgrade run
  baseline overall 93% pass · current overall 38% fail
❯ npx llmgrade diff --no-exit-code
  Critical failure: invented 45-day medical refund exception
  Merge gate: blocked
❯ npx llmgrade html
  Wrote llmgrade/report.html

Same result every time. You can hand-edit the generated fixture files and re-run. Because the scorer is tuned to this example, answers from other domains won't score meaningfully yet, and custom file paths are not wired up.

An eval case for a RAG support bot has three parts:

• Retrieved context — the doc snippets your retriever returns for the question.
• User question — what the customer actually asks.
• Answer — what your bot produced, scored only against the retrieved context.

The judge grades the answer only against the retrieved context — never outside knowledge. If the docs don't cover the question, the correct behavior is to say so and route to support, not to guess.

Each answer is scored on five weighted dimensions. The weights sum to 100% and produce the overall score (shown out of 10 in the demo).

LLMGrade isn't just a rubric page — the bundled example is an eval pack: source-truth facts, question variations, response failure modes, deterministic scoring rules, a hard gate, example cases, and a regression report. The evaluator parses the source into policy facts, extracts behavior flags from the answer, and judges whether the answer preserves the policy decision — not whether it matches one fixed string.

The policy facts

• Return window: 30 days
• Start point: from delivery, not purchase
• Item condition: unused
• Product state: original condition
• Exceptions: none listed
• Eligibility: all listed conditions must be met (AND, not OR)

Forbidden behavior: inventing exceptions, extending the window, weakening conditions, or promising eligibility the source doesn't support.

Rubric axes & score anchors

Hard gate (critical failure)

Any of these blocks the answer regardless of the other axes:

• Extends the window (45/60 days, business days, grace or holiday window)
• Measures the window from purchase instead of delivery
• Allows used items or weakens 'original condition' to 'good condition'
• Uses 'or' instead of 'and' across required conditions
• Invents medical, manager, holiday, gift, loyalty, store-credit, exchange, or support-agent exceptions
• Treats 'no exceptions listed' as permission to create one
• Promises eligibility the source doesn't support, or is gibberish/irrelevant

Stress-test matrix

A deterministic test suite (60+ cases across these categories) runs with npm run smoke against the same shared engine that powers the web demo and the local CLI.

These are not built yet. The demo above does not depend on them:

• Custom eval cases and custom file paths (today it's locked to the bundled fixture).
• Broader domains beyond the refund-policy example.
• Model-backed (LLM) rubric judging for arbitrary domains.
• Repo scanning and generated eval suites from your own app.
• GitHub Action enforcement — a required CI merge gate.
• Hosted run history.

This is a standard Next.js App Router project. To deploy:

1. 1Push this repository to GitHub.
2. 2In Vercel, click “Add New… → Project” and import the repo.
3. 3Framework preset auto-detects as Next.js — no build config needed.
4. 4Click Deploy. The demo works immediately, with no key.

❯ npm install
❯ npm run dev
  ▲ Next.js — http://localhost:3000

The interactive demo always runs on the deterministic judge and never needs a key. Separately, the /api/judge route can call Claude when ANTHROPIC_API_KEY is set (it falls back to the mock if the call fails). Model-backed judging is not wired into the demo flow yet.