LLMGrade
Prototype · app-specific evals for LLM apps

Catch LLM app drift
before merge

An independent prototype exploring evals for LLM applications. The working demo scores a RAG support-bot answer against its source policy and a visible rubric — and blocks a hallucinated refund-policy change.

Try the demo
No API key required Deterministic & offline Rubric-based scoring
llmgrade — local demo
❯ npx llmgrade init
  Scaffolds the bundled refund-policy eval
❯ npx llmgrade run
  baseline overall 93% pass · current overall 38% fail
❯ npx llmgrade diff
  ✖ critical failure: invented 45-day medical exception
  Merge gate: blocked (exit 1)

Local CLI demo uses the same bundled example offline.

What's real today

This is an early prototype. Here is exactly what runs now and what is still in development.

Working today
  • Interactive, deterministic web demo
  • Bundled refund-policy eval pack (facts, axes, hard gate, 60+ test cases)
  • Visible rubric + baseline-vs-candidate scoring
  • Critical-failure rule that hard-blocks the bad change
  • Local CLI prototype (init / run / diff / html) on the bundled fixture, inside the repo
  • Local HTML report generated by the CLI
In development
  • • Custom eval cases and file paths
  • • Broader domains beyond the refund-policy example
  • • Model-backed (LLM) rubric judging
  • • Repo scanning and generated eval suites
  • • GitHub Action enforcement (CI merge gate)
  • • Hosted run history

The regression it catches

Same question, one prompt change. The answer got more confident — and invented a policy the docs never state.

Baseline Pass

The policy only says returns are accepted within 30 days. I don't see a documented medical exception. Please contact support for special cases.

Grounded · refuses cleanly

After prompt change Fail

Yes. If you were sick, you qualify for a medical exception and can return the item up to 45 days after delivery.

Invented exception · critical failure

Run this eval yourself