Run the eval
Edit the source, question, and two outputs, then run the eval. LLMGrade scores both against the rubric and tells you if the candidate regressed.
A deterministic prototype scored on the RAG support-bot refund-policy example — comparing one app's behavior before vs. after a change. It runs locally; no hosted dashboard and no model-backed judging yet. See what's coming next.
- Interactive refund-policy eval
- Visible rubric + hard gate
- Runs in the cloned repo
- Bundled fixture only
- •Custom eval files
- •GitHub Action
- •Hosted history
LLMGrade gives you a starter rubric. Every result shows the exact rubric used. See the full Refund Policy Eval Pack — facts, axes, hard gate, and stress-test matrix.
View rubric JSON
{
"id": "rag-support",
"dimensions": [
{
"key": "groundedness",
"label": "Groundedness",
"weight": 0.35,
"description": "Does the answer stay supported by the source truth?"
},
{
"key": "policy_accuracy",
"label": "Policy accuracy",
"weight": 0.25,
"description": "Does the answer preserve the actual refund-policy requirements?"
},
{
"key": "completeness",
"label": "Completeness",
"weight": 0.15,
"description": "Does it include the key conditions needed to answer the user?"
},
{
"key": "refusal_quality",
"label": "Refusal / uncertainty quality",
"weight": 0.15,
"description": "Does it handle missing or unsupported information correctly?"
},
{
"key": "format_tone",
"label": "User usefulness & clarity",
"weight": 0.1,
"description": "Does it directly help the user without adding unsupported claims?"
}
],
"passGate": {
"overall": 0.8,
"groundedness": 0.85,
"policy_accuracy": 0.9
},
"diffGate": {
"maxOverallDrop": 0.08,
"maxGroundednessDrop": 0.1
},
"criticalFailureRule": "An answer hard-fails if it changes the return window (or measures it from purchase, or switches to business days), weakens 'unused' or 'original condition', joins required conditions with 'or' instead of 'and', invents an exception or support discretion, treats 'no exceptions listed' as permission, promises eligibility the source does not support, or is gibberish/irrelevant."
}LLMGrade scores each output against the visible rubric, out of 10. A copied source policy can be fully grounded but still incomplete if it doesn't answer the user's question. Critical failures, like inventing a policy exception, can block the candidate even if some dimensions score well.
Invalid or unrelated text is blocked as a non-answer, even if it doesn't hallucinate a policy.
V1 uses deterministic scoring for the refund-policy demo. Model-backed rubric judging for custom domains is a future V1.1 exploration.
Coming next: developer workflow
Today, this page previews the shape of a reusable eval case. The next step is letting developers save custom cases, run them locally against their own files, and eventually enforce regressions in GitHub before merge.
This is the eval case shape the local CLI reads. Copy it to see the format the developer workflow is built around.
llmgrade/cases/my-eval-case.jsonllmgrade/rubrics/rag-support.jsonPreview eval case JSON
{
"schemaVersion": 1,
"id": "my-eval-case",
"name": "Refund policy — medical exception",
"question": "Can I return this after 45 days if I was sick?",
"context": "Returns are accepted within 30 days of delivery. The item must be unused and in original condition. No exceptions are listed in the policy.",
"critical": true,
"rubric": "rag-support",
"criticalFailureRule": "An answer hard-fails if it changes the return window (or measures it from purchase, or switches to business days), weakens 'unused' or 'original condition', joins required conditions with 'or' instead of 'and', invents an exception or support discretion, treats 'no exceptions listed' as permission, promises eligibility the source does not support, or is gibberish/irrelevant.",
"answers": {
"baseline": "The policy only supports returns within 30 days, and the item must be unused. No medical exception is listed, so I can't confirm one — please contact support if you need the standard exception process.",
"current": "Yes, you can return it after 45 days if you were sick. Medical exceptions are allowed."
},
"expected": {
"decision": "blocked",
"baselinePass": true,
"candidatePass": false
}
}
Today the CLI runs the bundled refund-policy fixture, and the deterministic scorer is tuned to that example. Custom case paths and other domains are planned. See run the demo locally.
An answer hard-fails if it changes the return window (or measures it from purchase, or switches to business days), weakens 'unused' or 'original condition', joins required conditions with 'or' instead of 'and', invents an exception or support discretion, treats 'no exceptions listed' as permission, promises eligibility the source does not support, or is gibberish/irrelevant.
Overall ≥ 80%, groundedness ≥ 85%, policy accuracy ≥ 90%, and no critical failures.
Fail if overall drops more than 8 pts, groundedness drops more than 10 pts, or a critical case regresses from pass to fail.