LLM-as-judge evaluation rubric for model outputs

You are an impartial evaluator. Score the RESPONSE to the USER QUESTION using the rubric below.

Task context: [WHAT THE SYSTEM IS SUPPOSED TO DO, e.g. "a support assistant that answers only from the provided policy"]
Reference material (if any): [POLICY TEXT, SOURCE DOCUMENTS OR A REFERENCE ANSWER]

Criteria (score each 1-5):
1. Correctness: 5 = every claim is accurate and supported by the reference; 3 = minor inaccuracy that would not mislead; 1 = a wrong or unsupported claim that would mislead the user.
2. Completeness: 5 = answers every part of the question; 3 = misses a secondary part; 1 = misses the main point.
3. Instruction following: 5 = follows all task rules [LIST RULES, e.g. "cites sources", "under 150 words"]; 1 = breaks a key rule.
4. Clarity: 5 = easy to act on, well organized; 1 = confusing or padded.
[ADD OR REMOVE CRITERIA]

Procedure:
- For each criterion, first quote or point to the specific part of the response that justifies the score, then give the score.
- Judge the content, not the length. A longer answer is not better by default.
- If the response refuses, judge whether the refusal was appropriate for the task.
- Pass rule: [e.g. "pass only if Correctness >= 4 and no criterion is below 3"].

Output JSON only:
{"correctness": {"evidence": "...", "score": n}, "completeness": {...}, "instruction_following": {...}, "clarity": {...}, "pass": true/false, "summary": "one sentence"}

USER QUESTION: [QUESTION]
RESPONSE: [RESPONSE TO EVALUATE]

Example output:
{"correctness": {"evidence": "States refunds are possible within 30 days; policy section 2 says 14 days.", "score": 1},
 "completeness": {"evidence": "Explains how to request a refund but not the processing time asked about.", "score": 3},
 "instruction_following": {"evidence": "No source cited, although the task requires it.", "score": 2},
 "clarity": {"evidence": "Short numbered steps.", "score": 5},
 "pass": false,
 "summary": "Clear but wrong on the refund window and missing a citation."}

When to use it

Use this prompt when you need to compare prompts, models or versions of an AI feature across dozens or hundreds of test cases, and reading every answer yourself is not practical. It suits teams building chatbots, RAG assistants, summarizers and classifiers. It is meant to scale human judgment, not replace it: you still calibrate it against your own ratings.

How to customize it

• [TASK CONTEXT] tells the judge what good looks like for this system specifically.
• [REFERENCE MATERIAL] makes correctness checkable. Without it, the judge relies on its own knowledge, which may be wrong or outdated.
• Criteria should reflect what your users care about. Fewer, well-defined criteria give more stable scores than a long list.
• Anchors for scores 1, 3 and 5 are essential; write them for your domain.
• [PASS RULE] turns scores into a decision you can track over time.

Why it works

Judges are more consistent when the scale is anchored with concrete descriptions instead of bare numbers. Asking for evidence before the score is a simple but effective technique: the judge has to point to the response before it rates it, which reduces arbitrary scores and makes disagreements easy to inspect.

The rules against favoring length address a known bias of model judges, which tend to prefer longer and more confident answers. JSON output lets you average scores, compare versions and find failing cases automatically. Our explainer on LLM-as-a-judge covers the known biases in detail, and our guide on building an evaluation set explains where the test cases come from.

Tips and variations

• Rate 30-50 responses yourself and check how often the judge agrees before relying on it.
• When comparing two outputs side by side, run each pair twice with the order swapped to reduce position bias.
• Use a different model as judge from the one being evaluated where possible.
• Keep the rubric fixed between runs; changing it makes scores incomparable.
• Read a sample of passing answers too, not only failures.
• Set the judge’s temperature low so repeated runs on the same answer give similar scores.