Changing a prompt, switching models or adding a retrieval step can make an AI application better on some inputs and worse on others. Without measurement, you’re guessing. An evaluation set — a fixed collection of test cases with clear success criteria — turns those guesses into numbers. It is the single most valuable asset in an AI project.

Step 1 — Define what “good” means

Before collecting examples, write down the qualities that matter for your application. For a support assistant, for example:

  • Correctness: factually right according to policy.
  • Groundedness: based on provided documents, not invented.
  • Completeness: answers the whole question.
  • Tone: polite and on-brand.
  • Format: follows the required structure.
  • Safety: refuses what it should refuse.

Each quality becomes a metric.

Step 2 — Collect realistic test cases

The best sources:

  1. Real user inputs from logs or pilots (anonymized).
  2. Known failures — every bug becomes a test case.
  3. Edge cases: empty inputs, very long inputs, ambiguous questions, other languages, attempts to break rules.
  4. Synthetic cases generated with a model to fill gaps — reviewed by a person.

Start with 30–50 cases covering the main categories, then grow over time. A small, well-chosen set beats a large random one.

Step 3 — Write expected outcomes

For each case, record what a good answer must contain or do:

  • an exact answer (for classification or extraction);
  • key facts that must appear;
  • facts that must not appear;
  • a reference answer for comparison;
  • whether the system should refuse or ask for clarification.

Step 4 — Choose scoring methods

Use the cheapest reliable method for each criterion:

  • Code-based checks: exact match, JSON validity, regex, length limits. Fast and objective.
  • Model-based grading: a strong model judges qualities such as relevance or tone using a rubric — see LLM-as-a-judge.
  • Human review: the gold standard for nuanced quality; use it on a sample and to calibrate automated graders.

Step 5 — Run it on every change

Automate the evaluation so it runs whenever you change prompts, models, retrieval settings or code. Track scores over time and per category. A change that raises the average while breaking an important category is not an improvement.

Step 6 — Keep it alive

  • Add new failures from production regularly.
  • Remove or fix cases that turn out to be wrong.
  • Keep a separate held-out set you rarely look at, to avoid tuning prompts to the test.

Observability completes the loop

In production, log inputs, outputs, retrieved documents, latency, cost and user feedback. Sample conversations for review, detect drift and feed new problem cases back into the evaluation set.

Common mistakes to avoid

  • Testing only happy paths. Real users send messy, ambiguous and adversarial inputs; include them.
  • Relying solely on an AI judge. Calibrate automated grading against human judgments on a sample.
  • Changing the test set and the system at the same time. Change one thing per experiment so you know what caused the difference.
  • Looking only at the average. Break results down by category; regressions hide in averages.

Maintaining your evaluation set

Treat the evaluation set like code: version it, review changes, and add new cases whenever users report a failure. Retire cases that no longer reflect real use. Rerun the full set before every prompt, model or retrieval change, and track scores over time to catch regressions early.

Related: LLM-as-a-judge explained and the quick take on benchmark scores.

Bottom line

A small, realistic, versioned evaluation set is the single most useful tool for improving an LLM application safely over time.

Key takeaways

  • An evaluation set is a fixed collection of test cases with clear success criteria.
  • Start with 30–50 realistic cases, including known failures and edge cases.
  • Combine code checks, model-based grading and human review.
  • Run evaluations automatically on every change and track results per category.
  • Feed production failures back into the set to keep it relevant.

Frequently asked questions

How many test cases do I need?

Start with 30–50 to catch major issues, and grow to hundreds as the application matures and you need finer distinctions.

Can I use public benchmarks instead?

Public benchmarks help choose a model in general, but only your own cases reflect your users, documents and requirements.

How often should I run evaluations?

On every meaningful change to prompts, models, retrieval or code, and periodically in production to detect drift.