LLM evaluation, usually called evals, is the practice of testing a language model application against a fixed set of realistic tasks with clear success criteria, and scoring the results. Evals are to AI features what automated tests are to ordinary software: they show whether a change to a prompt, a model, a retrieval setting or a tool made the system better or worse. Without them, every release is a guess.
Evals test the whole application (the model plus its prompts, retrieval, tools and code), not only the model. Public benchmarks measure models in general; your evals measure whether your system does your job.
Key Parts of an Eval
Anthropic’s engineering guide on agent evals uses a vocabulary that has become common (Anthropic):
- Task: one test case with defined inputs and success criteria.
- Trial: one attempt at a task. Because outputs vary between runs, tasks are run several times.
- Grader: the logic that scores a result. A task can have several.
- Transcript (or trace): the full record of a trial, including tool calls and intermediate steps.
- Outcome: the final state, for example whether a booking really exists in the database, not only whether the agent said it does.
Three Kinds of Grader
| Grader | Good for | Weakness |
|---|---|---|
| Code-based (exact match, schema checks, unit tests, “did it call the right tool”) | Facts, formats, numbers, actions | Cannot judge tone or nuance |
| Model-based (“LLM as a judge” with a rubric) | Helpfulness, faithfulness to sources, style | Non-deterministic, must be checked against human judgement |
| Human review | The reference standard, and calibrating the other two | Slow and expensive |
A sound setup uses code wherever a check can be written as code, a model judge where it cannot, and people to read samples and confirm that the model judge agrees with them.
What to Measure
Common metrics depend on the application:
- Retrieval-augmented generation (RAG): whether the right passage was retrieved, whether the answer is faithful to it, and whether it answers the question.
- Agents: task success measured by outcome, number of steps, wrong or unsafe tool calls, cost and time per task.
- Extraction and classification: precision, recall and exact-match accuracy against labelled examples.
- Safety: refusal of out-of-scope requests, resistance to prompt injection, leakage of data the user should not see.
- Consistency: the same task passing on every trial rather than once out of several. The Anthropic guide notes that a 75% per-trial success rate gives only about 42% chance of passing three trials in a row, which matters for customer-facing agents.
Offline and Online Evaluation
- Offline evals run before release on a fixed dataset. They split into capability evals (hard tasks the system does not yet pass, giving a target) and regression evals (tasks it already passes, which should stay near 100%).
- Online evaluation scores samples of real production traffic, collects user feedback and flags failures. Those failures become new offline test cases. This loop is part of LLMOps.
Common Mistakes
- Starting too late. Teams tune prompts by eye for weeks, then cannot tell whether a model upgrade helped. OpenAI’s own guidance suggests considering writing evals before writing prompts (OpenAI model optimization), and the Anthropic guide suggests 20 to 50 tasks drawn from real failures as a starting point.
- Synthetic-only test sets. Questions a model invented are easier and cleaner than what real users ask.
- Trusting a model judge blindly. A judge model has its own biases, such as preferring longer answers. It needs calibration against human labels.
- Grading the words, not the outcome. An agent saying “refund issued” is not the same as a refund existing.
- Leaking the test set into prompts. Examples copied from the eval set into the prompt make scores meaningless.
- One aggregate score. A single number hides which kinds of question got worse.
Related Terms
Evals are how teams measure AI hallucination, check AI guardrails, decide whether fine-tuning is worth it, and compare models. They depend on tracing (AI observability) to see what happened inside a failed trial.
Evals in Practice
The first evals are written before the first prompt, from real questions and documents, including questions the system should refuse. Code checks cover every fact, number and action that can be checked mechanically, and people review real output before rollout for what code cannot judge; a model judge takes over part of the volume only after its grades have been compared with a person’s on the same answers. Every failure users report becomes a new test case, and experiments and production traces are tracked in LangSmith. See how this fits into LLMOps services.