Evals (LLM evaluation)

Conventional software returns the same output for the same input. Language models often answer the same question differently, and an answer can read fluently and still be wrong. Evals, short for evaluations, adapt testing to this situation. A set of test cases with expected results runs against the system, and defined criteria score the answers: by code, by people or by a second model. OpenAI describes evals as structured tests meant to ensure accuracy and reliability despite the nondeterministic nature of AI systems.

Evals are not benchmarks. A benchmark compares models on a public, standardized collection of tasks. A company eval tests your own task with your own cases: the terminology, formats, edge cases and quality standards of the business. A model can top the benchmarks and still fail at your task.

For executives, evals are how AI quality gets proven instead of asserted. They answer questions like: Is the new model at least as good as the old one for this task? Did the revised instruction make anything worse? Without evals, each of these decisions rests on the impressions of a few testers.

How evals work.

An eval has three parts. The test set contains inputs from the real task, each with the expected result or with criteria a good answer must meet. It is often called a golden set. The scoring criteria define what counts as a pass. The analysis rolls individual verdicts up into metrics, such as the share of correct answers or the number of answers containing made-up details.

Anthropic’s documentation names three design principles. An eval should mirror the real distribution of tasks and include edge cases. It should be gradable automatically where possible. And many test cases with a slightly weaker signal beat a few cases graded carefully by hand. Edge cases it lists include missing or irrelevant input, very long input and ambiguous cases where even humans would struggle to agree.

Success criteria have to be specific and measurable. “Good answers” is not a criterion. “The answer names the correct contract type and cites the relevant clause” is one. Most tasks need several criteria at once, such as factual accuracy, tone, handling of personal data, response time and cost.

Who grades: code, people or a model.

There are three ways to grade, and Anthropic’s documentation compares them. Code-based grading is the fastest and most reliable: Does the answer match the expected one exactly, does it contain a key phrase, is the format right? For tasks with a clear-cut answer, such as sorting into fixed categories, that is often enough. For judgments that require weighing things up, it is too rigid.

Human graders are the most flexible and deliver the highest quality, but they are slow and expensive. They need clear rubrics, or two experts will score the same answer differently.

The third way is a language model as grader, known as LLM-as-a-judge. It scores answers against a rubric, quickly and at scale. The documentation recommends detailed rubrics, a clear-cut verdict such as “correct” or “incorrect” or a score from 1 to 5, and letting the grader model reason before it scores. Above all, its reliability should be tested before it is used at scale. In practice the methods are combined: code for the clear-cut cases, a grader model for volume, people for spot checks and disputed cases.

LLM-as-a-judge has known weaknesses.

A model acting as grader is not neutral. In 2023, Lianmin Zheng and coauthors studied how well language models grade other models’ answers, using the 80 multi-turn questions of the MT-Bench test set and user votes from the Chatbot Arena platform. Strong models agreed with human preferences in over 80 percent of cases, about as often as humans agree with each other. The study also documents systematic biases: the answer shown first is favored, longer answers score better even when they are not better, and a model rates its own answers more favorably. Limited judgment on math and logic tasks comes on top.

That leads to rules. When comparing two answers, swap the order and count the verdict only if it holds both ways. Where possible, the grader should not be the same model as the one being tested. And its verdicts should be checked regularly against human verdicts on a sample. OpenAI’s documentation also recommends measuring a grader model’s agreement with human labels before trusting it.

Every change means running the test set again.

Evals are not a one-time test before launch. Language models change even when their name stays the same. In 2023, Lingjiao Chen, Matei Zaharia and James Zou compared the March and June 2023 versions of GPT-4 on the same tasks. The behavior of the same service, they found, can change substantially in a short time. On identifying prime numbers, GPT-4’s accuracy in their tests fell from 84 to 51 percent. Their conclusion: language models need continuous monitoring.

Then there are your own changes: a revised instruction, new documents in the knowledge base, a cheaper model. Each can break cases that used to work. A regression test catches this. The full test set runs against the old and the new version, and the switch happens only when the new one is at least on par on the criteria that matter.

OpenAI calls this continuous evaluation: evals run on every change, production cases where the system failed are added to the test set, and the set grows over time. The NIST AI Risk Management Framework states the principle more broadly: AI systems should be tested before deployment and regularly while in operation.

What evals cost and what drives it.

The biggest cost in evals is people’s time, not compute. The first test set takes the most effort: subject matter experts pick real cases, define the correct results and word the criteria so that two graders reach the same verdict. This work cannot be handed off to IT, because only the business side knows what a correct answer is.

Ongoing cost depends on three factors: the number of test cases, how often the evals run and the share of human grading. Code-based checks cost almost nothing. A grader model adds one more model call per graded answer. Human spot checks take staff time but are still needed to keep the grader model honest.

Frameworks and services exist for the technical side. In 2023, OpenAI released OpenAI Evals, an open-source framework with a collection of test tasks. Its hosted Evals platform in the developer dashboard, however, is being retired: according to OpenAI it has been read-only since October 31, 2026 and is scheduled to shut down on November 30, 2026 (as of November 2026). Test sets and criteria should therefore live in a format the company owns and can reuse with any tool.

How to recognize good evals.

Good evals share a few clear traits. The test cases come from real work and include hard and rare cases. For each criterion, what counts as a pass is written down. Results are documented per version, so anyone can trace which version was tested when and with what result.

For agents that carry out several steps on their own, looking at the final result is not enough. The path gets checked too: which tools the agent called, with which inputs, and whether it stopped at the right point. A correct result reached by a risky path is a finding.

Warning signs include a single overall score with no breakdown by criterion, a test set that has not grown since launch and a grader model whose verdicts were never checked against people. Another warning sign: the same test set is used both to tune the instruction and to approve it. If you keep adjusting the instruction until exactly these cases pass, you end up measuring the tuning, not the quality. That is why some cases are held back and used only for approval.

Where the term stops working

Evals measure only what is in the test set; for tasks and cases that are missing, a good result says nothing. They do not prove the absence of errors and replace neither the review of individual results nor security testing against deliberate attacks. They should be distinguished from benchmarks, the public comparison tests for models, from prompt engineering, whose effect they measure, and from analysis done with AI, such as sentiment analysis, where AI is the tool rather than the thing under test.