"It seems better" is not a metric. "It scored 87 on our golden set, up from 81" is a metric. Most LLM teams ship features on vibes. The successful ones ship on evals.
What an eval is, and isn't
An eval is an automated test for an LLM-powered behavior. It takes an input, runs your system, and produces a score. Like any test, it should be deterministic enough to run in CI, fast enough to run on every PR, and meaningful enough that a regression matters. It is not a one-off Notebook. It is not a screenshot of three good outputs.
The golden set
Start with 50 examples. Real ones, from your actual use case. For each: the input, the ideal output, and a rubric for what "correct" means. Grow it to 200 over the first quarter. Don't grow it past 1,000 in the first year — past that point, you're paying for it on every CI run, and the marginal example rarely buys you signal.
The single highest-leverage activity in your first month is sitting with a domain expert and writing the rubric. Not the examples — the rubric. "This answer is correct if it cites at least one applicable rule and does not contradict the source." That sentence is what you will spend a year refining.
Judges
Three kinds of judge:
- Deterministic. Schema match, citation presence, regex, numeric bounds. Cheap, reliable, your default.
- Embedding-based. Cosine similarity to the gold answer. Fast, useful for free-form text where exact match is too strict.
- LLM-as-judge. Another model evaluating the output against the rubric. Powerful, expensive, and itself subject to evaluation. Only use it where the first two can't reach.
CI integration
The point of an eval is to catch regressions before they ship. That means it runs in CI, on every PR. We block merge on more than a 2-point regression on the golden set. This sounds aggressive; it is aggressive. It is also the only way we have found to keep quality from drifting downward across hundreds of small changes.
Costs are real. A full eval at every PR adds dollars. We tier: a fast 50-example smoke set on every PR, the full 200 on merge to main, the long-tail set nightly. This keeps the median PR fast and the safety net wide.
Drift detection in production
The other side of the eval is production drift. Sample 0.5–1% of production traffic, score it with the same judges, and chart the score over time. When the score drops by more than two standard deviations across a day, you've shipped (or a model provider has silently shipped) something. We've caught vendor-side regressions this way three times in the past year.
What an eval cannot do
An eval cannot tell you whether your product solves the user's problem. It can only tell you whether your system is doing what you said it should. Connect your evals to a north-star metric — resolution rate, time-to-answer, user satisfaction — and review them together. A system that's getting better on evals while users are getting unhappier means your evals are measuring the wrong thing.