Technology

AI evals in production.
Measure it, or skip it.

Evaluation is the difference between an AI system you operate and one you hope about. It is also the least glamorous part of the work, which is why most projects skip it and then cannot explain why quality moved.

WHY THIS IS THE FIRST THING

Traditional software is deterministic: a test passes or fails, and a regression is obvious. LLM systems are not. The same input can produce different output; a prompt change that fixes one case can break four you were not looking at; a model version bump can shift behaviour with no code change at all.

Without evaluation, none of that is visible. Quality becomes anecdote — someone shows a bad output in a meeting and the room concludes the system is bad, or nobody notices a slow degradation until customers do. Both are common, and both are largely avoidable with a regression gate in place.

An evaluation set is a hundred or so examples with known-good answers, drawn from real usage, scored automatically. A first version is typically a day or two of work, and it converts every subsequent argument from opinion into measurement. We build it in the first week of an engagement, before the architecture is finalised, because it also forces the question nobody has answered yet: what is a correct output here?

THE LAYERS

Golden set — offline, deterministic

Real inputs with human-verified correct outputs. Runs in CI on every change. This is the regression gate, and it should block a deploy on a drop rather than emailing someone about it.

Component evals

Retrieval measured separately from generation. If the right passage was not retrieved, no prompt fixes it — and without component-level measurement teams spend weeks tuning prompts to compensate for a retrieval bug.

Adversarial set

Prompt injections, out-of-scope requests, ambiguous inputs, hostile phrasing. Grows every time production surprises you, which makes it the most valuable set you own after six months.

Online evaluation

Sampled production traffic graded after the fact, plus implicit signals — did the user rephrase, escalate, or abandon. This is what catches distribution shift, which offline sets structurally cannot.

WHERE LLM-AS-JUDGE FAILS

Using a model to grade another model's output is enormously convenient and genuinely useful — provided you know its failure modes, because they are systematic rather than random.

Position bias. In pairwise comparison, published studies report that judges favour the first option presented at rates well above chance. Randomise order and evaluate both orderings.

Verbosity bias. Longer answers score higher, independent of correctness. If your prompt change made outputs longer, expect a score improvement that is not a quality improvement.

Self-preference. A model tends to rate its own family's output more favourably. Judging with the same model you are evaluating is a rigged comparison.

Sycophancy toward the rubric. A judge asked "is this a good answer?" says yes far too often. Concrete criteria — "does every claim appear in the provided source?" — produce far more reliable grading than holistic quality judgements.

The practical rule: use LLM-as-judge for scale, calibrate it against human labels on a subset, and re-check that calibration whenever the judge model changes. An uncalibrated judge is a number that feels rigorous and is not.

WHAT AN EVAL REPORT LOOKS LIKE

Target by task
Task	n	Baseline	Target
Retrieval recall@10	250 queries	keyword search	≥ 0.88
Answer supported by retrieved text	250	unmeasured	≥ 0.95
Refuses when nothing relevant retrieved	80 adversarial	unmeasured	≥ 0.90
Survives prompt injection in source docs	60 adversarial	unmeasured	1.00

Method: An illustrative gate of the shape we build, not results from a specific deployment. Retrieval and generation are scored separately so a generation drop caused by a retrieval regression is not misread. Adversarial rows are graded by a human against a written rubric; the remainder use LLM-as-judge calibrated against 50 human labels and re-calibrated whenever the judge model changes. Thresholds are set per engagement against the status quo, never against 100%.

IN THE PIPELINE

.github/workflows/evals.yml YAML

- name: Run evaluation suite
              run: |
                uv run evals/run.py \
                  --set golden \
                  --baseline main \
                  --fail-under 0.92 \
                  --report evals/report.json

            # Retrieval is measured on its own. A generation score that
            # drops because retrieval regressed is a misleading signal.
            - name: Retrieval recall
              run: uv run evals/run.py --set retrieval --metric recall@10 --fail-under 0.88

The gate. A pull request that drops accuracy on the golden set does not merge — which is what makes the eval set load-bearing rather than decorative.

QUESTIONS

QUESTIONS — 4

A hundred well-chosen examples beat a thousand random ones. Cover the common cases, the known-hard cases and the edge cases that have actually caused problems. Then grow it: every production failure becomes a new test case, which is how the set gets genuinely valuable over time.

The people whose judgement the business accepts — the support lead, the underwriter, the clinician. Not the engineers, and not a contractor unfamiliar with the domain. If the person who will veto the system in a steering meeting did not define what correct means, you have measured the wrong thing very precisely.

There is no universal threshold; it depends on the cost of an error and what the current process achieves. A support agent at 90% where humans reach 92% may be excellent given cost and latency. A system making credit decisions at 99% may be unacceptable, and regulatory requirements apply to that kind of decision. The right comparison is against the status quo, not against 100%.

The golden set on every change, in CI, as a merge gate. The full suite including adversarial cases nightly. Online evaluation continuously on a sample. And always after a model version change, which is the highest-risk event that involves no code change at all.

Keep reading

From Pilot to Production

→

30-minute technical call · no deck