The question every agent team eventually cannot avoid: how do we know it works?
Traditional software answers this with tests. A function has defined inputs and expected outputs; you assert the relationship and the build fails when it breaks. That contract is what makes continuous delivery possible.
Agents break the contract in three places at once. The output is non-deterministic, so the same input yields different valid responses. The output is often not comparable by equality, because “the summary is accurate” is not a string match. And the failure modes are frequently invisible in the output itself — an agent that reached the right answer by calling a destructive tool three times and getting lucky has failed in a way no output assertion detects.
Benchmarks do not solve this. They measure general capability on standardised tasks, which is a genuinely useful signal for model selection and a nearly worthless one for answering whether your agent handles your invoices correctly. Closing that gap is what evaluation engineering is for.
The four things worth measuring
Most teams start by measuring output quality and stop there. That is one of four layers, and usually not the one where production failures originate.
Task success. Did the agent achieve the outcome? This requires an outcome definition sharper than teams usually have — which is why success definition belongs in the design phase, not the evaluation phase. Binary success is preferable to a graded score wherever you can define it, because graded scores invite drift in the grading.
Trajectory quality. How did it get there? Which tools did it call, in what order, how many times, and were any calls unnecessary or destructive? An agent that solves the task via a wasteful or dangerous path is a latent incident. Trajectory evaluation catches the class of failure where the answer is right and the process is wrong — and this is the layer most teams skip entirely.
Cost and latency. Tokens consumed, tool calls made, wall-clock time. These are correctness properties, not performance nice-to-haves: an agent that produces excellent results at four times the modelled cost is not shippable, and one that takes ninety seconds where users expect five has failed regardless of accuracy. Treat regressions here exactly as you treat accuracy regressions. The economics are part of the contract.
Safety and boundary adherence. Did the agent respect its autonomy borders? Did it escalate when it should have? Did it refuse what it should refuse, and attempt only what it was scoped to attempt? This layer needs adversarial inputs, not representative ones, which makes it structurally different from the other three.
A useful heuristic: if your eval suite reports a single number, you are measuring one of these four and blind to three.
Golden datasets: the unglamorous foundation
Everything downstream depends on having a curated set of cases with known-good outcomes. There is no way around building one, and no shortcut that survives contact with production.
Size matters less than composition. Teams delay evaluation because they think they need thousands of examples. In practice, 100–200 well-chosen cases outperform 2,000 scraped ones, because the value is in coverage of failure modes rather than volume. Aim for roughly: 40% representative happy path, 30% known edge cases, 20% cases that previously failed in production, 10% adversarial.
That last two categories are what make the suite improve over time. Every production incident becomes a permanent test case. This single practice does more for agent reliability than any tooling choice, and it is the mechanism by which an eval suite stops being a snapshot of your initial assumptions.
Label the outcome, not the output. For most enterprise tasks, do not store the expected response text. Store the expected decision and the expected tool trajectory. “This invoice should be approved; the agent should call lookup_supplier then validate_totals then approve; it should not call create_supplier.” This is checkable deterministically, survives model upgrades that change phrasing, and captures the trajectory layer for free.
Version and freeze it. A golden dataset that quietly changes cannot support regression testing, because you can no longer distinguish an agent regression from a dataset change. Version it alongside the agent, and refresh on a deliberate schedule rather than continuously.
Watch for contamination. Cases drawn from public sources may be in the model’s training data, which inflates scores meaninglessly. Enterprise teams have an advantage here: your own historical transactions are both more representative and uncontaminated.
Where LLM judges work, and where they mislead
Using a model to grade another model’s output is now standard practice, and it is genuinely useful within limits that are frequently ignored.
LLM-as-judge works well for evaluating properties that are hard to express as assertions but easy to recognise: is this summary faithful to the source, is this tone appropriate, does this response answer the question asked. It scales to volumes that human review cannot, and it is consistent in ways human reviewers are not.
It fails in specific, documented ways. Judges exhibit position bias, favouring the first option in pairwise comparisons. They exhibit verbosity bias, rating longer responses higher independent of quality. They exhibit self-preference, rating outputs from their own model family more favourably — which matters enormously if you are using the same model to generate and grade. And they are unreliable on tasks requiring domain expertise the judge lacks, where they produce confident, plausible, wrong grades.
Practical mitigations, in order of value:
- Use a different model family for judging than for generating. This is the cheapest and most effective single control.
- Give the judge a rubric, not a question. “Rate 1–5” produces noise. “Return
faithful: true|falseand cite the source span supporting each claim” produces something checkable. - Randomise position in pairwise comparisons and average across both orders.
- Calibrate against human labels. Grade 50 cases by hand, compare to the judge, and measure agreement. If agreement is below roughly 80%, the judge is not yet usable for that dimension. Re-calibrate whenever you change judge models.
- Never use a judge for safety-critical determinations. Boundary adherence and destructive-action correctness should be checked deterministically. A judge is a quality instrument, not a control.
The general principle: use deterministic checks wherever the property admits one, and reserve judges for genuinely subjective properties. Teams reach for judges too early because judges are easy to add, and end up with a suite whose failures they cannot diagnose.
Regression suites and CI
The point of all this is to run continuously, because agent systems regress from causes traditional software does not have.
Your agent can break without a code change. The provider updates the model behind your API. A tool’s response schema shifts. A context file grows past the point of usefulness. Retrieved documents change underneath a RAG pipeline. None of these appear in your diff.
This changes what CI needs to look like:
Run a fast subset on every commit, the full suite nightly. Full agent evals are slow and expensive — they make real tool calls and consume real tokens. A 30-case smoke suite on commit and the full 200-case suite nightly is the pattern that survives contact with developer patience.
Run the full suite on a schedule even with no commits. This is the control that catches provider-side model drift. An eval run triggered by nothing but the calendar is the only thing that distinguishes “our code broke” from “the model changed.”
Pin model versions in production, and test the next version deliberately. Auto-upgrading to a new model version is the same risk class as auto-upgrading a database major version. When a new model appears, run the full suite against it as a candidate.
Set thresholds on all four layers, with distinct budgets. Fail the build on a task-success drop beyond your noise floor, on any new destructive-tool call in the trajectory, on cost regression beyond a percentage, and on any safety-case failure at all. The last one should not have a tolerance.
Account for non-determinism honestly. Run each case multiple times and report the distribution, not a single result. A case that passes 7 times out of 10 is a different engineering situation from one that passes 10 out of 10, and a suite that reports both as “pass” is hiding your real reliability. Establish your noise floor before setting thresholds, or you will spend months chasing phantom regressions.
Online evaluation: the part that actually predicts production
Offline evals on a golden dataset tell you whether the agent handles cases you anticipated. Production tells you about the ones you did not, and the distribution of real inputs drifts continuously.
Three mechanisms are worth the effort:
Sample and grade live traffic. Route a small percentage of production interactions through the same judges and checks as your offline suite. This is the only measurement that reflects the actual input distribution, and the divergence between offline and online scores is itself a valuable signal — a wide gap means your golden dataset has stopped being representative.
Instrument implicit feedback. Escalation rate, human override rate, retry rate, and abandonment are all available without asking anyone anything, and they move before your quality metrics do. A rising override rate is the earliest reliable warning that an agent is degrading. This is where evaluation and observability become the same practice rather than adjacent ones.
Shadow-run changes before shipping them. For a significant prompt, model, or tool change, run the new configuration against live traffic without acting on its output, and compare trajectories against the incumbent. This catches the changes that improve offline scores and degrade real behaviour.
What evaluation cannot give you
Evals measure what you thought to measure. They are structurally blind to novel failure modes, which means a clean suite is evidence of the absence of known problems and not evidence of correctness. This is why evaluation supports autonomy borders rather than replacing them — the borders exist precisely for the failures the suite does not anticipate.
Evals also cannot resolve a missing success definition. A team that cannot articulate what a good outcome looks like will build an eval suite that measures something adjacent and reassuring. The failure is upstream, in design.
And they are not free. A serious eval suite costs real tokens on every run, and the discipline to maintain it competes with feature work. Budget it as infrastructure, because the alternative is discovering regressions in production where they cost considerably more.
Key takeaways
Agent evaluation replaces the deterministic input-output contract that makes traditional testing work, and it requires measuring four distinct layers: task success, trajectory quality, cost and latency, and safety or boundary adherence — a suite reporting a single number is blind to three of them. The foundation is a versioned golden dataset of 100–200 cases weighted toward edge cases, past production failures, and adversarial inputs, labelled with expected decisions and tool trajectories rather than expected output text so it survives model upgrades. LLM judges scale subjective quality assessment but carry position, verbosity, and self-preference biases, so use a different model family than the one generating, give rubrics rather than rating scales, calibrate against human labels to at least 80% agreement, and never use a judge for safety-critical determinations. Because agents regress without code changes when providers update models or tools shift underneath them, run a fast subset per commit, the full suite nightly and on a bare schedule, pin production model versions, set separate thresholds per layer with zero tolerance on safety, and report distributions rather than single results to account for non-determinism. Finally, offline evals only cover anticipated cases: sampling and grading live traffic, instrumenting escalation and override rates, and shadow-running changes are what turn evaluation from a pre-launch gate into a system that predicts production.