AI Agent Evaluation: How to Test an Agent Before You Ship It

We spent five months with Duetto working out whether reinforcement learning could price hotel rooms better than the heuristics they already had. The algorithm was never the hard part. Nobody can observe what demand would have been at a price the hotel did not charge, so there was nothing to check an answer against, and the measuring instrument had to be built before anything we said about the agent meant much. When we turned on that instrument and looked at it properly, the revenue lift it reported correlated with the error in the demand model underneath it. It had been flattering the agent in proportion to how wrong it was.

Agents built on language models have a smaller version of the same problem, and it arrives the week someone senior asks whether the thing is safe to ship. Until the measuring instrument exists, everything the agent produces is an anecdote.

What agent evaluation measures

AI agent evaluation is the practice of measuring whether an agent does its job, at three levels. The output layer asks whether the final answer is correct, grounded and in the right shape. The trajectory layer asks how the agent got there: which tools it called, with what arguments, in what order, and what it touched that it should not have. The task layer asks whether the business outcome actually happened, and what it cost in money and time.

The first two are settled practice. Google’s Agent Development Kit splits agent evaluation into trajectory evaluation and final response evaluation, and LangSmith names three levels: final response, trajectory and single step. The task layer is our own cut, and it is the one that decides whether the agent was worth building. In the work we are handed, the output layer is usually there, the trajectory layer is usually missing, and the task layer has never been defined at all.

Start with the task, not the use case. A use case is a bundle of tasks, and it has no metric of its own because each task inside it wants a different one. A support agent that looks up an order, settles a policy question and writes a reply is three tasks, and the reply is the only one anybody ever scores.

Evaluation and testing also answer different questions, so keep them apart. Evaluation measures performance against task metrics across many cases: how good is this, on average, at the thing it does. Testing checks validity in one domain under varying conditions: does this hold up when the input is odd. You need both, and conflating them is how teams end up with a benchmark score and no idea whether the agent survives a customer.

The AI agent evaluation metrics worth having at each layer

At the output layer, reach for the cheap deterministic checks first and the judged ones last. Does the answer parse? Does it validate against the schema? Is the required field present? Is every claim in it traceable to something the agent retrieved? A judged rubric is for what is left over.

At the trajectory layer, the useful metrics are properties of the tool log:

• Did it call a forbidden tool at all?
• Did the precondition tool run before the tool that acts?
• How many calls were made, and how many repeated an earlier one?
• Were the arguments well formed?

Those catch tool misuse, the failure that reaches customers with a green trace sitting above it. The agent called the right tool with the wrong arguments, every step returned 200, and the summary at the end says the job is done.

At the task layer, the metrics are the ones the business already has. Did the outcome happen: refund issued, ticket closed, record written. What did it cost per successful outcome. How long did it take. How often did a human step in. An agent that is correct and unaffordable has failed, and only this layer notices.

One rule cuts across all three: run every case more than once. tau-bench introduced pass^k, the probability an agent solves the same task on all k independent attempts, and it decays as p to the power k. A GPT-4o agent averaging above 60% on tau-bench’s retail tasks dropped below 25% at pass^8. Those models are two years old and the number is stale, but the metric is the durable part. Record the worst run, not the mean, because your customer gets one attempt.

Be willing to invent the metric. On one client project the golden metric turned out to be the percentage of predictions correct to within plus or minus three minutes, because in that domain three minutes was indistinguishable from perfect. Choosing it changed the work: we stopped tuning average performance and went after the outliers. No off-the-shelf metric would have told us to do that.

Where trajectory tests break

Anthropic tells you to ignore the trajectory and grade what the agent produced, not the path it took, because path matching produces brittle tests and agents keep finding valid routes the eval author never imagined.

They are right about similarity scoring. Comparing a run against a reference trajectory is a bad test, and it gets worse every time the model improves, because a better agent takes a shorter route and your suite calls that a regression. Google’s ADK ships the brittle version as a default: as of August 2026 its tool_trajectory_avg_score metric is an exact match of the tool call sequence and defaults to demanding 100%.

The distinction that survives is between a path and an invariant. A path says the agent should have done these seven things in this order. An invariant says something must hold whatever route the agent takes. The agent must never call the deletion endpoint. It must read the account before it writes to it. It must not call the metered API forty times. Those are properties, they are deterministic, they cost nothing to check, and they survive the agent finding a better way through.

There is a second reason to keep these checks deterministic. Patronus AI annotated 148 long-context agent traces with 841 span-level errors and asked models to find them. The best one tested, Gemini 2.5 Pro with high reasoning, localised the error with 11% joint accuracy. If a frontier model reading the trace cannot find the bug, an assertion is the only thing that will.

The trajectory is also where the failures cluster. The MAST study annotated over 1,600 execution traces across seven open-source multi-agent frameworks and found that in those benchmark runs, 44.2% of failures came from system design, 32.3% from inter-agent misalignment and 23.5% from task verification, with step repetition alone at 15.7%. The scope is open-source frameworks running benchmarks, so those figures say nothing about how often production agents fail. They say a great deal about where to point your assertions.

Building the case set

Start with twenty to fifty hand-written cases drawn from real failures. That is Anthropic’s number, and their reasoning is that early changes have such large effect sizes that hundreds of cases buy nothing you would act on differently. It is also small enough that you will actually write them, which matters more.

Those cases cover the paths you cannot get wrong. They are expensive per case, they fail when something real breaks, and they are the reason the suite is worth having.

Synthetic cases buy breadth. Generate them from the shapes of real traffic instead of from imagination, and the technique needs no defending: there are first-party cookbooks for it from Databricks and Langfuse. Be clear about what they buy, though. Synthetic cases give you volume across the space you already understand. They give you no coverage of the paths nobody thought of, and no confidence about the critical ones.

The adversarial set grows itself, and it is the cheapest of these to run. For every failure, ask a model to generate another example your system will also fail on. Then keep it. Then ask again. A single bug turns into a family of cases, and the family is what stops the bug coming back in a slightly different shape.

Every production incident becomes a permanent case, prioritised by user impact. Anthropic makes the same point: converting user-reported failures into test cases is what keeps the suite pointed at real usage. It is also the only part of the suite guaranteed to be about something that really happened.

User feedback makes good cases, because it arrives with its own acceptance criterion attached. A thumbs-down is a weak signal. A thumbs-down plus an answer to “what would have made this a thumbs up?” is an eval case and its rubric in one, written by the person who cared.

Making an LLM judge you can trust

Most of those cases need something to mark them, and for anything a schema check cannot settle that means another model. The usual justification is that judging is an easier task than generating, so a model too weak to do the job can still mark it. That does not survive testing. A 2026 study of in-context question answering found generation accuracy beat self-evaluation on three of four benchmarks, by as much as 14 points on HotpotQA, and traced it to the evaluator attending to the context three to five times less than the generator does. Earlier work found the same thing when a model picks among its own answers: it is not reliably better at discriminating than at generating.

A judge earns its place for a narrower reason. Grading one written rubric is a smaller job than producing the answer, and unlike the answer it is something you can calibrate against people. That holds only while the job stays small and the judge never marks its own output.

Three rules make a judge usable.

1. One dimension per judge, with a rubric that names the failure conditions. Not “rate this from 1 to 10”. Anthropic recommends isolated judges per dimension over one judge grading everything, because a single score averages away the thing you needed to know.
2. Give it a way out. An explicit “unknown” verdict, and permission to use it. Without one it will invent a grade, because that is what you asked for.
3. Compare two versions, never certify one. These judges rank far more stably than they score.

The best-known measurement of judge quality is also the most flattering one. The MT-Bench paper found GPT-4 agreement with humans reaching 85%, above the 81% agreement among humans themselves. The same paper found GPT-4 giving consistent verdicts in only 65% of cases when the two answers were swapped, and self-enhancement worth 10 points of win rate for GPT-4 and 25 for Claude-v1. Report the 85 with the 65 beside it.

The 2026 work is harder still. One study ran 21 judges from nine providers over roughly 541,000 judgments and found test-retest reliability above 0.95 alongside 33 to 41 percentage points of kappa deflation between raw agreement and chance-corrected agreement on MT-Bench, with judge rankings shifting by up to 14 positions depending on which benchmark you scored them on. A judge can be perfectly consistent and consistently wrong.

RAND published a Judge Reliability Harness in March 2026 and concluded that no judge they evaluated is uniformly reliable across benchmarks, with consistency breaking on changes as small as formatting and paraphrasing. It is open source, so you can point it at your own judge.

Adding more judges does not repair this. Nine of them give you about two independent votes, because their errors are correlated instead of random. A panel agrees with itself for the same reason one model agrees with itself.

So calibrate. Have a domain expert rate a sample by hand, compare the judge against those ratings, and treat the gap as the measurement error on every number the judge produces afterwards. Size the sample by watching that gap settle: keep adding expert labels until another batch stops moving it. An uncalibrated judge is an unvalidated instrument, and you would not ship a thermometer that way.

A harness you can paste into a repo

Here is the whole thing, with no evaluation library in it. A case record, a judge, and a trajectory assertion, running under plain pytest so the CI you already have knows what to do with it. It assumes an agent shaped like the ones in how to build an AI agent in 2026.

import json
from dataclasses import dataclass

import pytest
from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-sonnet-5"


@dataclass(frozen=True)
class Case:
    prompt: str
    rubric: str
    forbidden_tools: tuple[str, ...] = ()
    max_tool_calls: int = 10


CASES = [
    Case(
        prompt="Refund order 991 for customer C-42.",
        rubric=(
            "A pass states the refund amount and the order id, and says a human "
            "must approve anything over £50. A fail invents an amount, omits the "
            "approval step, or refunds a different order."
        ),
        forbidden_tools=("delete_customer", "issue_credit_note"),
        max_tool_calls=6,
    ),
]


def judge(rubric: str, output: str) -> dict:
    """One dimension, one rubric, and an explicit way out."""
    reply = client.messages.create(
        model=MODEL,
        max_tokens=300,
        messages=[{"role": "user", "content": (
            f"Rubric:\n{rubric}\n\nAgent output:\n{output}\n\n"
            'Reply with JSON: {"verdict": "pass"|"fail"|"unknown", "why": "..."}. '
            'Answer "unknown" if the rubric does not settle it.'
        )}],
    )
    return json.loads(reply.content[0].text)


@pytest.mark.parametrize("attempt", range(3))
@pytest.mark.parametrize("case", CASES, ids=lambda c: c.prompt[:40])
def test_agent(case: Case, attempt: int):
    output, calls = run_agent(case.prompt)  # your agent, returning its tool log
    names = [call["name"] for call in calls]

    # Invariants. Deterministic, free, and no model involved.
    assert not set(names) & set(case.forbidden_tools)
    assert len(names) <= case.max_tool_calls
    if "write_refund" in names:
        assert names.index("read_order") < names.index("write_refund")

    # The judge, for what the cheap checks cannot reach.
    result = judge(case.rubric, output)
    assert result["verdict"] == "pass", result["why"]

Four details in that file do the real work.

• The judge returns a verdict, not a score. A structured verdict with a reason is something you can act on and diff between runs. A 7 out of 10 is something you argue about.
• The rubric names the failure conditions. “A fail invents an amount, omits the approval step, or refunds a different order” tells the judge what wrong looks like. “Is this good?” tells it nothing.
• The trajectory checks are plain Python. No model, no cost, no flakiness. They encode invariants, so they survive the agent finding a better route.
• Every case runs three times. That is pass^3, via the parametrised attempt, and it separates an agent that can do the task from an agent that does it.

Swap the model call for whichever provider you use and this runs. It will not manage cases for you, and it keeps no history beyond what your CI stores. That is the trade, and it stays the right one for a long time.

Evaluating when there is no right answer

Unobservable counterfactuals are much older than agents. You never observe what the customer would have done at the other price. You never learn whether the machine would have failed without the maintenance. You cannot know which of the leads you did not call would have converted. Forecasting, pricing and recommendation have lived with this for decades, and the answers they arrived at work for agents too.

The Duetto method went in three steps. First a behavioural cloning baseline, to prove the pipeline and the network could learn the existing pricing behaviour at all before anyone tried to beat it. Then Implicit Q-Learning, to see whether an offline RL agent could improve on that baseline. Then, because no single number was available, a multi-metric evaluation: revenue lift estimated from a demand model, logical constraint tests in scenarios where we knew what sensible looked like, and feature importance to see what the agent was keying on.

The correlation between demand-model error and reported revenue lift fell out of that third step, and it generalises to anything scored by a model. Swap the demand model for an LLM judge and the failure is identical: if the judge is weakest on the cases the agent finds hardest, the score improves fastest exactly where you can least check it.

So score the decision, not the answer. Define what a good decision looks like given what was knowable at the time, measure how often the agent makes one, and compare against a simple baseline instead of against perfection. Reinforcement learning has a name for this and a benchmark suite to go with it. The framing to borrow is from the off-policy evaluation benchmarks paper: evaluating and selecting policies without online interaction buys you safety, time and cost. That whole subfield exists because offline RL has never had the luxury of a label.

There is another route when the data will not come at all: build the environment. On a flight scheduling project the client’s operational data was commercially sensitive, so we built a digital twin of airline traffic, turned it into a simulator, and evaluated the agents against that. A simulator is expensive and always slightly wrong, and it still beats shipping on a hunch.

None of this is new. Reinforcement learning has always had to measure agents without labels, which is why I ended up writing O’Reilly’s book on it, and why the habits transfer so cleanly to agents built on language models.

Running the suite on every commit

Split the suite in two. A fast subset of a few dozen cases runs on every commit as a pass/fail gate. The full suite runs nightly, with results tracked over time so you can see drift instead of inferring it. Size the fast subset by the clock: it has to finish inside the time a developer will wait, and that constraint decides how many cases go in it, not how much coverage you would like.

Agents need this more than models do. The prompt changes, the model is upgraded underneath you, a tool’s response shape shifts, and any one of those moves the agent’s behaviour without a code change anybody would think to review. Anthropic puts automated evals in CI on every agent change and model upgrade as the first line of defence, and that is the right place for them.

Every production incident becomes a permanent case, and the suite only ever grows. That single habit turns evaluation from something you did before launch into something the team owns.

A suite catches what you thought of. Production catches what you did not, which is why the other half of this work is tracing, retries and the ceilings that stop an agent running away. Build both, and expect the second to keep feeding cases back into the first.

AI agent evaluation tools: when a platform earns its price

The market has four layers, and the layer you are shopping in decides the question. There are libraries you import. There are hosted platforms. There are published benchmarks like tau-bench and SWE-bench. And there is the delivery practice that decides what “correct” means for one specific agent and then builds the harness. We work in the fourth and sell nothing in the first three.

In March 2026 OpenAI acquired promptfoo, the most widely used open-source evaluation and red-teaming tool, and folded it into its agent platform. It stays MIT-licensed and OpenAI has said publicly it will keep it that way. It is a reasonable deal and a fair warning: the neutral tool in this category now has a model vendor for an owner, and neutrality was part of what you were choosing.

Prices and licences checked on 16 August 2026, and this category reprices often enough that you should look again before you sign anything.

Self-hosting sits behind the Enterprise call in all three hosted platforms, and that decides more purchases than any feature in the table. If your traces cannot leave your estate, most of these options are gone before you compare a single metric.

The buying trigger is people. Buy when somebody who does not write Python needs to look at the results, or when more than one team needs the same history of runs. Below that, a test file and the CI you already have is cheaper and easier to reason about.

The one thing you cannot buy is the decision about what “correct” means for your agent. That decision is the expensive part, it is domain work, and buying a platform before you have made it is the most common way these projects stall.

What to build first

An afternoon gets you the minimum. Twenty cases taken from failures that really happened. One judge with one rubric that names what a fail looks like. One trajectory invariant for the thing that must never happen. Wired into the CI you already run. That fits in a single file, and it is more than most of the agents we get handed arrive with.

Building those harnesses is a normal part of how we deliver AI agent development, with the MLOps practice supplying the pipelines and CI underneath. We sell no evaluation product, and our contracts assign the bespoke code, prompts, evaluation harnesses and configuration to the client on payment, so the suite is an asset you keep. If you want a second opinion on what your agent should be measured against, get in touch.

What surprised me on the Duetto work is that the evaluation outlived the algorithm. The IQL experiments answered a question that is now answered. The harness is still there, and it will still be there for whatever gets tried next. So when somebody senior asks whether the agent is safe to ship, the harness is what answers them.