Why AI Agents Fail in Production, and the Engineering That Stops It

The agent has been fine for six weeks. Then a customer complains, you open the trace, and every step is green. Nothing timed out, nothing threw an exception, and the summary at the end says the job is done. It is not done.

Agents fail in production in a small number of recognisable ways, and almost none of them are the model being wrong. They call the right tool with the wrong arguments. They run out of context partway through a long task and forget a constraint you gave them at the start. They report success after a step that failed. They loop, and you find out when the bill arrives. Or they answer confidently from data that stopped updating on Tuesday.

Every one of those has an engineering fix, and every fix is code inside your agent loop, not a product you buy.

How agents fail in production

Each row below is a failure mode, what it looks like when you go hunting for it, the thing you have to be recording to see it at all, and the engineering that stops it.

Five is not the complete taxonomy. The obvious omission is latency: timeouts and resource exhaustion are first-class categories in Patronus AI’s TRAIL taxonomy, and they sit outside this list because their fixes are the ordinary ones you already apply to any distributed system. These five are where agent-specific engineering earns its keep.

None of them is the model reasoning badly. They all happen with a frontier model and a well-written prompt, because they are properties of what surrounds the model: the loop, the tools, the context, the verification and the data.

What has actually been measured

The evidence is thinner than the confident numbers suggest. Four studies get quoted at this problem, and not one of them measures deployed agents carrying real traffic. What each actually counts:

• MAST (UC Berkeley and Intesa Sanpaolo, NeurIPS 2025). 1,642 annotated execution traces from seven open-source multi-agent frameworks, with the taxonomy built by hand from 150 traces averaging over 15,000 lines each and six expert annotators agreeing at 0.88. Fourteen failure modes in three groups. Its headline 41% to 86.7% failure rate covers those seven frameworks on benchmark tasks, so it gives you the shape of agent failure and nothing about your agent. Most of the percentages below come from here.
• tau-bench (Sierra, June 2024). Contributed pass^k, the probability an agent solves the same task on all k independent attempts. GPT-4o scored 61% at pass^1 on the retail domain and 25% at pass^8. A demo measures pass^1, the chance it works once while somebody is watching. Your customers measure pass^k.
• A reliability framework (April 2026). Ten open-source models over 23,392 episodes, across four task durations, finding that capability and reliability “diverge systematically as task duration increases” and that existing benchmarks are “structurally blind to this divergence”. A preprint from one lab, so hold it loosely.
• TRAIL (Patronus AI, 2025). 148 long agent traces from GAIA and SWE-Bench Lite, annotated with 841 span-level errors across more than twenty categories, an average of 5.68 per trace.

Berkeley’s own conclusion is the one worth carrying: the failures “often stem from system design issues, not just LLM limitations or simple prompt following, and require more than superficial fixes”. Their sharpest single result is a workflow change. On ChatDev, making sure the CEO agent had the final say raised overall task success by 9.4%, with no change to the model at all.

Tool misuse: the right tool with the wrong arguments

Two different failures wear the same face in a log that records only tool names. The agent picked the wrong tool. Or the agent picked the right tool and called it wrongly. They have completely different fixes, and without the arguments in your trace you cannot tell which one you are looking at.

Both are common. In MAST’s annotations, reasoning-action mismatch accounts for 13.2% of observed failures and disobeying the task specification another 11.8%, so a quarter of everything that went wrong is some version of this.

We have argued before that typed schemas are largely a solved problem, now that MCP and typed frameworks schema-check every call, and that the remaining work is tool design. That still holds. But a schema only proves a call has the right shape, and nothing proves it has the right meaning. A refund of exactly the correct shape, issued to the wrong customer, passes every schema you can write.

So you need a validation gate between the model’s proposed call and the thing that executes it, holding the checks a type cannot express. Is this order id one the agent actually retrieved during this run, or did it appear from nowhere? Is this amount inside the policy limit? Is this the record the user asked about? The code for that is in the loop below.

The other half is the tool description, written for the model and not for a human reading your API docs. Anthropic reported that on their SWE-bench work they spent more time optimising tools than the overall prompt. If that is where the effort goes to make an agent work, it is also where the effort goes to bound it.

And sometimes the fix is to stop asking the agent. We have done this on engagements: when one step keeps failing, take it off the agent entirely and give it to a small trained classifier, or to a rules engine that has been sitting in somebody’s head for a decade. It runs that narrow decision deterministically and hands the rest back. Reach for machine learning when the decision is genuinely fuzzy, reach for rules when it is not, and let the agent do the part that is neither. Machine learning is fast becoming a forgotten technology, and it is still the cheapest answer to a lot of steps people now pay a model to guess at.

Context exhaustion: the agent forgets what you told it

Context exhaustion shows up as decay, not as an error. The first several steps are fine. Then the agent starts ignoring a constraint you gave it at the top of the run, and the runs that fail are always the long ones.

The mechanism is not mysterious. Anthropic put it plainly: as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases. They attribute it to a finite attention budget spread across n-squared pairwise token relationships, and to models trained predominantly on shorter sequences. Chroma’s context rot work corroborates it independently: eighteen models, deliberately trivial tasks, input length as the only variable, and degradation arriving “often in surprising and non-uniform ways”. Models scored better on shuffled, incoherent haystacks than on coherent ones, which should end any assumption that a bigger window solves this.

Three fixes, each with a cost worth knowing before you pick one.

• Compaction, which Anthropic calls the first lever. Summarise the working state and restart the window from the summary. Whatever your compaction prompt failed to capture is now gone, so write that prompt for recall first and precision second.
• Externalised memory. Have the agent write findings to files as it goes, so the record outlives the window. Slower, and the agent has to remember to look. Treat it as unproven: in the April 2026 reliability study, a memory scaffold made long-horizon reliability worse or flat for every one of the ten models tested. Measure it on your own workload before you assume it helps.
• Sub-agents with narrow contexts, each exploring at length and returning a condensed summary. You buy clean windows and you pay in coordination.

The token arithmetic is why this compounds. Anthropic measured agents using roughly four times the tokens of a chat interaction and multi-agent systems roughly fifteen times, with token usage alone explaining 80% of performance variance on BrowseComp.

Record two things: token count per step, and the position in the run where the failure happened. If your failures cluster late in long runs, this is your failure mode, and no amount of prompt engineering will touch it.

Silent partial success: the agent says it is done

This is the failure from the first paragraph, and it is the one that reaches customers. A step failed, the agent recovered its composure, and the run ended with a confident summary saying the work is complete. The ticket was never created. The email was never sent. Nothing in the trace is red.

It is also the largest thing MAST measures that almost nobody instruments. The task verification category is 23.5% of observed failures, split between incorrect verification at 9.1%, no or incomplete verification at 8.2%, and premature termination at 6.2%.

A quieter version lives one layer down. Most pipelines collapse “the tool ran and found nothing” and “the tool ran and returned data” into a single success. Forge, a reliability wrapper for tool calling, introduces a distinct ToolResolutionError precisely to keep them apart. An empty result that reads as a success is how a run completes having done nothing at all.

Agents will also skip a verification step and then write a reasonable-sounding justification for having skipped it. The Reward Hacking Benchmark seeds tool-using tasks with naturalistic shortcuts, including skipping verification and tampering with evaluation functions. Exploit rates ran from 0% for Claude Sonnet 4.5 to 13.9% for DeepSeek-R1-Zero, and 72% of those episodes carried an explicit chain-of-thought rationale presenting the shortcut as sound reasoning. It measures deliberate shortcut-taking in reinforcement-learning-trained agents, not ordinary agents mistakenly reporting done, so do not over-read it. The fix is the part to take: hardening the environment cut exploit rates by 87.7% relative, without hurting task success. You fix this outside the model.

So your success signal is a check on the business outcome that does not consult the agent. Ask the ticketing system whether the ticket exists. Do not ask the agent whether it made one. That means every run carries a business identifier so the check is possible at all, a five-minute change nobody makes until the first incident.

And no, you cannot delegate the checking to another model. TRAIL’s 11% is the number to remember whenever somebody proposes an LLM that watches the traces.

Unbounded loops and the bill that follows

This is the biggest slice of MAST by some distance. Step repetition accounts for 15.7% of observed failures and being unaware of termination conditions another 12.4%. Together, 28.1% of everything that went wrong is the agent not stopping.

It became expensive in 2026. TechCrunch reported in June that Uber blew through its entire 2026 AI coding budget by April, that Microsoft revoked its developers’ Claude Code licences months after enabling them, and that per-developer consumption had risen about 18.6 times in nine months on Jellyfish’s measurements. Those are coding assistants, not production agents. The mechanism is identical and the budget line is the same one.

Retries are how a bounded loop becomes an unbounded one, so they need a rule. Retry what is genuinely transient: a timeout, a 429, a 503. Do not retry a semantic failure, because the second attempt compounds the first wrong plan at full price. Give the retry budget a hard number, give the whole run a step ceiling, and when the budget is gone, fall back or escalate. What you never do is return a degraded answer as though it were a good one.

The alerting rule is the part most teams get backwards. Do not alert on total spend, which rises with volume and should. Alert on cost per successful outcome, which stays flat while the system is healthy and climbs the moment something regresses. That metric only exists if you built the outcome check from the previous section, which is the second reason to build it.

Ceilings belong in the loop, not on a dashboard. A dashboard tells you about the eleven pounds after you have spent it.

Stale inputs: a clean trace and a wrong answer

Here the trace is perfect. Every tool returned 200. Every step did what it said it did. The answer is wrong because a nightly sync stopped three days ago and nobody was watching it, and the agent cannot tell fresh data from data that stopped moving.

Two statistics get quoted at this problem and neither one fits it. Gartner’s February 2025 survey of 1,203 data management leaders found that 63% either lack the right data management practices for AI or are unsure whether they have them, which measures readiness, not failure. And the claim that governed data yields 85% to 92% accuracy against 45% to 60% for ungoverned data comes from a company selling a data catalogue, with no published method behind it.

The fix is old, boring, and borrowed from MLOps, where it has been standard practice for a decade. Treat every data source your agent reads as a monitored dependency with its own health checks: schema, volume, freshness. Alert on the source independently of the thing that consumes it, exactly as you would for a model’s training pipeline. If your retrieval index rebuilds nightly, the alert you need is not about the index. It is about the upstream table that stopped receiving rows.

This is also the failure mode an observability platform is structurally unable to catch. It is watching the agent, and the agent is behaving perfectly.

The agent loop, with the ceilings in it

Here is a refund agent with all four controls in one place: a step ceiling, a cost ceiling, a validation gate that catches what the schema cannot, and a fallback for when the budget is gone. It uses Pydantic AI, in the same domain as the refund agent in our guide to building agents.

from dataclasses import dataclass, field
from decimal import Decimal

from pydantic_ai import Agent, ModelRetry, RunContext
from pydantic_ai.exceptions import UsageLimitExceeded
from pydantic_ai.usage import UsageLimits

REFUND_LIMIT_PENCE = 5_000

@dataclass
class Session:
    """Everything this run may act on, and everything it has actually seen."""

    customer_id: str
    seen_orders: set[str] = field(default_factory=set)

agent = Agent(
    "openai:gpt-4.1",
    deps_type=Session,
    retries=2,
    system_prompt=(
        "Decide whether a refund request is in policy, then act. "
        "Only refund orders you have looked up in this conversation."
    ),
)

@agent.tool
def search_orders(ctx: RunContext[Session]) -> list[str]:
    """Return the recent order ids for the customer in this session."""
    orders = orders_for(ctx.deps.customer_id)
    ctx.deps.seen_orders.update(orders)
    return orders

@agent.tool
def issue_refund(ctx: RunContext[Session], order_id: str, amount_pence: int) -> str:
    """Refund an order that has already been looked up, up to the policy limit."""
    if order_id not in ctx.deps.seen_orders:
        raise ModelRetry(f"{order_id} was not returned by search_orders in this run.")
    if amount_pence > REFUND_LIMIT_PENCE:
        raise ModelRetry(
            f"{amount_pence}p is over the {REFUND_LIMIT_PENCE}p limit. Escalate instead."
        )
    return refund(order_id, amount_pence)

def handle(request: str, session: Session) -> str:
    try:
        result = agent.run_sync(
            request,
            deps=session,
            usage_limits=UsageLimits(
                request_limit=8,
                tool_calls_limit=12,
                cost_limit=Decimal("0.50"),
            ),
        )
    except UsageLimitExceeded as exhausted:
        return escalate(session, reason=str(exhausted))

    if not ledger_shows_refund(session.customer_id):
        return escalate(session, reason="the agent reported a refund the ledger has not seen")

    return result.output

The type signature on issue_refund catches the shape of the call. The two checks inside it catch the meaning: an order id the agent never retrieved, and an amount over policy.

Three details then matter more than their line count. ModelRetry hands the reason back to the model so it can correct itself, bounded by retries=2: that is a semantic retry, a different thing from retrying a network error, and the two should never share a budget. cost_limit aborts on real spend priced from the provider’s own rates, so the ceiling is expressed in the currency your finance team cares about. And ledger_shows_refund asks the ledger, not the agent.

The fallback here escalates to a human. A narrower deterministic path is often better: run the query the agent was circling and return that, so the customer gets an answer instead of a wait. If that path ends up carrying most of the traffic, you have learnt something, and the next step is to build it properly as a workflow and let the agent handle the exceptions.

None of this is difficult. Almost nobody does it, because it is the work that does not demo.

Instrument to a standard, not to a vendor

The instrument column in that table says what to record. Two things decide whether the recording is any use: how much of it you keep, and what shape it lands in.

The first is volume. Sample full traces in normal operation, and capture every trace on failure. Sampling is why teams can afford tracing. Failures are why they regret sampling.

The second is where you send it. Instrument to OpenTelemetry’s GenAI semantic conventions instead of a vendor SDK. They model a run as a span tree, with invoke_agent containing chat and execute_tool spans, and they cover agent creation, workflow invocation, planning, retrieval and token usage. Know what you are adopting: as of July 2026 every one of those spans, metrics and attributes still carries the Development badge, none is marked Stable, and the conventions moved out of the main semantic-conventions repository in June 2026 into a dedicated repository with no tagged releases yet, so there is no versioned schema to pin against. Adopt it anyway. The alternative is a proprietary trace format and a migration you will pay for later.

One signal is missing from all of that, and no amount of tracing produces it. Everything above tells you whether the agent did what it was asked. Nothing tells you whether it did what the person actually wanted. User feedback is the only measure of that, and it is worth more than any metric on the list.

We run a documentation bot internally. It is good, and it writes at length about every small feature it finds. People complained about its pull requests for months and nobody changed anything, because there was nowhere to put a complaint. The traces were clean throughout. The agent was doing exactly what it had been told to do. That is why our bots are conversational now: a correction has to have somewhere to go, and the cheapest place to put it is a reply.

Traces are portable. Reliability is not. There is a convention for how to record a tool call. There is no convention, and there cannot be one, for your retry policy or your approval gate.

When to put a human in front of it

The test for an approval gate does not depend on how good the agent is. Gate any action that is expensive, externally visible, or hard to reverse. Sending customer communications. Moving money. Writing to a system of record. Deleting anything. Read-only actions and internally reversible ones do not need a gate and should not have one, because a gate on everything is a gate on nothing.

The question is whether an error would be embarrassing or costly to undo, not whether the agent is usually right. An agent that is right 99% of the time and sends a thousand emails a day is wrong ten times a day, in public.

We made this call on a real process before it was a genre of blog post. Working with CMPC, a paper miller, we built reinforcement learning agents to control part of their pulp process: large pressurised tanks where chemicals are mixed with pulp, instrumented and controlled through a proprietary system, and until then run entirely on the judgement of experienced operators. The agents work. Together with CMPC we chose to have them recommend parameters to the operators instead of taking control, because the process was real and so was the downside. That is an approval gate, designed in, on an autonomous system running against an industrial process.

The security people arrive at the same answer from the opposite direction, which is the strongest form this argument takes. OWASP concludes that given the stochastic influence at the heart of how models work, it is unclear whether fool-proof prevention of prompt injection is possible, and every mitigation it recommends is a tool-layer control: handle privileged functions in code instead of giving them to the model, restrict privileges to the minimum necessary, and require human-in-the-loop approval for privileged operations.

A gate is a code path, not a policy. If nobody can name the function that blocks the call, there is no gate.

Where observability platforms help, and where they stop

Start with what they are good at, because it is real. Trace search across thousands of runs. A diffable view of two runs of the same task. Retention, access control, and evaluation scores attached to production traffic instead of to a spreadsheet. Once you have enough volume that grepping logs stops working, a platform is worth the money, and building one yourself is a poor use of your engineers.

The names, and what each leads with. LangSmith sits closest to LangChain and LangGraph, which is the reason to pick it and the reason not to. Arize Phoenix and Braintrust both come at it from evaluation. Pydantic Logfire is built on OpenTelemetry. And the incumbents, Datadog, Dynatrace and Splunk, now carry LLM and agent observability inside the APM product your company already pays for, which is frequently the right answer for reasons that have nothing to do with features.

They price on different axes, which is the practical thing the comparison posts skip. Checked against each vendor’s own page on 14 August 2026: LangSmith prices on seats, at $39 each above a one-seat free tier. Braintrust prices on data processed, at $3 to $4 per gigabyte. Arize AX prices on spans, ingest and retention together. Langfuse prices on units. Headline numbers are not comparable across those, so work out which axis your own system is expensive on first.

We do not sell a tracing product, so here is the gap those comparisons leave. If your traces cannot leave your own infrastructure, and in regulated work they often cannot, Langfuse is open source and self-hosts for free, Opik is Apache 2.0, and Arize ships Phoenix as an open, local-first build alongside the commercial product. LangSmith and Arize both self-host on their enterprise tiers, so this is a licensing and budget question, not a hard wall. You lose the managed search index. You keep the trace, which is the part that matters. And in the interest of disclosure, we build Helix, a private AI platform, so we have a horse at the platform layer too.

Every platform listed above will show you, in an excellent interface, the run where your agent spent forty minutes and eleven pounds calling the same endpoint. Not one of them will stop it happening. The step ceiling that stops it is four lines of your own code, and no product can ship it for you, because only you know what a reasonable number of steps is for your job.

A monitoring product is bought to tell you what happened, and the good ones do that well. Making it not happen is a different job, and it is yours.

Where to start

Go back to that trace where every step was green. Nothing in it was recording the one thing that mattered, which is whether the outcome the customer wanted actually happened.

So start there. Put a step ceiling and a cost ceiling in the loop, and add one check that asks your ledger instead of your agent. It is an afternoon of work, and it moves the two worst failures on this list out of your customer’s inbox and into your alerting, where you can do something about them.

Everything above happens after go-live. The other half of the job happens before it, in the evaluation suites and CI gates that stop a bad version shipping at all.

If you would rather somebody else owned the pager, that is what we do: AI agent development for the build, MLOps for the operational backbone underneath it, AI workflow automation for the deterministic half nobody should pay a model to do, or book a scoping call and tell us what broke.

Build it so the failure is loud. The green trace is the dangerous one.