Harness Engineering: What We Learnt Building AI Agents

I have brewed my own beer for twelve years, because the beer I can buy is brewed for the average drinker and I want one brewed for me. The dark mild I made in May tastes of raisins and cinnamon, yet its yeast came in a packet of S-04 that any homebrew shop sells.

The yeast does the heavy lifting, but what has improved my beer the most over those years is the equipment and the process around it. My biggest equipment upgrade came in June 2022, when I bought a Grainfather and a stainless steel fermenter, and my biggest process change came seven months later, when I started boiling my water before the mash to soften it.

An AI agent works the same way. The model is the yeast, and you buy it from the same few suppliers as everyone else. Harness engineering is improving everything around it: the tools in the agent’s sandbox are its equipment, and the loop that runs it is its process. Below are three examples of doing that on our own agents, and what each one taught us.

How we fixed our AI pull request reviewer

The reviewer I built runs on Helix, the agent platform the Winder.AI team co-develops, where a bot wakes on each pull request and hands the review to a coding agent in its own sandbox.

In August the reviewer approved almost everything and flooded a queue with retries, so Priya switched it off, and Karolis’s verdict was that “the performance is lacking”.

We started by editing the prompt, and that fixed what the reviewer wrote: within a few weeks, 122 of its latest 124 reviews followed the format we had asked for.

The same edits told the reviewer to read the code and stop testing it. It had been building the project and running scenarios through the code, doing real QA that CI already covers, and every review took far longer than it needed to.

The prompt also told the reviewer to skip drafts, and it obeyed, but only after spending two minutes working out that a pull request was still a draft, ten times on the same pull request. Editing the prompt could change what the reviewer said, but it could not change when the reviewer ran.

We fixed that with a small filter that sits in front of the bot and drops draft and closed pull requests before the bot ever wakes.

The last fix was to delete the rule that told the reviewer to approve only when CI was green, which had left it waiting on CI for 633 of the 817 seconds in one review. Now it reviews the code and nothing else.

The prompt is text the bot reads on every run, so it can only ask, and the model chooses whether to comply: the bot obeyed the rule about drafts, but spent two minutes deciding to each time. The filter is a constraint, a tiny deterministic pipeline that encodes our own thinking about which pull requests need a review. Harness engineering is choosing which of the two each decision needs.

What is harness engineering?

The filter made one of the bot’s decisions deterministic, and that is how I would define harness engineering: choosing the right level of determinism for the task at hand.

Determinism can enter an agent through its equipment or its process. The equipment is the deterministic tools the agent may call, such as a calculator, which returns the same answer every time so the model never does the sum itself. The process is the code around the agent, such as the filter in front of our reviewer, or a fixed sequence of steps the agent works through.

Pushing everything into code fails the other way, because most useful tasks need judgement that only a model can provide. Even a very deterministic process usually requires some judgement, such as understanding what a user meant. So the work is finding where judgement is still required, and adding determinism everywhere else to make the agent more reliable.

Birgitta Böckeler defines the harness as everything in an agent except the model, and she writes from the seat of a coding-agent user. Coding agents such as those in our comparison of agent harnesses ship with part of a harness built in, and she describes the outer harness you add: guides that steer the agent before it acts, and sensors that tell it afterwards whether it worked. To me that is the feedback around an agent that learns from its mistakes, which is closer to reinforcement learning than to the harness of an application.

My definition is simpler: a harness is a loop with one or more models inside it. The loop can depend on inputs and on checks, but it can be any process the task needs, and harness engineering is deciding how much of that loop to make deterministic.

Seeing the harness as a loop means a model can take more than one role inside it, and so can several models. The advice engine we are building looks like a loop with several models in it doing different jobs, some more agentic than others. Our pull request reviewer is a loop around a loop: the coding agent is one model wrapped with its inputs, outputs and context, and the bot’s filter and event handling form an outer loop around it. A multi-agent workflow is the same pattern at a larger scale, one loop containing several agents, and it is a harness too.

Han Lee argues that most harness code will dissolve into the next generation of models. He is right about code that patches a weakness the next model fixes, which is a reason to add to the harness only to fix a failure you have already seen. But a tax calculation should stay deterministic however good the model gets.

How a harness differs from an agent

The word “agent” gets used for two different things, which makes a harness hard to pin down. In everyday use it names a whole product, such as a coding agent or a customer-support agent. Anthropic uses it for a way of working: agents are “systems where LLMs dynamically direct their own processes and tool usage”, while workflows are “systems where LLMs and tools are orchestrated through predefined code paths”.

Under my definition the harness is the whole system, the loop and every model inside it, so the everyday agent is a harness. Anthropic’s distinction then describes parts of the loop: where a model decides what happens next, that part is agentic, and where code decides, it is a workflow. One harness can hold both, and every one of ours does.

So asking how agentic a harness should be is the same as asking how deterministic its loop should be.

How deterministic should the loop be?

Harnesses differ in how much they leave to the model. At one end the model runs a loop and decides everything, and at the far end the harness picks a playbook and runs it step by step while the model fills only the gaps. In the middle, the harness keeps asking the model to fill in a fixed set of fields, so code always knows what is still missing.

One way to place a harness on that range is to ask what a wrong answer costs and who catches it. A person reads every review our bot writes, so it can sit near the agentic end, while the advice engine’s answers have to be right and auditable, so it sits near the other.

A second way is to ask which steps can be proven correct, because plenty of steps in a task have one right answer that a few lines of ordinary software can compute, and an agentic harness will often write that software itself, by having the model generate code for the calculation. It writes the code afresh on every run, so across hundreds of runs some of those versions carry subtle bugs, and a bug in one step becomes a wrong final answer. Even good models slip when the numbers sit inside a word problem, and handing the sums to code fixes it.

So look for the steps that repeat, that have one right answer, and that sit at the centre of your problem. Those belong in a library of deterministic scripts or calculators that the harness calls, where each one is visible, tested and written once. A calculator can even be a trained model, such as the demand curve in our pricing algorithm walkthrough, built and tested like any other.

Determinism also makes a harness easy to explain, because when a script produced a number you can point at exactly what happened and why. That trace can become the cornerstone of the answer itself: this figure came from this rule, applied to these facts. Anyone auditing the answer can check that trace without having to trust the model.

Our agents where code runs the loop

We are building an advice engine for a regulated profession, and because its answers have to be right, each version has given the models less to decide. The rules and the calculations now sit in a deterministic layer that code runs, and the models handle the parts that need judgement, such as understanding the question and writing up the answer. The design is still changing and the profession’s own experts have yet to grade it, but every figure it produces comes from a rule we can point at.

We moved a decision from the model to code on an earlier project too, an agent that phones automated telephone menus and works its way through them. When the model chose which option to press, it kept repeating the path that had worked once, so now it lists every option and code chooses, trying new ones first. On one test menu that cut the exploration from 46 steps to 6.

Where to improve the harness first

The teams building harnesses have converged on one move: when a model keeps getting something wrong, they replace the instruction with something deterministic. Anthropic’s Claude Code documentation says “unlike CLAUDE.md instructions which are advisory, hooks are deterministic and guarantee the action happens”, and OpenAI’s harness team works to the same rule: “when documentation falls short, we promote the rule into code”. These are the places I would add determinism first, cheapest first.

Load less context and give tools a shape the model cannot misuse

An MCP tool we added made no difference, because one of the platform’s default skills told the agent to use the Helix command-line tool instead. When we changed the platform to load skills only when a task needs them, the average task fell from 16 turns to 6. OpenAI’s team learned the same lesson with its instruction file, which failed as one big manual and works as a short map of about 100 lines that points the agent at deeper documents.

Can Bölük changed nothing but the edit format offered to 16 models and raised their scores by about 15 points on average, so the shape of a tool matters too, and Anthropic found that letting the model search for tools beat loading every definition up front. Your own private code is context a model cannot have from training, which is why we built Kodit to give coding assistants a search over repositories that are new to them.

Put a static analyser on every output

A static analyser reads the output, applies fixed rules and says exactly what is wrong, which makes it the cheapest determinism you can add. When the SWE-agent team removed the syntax checker from their agent’s edit command, the share of issues it resolved fell from 18.0% to 15.0% on the same model, and a 2025 study that fed Bandit and Pylint results back to GPT-4o over ten rounds cut security issues from over 40% to 13%.

OpenAI’s team enforces its architecture with custom linters and structural tests, and writes each linter’s error message to tell the agent how to fix the problem. Every coding agent working on Kodit runs make check, which formats, vets, lints and tests the code before the work counts as done.

My drafts pass through Vale, a prose linter whose rules encode my style guide and catch the phrases I have banned, and PostHog’s documentation team reached the same split: “use Vale to detect issues, then use LLMs to help fix them”. Vale checks sentences and cannot tell whether the argument is right, so that judgement stays with a model or a person.

Turn instructions the model ignores into hooks

Our reviewer’s filter does the job of a hook: the prompt asked the bot to skip drafts, the bot obeyed slowly, and now code drops them before the bot wakes. The harnesses ship this as a feature. Claude Code and Codex both let a hook run when the agent tries to finish and block the turn until a check passes, Prime Agent returns a failed completion gate’s output to the agent for another attempt, and Hermes Agent blocks a list of dangerous commands even in its most permissive mode.

Give the agent ways to check its own work

An agent can only verify what it can see, so OpenAI’s team made its application readable to Codex: each change boots its own copy of the app, which the agent drives through the browser’s developer tools and checks against its own logs and metrics. Our telephone agent acts on systems we do not control, so we built a fake phone system that plays a menu from a file, and the agent’s tests use the same audio path as real calls.

Tests the agent writes for itself add little: across four models, encouraging or discouraging them moved resolve rates by under three points. The tests that help come from your own failures, and Anthropic suggests starting from 20 to 50 of them, running each more than once. Our guide to agent evaluation covers the method.

Consider adding a more formal loop around the harness

Every fix here started with a failure someone noticed, and a formal loop around the harness makes that noticing routine. Ours has Claude read the agent’s transcripts and classify the issues in them, and each recurring issue becomes a fix, a deterministic one wherever possible. Read the transcripts, because outputs only record the runs that produced something: our reviewer’s ten draft-checking runs posted nothing, so the GitHub history we scored it on skips them. Our article on agents failing in production covers tracing in more depth.

My writing pipeline runs the same loop, so when I flagged “that holds” as a phrase I would never use, it became a Vale rule the same day and the gate now blocks any draft that uses it. OpenAI’s team runs the loop on a schedule: its engineers used to spend every Friday cleaning up “AI slop”, so they wrote their “golden principles” as mechanical rules and set background Codex tasks to scan for deviations and open refactoring pull requests. Addy Osmani calls this a ratchet, where every agent mistake becomes a permanent fix.

Where judgement still belongs

Boiling the water before the mash has been a standard step in my brewing since January 2023. In May I skipped it for the dark mild, because darker malts handle hard water, and that call was mine to make.

A good harness works the same way, with code doing every step that has one right answer, the same way every time, and the model left with the calls that need judgement, such as what a user meant or whether a change hides a real bug. Drawing that line for your own task is harness engineering.