---
title: "Harness Engineering for Coding Agents: A Practical Team Setup"
date: "2026-11-16"
description: "Same model, different results: the harness decides. What a team controls around Claude Code or Codex, with hook configs, fast checks, and a setup checklist."
author: "am2rican5"
tags: ["AI", "Coding Agents", "Developer Tools", "Software Engineering"]
isActive: true
readingTime: "14 min read"
translationKey: "harness-engineering-coding-agents"
slug: "harness-engineering-coding-agents"
authorImage: "https://github.com/am2rican5.png"
---

## The Short Answer

A coding agent is a model plus a harness. Your team doesn't choose the model's weights, but it does own a large part of the harness: the instructions the agent reads, the checks it runs, the hooks that fire around its tool calls, and the gates a change has to pass before it merges. That part decides a lot of what you get back from the same model.

If you have one week to spend on it, spend it in this order:

1. **Sensors before guides.** Make the agent find out it's wrong, fast, with error messages written for it to act on.
2. **Don't let "looks done" end the turn.** Run verification when the agent tries to stop, not only in CI.
3. **Keep the instruction file a map.** Short, pointing at the checks and docs that hold the detail.
4. **Treat every control as an assumption with an expiry date.** Remove pieces one at a time when the model changes, and keep only what still moves a number.

## Four Nights Versus Two Hours

The clearest recent story about harnesses came out on November 11. Swami Sivasubramanian, in a guest post on Werner Vogels' blog announcing [AWS Emulator](https://www.allthingsdistributed.com/2026/11/aws-emulator-and-the-engineers-job-when-agents-write-the-code.html), describes using Kiro to rebuild the replication engine of Alf, a Paxos-based lock service that he and Al Vermeulen had originally built in about four months:

> I supervised the agent for four nights before I saw that it had no way to check its results. Once it had a spec and a test environment, the rebuild took about two hours.

Same tool, same task. What changed was a spec and something to test against.

The numbers from vendors point the same way, with the usual caveats. LangChain [moved its coding agent from 52.8% to 66.5% on Terminal-Bench 2.0](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) with `gpt-5.2-codex` held fixed, changing only the system prompt, tools, and middleware. That's one vendor's run on one 89-task benchmark that the harness was tuned on, so read it as a direction, not a forecast for your repo. Anthropic ran the same one-sentence prompt (a 2D retro game maker) [through a single agent and through a planner–generator–evaluator harness](https://www.anthropic.com/research/harness-design-long-running-apps): the solo run took 20 minutes and $9 and produced a game where nothing responded to input; the harness took 6 hours and $200 and produced one you could play.

One of the week's most-discussed complaints about AI-written code lands in the same place. Evan Merz's [*AI usage generates a fragile codebase*](https://evanxmerz.com/post/ai-usage-generates-a-fragile-codebase/) drew more than 600 points on [Hacker News](https://news.ycombinator.com/item?id=50438318). His first item is that AI writes shallow tests, which "creates massive gaps in the test coverage that are invisible to developers because they aren't captured by the line level coverage metric." That's a harness problem as much as a model problem: nothing in the loop was able to tell the agent its tests were weak.

## What "Harness" Means When You Didn't Build the Agent

The Wavestone AI Lab paper [*Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents*](https://arxiv.org/abs/2609.00006) read the source of eleven production coding agents, including Claude Code, Codex CLI, Gemini CLI, OpenHands, and Aider. It starts from one line, "An agent is a model plus a harness," and maps every harness onto seven subsystems: the agent loop, LLM integration, tools, memory and context, safety and permissions, orchestration, and extensibility. That's the vendor's side of the harness, and you mostly don't touch it.

Birgitta Böckeler's [*Harness engineering for coding agent users*](https://martinfowler.com/articles/harness-engineering.html) is about the other side: the **outer harness** a team builds around an agent it didn't write. (In an October update on [her own site](https://birgitta.info/posts/harness-engineering.html), she says she now leans toward calling this context engineering for coding agents, and keeping "harness engineering" for the builders. The vocabulary is still moving; the practice isn't.) Her model has two axes that are worth adopting as-is:

- **Guides** act before the agent does: instruction files, skills, templates, codemods. **Sensors** act after: tests, linters, type checks, structural tests, AI review.
- Each can be **computational** (deterministic, fast, run by the CPU) or **inferential** (an LLM judging something, slower and non-deterministic).

Guides alone give you an agent that follows rules and never learns whether they worked. Sensors alone give you an agent that keeps repeating the same mistakes and getting caught.

Mitchell Hashimoto's version, from [the post](https://mitchellh.com/writing/my-ai-adoption-journey) where he first used the term, is the operational one: "anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again." He names two forms, a line in `AGENTS.md` and "actual, programmed tools," usually paired.

## Sensors First

**If you can only invest in one side, invest in sensors.** A guide asks the model to remember something. A sensor tells it, at the moment it matters, that it got something wrong, and models are good at fixing errors they can see. Hashimoto found the same thing early: "If you give an agent a way to verify its work, it more often than not fixes its own mistakes and prevents regressions."

The exception: a codebase with no types, no tests, and no module boundaries. Böckeler calls this *harnessability*. A strongly typed language gives you a sensor for free; a legacy app full of implicit behavior doesn't, and "the harness is most needed where it is hardest to build." There, a few good guides buy time while you build the first sensors.

Three properties separate a useful sensor from a CI job nobody reads.

**It runs inside the loop, not after the PR.** Böckeler's "keep quality left" applies doubly to agents: the cheaper the check, the earlier it should run, ideally right after the edit. In Claude Code, that's a `PostToolUse` hook on file edits. Project hooks live in `.claude/settings.json`, which you can commit:

```json
{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/check-file.sh",
            "args": []
          }
        ]
      }
    ]
  }
}
```

**It's scoped to what just changed.** Run the linter on the edited file, not the whole repo, so the hook takes seconds and nobody is tempted to switch it off:

```bash
#!/bin/bash
# .claude/hooks/check-file.sh: lint the file the agent just edited.
# Exit 2 on PostToolUse shows stderr to the agent (the edit already happened).
file=$(jq -r '.tool_input.file_path // empty')
case "$file" in
  *.ts|*.tsx) ;;
  *) exit 0 ;;
esac

if ! out=$(npx eslint --max-warnings=0 "$file" 2>&1); then
  echo "ESLint failed on $file. Fix the cause; do not disable the rule or add an ignore comment." >&2
  echo "$out" >&2
  exit 2
fi
exit 0
```

**Its failure message is written for the agent.** This is the cheapest high-leverage change on the list. OpenAI's team, writing about [a product built with no hand-written code](https://openai.com/index/harness-engineering/), enforces its architecture with custom lints: "Because the lints are custom, we write the error messages to inject remediation instructions into agent context." Böckeler calls the same thing "a positive kind of prompt injection." You don't need a custom linter to start. Most linters let you attach a message to a rule:

```javascript
// eslint.config.js (excerpt)
export default [
  {
    files: ["src/**/*.ts"],
    rules: {
      "no-restricted-imports": ["error", {
        patterns: [{
          group: ["**/infra/db/**"],
          message:
            "Domain code must not import the DB layer. Use the repository in src/repo/ " +
            "(see docs/architecture.md#layers) and add a method there if one is missing.",
        }],
      }],
    },
  },
];
```

A human reading "Unexpected import" knows what to do. An agent fixing a red build at speed may delete the import and inline a query. The message tells it which fix you want.

A few practical notes on hooks. Exit code 1 doesn't block anything in Claude Code; a hook that should stop an action needs exit code 2 or a JSON decision. A hook that crashes or has a typo in its path is, by default, a non-blocking error, so a broken policy hook fails open; Claude Code's [hooks reference](https://code.claude.com/docs/en/hooks) documents `"onFailure": "block"` (v2.1.295+) for hooks that must fail closed, and, because its `if` filter is best-effort, recommends the permission system for hard allow/deny rules. Hook schemas change between releases, so treat the snippets here as starting points and check them against the reference for your version.

The same configuration ideas travel. The Wavestone paper found lifecycle hooks in nine of the eleven agents it studied by July 2026, and Codex's hook events use names that are near-verbatim Claude Code's (`PreToolUse`, `PostToolUse`, `Stop`, and so on). The scripts are the durable part: a hook that runs `check-file.sh` is easy to port, and the script itself is just a lint command.

## Don't Let "Looks Done" End the Turn

LangChain's trace analysis found that the most common failure was mundane: "the agent wrote a solution, re-read its own code, confirmed it looks ok, and stopped." Their fix was a `PreCompletionChecklistMiddleware` that intercepts the agent before it exits and sends it back for a verification pass.

Claude Code has the same hook point. A `Stop` hook can return `decision: "block"` with a reason, and the agent keeps working with that reason as feedback:

```json
{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/verify-before-stop.sh",
            "args": [],
            "timeout": 600
          }
        ]
      }
    ]
  }
}
```

```bash
#!/bin/bash
# .claude/hooks/verify-before-stop.sh: run tests when the agent tries to finish.
input=$(cat)

# Already continuing because of this hook once? Let the turn end; a human takes it from here.
[ "$(jq -r '.stop_hook_active' <<<"$input")" = "true" ] && exit 0

# Nothing changed, nothing to verify.
[ -z "$(git status --porcelain)" ] && exit 0

if ! out=$(npm test --silent 2>&1); then
  reason=$(printf 'Tests are failing. Read the output, fix the cause (not the test), and rerun before finishing:\n%s' "$(tail -n 40 <<<"$out")")
  jq -n --arg r "$reason" '{decision: "block", reason: $r}'
fi
exit 0
```

Two guards matter. `stop_hook_active` is `true` when the agent is already continuing because of a stop hook; checking it keeps the hook from looping on a condition the agent can't fix. Claude Code also caps stop-hook continuations at eight in a row. This script takes the stricter route: one forced retry, then the turn ends and the failing output is there for a person. Loosen it if your test suite is reliable and your tasks are long.

Keep the stop check to things that are fast and deterministic: unit tests for the touched package, a type check, the build. Slow suites belong in CI, where the PR template from [the risk-first review checklist](/blog/review-ai-generated-pull-requests) asks the author to paste what ran and what it returned.

## Guides: A Map, Not a Manual

OpenAI's team tried "one big `AGENTS.md`" and it failed in predictable ways. It crowded out the task, it rotted, and, in their words, "Too much guidance becomes non-guidance." Their replacement is a file of roughly 100 lines that works as a table of contents, pointing at a structured `docs/` directory, with linters and CI jobs that check the docs stay current and cross-linked. Their line for it: "give Codex a map, not a 1,000-page instruction manual."

That lines up with the research covered in [Does AGENTS.md Help Coding Agents?](/blog/does-agents-md-help-coding-agents): context files don't make the agent more capable, they remove decisions it would otherwise have to make. In a harness, the instruction file has one more job: telling the agent which sensors exist. "Run `pnpm check:content` before finishing a content change" is a guide pointing at a sensor, and it saves the agent a red run.

Skills are the other common guide. The same Wavestone paper counts `SKILL.md` skills in nine of eleven agents. A skill earns its place when it packages a procedure the agent gets wrong without it, like a release process or a migration recipe, ideally with a script attached. Google's [modern-web-guidance](https://github.com/GoogleChrome/modern-web-guidance) skills, which did well on [Hacker News](https://news.ycombinator.com/item?id=50443812) last week, are a public example of the type: they push agents off dated JavaScript patterns before they write any code.

## Where Harnesses Stop Helping

It would be convenient if enough sensors made supervision unnecessary. The evidence says otherwise, and the failures are specific.

Self-evaluation is lenient. Anthropic's evaluator agent needed several rounds of tuning before it graded usefully. "Out of the box, Claude is a poor QA agent," the post says. "In early runs, I watched it identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway." Separating the agent that grades from the agent that builds helped, but the evaluator still missed bugs in deeply nested features.

Behavior is the hard part. Böckeler sorts harnesses by what they regulate. Maintainability harnesses are the easiest, because linters, complexity checks, and coverage tools already exist. The behaviour harness, checking that the application does what users need, is in her words "the elephant in the room": most people giving agents high autonomy check that an AI-written test suite is green and well covered, which "puts a lot of faith into the AI-generated tests." That's exactly the gap Merz describes. Mutation testing and approved fixtures help in places; neither is a general answer yet.

And no sensor catches a misunderstood request. Her mapping of agent failure modes is the sobering part: computational sensors catch duplication, complexity, and architecture drift; LLM sensors partly catch over-engineering and brute-force fixes; neither reliably catches misdiagnosis or misunderstood instructions. "Correctness is outside any sensor's remit if the human didn't clearly specify what they wanted in the first place."

So the harness doesn't replace review. It changes what review is for. Her own goal for a harness is the right one: "A good harness should not necessarily aim to fully eliminate human input, but to direct it to where our input is most important." In practice that means intent, decisions, and behavior, which is where the [review checklist](/blog/review-ai-generated-pull-requests) spends its time.

## Multi-Agent Harnesses Are a Budget Decision

The $9-versus-$200 comparison is easy to misread as "use more agents." Anthropic's own conclusion is narrower. After Opus 4.6 shipped, many tasks that had needed the evaluator fell within what the generator did well alone, and the evaluator became overhead there. "It is worth the cost when the task sits beyond what the current model does reliably solo."

The Wavestone paper's recommendation for harness builders is to stay single-agent until a task has a real parallel exploration phase, citing earlier work (Hadfield et al.) that multi-agent systems use about 15× more tokens than a chat baseline. For a team, the translation is: a planner and an independent reviewer pay off for long, greenfield, or high-risk work; for a normal bug fix, a fast test suite and a stop hook do more per dollar.

## Every Control Is an Assumption With an Expiry Date

The most useful sentence in Anthropic's post is about maintenance: "every component in a harness encodes an assumption about what the model can't do on its own." Those assumptions go stale. Their earlier harness depended on context resets because Sonnet 4.5 wrapped up work early as its context filled; Opus 4.5 "largely removed that behavior," and the resets came out. LangChain says the same about its own loop-detection and time-budget nudges: "These guardrails will almost surely dissolve over time."

How you remove them matters. Anthropic's first attempt to simplify was to cut the harness back radically, and it couldn't reproduce the original results or tell which pieces had been load-bearing. What worked was removing one component at a time and checking the effect each time.

Orange ITS's [*Harness Engineering: Same Model, Better Results, and the Scaffolding You Can Delete*](https://www.orange-its.ch/en/insights/harness-engineering) turns this into a habit: label each control **durable** or **expiring** when you add it.

| Control | Type | Runs | Durable or expiring |
|---|---|---|---|
| Lint on edited file, with remediation messages | Computational sensor | `PostToolUse` hook | Durable |
| Tests on stop for touched package | Computational sensor | `Stop` hook | Durable |
| Architecture rules (import boundaries, layer tests) | Computational sensor | Hook + CI | Durable |
| Permissions and sandbox rules | Computational guide | Agent settings | Durable |
| `AGENTS.md` lines for commands and generated paths | Inferential guide | Every session | Review when tooling changes |
| "Re-read the task before finishing" prompt lines | Inferential guide | Every session | Expiring: retest on model upgrade |
| Loop or retry nudges | Computational sensor | Hook / middleware | Expiring: retest on model upgrade |
| Separate AI reviewer or evaluator | Inferential sensor | PR / end of run | Depends on task difficulty; retest |

Then, on each model upgrade, take one expiring control out, rerun a small set of tasks you know the answer to (the afternoon eval from the [AGENTS.md post](/blog/does-agents-md-help-coding-agents) works here), and keep the control only if a number gets worse without it.

One more cost to plan for is drift. OpenAI's team found that agents copy whatever patterns already exist in the repo, good or bad, and the team "used to spend every Friday (20% of the week) cleaning up 'AI slop.'" Their fix was to write the rules down as mechanical checks and run recurring cleanup agents that open small refactoring PRs. Böckeler lists the same category as continuous drift sensors: dead code, test quality, dependency scans, run against the codebase outside any single change.

## The Checklist

**Sensors**
- A fast lint or type check runs on every file edit, scoped to that file
- Failure messages say how to fix the problem, not only what's wrong
- A stop check runs the touched package's tests and the build before the agent finishes
- Stop hooks respect `stop_hook_active` and end after a bounded number of retries
- Policy hooks that must fail closed use exit code 2 and `onFailure: "block"`; hard allow/deny rules live in permissions
- Architecture boundaries are enforced by lint rules or structural tests, not by prose

**Guides**
- The instruction file is short and points to docs and checks instead of repeating them
- It names the commands the agent should run to verify its own work
- Skills exist only for procedures the agent gets wrong without them

**Review and cost**
- Humans review intent, decisions, and behavior; the harness handles what's mechanical
- Separate evaluator or reviewer agents are reserved for tasks beyond what the model does reliably alone
- A recurring job checks for drift (dead code, test quality, stale docs)

**Maintenance**
- Every control is labeled durable or expiring
- On model upgrades, expiring controls come out one at a time against a small eval
- When the same mistake shows up twice, it becomes a check or a line in the instruction file

## Sources

- Swami Sivasubramanian, [*AWS Emulator and the engineer's job when agents write the code*](https://www.allthingsdistributed.com/2026/11/aws-emulator-and-the-engineers-job-when-agents-write-the-code.html) (All Things Distributed, Nov 11, 2026)
- Vivek Trivedy, [*Improving Deep Agents with harness engineering*](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) (LangChain, Feb 17, 2026)
- Anthropic, [*Harness design for long-running application development*](https://www.anthropic.com/research/harness-design-long-running-apps) (Mar 24, 2026)
- Paul Barbaste, Tristan Darrigol, Germain Vu, Tom Wiltberger, [*Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — A Source-Code Study of Eleven Systems*](https://arxiv.org/abs/2609.00006) (arXiv, Jul 15, 2026)
- Birgitta Böckeler, [*Harness engineering for coding agent users*](https://martinfowler.com/articles/harness-engineering.html) (martinfowler.com, Apr 2, 2026; [October update](https://birgitta.info/posts/harness-engineering.html))
- Mitchell Hashimoto, [*My AI Adoption Journey*](https://mitchellh.com/writing/my-ai-adoption-journey) (Feb 5, 2026)
- Ryan Lopopolo, [*Harness engineering: leveraging Codex in an agent-first world*](https://openai.com/index/harness-engineering/) (OpenAI, Feb 11, 2026)
- Evan X. Merz, [*AI usage generates a fragile codebase*](https://evanxmerz.com/post/ai-usage-generates-a-fragile-codebase/) (Nov 7, 2026; [HN discussion](https://news.ycombinator.com/item?id=50438318))
- Orange ITS, [*Harness Engineering: Same Model, Better Results, and the Scaffolding You Can Delete*](https://www.orange-its.ch/en/insights/harness-engineering) (Nov 7, 2026)
- Anthropic, [Claude Code hooks reference](https://code.claude.com/docs/en/hooks)
- Google Chrome, [modern-web-guidance](https://github.com/GoogleChrome/modern-web-guidance)
