Coding Agents Can Finish the Task, but How Much Work Did They Waste?
A 50,000-run study shows pass rates hide wasted edits, repeats, and unrelated changes; harnesses need process-cost metrics.
Key Points Summary
- Pass/fail benchmark scores alone can be misleading because two coding-agent runs may both produce a working patch, yet differ greatly in wasted steps, repeated commands, reverted edits, and unrelated changes.
- The study used about 50,000 simulated runs over real Python repositories and unittest suites, simulating only the agent’s decision-making so harness rules could be compared without live-model differences, API costs, or tool-use variation.
- It separates “final patch size” from “total work,” defining wasted work as actions that do not move the run closer to the final passing state, while counting necessary reverts and harness overhead separately.
- Three harnesses were compared: basic, structured, and adaptive; the adaptive harness monitors the agent’s trajectory and applies rules such as diagnosing repeated failures, inspecting accumulated diffs, changing strategy after no progress, verifying messy passes, and flagging unrelated edits.
- The findings are limited to what the simulation can show: they reveal how harness rules behave under modeled agent profiles, not how real agents such as Claude Code or Codex would actually perform.
If you judge an AI coding agent only by whether its final patch passes the tests, you are missing most of the story. That is the finding from a new simulation study of roughly 50,000 runs, built on real Python repositories and real unittest suites, in which the agent's decision-making was simulated rather than driven by a live model. The experiment compared three harnesses — basic, structured and adaptive — and found that the cheapest-looking option was also the most misleading.
What the experiment actually measured
The study starts from a familiar benchmark shape: a real GitHub issue, a real codebase, and a patch that either fixes the problem or does not. The benchmark referenced has 2,294 problems drawn from 12 Python repositories. Its headline score is simple: did the patch solve the issue?
That score has a blind spot. It only looks at the final result. It does not care how many steps the model took to get there, how many commands it repeated, or how much unrelated code it touched along the way.
The author separated two things that a pass rate blends together. Final patch size is what differs between the starting repository and the submitted patch. Total work is everything the agent did before arriving there, including edits it later reverted. A pass rate scores only the first.
The example given is blunt: one run finds the failing test, tracks down the bug, makes the change and gets a green run. Another gets there too, but first edits the wrong file, backs the change out, reruns the same command several times, modifies unrelated code, and then runs tests that were already passing. Both count as successes. They do not look alike.
Why the agent was simulated
The author had two hard constraints: no API keys and no downloaded models, plus a desire for reproducibility. There was a methodological reason too. Running a real agent would also measure differences in the model, its tool use and its surrounding environment, plus API costs — making it hard to tell whether a result came from the workflow under test or from the particular agent chosen.
So only the agent's decisions were simulated, using a seeded Python policy. The repositories, edits, diffs, test runs and failures are real. The code actually runs. What was simulated is the part that decides what to do next.
This limits the claims. The study can test how a harness rule behaves when the agent behaves in a particular way. It cannot say how Claude Code or Codex behaves, because neither was run.
The tasks and behaviour profiles
Ten small bugs were created, each in its own temporary repository. They cover an off-by-one range, a boundary comparison, an empty-list crash, a wrong formula, a missing `strip`, lost ordering, a slice error, a wrong variable, swapped branch order and a whitespace split.
Each repository has a unittest suite and a known reference fix, plus a plausible wrong fix in the correct file and harmless changes in unrelated files as decoys. Tests run in a subprocess, so the agent faces a real repository rather than a list of predefined answers.
The base policy is ordinary: run tests, look at the failure, make a change, run tests again, stop when things pass. Four profiles add fixed probabilities of deviation:
| Profile | Wrong edit | Repeats failing command | Reverts | Quits early | Unneeded edit after pass |
|---|---|---|---|---|---|
| careful | 0.05 | 0.05 | 0.02 | 0.03 | 0.05 |
| normal | 0.15 | 0.15 | 0.08 | 0.08 | 0.15 |
| sloppy | 0.30 | 0.30 | 0.20 | 0.18 | 0.30 |
| loop_prone | 0.10 | 0.55 | 0.15 | 0.05 | 0.40 |
These numbers are hypothetical. They were not derived from measuring real agents, so the absolute values are not predictions of what any live coding agent would do. Two further controls matter: compliance sets how often the agent follows a harness instruction rather than ignoring it, and plan_benefit sets how much a forced planning step reduces the chance of a wrong edit.
Runs are paired — the agent's random stream is tied to profile, task and seed — so comparisons start from the same agent behaviour. Success is checked after the run against the final repository state, using a check the agent never sees.
What counts as wasted work
Not every failed test is waste. A failed test can be part of debugging. An action counts as wasted when it does not move the run closer to the final state. That includes repeating a command without any repository change, rerunning a test when no relevant file has changed, reading the same contents again, editing a part of the repository unrelated to the tests, making an edit that changes nothing, or making a change that is later undone.
A revert is treated carefully. The revert itself is not counted as waste if it was needed to recover; the bad change is what counts. Harness work — planning, review, gates, cleanup — is counted separately as harness turns, so the cost of added structure stays visible.
The recorder stores only what happened. An analyzer works out waste afterwards, so the definition can change without rerunning every experiment. "Relevant" means a file the test suite actually imports, which the harness can infer from the tests without knowing the correct fix.
The three harnesses and the five rules
The basic harness lets the agent work until it says it is done. The structured harness forces plan, implement, test and review. The adaptive harness watches the trajectory, steps in with five rules, and refuses to let the agent finish while tests fail.
The five adaptive rules were locked before the runs: R1, if the same command failed twice with no state change, diagnose before editing again. R2, if the same file was modified twice or two reverts happened, inspect the accumulated diff. R3, after three turns with no state change, change strategy. R4, if tests pass after a messy run, verify by reviewing the diff. R5, if an edit touches a file the tests do not import, justify it or undo it. A gate refuses to let the agent finish while tests fail. Two of the three harnesses use that gate.
Result 1: the cheapest harness was misleading
The main comparison used 20 seeds, 10 tasks and 200 runs per cell at compliance 0.5.
| Profile | Harness | Success | Turns | Wasted turns | Harness turns |
|---|---|---|---|---|---|
| careful | basic | 92.5% | 5.20 ± 0.19 | 0.29 ± 0.11 | 0.00 |
| careful | structured | 100.0% | 7.66 ± 0.20 | 0.44 ± 0.14 | 2.12 |
| careful | adaptive | 100.0% | 5.76 ± 0.26 | 0.28 ± 0.10 | 0.42 |
| normal | basic | 77.5% | 5.25 ± 0.27 | 0.63 ± 0.16 | 0.00 |
| normal | structured | 100.0% | 8.82 ± 0.33 | 1.31 ± 0.25 | 2.35 |
| normal | adaptive | 100.0% | 6.62 ± 0.34 | 0.74 ± 0.17 | 0.86 |
| sloppy | basic | 51.5% | 5.62 ± 0.41 | 1.44 ± 0.27 | 0.00 |
| sloppy | structured | 100.0% | 11.91 ± 0.62 | 3.15 ± 0.37 | 3.10 |
| sloppy | adaptive | 100.0% | 9.95 ± 0.68 | 1.97 ± 0.29 | 2.83 |
| loop_prone | basic | 89.0% | 7.45 ± 0.46 | 2.46 ± 0.37 | 0.00 |
| loop_prone | structured | 100.0% | 11.59 ± 0.56 | 3.85 ± 0.43 | 2.48 |
| loop_prone | adaptive | 100.0% | 9.04 ± 0.56 | 2.29 ± 0.31 | 1.97 |
At first glance the basic harness looks like the winner. It has the fewest turns in every profile, because it never spends turns on planning, review or gates. But its success rate collapses as the agent gets sloppier: 92.5% for careful, 77.5% for normal, 51.5% for sloppy, and 89.0% for loop_prone. The structured and adaptive harnesses reach 100.0% in every cell.
That is the first result: a harness can look cheapest on turns while quietly failing on a large share of runs. If you only read the turn count, you would pick the wrong one.
What the structured harness costs
The structured harness buys its 100.0% success rate with harness turns: 2.12 for careful, 2.35 for normal, 3.10 for sloppy, 2.48 for loop_prone. Those are turns the agent did not choose, added by the workflow. Its total turns are the highest in every row, and its wasted turns are also the highest — 3.85 for loop_prone against 2.29 for adaptive.
The adaptive harness sits between the two. It reaches the same 100.0% success rate as structured, but with fewer harness turns in every profile: 0.42, 0.86, 2.83 and 1.97. Its wasted turns also stay below structured's in every profile. Compared with basic, it spends a modest number of extra turns to close the success gap.
The design distinction comes from Anthropic's framing: a workflow has the steps worked out in advance, while an agent decides what to do next, and it is usually better to start simple rather than add machinery upfront. The structured harness tells the agent what to do. The adaptive harness waits for certain things to happen and then reacts.
Similar ideas exist in tools people already use. Claude Code has hooks that run a command at a particular point in a session, including formatting a file after an edit or stopping a command before it runs. SWE-agent found that the setup around the model matters: how it edits files, moves around the repository and runs tests can affect the result. The adaptive harness here is much smaller than either system — just a few fixed rules reacting to a simulated agent.
The prediction that was wrong
The author made a prediction before running the experiment, and it turned out to be wrong. That admission is worth noting because it is the kind of detail that usually gets edited out of benchmark write-ups. The article does not restate the prediction in the excerpt provided, but it flags the miss as one of the standout outcomes alongside the three main results.
Why this matters if you run coding agents
Two practical points fall out of the data.
First, pass/fail is not enough. A run that passes on turn 4 and then spends turns 5 through 10 on work nobody asked for is scored identically to a clean five-turn run. In the opening example, the agent changed eight lines across three files while the reference fix was two lines in one file — six of the eight changed lines were unnecessary. Extra work can survive into the final patch, where it becomes someone's review burden and future maintenance cost.
Second, the harness changes the outcome. The same simulated agent behaviour produced 51.5% success under the basic harness and 100.0% under structured and adaptive in the sloppy profile. Whether the agent is allowed to declare victory while tests fail is not a detail; it is the difference between half the runs failing and none of them failing.
The caveat is the one the author repeats: the agent is simulated. The numbers describe how harness rules interact with a seeded policy, not how Claude Code or Codex performs. The sweeps across profiles exist partly to show how much results move when the assumptions move.
The next question is the obvious one. If a few fixed rules can close a success gap this large in simulation, the same rules are cheap to try against a real agent on a real repository — and the recorder-plus-analyzer design means the waste definition can be adjusted without rebuilding the whole experiment.
Comments (0)