Model Comparisons

This ablation of a local model in .NET asks whether the harness changes the outcome or only the cost.

Agent harnesses may change AI coding costs far more than results; .NET ablation and 2026 studies show cost and outcomes should be judged separately.

Key Takeaways

  • Agent harnesses can change token cost by up to ~40x while producing only small differences in solved-task rates.
  • Harnesses should be judged on two separate axes: result quality, meaning tasks solved, and cost, meaning tokens, time, or money spent.
  • Some harness effects are real but uneven: pass rates may be similar overall, yet a harness can strongly improve one task category and produce consistent failure patterns.
  • Benchmark designers increasingly treat harnesses as confounders, often fixing the harness to measure model capability more cleanly.
  • Harness design and model training are intertwined: a better harness can lift weaker models, while poorly matched fine-tuning or post-training can reduce agent performance.

Agent Harnesses Can Change Cost 40 Times Without Moving Results

A writer who published two pieces arguing that the agent harness, not the model, is the new bottleneck in coding agents is revising that claim after reviewing February 2026 counter-evidence from METR, Sentient Labs, DeepSWE, and practitioner reports. The writer also built a local .NET ablation rig to test which harness pieces move the number of solved tasks and which pieces only move the token bill. For readers evaluating AI agents, the main lesson is to judge a harness on two axes, result and cost, and not infer one from the other.

Two axes, one correction

The trigger was Jaroslaw Wasowski’s Level Up Coding article, “Harness Engineering: When Agent Harness Moves the Result and When It Only Moves the Cost.” The article is member-only, and the writer could read only the opening, so the primary sources were checked directly. Wasowski’s central point was that a harness should be judged on two separate axes: whether it changes the result, meaning more tasks are actually solved, and whether it changes the cost, meaning more tokens, minutes, or money for the same solved tasks. He also revised his earlier view that the harness is a moat worth hiding behind.

METR’s February 2026 research note, “Measuring Time Horizon Using Claude Code and Codex,” tested whether vendor harnesses beat METR’s own default scaffolds when measuring how long a task a model can complete. METR’s defaults are ReAct, a plain act and observe loop, and Triframe, which uses advisor, actor, and rater subagents to choose each action. The note reported that Opus 4.5 with Claude Code beat Opus 4.5 with ReAct in 50.7% of bootstrap samples, which is close to a coin flip. GPT-5 with Codex beat GPT-5 with Triframe in only 14.5% of bootstrap samples, and neither direction was statistically significant. Raising the Opus 4.5 token budget from 8 million to 32 million for the Claude Code runs barely moved the preliminary time horizon.

METR describes the note as a research note, not a vetted publication. It covers two models. It also flags a mismatch: the vendor harnesses are built for interactive use with a human, while the evaluation was fully autonomous. Wasowski’s intro places a token gap of up to 40 times next to the METR finding. The writer could not find a 40-times figure inside the METR note and traced that number to the next paper, while noting that Wasowski’s full text may attribute it more precisely.

Same tasks, wildly different bills

The clearest experiment in the debate is “The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation” by Naman Vats and Oleg Golev at Sentient Labs, arXiv 2607.22585. It is preliminary work under review at the DL4C workshop at ICML 2026. The study compared three harnesses, Goose, OpenCode, and OpenHands-SDK, across two models, Qwen 3.6 Plus and MiniMax M2.5, through OpenRouter. It used 50 tasks sampled from Terminal-Bench Pro across eight categories, for 300 trials in total, with the same sandbox and a 900 second wall-time cap.

The pass rates sat between 38% and 50%. Within one model, the spread across harnesses was 2 points for Qwen and 8 points for MiniMax. Most pairwise differences were not distinguishable from zero. Tokens per solved task differed by roughly 40 times between harnesses. Goose used around 28,000 tokens per solve for Qwen and 37,000 for MiniMax. OpenCode used around 1.15 million and 1.55 million. Upgrading the model changed token cost by only 1.0 times to 1.3 times. In other words, the harness moved the bill far more than the model did.

Two secondary findings matter more than the headline. First, failure fingerprints were properties of the harness and stayed consistent across both models. Goose tended to stop when stuck, OpenHands-SDK pushed toward closure and sometimes submitted plausible wrong answers, and OpenCode tended to run into the time cap. Second, on the IMPL category, OpenHands-SDK solved 69% against 38% for the other two. The averaged pass-rate band hides at least one case where the harness moved the result, but only for one kind of work. The authors also note that with 50 tasks, every task is worth 2 points, so pass-rate differences should be read as descriptive.

Benchmarks that pin the harness still

DeepSWE, by Datacurve, arXiv 2607.07946, is a benchmark of 113 long-horizon tasks across 91 open-source repositories. Its relevant design choice is that every model is evaluated inside the same mini-swe-agent harness, with the same bash tool and the same prompt, explicitly so the leaderboard reflects model capability rather than scaffolding. The authors acknowledge the trade-off: real developers use Codex CLI or Claude Code, not mini-swe-agent.

That choice is telling. Benchmark builders decided the harness is enough of a confounder that it must be held still to measure anything else. That is not the same as saying the harness does not matter. It means the harness matters enough that it has to be fixed to see the model.

Wasowski’s companion piece, “Harness Engineering: Model-Harness Fit: Why a Better Teacher Makes a Worse Agent,” summarizes a Salesforce preprint cited as arXiv 2609.09134. The writer could not locate that preprint directly, so the numbers are secondhand through Wasowski’s summary. As described, a harness built around a weaker model lifted its success on seven agentic tasks from 29.2% to 78.0% without touching the weights. Fine-tuning the same model on a stronger expert’s full trajectories dropped it to 63.1%. Correcting single bad turns instead, one per failed rollout, brought it to 79.7%.

A separate ICML 2026 paper, “The Interplay of Harness Design and Post-Training in LLM Agents” by Kyungmin Kim and co-authors, reaches a compatible conclusion on an ALFWorld-based benchmark: harness and post-training cannot be separated, a harness bolted on after training recovers little, and for the smaller model the harness was a prerequisite for post-training to help at all.

Put those results next to the Scaffold Effect and a more careful picture forms. Swapping one reasonable harness for another, on a capable model, mostly moves cost. Building a harness that fits a specific, weaker model can move the result enormously. Those are different claims, and the writer had been using evidence for the first claim while arguing the second.

Practitioners measured their own setups

Two practitioner pieces from the same weeks did something more useful than argument: they measured their own systems and reported where the results disappointed them. Both are member-only, and the figures below are author-reported from their own runs.

Fareed Khan’s “Fixing Over-Engineering in LLM Coding Agents” built a proportional-code system around Qwen3.8–27B, aimed at agents that write far more code than a ticket needs. His opening example is a request that needed one database constraint and received 56 new lines, including a DELETE command that silently removes members. Across 34 trap tickets, an unguided agent’s passing answers added a median of 1.5 times the reference code, which the intro calls typical. With his full system, on 25 held-out tickets, that dropped to 0.88 times. Of the 11 pieces he designed, only 4 shipped, and his first hypothesis missed. Seven of eleven pieces did not earn their place. That is what an honest ablation looks like when run.

Pasquale Molinaro’s “I Had a Free Qwen3.8–27B Agent Copy 10 Web Pages. It Spent 64% of Its Time Measuring Pixels” ran Qwen3.8–27B at Q4_K_M on two free Kaggle T4s. The task was rebuilding ten pages from the Design2Code benchmark under three conditions: a one-shot copy, a one-shot copy plus one revision, and an agent that starts from the one-shot page with a renderer and 20 minutes per page. The agent was the only condition that could look at its own output, so the writer expected it to win clearly. Instead, the agent issued 237 measuring commands, and 64% of its 169 minutes went to measuring pixels. Most of the score gain traced back to a single page, through a change that cannot even be seen in a browser.

That last point matters for readers. An aggregate score went up, and the honest explanation was one task and an invisible change. If the author had stopped at the aggregate, the result could have been published as a harness win.

Re-reading the original harness bottleneck claims

The writer’s first piece, “Harness Engineering Is the New Bottleneck,” used LangChain moving a fixed model from 52.8% to 66.5% on Terminal-Bench 2.0 by rebuilding the harness as its strongest evidence. It flagged that this was one receipt, not two. What it did not ask was what kind of harness change it was. In light of the model-harness fit papers, that result reads less like harnesses beat models and more like one team fitted one harness to one model and one benchmark.

The eval loop piece already contained both axes, but the writer did not name them. A semantic search tool kept the pass rate flat and cut tokens by 41%. That was a pure cost move. A self-review step moved the pass rate by 13 points and held at 68% on a rerun. That looked like a result move. But the writer scored 12 issues and did not pair the comparison by issue, so each issue is worth about 8 points. The writer still expects the review step to be real because it gives the model information it did not have before, such as test output plus a fresh look at the diff, but would no longer state it as confidently.

The revised position is that harness pieces reliably move cost and only conditionally move the result. The working guess is that a piece moves the result when it feeds the model a signal it could not get otherwise, such as a compiler error, a failing check, or a rendered page. It mostly moves cost when it adds ceremony, such as a plan, another loop, or more room to wander.

What the local rig will test

The next step is a small .NET ablation rig built against a local model through Ollama or llama.cpp. The rig uses no software development kit. It talks to an OpenAI-compatible /chat/completions endpoint with a plain HttpClient, including an endpoint exposed by Ollama on port 11434. The goal is to switch individual harness pieces off and see which ones move the result and which ones only move the bill.

If harness pieces only move cost on a small local model, the strongest version of the original claim is in trouble. If they move the result, the question becomes which pieces provide signals the model could not otherwise access. For readers choosing agent stacks, the practical takeaway is to measure result and cost separately, run ablations on their own tasks, and treat aggregate score gains as hypotheses until the underlying task-by-task evidence is visible.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.