Software Engineering Duel: Toward a Self-Scaling Arena for Coding Agents

Meta description
SWE-Duel coding-agent benchmark has Red and Blue agents create and solve bug tasks at evaluation time, ranking GPT-5.5 and Claude Opus 4.8 over 1,500 turns.

Tags
coding agents, SWE-Duel, benchmark, software engineering

Featured image
Abstract friendly illustration of two stylized code agents passing a sealed envelope of tests across a split repository, no real people, no logos, no text.

SWE-Duel is a coding-agent benchmark built for developers and researchers who want software-engineering tests that do not go stale. Instead of relying on a fixed list of tasks, it has coding agents create tasks for each other at evaluation time. The experiment covers 11 model-harness pairs, meaning a model plus its agent execution tool, over 1,500 turns on 15 repositories in five languages.

Static benchmarks leak and saturate

The core problem is simple: static coding benchmarks do not last long. Their tasks can leak into training data, and stronger agents can saturate them by solving all available items. Once a benchmark is saturated, it may no longer separate capable agents.

SWE-Duel changes the setup by generating tasks during evaluation. Because the tasks are created in the evaluation run itself, there is nothing to leak in advance. The source says difficulty grows with the agents themselves, rather than being fixed in advance.

Red writes a feature, Blue reviews a hidden bug

Each duel uses two agents on a real repository. They take turns as Red and Blue.

Red starts by adding a new feature. Then Red hides a subtle bug in that feature and prepares sealed tests that trigger it. The task only counts if a fresh session of the same agent can fix the bug blind. That rule is meant to avoid unsolvable tasks: if Red cannot produce a task that a fresh version of itself can solve, the task does not count.

Blue then reviews the pull request without being told there is a bug. Blue wins only if its revision passes three checks: the feature tests, the existing tests, and the hidden bug tests. After that, Red and Blue swap roles.

The design tests two related but different skills at once. Red must create a hard but fair task. Blue must detect and fix a problem it has not been warned about.

GPT-5.5 leads solving, Claude Opus 4.8 leads task writing

Across the 1,500 turns, the results highlight two different strengths. GPT-5.5 plus Codex CLI ranks first as the strongest solver, fixing 98.3% of the bugs it reviews. That high rate suggests the combination is very effective at catching and correcting hidden issues in this benchmark format.

Claude Opus 4.8 plus mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate. A Red win rate of 27.3% means that, when this pair wrote the task as Red, Blue failed to pass the full set of tests 27.3% of the time. It does not mean the task was unsolvable, because the benchmark only counts tasks that a fresh session of the same agent can fix blind.

Difficulty appears to scale with model generations

The source also reports a family-level comparison. Within one model family, tasks written by the newest version beat a fixed reference solver far more often than tasks from older versions: 42.9% versus 4.0%. This suggests that newer models may be able to write tasks that are more difficult for a static solver to solve.

For readers, the practical takeaway is that SWE-Duel is not just another leaderboard. It is an evaluation method that tries to keep coding-agent tests fresh by making the agents the task creators. The next questions are how well this format works beyond 11 model-harness pairs, 15 repositories, and five languages, and how much the Red and Blue roles can reveal about real software-engineering behavior.