WideSWE: Can Coding Agents Synchronize Modifications Across Multiple Repositories?
WideSWE evaluates coding agents on 120 real multi-repo tasks, showing low full success and failures in discovery, execution, and completeness.
Key Points
- WideSWE evaluates coding agents on 120 real tasks across 103 software ecosystems, showing that agents remain unreliable when changes must be coordinated across multiple repositories.
- Full task success ranged from 10.83% to 42.50% across seven configurations, with the best result coming from Codex CLI paired with GPT-5.6-sol.
- Agent failures follow three main patterns: missing necessary changes, leaving changes unfinished, and modifying the right repositories without fully satisfying the request.
- Joint cross-repository execution helps more than repo-by-repo execution, especially for verification, because related repository context can guide correct implementation.
- The benchmark suggests treating agent output as a first draft requiring cross-repo verification, while pointing to better planning and verification loops as key improvement areas.
A new benchmark called WideSWE shows that today's coding agents remain far from reliable when a single task requires coordinated changes across several code repositories. The benchmark, built from 103 software ecosystems, contains 120 real-world tasks split evenly between 60 bug fixes and 60 features. Across seven agent configurations, full task success ranged from 10.83% to 42.50%. The best result came from pairing Codex CLI with GPT-5.6-sol.
The work matters because most coding-agent evaluations still measure performance inside one codebase. Real software work often does not respect those boundaries. A feature may need a library change, a dependent service update, and matching tests in three different repositories before anything actually works.
What WideSWE measures and how it was built
The researchers mined and reviewed changes across 103 software ecosystems to assemble 120 tasks grounded in actual development history. Each task is balanced between bug fixes and feature requests, with 60 of each. Prompts were derived from related issues and pull requests rather than written from scratch, which keeps the tasks closer to what developers actually face.
A notable design choice involves the hidden tests. The team systematically reviewed and adapted these tests so that multiple correct implementations can pass, while still preserving required behavior and regression checks. That reduces the risk of penalizing a valid solution simply because it differs from the original patch.
Where agents break down
The reported success rates are the headline finding. Full task success across the seven configurations spans 10.83% to 42.50%, meaning even the strongest setup fails more than half the time. The highest rate belongs to the configuration combining Codex CLI with GPT-5.6-sol.
The trajectory analysis adds more useful detail than the single number. The researchers observed three recurring failure patterns:
- Agents fail to identify the changes that are actually necessary.
- Agents recognize the needed changes but leave them unfinished.
- Agents modify the required repositories without fully satisfying the request.
These are different problems, and they suggest different fixes. The first is a discovery failure. The second is an execution or persistence failure. The third is a completeness failure, where the agent touches the right places but does not close the loop.
Does one repository at a time help?
The team also tested whether working on one repository at a time can reduce these difficulties, comparing it with joint execution under identical prompts.
Independent execution mainly helped recover omitted work. It was less effective at correcting implementations that had already been attempted but did not succeed. Joint execution, by contrast, could use information from related repositories to guide both implementation and verification. That suggests cross-repository context is not just convenient but functionally useful for checking whether a change is correct.
What this means for developers and tool builders
For anyone using coding agents today, the practical takeaway is caution around multi-repo work. A task that looks simple in a prompt may require changes in several places, and the benchmark shows agents frequently miss necessary edits or stop short of completing them. Treating agent output as a first draft that needs verification across repositories is the safer approach.
For tool builders, the results point to two areas worth attention. Better cross-repository reasoning during planning could address the discovery failures. Better verification loops could address the completeness failures. The comparison between independent and joint execution hints that feeding related repository information into the process helps, particularly for verification.
The benchmark code is available, which gives other researchers a way to reproduce and extend the results. The next question is whether newer agent designs can push success rates meaningfully above the current 42.50% ceiling, and whether the failure patterns shift as models improve.
Comments (0)