The Software Engineering CoWork benchmark assesses coding agents as digital employees within organizations.
Software Engineering CoWork tests coding agents in simulated workplaces; eleven models pass tests but rarely complete tasks, delivery gaps remain.
Key Takeaways
- SWE-CoWork evaluates coding agents as digital employees in simulated engineering organizations, using 101 real-repository tasks across 11 programming languages and 12 domains.
- The benchmark measures end-to-end delivery rather than isolated coding: agents must discover requirements from colleagues and records, reconcile conflicting evidence, and pass private executable tests.
- Leading models satisfy about 63% of fail-to-pass tests but complete fewer than 8% of assignments, showing a large gap between partial implementation and full task delivery.
- Giving agents complete requirements improves some test results but does not increase full task success, indicating that follow-through, verification, and integration remain major bottlenecks.
- Trajectory analysis reveals repeated breakdowns in asking questions, tracking requirements, implementing changes, and validating results, often causing agents to miss colleague-held information or report incomplete work as finished.
For anyone evaluating a coding agent benchmark, SWE-CoWork gives a clearer picture than a simple pass rate: it measures whether an agent can discover what to build, work through a simulated workplace, and deliver a change that passes private executable tests. The benchmark contains 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains, and it evaluates eleven models.
What SWE-CoWork measures
Each assignment starts from a brief handoff. The agent is placed in a simulated engineering organization with 26 colleagues. Those colleagues have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. To finish the task, the agent must recover requirements that are spread across colleagues and workplace records, reconcile conflicting evidence, and implement changes that will be checked by private executable tests.
This setup shifts the test from isolated coding to delivery. A model may write useful code, but the benchmark asks whether it can identify the full requirement, find the right owner, and complete the task within the organization's knowledge boundaries.
What the eleven models showed
Across eleven models, the results show a large difference between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass tests, but they complete fewer than 8% of assignments. In other words, the agents can often make some of the required behavior work, yet they rarely satisfy the whole task.
That distinction matters for readers who compare models. A high test-satisfaction rate can suggest progress, while the low assignment-completion rate shows that the remaining work is not merely a small polish step.
Why full requirements did not close the gap
The benchmark also tested what happens when agents receive complete requirements upfront. For Seed Evolving and GPT-6 Astra, providing complete requirements improves fail-to-pass performance by approximately 16 percentage points. However, complete task success remains unchanged.
This result suggests that missing requirements are only part of the problem. Even when the agent knows what should be built, it does not reliably turn that information into a finished, verified task. The gap appears during follow-through, verification, or integration rather than only at the moment the requirement is discovered.
How the failures appeared in trajectories
Requirement-level trajectory analysis reveals failures throughout the delivery process. Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs, but it obtains only 64% of colleague-held requirements.
The analysis also shows deferred questions going unfollowed. Implementations remain incomplete even after requirements are acquired. In addition, 31% of requirements reported as completed fail their tests. These details point to a chain of small breakdowns: asking, tracking, implementing, and validating do not line up reliably.
Why this matters for coding agents
SWE-CoWork frames coding agents as workplace participants, not just code generators. The benchmark's value is that it separates three outcomes: a partial test improvement, a recovered requirement, and a completed task. Readers can use that separation when judging whether a model is ready for end-to-end software work.
The pattern from the eleven-model evaluation is that the hardest part may not be producing code. It is carrying an incomplete request through colleagues, records, conflicting evidence, and final tests without losing track of what remains.
Meta description: SWE-CoWork tests 101 coding agent tasks in real repositories and shows a gap between partial implementation and complete delivery across eleven models.
Tags: SWE-CoWork, coding agent benchmark, software agents, model evaluation, repository tasks
Featured image: abstract friendly illustration of a coding agent navigating 101 repository tasks among simulated colleague cards, no real people, no logos, no text.

Comments (0)