The Co-Design Benchmark evaluates multimodal collaborative coding for agentic models.
Co-Design Bench evaluates coding agents on 273 multimodal app editing tasks, using acceptance criteria and evidence to expose partial success gaps.
Key Takeaways
- Co-Design Bench evaluates coding agents by testing whether they can revise real, running applications to meet explicit user acceptance criteria.
- It includes two tracks: Multimodal Editing for local changes guided by screenshots or media, and Repository Editing for connected changes across an app’s components, routes, state, and data flow.
- The benchmark uses a judge-then-refute protocol that checks hidden criteria with source code and browser evidence to catch stubs, hardcoded outputs, or incomplete features.
- Results show that strong models can pass many individual criteria while still fully completing only a small share of tasks, especially when requests require broader coordinated repository edits.
Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models is a benchmark for coding agents that must revise real applications from user requests. It is useful for developers and AI researchers because it checks whether a running application satisfies explicit acceptance criteria.
Real user requests set the starting point
The benchmark was built from 26,314 real user coding sessions and 37,463 deduplicated queries. It contains 273 tasks and 2,331 expert-authored acceptance criteria. Every task begins with an application that can run. The agent must interpret the user's intent, apply the needed edits, and meet all acceptance criteria.
This setup evaluates changes inside a connected application rather than a single snippet. A change in one place may affect text, layout, state, data flow, or totals elsewhere.
Two tracks cover localized and repository-wide edits
Co-Design Bench has two complementary tracks. The 100 Multimodal Editing tasks require local changes based on annotated screenshots, and they may include image, audio, video, or document inputs. The 173 Repository Editing tasks ask for connected changes across components, routes, state, data flow, and content.
Examples include recreating different text animations seen in a video, fixing a dashboard using a receipt and updating totals that depend on it, or coordinating layout and interaction changes across an app. Task types include Grounded Understanding, UI Transformation, and Functional Implementation, and one request may combine several of these capabilities.
Judge-then-refute protocol looks for working features
Evaluation follows a reference-free judge-then-refute protocol. It checks hidden acceptance criteria with source code and browser evidence. A second verifier questions any positive judgment that may rely on stubs, hardcoded outputs, unreachable functionality, or requirement mismatches.
For visual claims, evidence audits require screenshots; for interaction claims, they require recorded browser actions. The benchmark does not require a gold implementation or target screenshot, which keeps the focus on whether the delivered application behaves as requested.
Results show partial success can hide incomplete delivery
Testing used 10 model endpoints in a shared agentic harness. Among the endpoints, gpt-6-astra performed best, passing 69.9% of criteria per task on average. However, it fully satisfies only 26.4% of tasks. That gap matters: an agent can pass many checks while still missing the complete revision a user asked for.
Qwen3.8-27B records 64.4% overall strict completion, placing fourth overall. It scores 64.5% on Repository Editing, second on that track in this evaluation, and tops the text-only Multimodal Editing subset at 77.9%. The source notes that rankings vary with the editing setting, and the small repository-track score differences do not establish a general model advantage.
For every endpoint tested, criterion completion falls lower on repository requests that span three or four edit types than on requests that span one or two. This suggests that broader coordinated changes remain harder than more localized edits.
A separate subset keeps evaluation scope clear
The ME-96 subset has a different scope: it contains 96 Multimodal Editing tasks and 521 acceptance criteria, separate from the full 273-task paper evaluation. Builders reading results should check which task set is being discussed before comparing scores.
The release points to project and demos, a paper, code and evaluation, and a dataset. It welcomes researchers and builders working on coding agents, multimodal grounding, and reliable application editing to try the benchmark and share findings.
Meta description: Co-Design Bench evaluates coding agents on 273 real multimodal application editing tasks with acceptance criteria and working app checks for builders.
Tags: coding agents, multimodal editing, application editing, benchmark, AI evaluation
Featured image: An abstract friendly illustration of a chat window, a screenshot, a video clip, and code files connecting to a working app interface, no real people, no logos, no text.

Comments (0)