Benchmarks for coding agents should mirror the task workflows of their users.
Real coding agent sessions mix planning, execution, and review. SWE-TaskFlow aligns benchmarks to these workflows; multi-step interaction raises cost.
Key Takeaways
- Real coding-agent sessions in JetBrains IDEs involve varied task types and frequent switches, unlike discrete issue-derived benchmark tasks.
- No single interaction distribution is universally realistic; benchmark creators should define a target use case and calibrate against real session Task Flows.
- SWE-TaskFlow reshapes existing benchmarks into session-like interactions through prompt splitting and repository quality checks while preserving verified tasks and tests.
- A pilot on SWE-Bench Pro shows step-by-step interaction can roughly double agent cost without reliably changing resolve rates, making workflow realism central to evaluation.
For developers and teams working on coding agents evaluation, a study of 4,782 agent sessions in JetBrains IDEs shows how real software engineering interactions differ from issue-derived benchmark tasks and why SWE-TaskFlow can help align tests with actual use.
Real sessions reveal a wider mix of tasks
Coding agents evaluation generally tries to be as realistic as possible. The study collects agent sessions from real software engineers in JetBrains IDEs. It calls this set Production Sessions. Because the subject is interactive agents, the analysis studies sessions with at least three user messages. That subset is 33% of the sample.
These longer sessions differ from issue-derived benchmark tasks in two ways. First, user requests span a much broader mix of task types: questions about the project's code, planning, review, refactoring, and execution. Second, users switch between those types throughout a session.
That pattern matters because many evaluations are built from discrete issue-derived tasks. Real sessions can start with a question, move into planning, shift to execution, and end with review. A benchmark that only measures whether a final task is solved may miss the interaction shape that users actually experience.
Why single benchmark distributions are not enough
Long-session samples from three public interaction corpora exhibit markedly different Task Flows. Here, Task Flows are the distributions of session lengths, task types, and type-to-type transitions.
The finding suggests no single interaction distribution is universally realistic. Different corpora show different ways that coding sessions evolve. For benchmark creators, the practical lesson is to name a target use case first, then calibrate the evaluation against measurements from that use case.
Instead of asking only whether an agent can solve a task, an evaluation should also ask what kind of interaction pattern the task is meant to represent.
SWE-TaskFlow reshapes tests without changing tasks
The study presents SWE-TaskFlow, an approach for transforming any issue-derived benchmark. It preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository quality assurance.
Prompt splitting can turn one broad user request into several prompts, creating a more session-like path. Verifiable repository quality assurance helps keep the interaction grounded in the repository while moving toward the desired flow.
The approach also uses a TaskFlow Alignment Score for selecting among generated trajectories. The score helps choose interaction paths that match the target Task Flow. It gives benchmark builders a way to keep the original test content intact while changing how users and agents move through the session.
Pilot shows interaction protocol changes cost
The study includes a pilot on 700 SWE-Bench Pro tasks. It tests what happens when agents solve the task sequentially in several steps rather than in one interaction.
The result is straightforward: solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate. The extra steps raise the work required to complete the task, but they do not reliably make the task easier or harder to resolve.
This places the interaction protocol itself as an important dimension of evaluation. Two benchmarks may use the same tasks and tests, yet require very different interaction paths and cost. If an evaluation ignores the session pattern, it may understate what it takes to use an agent in a realistic workflow.
What this means for future evaluations
For readers who care about coding agents, the takeaway is practical. Realistic evaluation needs more than harder tasks or more tests. It needs a clear target use case, measurements from real sessions, and a way to preserve verified task content while reshaping interaction.
SWE-TaskFlow points toward that middle ground. It keeps issue-derived benchmarks usable, but calibrates them toward the task flows that software engineers show in production-like sessions. The pilot on 700 SWE-Bench Pro tasks also gives a concrete warning: changing the interaction protocol can double agent cost even when the resolve rate remains broadly stable.
Meta description: JetBrains IDE study of 4,782 coding agent sessions shows SWE-TaskFlow can calibrate benchmarks toward realistic user interaction task flows.
Tags: coding agents, SWE-TaskFlow, JetBrains IDEs, software engineering evaluation, benchmark calibration
Featured image: abstract, friendly illustration of coding agent sessions and task flow paths, no real people, no logos, no text.
Comments (0)