The reason we've stopped using SWE-bench Verified for evaluation

Meta description: Why SWE-bench Verified is losing reliability as a coding benchmark, and why many researchers now point to SWE-bench Pro instead.

Tags: AI benchmarks, SWE-bench, coding LLMs, model evaluation, software engineering

Featured image: A friendly abstract illustration of a cracked measuring ruler next to glowing code blocks, rendered in soft pastel geometric shapes.

SWE-bench Verified Is Contaminated: Why SWE-bench Pro Is Gaining Ground

If you follow AI coding models in any capacity—whether you're a developer choosing a model, a researcher tracking progress, or a writer trying to make sense of leaderboard claims—this article is for you. A growing body of analysis suggests that SWE-bench Verified, one of the most widely cited benchmarks for software engineering ability, is no longer giving us a trustworthy picture of how good frontier models actually are at real coding work. Below, we break down the two core problems: flawed tests and training data leakage. We also look at the proposed alternative, SWE-bench Pro, and what its rise means for how you should read benchmark numbers going forward.

What SWE-bench Verified Was Supposed to Measure

Benchmarks exist to answer a simple question: can this model actually do the job? SWE-bench, and later its curated subset SWE-bench Verified, tried to answer that question for software engineering. The format is appealingly concrete. A model is handed a real GitHub issue from a real open-source repository and asked to produce a patch that resolves it. The patch then runs against the project's test suite, and the model passes or fails based on whether the tests go green.

This was a genuine step up from earlier benchmarks that measured little more than whether a model could complete a function from a docstring. Real issues involve navigating an unfamiliar codebase, understanding existing conventions, and making changes that don't break anything else. For a while, SWE-bench Verified scores tracked reasonably well with what practitioners experienced when using models on real code.

The trouble is that the benchmark's environment has changed around it—and the benchmark hasn't kept up.

Problem One: The Tests Themselves Are Flawed

A benchmark is only as good as its grading. The analysis behind the growing skepticism points to problems with the tests used to validate patches in SWE-bench Verified.

Some test cases are weak in ways that let models pass without genuinely solving the underlying issue. A patch can satisfy a narrow assertion while missing the actual bug, or fail to handle edge cases a competent engineer would catch. When the grading instrument has gaps, high scores stop meaning what we assume they mean.

There's also a subtler issue: benchmark curation is hard at scale. SWE-bench Verified was created precisely because the original SWE-bench contained noisy examples—issues that were ambiguous, untestable, or dependent on quirks of a specific repository. The Verified subset filtered many of those out, but filtering is an imperfect process, and residual flaws remain. When a single imperfect test determines whether a model "solves" a task, small defects in that test can shift scores meaningfully across an entire leaderboard.

None of this means every failing test is broken or every passing patch is hollow. It means the measurement error is large enough that we should stop treating small score differences between frontier models as meaningful signals of real-world capability.

Problem Two: Training Data Leakage

The second problem is more structural: contamination. Most of the issues and repositories in SWE-bench come from public open-source projects. Those projects, their issue trackers, and their merged pull requests are all over the public internet—which means they are very likely sitting inside the training data of every major frontier model.

The consequence is uncomfortable. A model may not be solving a SWE-bench problem so much as recalling it. If the exact issue, or the actual commit that fixed it, appeared during training, the model can reproduce a solution that looks like reasoning but is closer to memorization. Researchers have proposed various detection methods for this kind of leakage, and the general finding across the field is that contamination is hard to rule out and plausibly widespread for public benchmarks that have existed for years.

This is not an accusation aimed at any particular model developer. Training on public code is normal and, in many cases, intended. The problem is that we then turn around and use that same public data as a measuring stick. A ruler that has been stretched to match the thing it measures can't tell you anything useful about length.

The practical upshot: recent jumps in SWE-bench Verified scores may overstate genuine progress in coding ability. Some of the improvement is likely real—models are visibly better at coding than they were two years ago. But the benchmark can no longer cleanly separate real skill from memorized answers.

The Case for SWE-bench Pro

This is where SWE-bench Pro comes in. It was designed with the weaknesses of the Verified benchmark explicitly in mind. The central idea is to reduce both measurement noise and contamination risk, so that scores reflect genuine problem-solving rather than test-taking.

While details of construction vary, the broad design goals behind SWE-bench Pro address the two failure modes described above: more carefully validated test cases, so that passing a task means the underlying issue was genuinely resolved, and a task pool constructed to be less exposed to the training corpora of current frontier models. The aim is a benchmark where the gap between "scored well" and "can actually do the job" is as small as possible.

The tradeoff is worth acknowledging. Newer, cleaner benchmarks have smaller ecosystems: fewer historical results to compare against, less familiarity among practitioners, and less continuity with older score trends. SWE-bench Verified has years of data behind it, which is exactly why it remains widely quoted even by people who no longer fully trust it. SWE-bench Pro's credibility will grow as more models are evaluated on it and its results continue to correlate with real-world performance.

How to Read Coding Benchmarks Today

Until the ecosystem settles, a few practical habits will serve you well:

• Treat benchmark scores as one signal, not the verdict. Pair leaderboard numbers with hands-on evaluation on tasks that resemble your actual work.
• Be skeptical of small differences. If two models are within a few points on SWE-bench Verified, that gap may be entirely within the benchmark's noise and contamination.
• Weight fresh, contamination-aware benchmarks more heavily over time. As SWE-bench Pro and similar efforts accumulate results, they deserve increasing trust.
• Watch for disclosure. Model providers and evaluators who openly discuss contamination risks and test validity are generally more credible than those who only cite their best number.

Who should use it / who should skip it

Who should pay attention to SWE-bench Pro: researchers comparing frontier models, teams making model-selection decisions where coding ability is central, and anyone publishing or reporting on AI progress who wants their claims to hold up over time.

Who can skip it for now: casual users who just want a good coding assistant. If your decision is "which model should I use in my editor," trying two or three candidates on your own codebase will tell you more than any leaderboard—contaminated or not.

The bigger takeaway applies to everyone: SWE-bench Verified's problems are a reminder that benchmarks are tools with expiration dates. When the measuring stick and the training data come from the same place, progress claims deserve a second look.