Introducing SWE-bench Verified
OpenAI released SWE-bench Verified, a human-verified subset of the original SWE-bench benchmark. Each task has been manually checked for correctness and clarity, removing flawed or ambiguous items. This curated dataset provides developers and researchers with a more trustworthy way to compare AI models' ability to fix real software bugs, reflecting genuine problem-solving skills rather than artifacts of bad data.
OpenAI has rolled out a new, human-verified subset of SWE-bench designed to give a more trustworthy picture of how well AI models can fix genuine software problems.
The original SWE-bench benchmark has been widely used to measure how models handle real issues drawn from actual code repositories. However, its dataset contained items that were flawed or ambiguous, making it harder to draw solid conclusions from scores. This curated subset addresses that weakness by having people manually check each task for correctness and clarity.
The result is an evaluation set where every problem has been validated by human reviewers, so scores on it reflect genuine problem-solving ability rather than artifacts of bad data. Developers and researchers can use this refined benchmark to compare models on realistic software engineering work with greater confidence in what the numbers actually mean.
In short, this release makes it easier to answer a simple question reliably: can an AI model actually resolve a real bug the way a competent engineer would?
Comments (0)