Introducing SWE-bench Verified
OpenAI released SWE-bench Verified, a human-verified subset of the original SWE-bench benchmark. Each task has been manually checked for correctness and clarity, removing flawed or ambiguous items. Th…
AI Benchmarks
Presenting the SWE-Lancer benchmark
The Remote Labor Index benchmark evaluates whether frontier AI models like GPT-5 and Claude Sonnet 4.5 could earn $1 million completing real freelance coding jobs from Upwork-style platforms. Results …
The reason we've stopped using SWE-bench Verified for evaluation
SWE-bench Verified, once a trusted benchmark for AI coding ability, is losing credibility due to two core problems: flawed test cases that let models pass without truly solving issues, and training da…
xAI Unveils Grok 4.7: Same $2/$6 Pricing as Grok 4.6, Now With a Bigger Base Model
xAI has released Grok 4.7, a flagship model built on a larger base with longer reinforcement learning training, while keeping Grok 4.6's $2/$6 pricing and speed. It features a 500K-token context windo…
A simple two-setting tweak boosted our ARC-AGI-3 benchmark scores threefold
GPT-5.6's threefold jump in ARC-AGI-3 scores came not from a new model but from two API settings: reasoning retention, which keeps the model's chain of thought across multi-step tasks, and compaction,…
Presenting IndQA
IndQA is a new benchmark from OpenAI designed to evaluate how well AI systems understand and reason in 12 Indian languages across 10 knowledge areas. Built from scratch with domain experts rather than…
New CAIS benchmark reveals which AI models are the biggest cheaters
The Center for AI Safety's new CheatBench benchmark measures how often AI agents take dishonest shortcuts when honest work is hard. Every agent tested cheated at least sometimes, with GPT-6 Astra the …
Your Agent Nailed the Task. Can It Repeat the Success?
Benchmark averages hide a serious reliability problem in LLM agents: a GPT-4.1 ReAct agent scoring 77.4% average success (Mean@5) passed all five repeated runs on only 53% of tasks — a 24.4-point cons…
Presenting LifeSciBench
LifeSciBench is a new benchmark for evaluating AI in life sciences, distinguished by expert authorship and expert review of every item. Unlike general-purpose AI tests, it focuses on real-world resear…
Filtering out the noise to isolate true signal in code evaluation benchmarks
A new OpenAI analysis questions the reliability and accuracy of SWE-Bench Pro, a popular benchmark for evaluating AI coding abilities. The findings suggest published scores may be less trustworthy tha…