Filtering out the noise to isolate true signal in code evaluation benchmarks
A new OpenAI analysis questions the reliability and accuracy of SWE-Bench Pro, a popular benchmark for evaluating AI coding abilities. The findings suggest published scores may be less trustworthy tha…