BenchMIRT is a new auditing method for LLM benchmarks that applies multidimensional Item Response Theory (MIRT) at the individual prompt level to uncover which underlying capabilities—such as safety o…
HealthBench is OpenAI's new benchmark evaluating how AI models perform in realistic healthcare scenarios. Developed with over 250 physicians, it tests accuracy, safety, context awareness, and communic…
LifeSciBench is a new benchmark for evaluating AI in life sciences, distinguished by expert authorship and expert review of every item. Unlike general-purpose AI tests, it focuses on real-world resear…
A new OpenAI analysis questions the reliability and accuracy of SWE-Bench Pro, a popular benchmark for evaluating AI coding abilities. The findings suggest published scores may be less trustworthy tha…
