What Should Your Pipeline Return When the Expected Result Is Nothing?
Reliability guardrails can force LLM pipelines to invent answers when none is correct; abstention should be explicit, validated, and evaluated.
AI evaluation
The reasons behind hallucinations in language models
OpenAI research reveals that AI hallucinations are not random glitches but predictable outcomes of evaluation systems that reward guessing over admitting uncertainty. Like students guessing on multipl…
BenchMIRT: What Do LLM Benchmarks Really Measure?
BenchMIRT is a new auditing method for LLM benchmarks that applies multidimensional Item Response Theory (MIRT) at the individual prompt level to uncover which underlying capabilities—such as safety o…
Meet HealthBench
HealthBench is OpenAI's new benchmark evaluating how AI models perform in realistic healthcare scenarios. Developed with over 250 physicians, it tests accuracy, safety, context awareness, and communic…
Presenting LifeSciBench
LifeSciBench is a new benchmark for evaluating AI in life sciences, distinguished by expert authorship and expert review of every item. Unlike general-purpose AI tests, it focuses on real-world resear…
Filtering out the noise to isolate true signal in code evaluation benchmarks
A new OpenAI analysis questions the reliability and accuracy of SWE-Bench Pro, a popular benchmark for evaluating AI coding abilities. The findings suggest published scores may be less trustworthy tha…