How the UK AISI and EvalEval Are Working to Ensure Reproducible Benchmark Results
The UK AI Security Institute and EvalEval are deepening their partnership by publishing verified, reproducible AI evaluation results through EvalEval's Evaluation Cards platform. AISI released benchmark data for models including GPT-5.4 and Claude Opus 4.6 across HealthBench, FrontierMath, SWE-Bench Pro, and more, alongside a new paper on how inference compute affects benchmark performance, promoting standardised, transparent evaluation reporting.
The UK AI Security Institute (AISI) and EvalEval are expanding their collaboration by putting shared reporting infrastructure into real-world use. For researchers, model developers, and policy analysts tracking AI capabilities, this partnership means easier access to verified, reproducible benchmark results. AISI has released detailed evaluation data—including results for frontier models like GPT-5.4 and Claude Opus 4.6—through EvalEval's Evaluation Cards platform, alongside its new paper on how inference compute shapes benchmark performance.
Why Reproducible Evaluation Reporting Matters
As AI systems are deployed faster and more widely, evaluations have become a key source of evidence about how models actually perform. But there's a problem: results get published across countless formats, platforms, and outlets, often with too little detail to reproduce them. And rerunning evaluations can be extremely expensive, making independent verification difficult.
EvalEval's mission is to fix this ecosystem. It offers a shared reporting schema called Every Eval Ever (EEE) and an open platform called Evaluation Cards, which bring evaluation results and the context needed to interpret them into one consistent structure.
This collaboration builds on work AISI has already done to strengthen the field—making evaluations more efficient with OptStop, more statistically rigorous with HiBayES, and more standardised in areas like transcript analysis and capability elicitation. Together, the two organisations are identifying where evaluation reporting falls short and building the shared infrastructure needed to close those gaps.
What AISI Is Making Public
Transparency at the transcript level matters for more than just reproducibility—it also supports deeper analysis and diagnosis. In this new phase of the partnership, AISI is publishing publicly reported evaluation methods and findings through Evaluation Cards where appropriate.
The release includes verified results, context, and configuration details for the five benchmarks featured in the paper's main experiment:
- HealthBench
- FrontierMath
- Humanity's Last Exam
- SWE-Bench Pro
- Terminal-Bench 2.0
These results span six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. Two related cyber evaluations—Cyber CTFs and The Last Ones—are also included, though they cover a different, partially overlapping set of models.
All of this data accompanies AISI's paper, How Inference Compute Shapes Frontier LLM Evaluation, which examines how benchmark performance depends on inference-time compute and the evaluation protocol used. One notable finding: performance on Humanity's Last Exam shifts noticeably depending on these factors. When models received correctness feedback from an oracle after each attempt, they kept solving additional tasks as token use increased.
What Open Results Mean for the Field
When results are released openly along with setup information, researchers can dig into individual studies more closely and compare findings across the wider ecosystem. Where other published reports lack these details, verified releases like AISI's act as reference points—for instance, helping researchers understand how setup choices may influence reported performance. The more evaluators adopt the EEE schema, the more open comparisons can support broader, more reliable meta-research across the field.
AISI hopes this kind of adoption will encourage further standardisation and sharing with EvalEval and other AI evaluation organisations.
How You Can Contribute
The collaboration invites different groups to get involved:
- Model developers can report verified evaluation results.
- Evaluation developers can publish benchmarks and run data using the Every Eval Ever schema.
- Evaluation, governance, and policy researchers can explore Evaluation Cards by benchmark or model, or use the platform to assess the overall state of evaluation reporting.
About the Organisations Involved
The EvalEval Coalition is a research community building scientifically grounded research and robust deployment infrastructure for the evaluation ecosystem. Its flagship projects—Every Eval Ever, a shared schema and repository for evaluation results, and Evaluation Cards, which combines benchmark metadata, evaluation-run data, and model metadata into interpretable records—help clarify when seemingly similar scores were actually produced under meaningfully different conditions.
The UK AI Security Institute sits within the UK government's Department for Science, Innovation and Technology. Its mission is to give governments a scientific understanding of the risks posed by advanced AI, through research and infrastructure that examine capabilities and impacts, test mitigations, and inform policy.
Comments (0)