FrontierPhysics evaluates agents for end-to-end physics research.
FrontierPhysics tests AI agents on 56 physics tasks with verifiers and rubrics, revealing gaps in reasoning, planning, and publishable output.
Key Takeaways
- FrontierPhysics is a benchmark for testing whether AI agents can complete full physics research workflows, not just answer isolated questions.
- It uses 56 expert-designed tasks across nine physics domains, covering literature review, planning, experimentation, and manuscript production.
- Evaluation combines deterministic verifiers with expert rubrics to judge both correctness and whether outputs are publishable.
- Testing across 22 models showed no configuration performed strongly enough, revealing a major gap between current agents and reliable scientific research.
- The main weakness is not missing physics knowledge, but insufficient rigorous reasoning and expert intuition in open-ended research tasks.
Meta description: FrontierPhysics tests AI agents on 56 expert-designed physics tasks using deterministic verifiers and rubrics for correctness and publishable research.
Tags: FrontierPhysics, AI agents, physics benchmark, research evaluation
Primary keyword: FrontierPhysics
Featured image: abstract friendly illustration of physics diagrams, research notebooks, and glowing agent nodes, with no real people, logos, or text.
FrontierPhysics is a benchmark designed for physicists, AI developers, and research evaluators who need to know whether AI agents can carry out complete physics work, not only answer isolated questions. It uses 56 expert-designed tasks to test literature review, research planning, experimentation, and manuscript production, with deterministic verifiers and expert rubrics deciding whether the output is correct and publishable.
Why Existing Physics Benchmarks Fall Short
As AI agents take on a larger role in physics research, earlier physics benchmarks have mostly measured narrow skills. They evaluate fixed questions, isolated sub-tasks, or paper reproduction when solutions are already given. That approach can show whether a system knows facts or can follow a prescribed solution path, but it rarely checks whether the resulting research is sound, useful, and close to publishable quality.
What FrontierPhysics Tests
FrontierPhysics was built by 42 physicists using tasks drawn from their own research. It covers nine domains across theoretical, computational, and experimental physics. Each task asks an agent to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts.
The benchmark is end-to-end because it moves from background reading to final scientific writing. It does not stop at a short answer or a code snippet. Instead, it tests whether an agent can sustain a research process across multiple stages.
How Scores Are Judged
FrontierPhysics combines deterministic verifiers with expert-authored rubrics. Deterministic checks can evaluate clear technical outputs, while rubrics written by physicists assess research quality. Together, they aim to answer two questions at once: is the work correct, and could it be considered publishable?
What the Results Reveal
The benchmark was tested across 22 models in 26 configurations. No configuration passed more than 46 of its 168 rollouts. That result shows a wide gap between current frontier agents and fully reliable scientific research.
Manual review of agent trajectories points to a specific weakness. The main problem is not missing physics knowledge or simple execution errors. Frontier agents most often fall short in rigorous reasoning and expert intuition, which are needed when a task does not have one obvious path from start to finish.
Why This Matters
For readers building or assessing scientific AI agents, FrontierPhysics provides a stricter signal than question-answer accuracy alone. It highlights where agents are still fragile: planning, judgment, and reasoning under expert standards. As agents enter research workflows, this kind of end-to-end evaluation becomes a useful checkpoint for whether they can support real physics work rather than only reproduce known solutions.

Comments (0)