OpenFrontiers Introduces a Collection of Open Research Problems and a Benchmark for Scientific Rediscovery
OpenFrontiers pairs 12,151 research problems with a 200-problem Lean 4 benchmark to measure LLM agents' true rediscovery versus web retrieval.
Key Takeaways
- OpenFrontiers introduces a cross-disciplinary corpus of 12,151 open research problems, including statements, context, and progress information.
- It provides a formal Lean 4 benchmark of 200 math problems, with quality-filtered subsets such as HQ-50 for more reliable evaluation.
- The benchmark tests LLM agents in closed-book, frozen-retrieval, and live-web settings to separate true rediscovery from online solution retrieval.
- Early results show closed-book rediscovery is rare, while live-web success often reflects retrieval of published solutions rather than independent reasoning.
Meta description: OpenFrontiers pairs 12,151 research problems with a 200-problem Lean~4 benchmark to test LLM agents on closed-book and live-web research problem solving.

Tags: OpenFrontiers, LLM benchmark, research problems, Lean~4, AI evaluation
Featured image: Abstract, friendly illustration style: open books, mathematical symbols, network nodes, and a sandboxed evaluation workspace; no real people, no logos, no text.
The OpenFrontiers benchmark and corpus were introduced to address a gap in research-level problem solving with large language models. While LLMs have shown increasing promise on complex scientific questions, the field still lacked a comprehensive collection of such problems and a clear way to evaluate them. For readers tracking AI progress, the resource offers a cross-disciplinary map of open research problems and a formal way to test whether LLM agents can rediscover solutions rather than retrieve them.
A corpus for tracking open scientific problems
The corpus records 12,151 problems. For each entry, it stores the problem statement, context, progress, and other information that can help researchers understand where a problem stands. This makes OpenFrontiers more than a list of difficult questions: it is a cross-disciplinary resource for studying unresolved research problems and the AI systems that attempt to engage with them.
A formal math benchmark with quality filters
The benchmark contains 200 problems: 110 resolved in 2026 and 90 still open. Each problem is represented by a Lean~4 statement, and each statement is checked by automated probes and two independent reviews. Lean~4 is used here as a formal proof language, so a solution can be expressed in a way that supports precise checking rather than informal human judgment alone.
A subset called HQ-50 applies further filters based on quality score and other quality-related factors. This smaller set gives the evaluation a more consistent foundation, reducing the risk that weakly specified or poorly checked problems dominate the results.
Separating rediscovery from web retrieval
The evaluation protocol grades formal proofs with two independent checkers in an isolated sandbox. It compares three settings: closed-book, frozen-retrieval, and live-web. In plain terms, the test asks whether an LLM agent can produce a correct proof without outside help, with a controlled retrieval resource, or while browsing live web content.
Seven LLM agents were evaluated on the HQ-50 subset under a standardized budget. The results suggest that closed-book rediscovery is rare: at most 7 of 40 resolved problems were rediscovered without external lookup. By contrast, live-web success mostly reflects retrieval of published solutions rather than independent mathematical rediscovery.
That distinction matters for readers following AI research. A model that finds a published answer online may still be valuable as a research assistant, but it is not the same as a model that reconstructs the solution from first principles. OpenFrontiers makes that difference easier to measure.
What it means for future AI research
By linking a broad research-problem corpus to a formally graded evaluation subset with documented solutions, OpenFrontiers provides a foundation for measuring research-level rediscovery. It also supports study of progress toward solving currently open scientific problems. The dataset and benchmark are available in the Hugging Face collection at https://huggingface.co/collections/FlippyDora/openfrontiers.
Comments (0)