Tool Reviews

Are LLMs reliable enough to assess whether a research idea is novel?

Allen Institute benchmark tests LLM novelty judges using top conference peer reviews. Prompt wording caused accuracy swings, limiting reliability.

Are LLMs reliable enough to assess whether a research idea is novel?

Key Takeaways

  • Ai2’s “Old Ideas, Novel Problems” study finds that LLM-based novelty judges are highly inconsistent; small prompt changes can drastically alter their verdicts, sometimes dropping accuracy below chance.
  • The benchmark is built from ICLR peer reviews that explicitly discuss originality, creating validated high- and low-novelty human ideas rather than relying on paper acceptance status.
  • Adding retrieval, more reasoning, or specialized novelty pipelines did not consistently improve judgments and often increased cost, while simple prompted baselines performed better.
  • This matters because AI systems for scientific discovery increasingly rely on LLMs to decide whether a generated research idea is genuinely novel or merely repackaged.

The Allen Institute for AI (Ai2) has released Old Ideas, Novel Problems, a study showing that large language models used as "novelty judges"—systems that assess whether a research idea is genuinely new—give wildly inconsistent verdicts. In one experiment, tweaking just a few words in a prompt shifted GPT-5.4's accuracy at identifying the more novel of two ideas from 93.5% to 40.9%, dropping below chance. The release includes a benchmark, data, code, and an automated pipeline so other researchers can build fresh evaluation sets from peer reviews.

Are LLMs reliable enough to assess whether a research idea is novel?

The work matters because Ai2 and others are building systems meant to automate parts of scientific discovery. DataVoyager explores and analyzes structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence across large bodies of literature into candidate theories. Together with Asta, these projects aim to move researchers from understanding prior work to proposing, testing, and refining new ideas. But generating ideas is only half the problem—the systems also need to tell a genuinely novel direction from one that repackages existing work, and that judgment is increasingly delegated to another LLM.

How the benchmark was built

To test novelty judges, the team needed examples with strong evidence about originality. Rather than treating ICLR 2026 acceptance as a proxy for novelty, they mined the peer reviews themselves for passages where reviewers explicitly discussed a contribution's originality—a key distinction, since a paper can be accepted without being especially novel and a rejected paper can still contain an original contribution.

From that process they built two pools. The high-novelty pool contains 154 human-authored ideas drawn from accepted papers in the top-rated 10% of their ICLR research area, where reviewers positively identified novelty and no reviewer disputed it. The lower-novelty pool contains 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none positively identified novelty. In a separate check of the mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases, and none of the disagreements would have changed whether a reviewer counted as praising or disputing novelty.

Those two pools form the Human-Only setting. Because novelty judges are increasingly used on AI-generated ideas, the team added a Human+Generated setting: the same 154 validated human ideas paired with 154 ideas from a simple LLM generator that received only a research area and an instruction to propose something novel—no literature search, tools, iteration, or feedback. The generated ideas are assumed less novel, a weak label rather than a verified judgment for each idea. As a check, an expert reviewed 30 pairs weighted toward cases the strongest judge got wrong, preferred the human idea in all 30, and judged none of the generated ideas novel. The researchers also rewrote human and AI ideas into a common structure to test whether judges were merely reacting to writing style.

Are LLMs reliable enough to assess whether a research idea is novel?

Small prompt changes, large verdict swings

The team tested six judges—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—in both pointwise (absolute novelty ratings) and pairwise (which of two ideas is more novel) formats, varying one factor at a time: prompt, reasoning budget, access to related papers, idea formatting, number of judgments combined, and which model generated the AI ideas.

Prompting produced the biggest shifts. GPT-5.4 saw the same Human+Generated pairs under two prompts: one stating that reviewers had found exactly one idea in each pair novel and asking which, the other simply asking which idea was novel. The simpler prompt cut accuracy from 93.5% to 40.9% and flipped the verdict on more than half the pairs. On the Human-Only dataset the same change moved accuracy by only about seven points—suggesting a judge can look stable on human research while behaving very differently on AI-generated ideas. Human ideas weren't immune, though: the same change moved Claude Opus 4.6 accuracy by 21 points.

No prompt was universally better. Dropping an instruction warning judges not to treat trivial or incremental contributions as novel improved accuracy for some judges and reduced it for others—a prompt that works for one model may not transfer to another.

The team also tested a common use case: comparing ideas from two AI generators, pairing Claude Opus 4.5 with Claude Sonnet 4.5 and GPT-5.4 with GPT-5.1. Across the tested prompt changes, roughly a quarter to a third of verdicts changed on average across the six judges—measurable disagreement without assuming which generator produces more novel ideas.

More search and reasoning didn't help

Adding retrieval and reasoning did not consistently improve judgments. GPT-5.6 Sol, given an extra-high reasoning budget and live iterative search restricted to arXiv, scored 0.78 macro-F1—11 points behind Claude Opus 4.6 at 0.89, roughly level with simpler Claude Opus 4.5 and GPT-5.4 setups—while costing about 10× more per decision at current API prices. Macro-F1 here measures how well a judge identifies novel and non-novel ideas; higher is better.

Are LLMs reliable enough to assess whether a research idea is novel?

Purpose-built evaluators fared no better despite costing far more. On Human+Generated, a simple Claude Opus 4.6 prompt with low reasoning and majority voting across three calls reached 0.854 macro-F1 for $2.59 total. Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66, and Sakana AI's AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both dedicated evaluators used Claude Opus 4.6 with one call per decision. The simple prompted baseline was both more accurate and considerably cheaper than either system.

Notably, many core findings don't depend on the labels at all: if two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.

What comes next

The results point to LLM-based novelty evaluation as a major open problem. Judges that appear reliable on human-authored research may behave differently on AI-generated ideas, prompts that help one model can hurt another, and more search or compute does not necessarily yield more consistent judgments.

The authors argue future work should test judges on the kinds of ideas they will actually evaluate, measure sensitivity across prompts and models, improve how prior literature is incorporated, and build benchmarks that evolve as the literature grows and AI systems improve. The released benchmark and reusable data pipeline are positioned as a starting point that researchers can apply to new peer reviews, other scientific communities, and future generations of novelty judges—giving anyone building automated ideation systems a way to measure how much their evaluator can actually be trusted.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.