Tool Reviews

PremiseCheck Efficiently Fact-Checks Human and AI Peer Reviews

PremiseCheck decomposes peer review comments into paper-checkable premises, flagging factual errors at lower cost and latency than LLM review.

PremiseCheck Efficiently Fact-Checks Human and AI Peer Reviews

Key Takeaways

  • PremiseCheck flags possible factual errors in human or AI peer reviews by rewriting each comment into explicit premises and checking those premises against the paper.
  • It separates language interpretation from factual verification, allowing the premise-extraction step to run without reading the paper and keeping cost and latency low.
  • On the ReviewFact benchmark, PremiseCheck achieves per-comment AUROC comparable to a holistic GPT-5.5 approach at about half the cost, with gains mainly in cheaper, faster triage.
  • The tool is best used to rank and prioritize review comments that may misstate a paper, not to make final judgments about reviewer criticism, fairness, or importance.

Meta description: PremiseCheck flags factual errors in peer reviews by turning comments into explicit premises, matching GPT-5.5 accuracy at lower cost and latency.

PremiseCheck Efficiently Fact-Checks Human and AI Peer Reviews

Tags: peer review, PremiseCheck, ReviewFact, GPT-5.5, AUROC

Featured image: An abstract friendly illustration of a review comment being checked against a paper, with no real people, no logos, and no text.

PremiseCheck is for people who deal with research review comments and need to know which comments may misstate what a paper says. Review comments, written by humans or AI, can be vague, compressed, or wrong about a paper. PremiseCheck flags possible factual errors cheaply: a large language model, LLM, rewrites each comment as explicit premises without reading the paper, and a small decision model checks each premise against the paper. The project reports comparable per-comment AUROC, a ranking score, to a holistic GPT-5.5 approach at about half the cost, with the main gain coming from lower cost and latency. A single mistaken statement about a paper can waste author time, so a cheap flag can help readers decide where to look first.

It checks premises, not whole comments at once

The design separates two tasks. First, a comment is rewritten into explicit premises: the factual claims the comment appears to make about the paper. That rewrite does not require reading the paper, so it can be run cheaply. Second, a small decision model checks each premise against the paper. Because the paper is only read in the checking step, the workflow separates language interpretation from factual verification, which helps keep the first step cheap. The system scores the comment by its weakest premise and names the premise the paper contradicts. That makes the flag easier to act on than a score without a named claim.

PremiseCheck Efficiently Fact-Checks Human and AI Peer Reviews

ReviewFact tests accuracy, cost, and latency together

The evaluation uses ReviewFact, a dataset of 9,267 review comments on 217 papers from natural language processing, machine learning, and the natural sciences. The comments are labeled for factual errors under one definition, from expert annotations in ReviewCritique, ReviewScore, and PeerReview Bench. The benchmark also measures cost and latency alongside accuracy, because a review-checking tool only becomes practical if it can run over many comments without becoming expensive or slow.

AUROC: ranking likely errors, not proving them

AUROC, or area under the receiver operating characteristic curve, is a ranking score: a higher value means the tool is better at placing comments that contain factual errors above comments that are likely correct. It does not prove that a flagged comment is wrong. For users, the practical question is whether the tool puts risky comments high enough in a queue, not whether every flag is certain.

Per comment, PremiseCheck reaches AUROC comparable to holistic GPT-5.5 in all three fields at about half the cost. On natural language processing reviews with a no-reasoning decomposer, the project reports 63.1 versus 63.2 AUROC, with about 25% of the cost and 45% of the latency. The close AUROC values suggest the main benefit is a cheaper way to get a similar signal.

PremiseCheck Efficiently Fact-Checks Human and AI Peer Reviews

Fusing scores helps at review level

At the level of a full review, fusing batched PremiseCheck scores with GPT-5.5 scores gives 71.5 AUROC at 28% of the reference cost. A full review may contain many comments. Review-level scoring helps decide whether a whole review deserves attention, while per-comment flags help locate the problem inside it. This matters because reviewers and authors usually need a prioritized list of comments worth a second look, not a final verdict on every sentence. A review-level score can guide attention, while the named premise points to the claim to check.

Who should use it / who should skip it

Use PremiseCheck if you need a cheap way to rank review comments by factual risk and see which premise the paper contradicts. Skip it if you need a final judgment on reviewer criticism, or if you expect the tool to answer fairness, importance, or usefulness questions. For repeated screening of many comments, the lower cost may matter more than a small accuracy difference.

The task is hard: both systems score only about 63 AUROC on natural language processing data. The machine learning and natural sciences subsets are small, with wide intervals, so field-level comparisons should be read cautiously. The benchmark also does not cover judgment errors. PremiseCheck is meant to point people to comments worth checking, not to replace reviewers.

The code is at https://github.com/OpenLAIR/PremiseCheck, the data at https://huggingface.co/datasets/songdj/ReviewFact, and the project page at https://openlair.github.io/premisecheck/. Questions and counterexamples are welcome.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.