This large-scale study of paper revision trajectories examines whether AI reviewers can see your efforts to improve your paper.
Study finds AI reviewers often miss major revision gains; scores stay flat, agreement is weak, and negative comments can reverse improvements.
Key Takeaways
- PaperTrajectory-2026 analyzes 6,959 paper revision trajectories from five AI conferences across 14 revision dimensions.
- Eight AI reviewers often leave scores unchanged even when revisions are annotated as major improvements.
- Reviewer agreement on score changes is weak, so the same revision may be interpreted very differently.
- Brief negative user opinions can erase or reverse score gains, making final scores less reliable.
- AI review evaluation should measure score stability and ensure reviewers properly credit added revision evidence.
PaperTrajectory-2026 Finds AI Reviewers Often Miss Author Revision Improvements
Meta description: PaperTrajectory-2026 evaluates eight AI reviewers on 6,959 trajectories; major revisions often score unchanged.
Tags: AI reviewers, PaperTrajectory-2026, revision dimensions, score stability
Featured image: abstract paper trajectory illustration, no people, logos, text
PaperTrajectory-2026 covers 6,959 paper trajectories from five AI conferences, automatically annotated across 14 revision dimensions. It evaluates eight AI reviewers and finds most revisions marked as major improvements receive unchanged scores.
Scores Often Stay Flat After Major Revisions
Most revisions annotated as major improvements did not move scores, suggesting reviewers may not reward identified improvements.
Agreement Is Weak Across Reviewers
Reviewers weakly agree on score-change direction; the same revision may be read differently, with no clear rise, fall, or stay signal, making scores unreliable records of improvement.
A Negative User Opinion Can Reverse a Gain
Even higher scores can be erased or reversed by brief negative user opinions, so final scores may reflect comments rather than revised evidence.
The Need for Stable Judgments
Reviewers should credit added revision evidence and keep scores stable when evidence is unchanged; evaluation should measure score stability, not only output numbers.
For readers, current AI reviewer scores may not track revision improvements as reliably as authors might hope.

Comments (0)