Evaluating early-stage research judgment helps determine whether research agents have taste.
Six frontier AI models judged early-stage research ideas; rankings beat random, but only 73 of 415 experiments were beyond source papers.
Key Takeaways
- A study tested frontier AI models as research agents to judge whether their own proposed NLP research ideas had taste.
- Of 415 experiments, 194 were judged sound, valuable, and new, but only 73 were also non-obvious and beyond the source paper.
- The models’ top-ranked choices and forecasts were better than random or base-rate benchmarks, but not clearly better than a second model reading only the paper and plan.
- The results suggest AI agents can generate many reasonable proposals, but self-ranking and forecasting are only partly useful and not yet reliable for identifying the best idea.
A recent study asked six frontier models to act as research agents and judge whether their own research ideas had taste. The models were given seven recent natural language processing papers and, working either alone or in teams, had to turn each paper into novel research. Each model proposed five small experiments, set a threshold for success, predicted whether each experiment would pass, and ranked the ideas.
What the test measured
The test focused on whether the models could identify good ideas after they were tested. A separate agent attempted every experiment, including the lowest-ranked ones, and three model auditors judged the results. Of 415 experiments, 194 were judged sound, valuable and new. Only 73 of those also went beyond the paper that inspired them and were non-obvious.
Did the models choose well?
The agents' first choices matched the auditors' most valued experiment more often than a random pick. However, the results did not show that the first choice beat the experiment the model had written first. Their forecasts beat a base rate, but were not measurably better than those of a second instance of the same model that read only the paper and the plans.
What readers should take from it
For readers who use AI to suggest studies, the experiment gives a concrete measure of taste: not whether ideas sound new, but whether they are judged sound, valuable, new, and beyond the source paper. It suggests that frontier models can produce many reasonable proposals, while only a smaller share appear genuinely beyond the seed paper.
It also suggests that self-ranking and forecasting are partly useful: they are better than random, yet not clearly better than a simpler model reading the same plan. When an AI research agent says an idea is promising, the signal may be real, but it may not yet be a reliable guide to the best idea. Further questions were invited on X at https://x.com/artkanke?s=11.

Comments (0)