When AI Judges Research Novelty, Small Prompt Changes Matter

A new Ai2 study finds that LLM judgments of research novelty can shift sharply with small changes to the evaluation setup. In tests across six judges, some prompt variations changed verdicts on more than half of the same idea pairs—and could push performance below chance. The team also reports that added retrieval and larger reasoning budgets did not reliably solve the problem.

The benchmark combines research ideas labeled using peer-review signals with AI-generated ideas. That makes the results a warning about a specific evaluation task, not proof that every AI assessment is unreliable. For teams using AI to screen proposals, a useful next check is to test the judge on the kinds of ideas it will actually see, vary prompt wording, and compare its decisions with human review. A single score can conceal a fragile evaluation.

Read Ai2’s study and benchmark. What checks would you require before using an AI novelty score to prioritize research?