Can LLMs Reliably Judge Research Novelty? NoveltyBench Says Not Yet

Researchers at Allen AI have released NoveltyBench, a benchmark designed to evaluate whether large language models can accurately assess the novelty of research ideas relative to existing literature. The results indicate that current LLMs perform inconsistently on this task, showing particular weakness when ideas are superficially similar to prior work but meaningfully distinct in contribution. This has direct implications for the growing use of AI in automated literature review, research ideation tools, and AI-assisted peer review pipelines. Developers and researchers building AI tools for scientific workflows should treat LLM novelty judgments as unreliable without human verification, at least under current model generations. The benchmark itself is publicly available, enabling teams to evaluate their own models or prompting strategies against this capability gap.