ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Sat, 12 Dec 2026 00:00:00 +0000

40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026), NeurIPS’26 Workshop on AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI, Atlanta, USA, December 9-13, 2026.

Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide given the growing number of submissions and time constraints. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles – the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date – demonstrate ARA’s generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. These results highlight ARA’s potential to complement human review at scale and support next-generation peer review. ARA consistently achieves 61% accuracy on three reproducibility assessment benchmarks, highlighting its potential to complement human review at scale and enabling next-generation peer review.

Kevin Riehl and Andres L. Marin contributed equally to this work and share first authorship.

Code and data on GitHub.

Georgios Fontaras | Kevin Riehl

ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review