ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Sat, 12 Dec 2026 00:00:00 +0000

40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026), NeurIPS’26 Workshop on AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI, Atlanta, USA, December 9-13, 2026.

Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide given the growing number of submissions and time constraints. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles – the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date – demonstrate ARA’s generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. These results highlight ARA’s potential to complement human review at scale and support next-generation peer review. ARA consistently achieves 61% accuracy on three reproducibility assessment benchmarks, highlighting its potential to complement human review at scale and enabling next-generation peer review.

Kevin Riehl and Andres L. Marin contributed equally to this work and share first authorship.

Code and data on GitHub.

Multi-Agent Reinforcement Learning of Karma Bidding Strategies

Sat, 11 Jul 2026 00:00:00 +0000

43rd International Conference on Machine Learning (ICML 2026), New Frontiers in Game-Theoretic Learning Workshop, Seoul, South Korea, July 11, 2026.

Capacity-constrained shared infrastructure systems require demand management mechanisms that balance efficiency and fairness. Karma mechanisms address this challenge using an artificial, non-tradable currency that enables decentralized allocation through repeated bidding, but computing equilibrium strategies in such settings is difficult due to unknown population dynamics, stochastic demand, and computation time in practice. Here, we study suitable regimes to prove convergence towards stochastic Nash equilibria when following a multi-agent reinforcement learning approach. Computational case studies demonstrate empirically that learned policies closely approximate equilibrium behavior, and further assess impact of learning algorithm and policy initialization on convergence speed. The work highlights the practical potential of the approach for real-world decision support in repeated access-allocation settings.

ICML conference page · Paper on OpenReview · Source code on GitHub

Patrick Langer | Kevin Riehl

ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Multi-Agent Reinforcement Learning of Karma Bidding Strategies