ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Sat, 12 Dec 2026 00:00:00 +0000

40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026), NeurIPS’26 Workshop on AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI, Atlanta, USA, December 9-13, 2026.

Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide given the growing number of submissions and time constraints. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles – the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date – demonstrate ARA’s generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. These results highlight ARA’s potential to complement human review at scale and support next-generation peer review. ARA consistently achieves 61% accuracy on three reproducibility assessment benchmarks, highlighting its potential to complement human review at scale and enabling next-generation peer review.

Kevin Riehl and Andres L. Marin contributed equally to this work and share first authorship.

Code and data on GitHub.

Claude Coke: Prevent Automated Crime by Agents

Wed, 09 Dec 2026 00:00:00 +0000

40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026), Position Paper Track, Sydney, Australia, December 6-12, 2026.

We show that contemporary LLM-agent stacks can automate serious criminal activity at scale by composing capabilities that are already present, and that relevant technical safeguards and governance controls need to be in place before becoming operationally viable. The argument rests on three empirical findings that are usually studied separately but together close a loop that automates illicit purchasing, whether by accident or by design. (i) Autonomous LLM agents fail catastrophically at multi-agent-economy coordination tasks, with bankruptcy as the modal outcome and not a rare tail case. (ii) The rate at which agents reach for clearly harmful tools moves with common contextual manipulations. (iii) Autonomous browser control has become general enough that some frontier agents can navigate, populate, and complete checkout flows on simulated illicit-product storefronts under light obfuscation, with completion varying across tested model configurations. Combining these findings requires no new model capability: in our experiments, some LLM agents autonomously completed simulated dark-web purchases of drugs, shotguns, and hitman services, while others refused. We propose and discuss three interventions: mandatory pre-deployment safety testing, hard human-confirmation gates at financially material steps, and verifiable agent identity infrastructure.

NeurIPS conference page · Paper on OpenReview

The manuscript source and figure assets are maintained separately from the selected run records, analysis scripts, and reproduction notebooks. At the time of writing, access to the data repository is restricted; linking to it here does not constitute the redacted public release discussed in the manuscript.

LLM Agents | Kevin Riehl

ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Claude Coke: Prevent Automated Crime by Agents