Assessing the effects of a precommitment policy applied during peer review

Haber, Noah A. and Errington, Timothy M. and Daley, Macie and Whaley, Paul and Nosek, Brian A. (2024) Assessing the effects of a precommitment policy applied during peer review. Evidence-Based Toxicology, 2 (1): 2386955. ISSN 2833-373X

Abstract

Improving journal policies and practices is hampered by a lack of experimentation and evidence. Implementing high-powered journal policy experiments to test solutions is difficult because of status quo biases, logistical difficulties, sample size requirements, and ethical considerations. In this article, we introduce a new policy and a new framework for experimental implementation at scale. The policy, called “Registered Revisions”, triggers during traditional peer review when reviewers ask for additional data and/or analysis. Authors are asked to make a Revision Plan for how they will address these comments. If the Revision Plan and responses to standard questions are approved, editors agree to make an in-principle acceptance for publication decision. In other words, the article will be accepted as long as the Revision Plan is carried out as specified, regardless of the results. Registered Revisions enables authors and editors to gain experience of preregistering their methods without them having to proactively commit to a full, two-stage Registered Reports-style publication process. The experimental framework is a meta experiment involving a collaboration of many journals. Each journal implements an experiment based on a shared protocol, logistical guidance, materials, and infrastructure. The journals can use and publish their experiments independently, and also contribute the data and findings to a prospective meta-analysis. This approach presents a potential path forward for evidence-informed journal policy reform.