Assessing computational reproducibility in Behavior Research Methods

Ellis, David and Towse, John and Brown, Olivia and Cork, Alicia and Davidson, Brittany and Devereux, Sophie and Hinds, Joanne and Ivory, Matthew and Nightingale, Sophie and Parry, Douglas and Piwek, Lukasz and Shaw, Heather and Towse, Andrea (2024) Assessing computational reproducibility in Behavior Research Methods. Behavior Research Methods. ISSN 1554-351X (In Press)

Text (AAM_Stage_2_manuscript_revision_complete_version_final_revision)
AAM_Stage_2_manuscript_revision_complete_version_final_revision.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (981kB)

Abstract

Psychological science has thrived thanks to new methods and innovative practices. Journals, including Behavior Research Methods (BRM), continue to support the dissemination and evaluation of research assets including data, software/hardware, statistical code, and databases of stimuli. However, such research assets rarely allow for computational reproducibility, meaning they are difficult to reuse. Therefore, in this preregistered report, we explore how BRM’s authors and BRM structures shape the landscape of functional research assets. Our broad research questions concern: (1) How quickly methods and analytical techniques reported in BRM can be used and developed further by other scientists; (2) Whether functionality has improved following changes to BRM journal policy in support of computational reproducibility; (3) Whether we can disentangle such policy changes from changes in reproducibility over time. We randomly sampled equal numbers of papers (N = 204) published in BRM before and after the implementation of policy changes. Pairs of researchers recorded how long it took to ensure assets (data, software/hardware, statistical code, and materials) were fully operational. They also coded the completeness and reusability of the assets. While improvements were observed in all measures, only changes to completeness were altered significantly following the policy changes (d = .37). The effects varied between different types of research assets, with data sets from experiments/surveys showing the largest improvements in completeness and reusability. Perhaps more importantly, changes to policy do appear to have improved the life span of research products by reducing natural decline. We conclude with a discussion of how, in the future, research and policy might better support computational reproducibility within and beyond psychological science.