Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences

Christie, A.P. and Abecasis, D. and Adjeroud, M. and Alonso, J.C. and Amano, T. and Anton, A. and Baldigo, B.P. and Barrientos, R. and Bicknell, J.E. and Buhl, D.A. and Cebrian, J. and Ceia, R.S. and Cibils-Martina, L. and Clarke, S. and Claudet, J. and Craig, M.D. and Davoult, D. and De Backer, A. and Donovan, M.K. and Eddy, T.D. and França, F.M. and Gardner, J.P.A. and Harris, B.P. and Huusko, A. and Jones, I.L. and Kelaher, B.P. and Kotiaho, J.S. and López-Baucells, A. and Major, H.L. and Mäki-Petäys, A. and Martín, B. and Martín, C.A. and Martin, P.A. and Mateos-Molina, D. and McConnaughey, R.A. and Meroni, M. and Meyer, C.F.J. and Mills, K. and Montefalcone, M. and Noreika, N. and Palacín, C. and Pande, A. and Pitcher, C.R. and Ponce, C. and Rinella, M. and Rocha, R. and Ruiz-Delgado, M.C. and Schmitter-Soto, J.J. and Shaffer, J.A. and Sharma, S. and Sher, A.A. and Stagnol, D. and Stanley, T.R. and Stokesbury, K.D.E. and Torres, A. and Tully, O. and Vehanen, T. and Watts, C. and Zhao, Q. and Sutherland, W.J. (2020) Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nature Communications, 11 (1): 6377. ISSN 2041-1723

Abstract

Building trust in science and evidence-based decision-making depends heavily on the credibility of studies and their findings. Researchers employ many different study designs that vary in their risk of bias to evaluate the true effect of interventions or impacts. Here, we empirically quantify, on a large scale, the prevalence of different study designs and the magnitude of bias in their estimates. Randomised designs and controlled observational designs with pre-intervention sampling were used by just 23% of intervention studies in biodiversity conservation, and 36% of intervention studies in social science. We demonstrate, through pairwise within-study comparisons across 49 environmental datasets, that these types of designs usually give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.