Natural experiments for the evaluation of place-based public health interventions:a methodology scoping review

Albers, Patricia and Rinaldi, Chiara and Brown, Heather and Mason, Kate and d’Apice, Katrina and McGill, Elizabeth and McQuire, Cheryl and Craig, Peter and A Laverty, Anthony and Beeson, Morgan and Campbell, Mhairi and Egan, Matt and Gibson, Marcia` and Fuller, Maxwell and Dillon, Amy and Taylor-Robinson, David and Jago, Russell and Tilling, Kate and Barr, Benjamin and Sniehotta, Falko and Hickman, Matthew and Millet, Christopher and de Vocht, Frank (2023) Natural experiments for the evaluation of place-based public health interventions:a methodology scoping review. Frontiers in Public Health, 11. ISSN 2296-2565 (In Press)

Text (NE_Scoping_Review_manuscript_Frontiers_PH_PA)
NE_Scoping_Review_manuscript_Frontiers_PH_PA.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (375kB)

Abstract

Place-based public health evaluations are increasingly making use of natural experiments. This scoping review aimed to provide an overview of the design and use of natural experiment evaluations (NEEs), and an assessment of the plausibility of the as-if randomisation assumption. A systematic search of three bibliographic databases (Pubmed, Web of Science and Ovid-Medline) was conducted in January 2020 to capture publications that reported a natural experiment of a place-based public health intervention or outcome. For each, study design elements were extracted. An additional evaluation of as-if randomisation was conducted by twelve 12 of this paper’s authors who evaluated the same set of 20 randomly selected studies and assessed ‘as-if’ randomisation for each. 366 NEE studies of place-based public health interventions were identified. The most commonly used NEE approach was a Difference-in-Differences study design (25%), followed by before-after comparisons studies (23%) and regression analysis studies. 42% of NEEs had likely or probably as-if randomisation of exposure (the intervention), while for 25% this was implausible. An inter-rater agreement exercise indicated poor reliability of as-if randomisation assignment. Only about half of NEEs reported some form of sensitivity or falsification analysis to support inferences. NEEs are conducted using many different designs and statistical methods and encompass various definitions of a natural experiment, while it is questionable whether all evaluations reported as natural experiments should be considered as such. The likelihood of as-if randomisation should be specifically reported, and primary analyses should be supported by sensitivity analyses and/or falsification tests. Transparent reporting of NEE designs and evaluation methods will contribute to the optimum use of place-based NEEs.