High quality direct replications matter: response to Williams (2014)

Corker, Katherine S. and Lynott, Dermot and Wortman, Jessica and Connell, Louise and Donnellan, M. Brent and Lucas, Richard E. and O'Brien, Kerry S. (2014) High quality direct replications matter: response to Williams (2014). Social Psychology, 45 (4). pp. 324-326. ISSN 1864-9335

Abstract

We respond to Williams’(2014) comments on our three failures to replicate of Study 2 from Williams and Bargh (2008). We clarify our conclusions on this topic, making clear that although the results of our studies cast doubt on the specific effect reported in Williams and Bargh (i.e., that instant hot and cold packs influence choice of reward for self or friend), a more complete understanding of the embodiment hypothesis in question requires consideration of relevant conceptual replications. Accordingly, we consider the strength of the evidence in the conceptual replications that Williams identifies and find that small samples appear to be the norm. We conclude that in order for researchers to move forward, future studies must take seriously issues of power, researcher degrees of freedom, and file drawer problems. Doing so will ensure that future studies are more informative tests of this hypothesis.