Is the finding too good to be true?:Moving from “more is better” to thinking in terms of simple predictions and credibility

Youngstrom, Eric. A. and Salcedo, Stephania and Frazier, Thomas W. and Perez Algorta, Guillermo (2019) Is the finding too good to be true?:Moving from “more is better” to thinking in terms of simple predictions and credibility. Journal of Clinical Child and Adolescent Psychology, 48 (6). pp. 811-824. ISSN 1537-4416

[img] Text (881_Too_good_to_be_true_R1_no_field_codes_ (1))
881_Too_good_to_be_true_R1_no_field_codes_1_.pdf - Accepted Version
Restricted to Repository staff only until 16 October 2020.
Available under License Creative Commons Attribution-NonCommercial.

Download (357kB)

Abstract

In 2018, De Los Reyes and Langer expanded the scope of the Evidence Base Updates series to include reviews of psychological assessment techniques. In keeping with the goal of offering clear "take-home messages" about the evidence underlying the technique, experts have proposed a rubric for evaluating the reliability and validity support. Changes in the research environment and pressures in the peer review process, as well as a lack of familiarity with some statistical methods, have created a situation where many findings that appear “excellent” in the rubric are likely to be “too good to be true,” in the sense that they are unlikely to generalize to clinical settings or are unlikely to be reproduced in independent samples. We describe several common scenarios where published results are often too good to be true, including internal consistency, inter-rater reliability, correlation, standardized mean differences, diagnostic accuracy, and global model fit statistics. Simple practices could go a long way towards improving design, reporting, and interpretation of findings. When effect sizes are in the “excellent” range for issues that have been challenging, scrutinize before celebrating. When benchmarks are available base on theory or meta-analyses, results that are moderately better than expected in the favorable direction (i.e., Cohen’s q≥+.30) also invite critical appraisal and replication before application. If readers and reviewers pull for transparency and do not unduly penalize authors who provide it, then change in research quality will be faster and both generalizability and reproducibility are likely to benefit.

Item Type: Journal Article
Journal or Publication Title: Journal of Clinical Child and Adolescent Psychology
Additional Information: This is an Accepted Manuscript of an article published by Taylor & Francis in Journal of Clinical Child & Adolescent Psychology on 16/10/2019, available online: https://www.tandfonline.com/doi/full/10.1080/15374416.2019.1669158
Uncontrolled Keywords: /dk/atira/pure/subjectarea/asjc/3200/3204
Subjects:
Departments: Faculty of Health and Medicine > Health Research
ID Code: 136691
Deposited By: ep_importer_pure
Deposited On: 25 Sep 2019 08:50
Refereed?: Yes
Published?: Published
Last Modified: 14 Dec 2019 05:11
URI: https://eprints.lancs.ac.uk/id/eprint/136691

Actions (login required)

View Item View Item