A permutation test for assessing the presence of individual differences in treatment effects

Chang, C. and Jaki, T. and Sadiq, M.S. and Kuhlemeier, A. and Feaster, D. and Cole, N. and Lamont, A. and Oberski, D. and Desai, Y. and Lee Van Horn, M. (2021) A permutation test for assessing the presence of individual differences in treatment effects. Statistical Methods in Medical Research, 30 (11). pp. 2369-2381. ISSN 0962-2802

Text (Permutation final)
Permutation_final.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.

Download (635kB)


An important goal of personalized medicine is to identify heterogeneity in treatment effects and then use that heterogeneity to target the intervention to those most likely to benefit. Heterogeneity is assessed using the predicted individual treatment effects framework, and a permutation test is proposed to establish if significant heterogeneity is present given the covariates and predictive model or algorithm used for predicted individual treatment effects. We first show evidence for heterogeneity in the effects of treatment across an illustrative example data set. We then use simulations with two different predictive methods (linear regression model and Random Forests) to show that the permutation test has adequate type-I error control. Next, we use an example dataset as the basis for simulations to demonstrate the ability of the permutation test to find heterogeneity in treatment effects for a predicted individual treatment effects estimate as a function of both effect size and sample size. We find that the proposed test has good power for detecting heterogeneity in treatment effects when the heterogeneity was due primarily to a single predictor, or when it was spread across the predictors. Power was found to be greater for predictions from a linear model than from random forests. This non-parametric permutation test can be used to test for significant differences across individuals in predicted individual treatment effects obtained with a given set of covariates using any predictive method with no additional assumptions.

Item Type:
Journal Article
Journal or Publication Title:
Statistical Methods in Medical Research
Additional Information:
The final, definitive version of this article has been published in the Journal, Statistical Methods in Medical Research, 30 (11), 2021, © SAGE Publications Ltd, 2021 by SAGE Publications Ltd at the Statistical Methods in Medical Research page: https://journals.sagepub.com/doi/10.1177/09622802211033640 on SAGE Journals Online: http://journals.sagepub.com/
Uncontrolled Keywords:
ID Code:
Deposited By:
Deposited On:
05 Nov 2021 09:40
Last Modified:
20 Sep 2023 01:46