Gandy, Axel and Hahn, Georg (2017) QuickMMCTest : quick multiple Monte Carlo testing. Statistics and Computing, 27 (3). pp. 823-832. ISSN 0960-3174
quick.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.
Download (144kB)
Abstract
Multiple hypothesis testing is widely used to evaluate scientific studies involving statistical tests. However, for many of these tests, p values are not available and are thus often approximated using Monte Carlo tests such as permutation tests or bootstrap tests. This article presents a simple algorithm based on Thompson Sampling to test multiple hypotheses. It works with arbitrary multiple testing procedures, in particular with step-up and step-down procedures. Its main feature is to sequentially allocate Monte Carlo effort, generating more Monte Carlo samples for tests whose decisions are so far less certain. A simulation study demonstrates that for a low computational effort, the new approach yields a higher power and a higher degree of reproducibility of its results than previously suggested methods.