Reproducibility in forecasting research

Boylan, John and Goodwin, Paul and Mohammadipour, Maryam and Syntetos, Aris (2015) Reproducibility in forecasting research. International Journal of Forecasting, 31 (1). pp. 79-90. ISSN 0169-2070

Text (Reproducibility IJF Boylan et al)
Reproducibility_IJF_Boylan_et_al.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial-NoDerivs.
Download (1MB)

Abstract

The importance of replication has been recognised across many scientific disciplines. Reproducibility is a necessary condition for replicability because an inability to reproduce results implies that the methods have been insufficiently specified, thus precluding replication. This paper describes how two independent teams of researchers attempted to reproduce the empirical findings of an important paper, “Shrinkage estimators of time series seasonal factors and their effect on forecasting accuracy” (Miller & Williams, 2003, IJF). The teams of researchers proceeded systematically, reporting results before and after receiving clarifications from the authors of the original study. The teams were able to approximately reproduce each other’s results but not those of Miller & Williams. These discrepancies led to differences in the conclusions on conditions under which seasonal damping outperforms Classical Decomposition. The paper specifies the forecasting methods employed using a flowchart. It is argued that this approach to method documentation is complementary to the provision of computer code, as it is accessible to a broader audience of forecasting practitioners and researchers. The significance of this research lies not only in its lessons for seasonal forecasting but, more generally, in its approach to the reproduction of forecasting research.