Davydenko, A and Fildes, R A and Trapero Arenas, J R (2010) Measuring the Accuracy of Judgmental Adjustments to SKU-level Demand Forecasts. Working Paper. The Department of Management Science, Lancaster University.
Abstract
The paper shows that due to the features of SKU (stock-keeping unit) demand data wellknown error measures previously used to analyse the accuracy of adjustments are generally not advisable for the task. In particular, percentage errors are affected by outliers and biases arising from a large number of low actual demand values and correlation between forecast errors and actual outcomes. It is also shown that MASE is equivalent to the arithmetic average of relative mean absolute errors (MAEs) and inherently is biased towards overrating the benchmark method. Therefore existing measures cannot deliver easily interpretable and unambiguous results. To overcome the imperfections of existing schemes a new measure is introduced which indicates average relative improvement of MAE. In contrast to MASE the proposed scheme is based on finding the geometric average of relative MAEs. This allows objective evaluation of relative change in forecasting accuracy yielded by the use of adjustments. Empirical analysis employed a large number of observations collected from a company specialising on manufacturing of fast-moving consumer goods (FMCG). The results suggest that adjustments reduced MAE of baseline statistical forecast on average by approximately 10%. Using a binomial test it was confirmed that adjustments improved the accuracy of forecasts significantly more frequently rather than they reduced it.