Measuring Forecasting Accuracy: Problems and Recommendations (by the Example of SKU-Level Judgmental Adjustments)

Abstract

Forecast adjustment commonly occurs when organizational forecasters adjust a statistical forecast of demand to take into account factors which are excluded from the statistical calculation. This paper addresses the question of how to measure the accuracy of such adjustments. We show that many existing error measures are generally not suited to the task, due to specific features of the demand data. Alongside the well-known weaknesses of existing measures, a number of additional effects are demonstrated that complicate the interpretation of measurement results and can even lead to false conclusions being drawn. In order to ensure an interpretable and unambiguous evaluation, we recommend the use of a metric based on aggregating performance ratios across time series using the weighted geometric mean. We illustrate that this measure has the advantage of treating over- and under-forecasting even-handedly, has a more symmetric distribution, and is robust.

Empirical analysis using the recommended metric showed that, on average, adjustments yielded improvements under symmetric linear loss, while harming accuracy in terms of some traditional measures. This provides further support to the critical importance of selecting appropriate error measures when evaluating the forecasting accuracy. The general accuracy evaluation scheme recommended in the paper is applicable in a wide range of settings including forecasting for fashion industry.

This paper is an extended version of Davydenko and Fildes [8] which appeared in the International Journal of Forecasting

Appendix 1 Alternative Representation of MASE

According to Hyndman and Koehler [20], for the scenario when forecasts are made from varying origins but with a constant horizon (here taken as one), the scaled error is defined as¹

$$ {q}_{i,t}=\frac{e_{i,t}}{{\mathrm{MAE}}_i^{\mathrm{b}}},\kern0.5em {\mathrm{MAE}}_i^{\mathrm{b}}=\frac{1}{l_i-1}{\displaystyle \sum}_{j=2}^{l_i}\left|{Y}_{i,j}-{Y}_{i,j-1}\right|, $$

where MAE ^b _i is the MAE from the benchmark (naïve) method for series i, e _i,t is the error of a forecast being evaluated against the benchmark for series i and period t, l _i is the number of elements in series i, and Y _i,j is the actual value observed at time j for series i.

Let the mean absolute scaled error (MASE) be calculated by averaging the absolute scaled errors across time periods and time series:

$$ \mathrm{MASE}=\frac{1}{{{\displaystyle \sum}}_{i=1}^m{n}_i}{\displaystyle \sum}_{i=1}^m{\displaystyle \sum}_{t\in {T}_i}\frac{\left|{e}_{i,t}\right|}{{\mathrm{MAE}}_i^{\mathrm{b}}}, $$

where n _i is the number of available values of e _i,t for series i, m is the total number of series, and T _i is a set containing time periods for which the errors e _i,t are available for series i.

Then,

$$ \mathrm{MASE}=\frac{1}{{{\displaystyle \sum}}_{i=1}^m{n}_i}{\displaystyle \sum}_{i=1}^m{\displaystyle \sum}_{t\in {T}_i}\frac{\left|{e}_{i,t}\right|}{{\mathrm{MAE}}_i^{\mathrm{b}}} $$

$$ =\frac{1}{{{\displaystyle \sum}}_{i=1}^m{n}_i}{\displaystyle \sum}_{i=1}^m\frac{{{\displaystyle \sum}}_{t\in {T}_i}\left|{e}_{i,t}\right|}{{\mathrm{MAE}}_i^{\mathrm{b}}} $$

$$ =\frac{1}{{{\displaystyle \sum}}_{i=1}^m{n}_i}{\displaystyle \sum}_{i=1}^m{n}_i\frac{\frac{1}{n_i}{{\displaystyle \sum}}_{t\in {T}_i}\left|{e}_{i,t}\right|}{{\mathrm{MAE}}_i^{\mathrm{b}}} $$

$$ =\frac{1}{{{\displaystyle \sum}}_{i=1}^m{n}_i}{\displaystyle \sum}_{i=1}^m{n}_i{r}_i,\kern0.75em {r}_i=\frac{{\mathrm{MAE}}_i}{{\mathrm{MAE}}_i^{\mathrm{b}}}, $$

where MAE _i is the MAE for series i for the forecast being evaluated against the benchmark.