Forecasting Process Summary
Performance Step
The previous steps are used to forecast the future. This ex-post forecast evaluation judges the performance of the forecasting model. After forecasting future periods, the actual data becomes available as time passes. For example, suppose that monthly forecasts are computed for the next three months into the future. After three months pass, the actual data are available. The forecasts made three months ago can now be compared to the actual data of the last three months.
The availability of the new data begs the following questions:
How well are you forecasting?
Why are you forecasting poorly?
If you were forecasting well before, what went wrong?
Some useful measures of forecast performance are the statistics of fit described in the section Statistics of Fit. When the statistics of fit are used for performance measures, the statistics are computed from the previous predictions and the newly available actual data in the forecast horizon. For example, the MAPE can be computed from the previous predictions and the newly available actual data in the three-month forecast horizon.
Another useful measure of forecast performance is determining whether the newly available data fall within the previous forecasts’ confidence limits. For example, performance could be measured by whether or not the newly available actual data fall outside the previous forecasts’ confidence limits in the three-month forecast horizon.
If the forecasts were judged to be accurate in the past, a poor performance measure (such as actual data outside the confidence limits) could also be indicative of a change in the underlying process. A change in behavior, an unusual event, or other departure from past patterns might have occurred since the forecasts were made.
Such departures from past trends might be normal and indicate the need to update the forecasting model selection for this variable, or they can be a warning of special circumstances that warrant further investigation.
Large departures from forecast can sometimes reflect data errors, changes in policies or data definitions (for example, what exactly is counted as sales), fraud, or a structural change in the market environment.