The previous steps are used to forecasta numerical prediction of a future value for a specified time period for each unique combination of BY variable values the future. This ex-post forecast evaluation judges the performance of the forecasting model. After forecasting future periods, the actual data becomes available as time passes. For example, suppose that monthly forecasts are computed for the next three months into the future. After three months pass, the actual data are available. The forecasts made three months ago can now be compared to the actual data of the last three months.
The availability of the new data begs the following questions:
Some useful measures of forecast performance are the statistics of fit described in the section Model Selection Criteria. When the statistics of fit are used for performance measures, the statistics are computed from the previous predictions and the newly available actual data in the forecast horizonthe number of intervals into the future, beyond a base date, for which analyses and predictions are made.. For example, the MAPE can be computed from the previous predictions and the newly available actual data in the three-month forecast horizon.
Another useful measure of forecast performance is determining whether the newly available data fall within the previous forecasts’ confidence limitsthe upper and lower values of a (usually 95%) confidence interval. In repeated sampling, approximately (1-alpha) 100% of the resulting intervals would contain the true value of the parameter that the interval estimates (where alpha is the confidence level associated with the interval).. For example, performance could be measured by whether the newly available actual data fall outside the previous forecasts’ confidence limits in the three-month forecast horizon.
If the forecasts were judged to be accurate in the past, a poor performance measure (such as actual data outside the confidence limits) could also be indicative of a change in the underlying process. A change in behavior, an unusual eventan incident that disrupts the normal flow of any process that generates the time series. Examples of events are holidays, retail promotions, and natural disasters., or other departure from past patterns might have occurred since the forecasts were made.
Such departures from past trends might be normal and indicate the need to update the forecasting model selection for this variable, or they can be a warning of special circumstances that warrant further investigation.
Large departures from forecast can sometimes reflect data errors, changes in policies or data definitions (for example, what exactly is counted as sales), fraud, or a structural change in the market environment.