The HPFOREST Procedure

The HPFOREST procedure computes the average square error measure of prediction error. For a binary or nominal target, PROC HPFOREST also computes the misclassification rate and the log-loss.

The average square error for an interval, the average square error for a nominal target, the misclassification rate, and the log-loss are defined respectively as

where is the target prediction of observation i, equals 1 if the nominal target value j occurs in observation i or equals 0 of it does not occur there, is the predicted probability of nominal target value j for observation i, N is the number of observations, J is the number of nominal target values (classes), and is truncated away from 0 and 1:

The definitions are valid whether is the usual model prediction, the out-of-bag prediction, or the no-model prediction. The three predictions result in three different estimates of ASE. The model has some predictive ability if the out-of-bag estimate of fit is smaller than the no-model estimate. The ASE that is based on the usual model predictions of the original training data is usually optimistic and smaller than what its value will be on future data.