The HPFOREST Procedure

The difference between loss reduction and Breiman’s method can be illustrated with a simple example. Suppose X and Z are correlated and . Should Z be assigned any importance? One answer is no: Z is not even in the formula that generates Y. The other answer is yes: observing Z provides information about Y, and therefore Z is helpful in explaining the variation of Y. An input has conditional importance if it is needed for prediction even after values of the other variables are given; Z has no conditional importance. An input has marginal importance if it is predictive of Y by itself; Z has marginal importance.

The degree to which correlations determine the final importance values depends both on the algorithm for importance and on the algorithmic parameters for the model. In the current example, if VARS_TO_TRY=2, then X and Z compete in every node, PROC HPFOREST selects X to split almost all the nodes, and all importance measures assign Z negligible importance. If VARS_TO_TRY=1, then PROC HPFOREST must use Z to split some nodes, and it assigns some importance to Z. In general, loss reduction assigns more importance to correlated variables than Breiman’s method, and Breiman’s method assigns more importance than a conditional permutation method that is introduced in Strobl et al. (2008) and summarized in the section Strobl’s Method.

Figure 7.15 shows the result from generating 25 samples at each of several correlation values that range from 0 to 1. Each sample has 500 bivariate correlated normal observations. PROC HPFOREST is run with VARS_TO_TRY=1 and outputs the proportion of total importance that is assigned to Z. Figure 7.15 has a point for each of three measures of importance that are evaluated on each sample. Unless the correlation equals 1, Breiman’s method and Strobl’s method assign little importance to Z. All methods choose X as the most important variable unless they are perfectly correlated.

Figure 7.15: Z’s Proportion of Total Importance

Archer and Kimes (2008) present a simulation that compares the variable importance methods. To emulate genomic data, correlated variables appear in groups, and at most one variable in a correlated group is in the regression equation that generates the target. Both forest methods are about as good as using a method of variable importance from regression. This is consistent with Figure 7.15, which deems X the most important variable by every method and every correlation value except a correlation of 1.