The HP4SCORE Procedure

This example compares loss reduction variable importance with random branch assignments (RBA). Measures of importance often inflate the importance of correlated variables at the expense of independent variables. RBA importance mitigates this effect more than other measures (Neville and Tan 2014).

The input variables , 1, 2, …12, are generated as normal(0,1), with correlation of 0.9 between pairs of to and all other pairs independent. The target variable Y is generated as

The following code generates 10,000 observations for training and 10,000 observations for validation:

   data cov;
      input x1-x6;
      datalines;
      1   0.9 0.9 0.9 0.9 0.9
      0.9 1   0.9 0.9 0.9 0.9
      0.9 0.9 1   0.9 0.9 0.9
      0.9 0.9 0.9 1   0.9 0.9
      0.9 0.9 0.9 0.9 1   0.9
      0.9 0.9 0.9 0.9 0.9 1
      ;
   run;

%macro make_data(n=10000, nx=12, seed=54321, out=corr);
proc simnormal data=cov
   out = osim(drop=Rnum)
   numreal = &n
   seed = &seed.;
   var x1-x6;
run;

data uncorr;
   call streaminit(&seed.);
   do i=1 to &n.;
      %do j = 7 %to &nx.  ;
         x&j. = rand('normal', 0, 1);
      %end;
      e = rand('normal', 0, 1);
      output;
   end;
run;

data &out.;
   keep x: y;
   merge osim uncorr;
   y = 4*x1 + 4*x2 + 2*x3 + 2*x4
     - 4*x7 - 4*x8 - 2*x9 - 2*x10 + e;
run;
%mend;

%make_data(n=10000, nx=12, seed=54323, out=train);
%make_data(n=10000, nx=12, seed=65437, out=valid);

data train;
   set train;
   is_valid = 0;
data valid;
   set valid;
   is_valid = 1;
data train_valid;
   set train valid;
run;

The following code runs PROC HPFOREST on the simulated data and outputs a table of loss reduction variable importance:

proc hpforest data=train_valid
   maxdepth=6 maxtrees=100 vars_to_try=8 leafsize=10;
   input x:/level=interval;
   target y/level=interval;
   partition var=is_valid (train=0, valid=1);
   ods output VariableImportance=loss_reduction_importance;
   save file="simModel.sav";
quit;

proc print data=loss_reduction_importance;
run;

In the following code, PROC HP4SCORE imports the forest model that was created by PROC HPFOREST and computes the RBA importance based on the validation data:

proc hp4score data=valid;
   ods output VariableImportance=rba_importance_valid;
   performance threads=1;
   importance file="simModel.sav" role=valid out=_NULL_
   var=(x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12);
run;

proc print data=rba_importance_valid;
run;

Output 3.2.1 shows the loss reduction variable importance from PROC HPFOREST. The variables are ranked in the correct order: X1, X2, X7, and X8 are ranked higher than X3, X4, X9, and X10, which in turn are ranked higher than the variables that do not determine the target Y. However, the bias for correlated input variables is apparent in every loss measure. For example, the importance of X1 and X2 based on the mean square error (MSE) is 67 and 45 respectively, compared to 11 for X7 and X8 even though X7 and X8 are equally important in determining the target Y. The ratio is for the mean square error (MSE) and 3.3 for the average absolute error (AAE). For validation data, these values increase to 5.7 and 3.8.

Output 3.2.2 shows the RBA variable importance based on the validation data from PROC HP4SCORE. Again the variables are ranked correctly and the bias for correlated input variables is apparent. However, the bias is much smaller in RBA than in loss reduction. The values for X1 and X2 divided by those for X7 and X8 is 2.2 instead of 5.2 for MSE, and is 1.7 instead of 3.3 for AAE. Although not perfect, the RBA is evidently much less influenced by correlations and this is the reason for recommending them. Neville and Tan (2014) present evidence that the RBA method is also less influenced than Breiman’s method (Breiman and Cutler 2003) as coded in the randomForest() function in R (Liaw and Wiener 2002).