The HP4SCORE Procedure

This example illustrates how to plot the importance of individual observations against the values of the variable being evaluated. The plot can suggest whether certain values are more influential than others. The importance of a single observation is its contribution to the importance of the variable and can be computed from information in the output data set.

This example uses a binary target variable SPAM, which can take the value 0 or 1. PROC HP4SCORE creates variable names P_SPAM0 and P_SPAM1 for the posterior probabilities. The margin function equals the posterior probability of the actual value minus the posterior probability of the other value: P_SPAM0 minus P_SPAM1 for an observation in which SPAM has the value 0. To compute the importance of variable X for an observation, subtract the margin for the row that has Variable equal to X from the margin of the preceding row that has Variable blank. This is done in a DATA step in the example.

The example uses the Spambase data that are available from the UCI Machine Learning Repository (Lichman 2013) at http://archive.ics.uci.edu/ml/datasets/Spambase.^[1]The following SAS statements create a SAS data set from a URL. The observations represent emails, and the target is 1 if the email is an unsolicited commercial email. The value of an input variable whose name is of the form WF_string is the percentage of words in the email that match string. The value of an input variable whose name is of the form CF_char is the percentage of characters in the email that match char. An ID variable is added and is used to merge the original values of the input variables with the predicted values.

   data spambase;
      %let url=//archive.ics.uci.edu/ml/machine-learning-databases;
      infile "http:&url/spambase/spambase.data" device=url delimiter=',';
      input wf_make       wf_adress     wf_all        wf_3d      wf_our
            wf_over       wf_remove     wf_internet   wf_order   wf_mail
            wf_receive    wf_will       wf_people     wf_report  wf_addresses
            wf_free       wf_business   wf_email      wf_you     wf_credit
            wf_your       wf_font       wf_000        wf_money   wf_hp
            wf_hpl        wf_george     wf_650        wf_lab     wf_labs
            wf_telnet     wf_857        wf_data       wf_415     wf_85
            wf_technology wf_1999       wf_parts      wf_pm      wf_direct
            wf_cs         wf_meeting    wf_original   wf_project wf_re
            wf_edu        wf_table      wf_conference
            cf_semicolon  cf_parenthese cf_bracket    cf_exclamation
            cf_dollar     cf_pound
            average       longest       total
            spam;
      id = _n_;
   run;

The following code runs PROC HPFOREST on the Spambase data set; saves a model in a binary file spamModel.sav; and saves the loss reduction variable importance in a SAS table, loss_reduction_importance.

proc hpforest data=spambase maxtrees=100 vars_to_try=26;
   input w: c: average longest total/level=interval;
   target spam/level=binary;
   id id;
   ods output VariableImportance=loss_reduction_importance;
   save file="spamModel.sav";
quit;

The following statements print the first five observations in loss_reduction_importance, which are shown in Output 3.3.1:

proc print data=loss_reduction_importance(obs=5);
run;

The most important variables that are based on loss reduction are cf_dollar, wf_your, and wf_remove. The following statements compute the RBA variable importance on these three variables. The output data set (named Scored) contains four observations for each observation input from the Spambase data set. One output observation contains the standard predictions, and the other three contain modified predictions: one modification for each of the three variables being evaluated. The ID variable is used later to merge the original values of these variables into the Scored data set. The THREADS=1 option in the PERFORMANCE statement ensures that the same random branch assignments are made if you run the code again.

proc hp4score data=spambase;
   ods output VariableImportance=rba_importance;
   id id;
   performance threads=1;
   importance file="spamModel.sav" out=scored
      var=(cf_dollar wf_your wf_remove);
run;

The following DATA step uses the margin loss function to compute the importance of each observation. For a binary target variable, the margin loss function is simply the predicted probability of the actual target value minus the predicted probability of the other target value.

data importance;
   set scored;
   retain baseline 0;
   keep id importance variable;
   if variable="" then baseline=P_spam1 - P_spam0;
   else if variable ne "" then do;
      importance = baseline-(P_spam1 - P_spam0);
      if spam = 0 then importance = -importance;
      output;
   end;
run;

The following macro merges the importance of variable Varname for an observation with the observed value of Varname and then plots the result:

%macro actual_vs_importance(varname=, dataout=);
data actual;
   set spambase;
   keep &varname id;
data varimp;
   set importance;
   keep id importance;
   if variable="&varname" then output;
proc sort data=varimp; by id;
data &dataout.;
   merge actual varimp;
   by id;
run;

proc sgplot data=&dataout.;
   title "&varname importance vs &varname values";
   scatter y=importance x=&varname;
   reg y=importance x=&varname / degree=3 lineattrs=(color=orange thickness=3);
   refline 0;
run;

%mend;

%actual_vs_importance(varname=cf_dollar, dataout=cf_dollar_imp_vs_actual);
%actual_vs_importance(varname=wf_your, dataout=wf_your_imp_vs_actual);
%actual_vs_importance(varname=wf_remove, dataout=wf_remove_imp_vs_actual);

Figure 2 shows the importance of variable cf_dollar for each observation plotted against the values of cf_dollar. Most of the points have positive importance, indicating that all values of cf_dollar are important. A polynomial regression of degree 3 is superimposed. The curve rises slowly from cf_dollar to about 1.8, suggesting that values from 0 to 1.8 are increasingly influential for the prediction, and values greater than 1.8 have about the same influence. However, conclusions about values greater than 1.8 might be unreliable because the observations are relatively few.

Figure 2: The Importance versus Actual Values of cf_dollar

Figure 3 shows the importance of the variable wf_your for each observation plotted against the values of wf_your. The large number of observations that have values of wf_your up to 6 suggests that the rise of influence of wf_your is more reliable than the rise of influence for cf_dollar. However, the downturn of influence for larger values should probability be ignored because it is based on few observations.

Figure 3: The Importance versus Actual Values of wf_your

Figure 4 is the corresponding plot for the variable wf_remove and shows more dramatically what can go wrong when extreme values of the variable are rare. The polynomial regression curve rises steeply at the right side of the plot. This rise is a random result and does not represent a real relationship. It is based on two observations. The importance values of those two observations can change significantly with different values of the SEED= option in the PROC HP4SCORE statement. A proper study of the relationship between the importance of a variable and the values of the variable would require you to run PROC HP4SCORE several times with different values of the SEED= option.

Figure 4: The Importance versus Actual Values of wf_remove