The HPFOREST Procedure

Using the original training data to evaluate a forest model is poor practice because the forest predicts the training data much better than it predicts similar data withheld from training. Using the out-of-bag data is better practice because, with enough trees, the fit of a forest to the out-of-bag data converges to what the fit would be on similar data withheld from training. With only a few trees, the fit to the out-of-bag data is worse than what the fit would be on withheld data. Consequently, the training and out-of-bag data provide lower and upper bounds to what the error rate will be when the forest is applied to new data.

This example illustrates the difference between the misclassification rates estimated from the training and out-of-bag data. The HPFOREST procedure is run on the Spambase data. The target, SPAM, has two values: 0 indicates a legitimate e-mail, 1 indicates spam. The number of trees is set large enough for the out-of-bag misclassification error rates to converge (MAXTREES=200 or 500).

The following SAS statements create a SAS data set from an URL:

   data spambase;
      %let url=//archive.ics.uci.edu/ml/machine-learning-databases;
      infile "http:&url/spambase/spambase.data"
            device=url delimiter=',';
      input wf_make wf_adress wf_all wf_3d wf_our
            wf_over wf_remove wf_internet wf_order wf_mail
            wf_receive wf_will wf_people wf_report wf_addresses
            wf_free wf_business wf_email wf_you wf_credit
            wf_your wf_font wf_000 wf_money wf_hp
            wf_hpl wf_george wf_650 wf_lab wf_labs
            wf_telnet wf_857 wf_data wf_415 wf_85
            wf_technology wf_1999 wf_parts wf_pm wf_direct
            wf_cs wf_meeting wf_original wf_project wf_re
            wf_edu wf_table wf_conference
            cf_semicolon cf_parenthese cf_bracket cf_exclamation
            cf_dollar cf_pound
            average longest total spam;
   run;

proc hpforest data=spambase maxtrees=200;
   input w: c: average longest total/level=interval;
   target spam/level=binary;
   ods output FitStatistics=fitstats(rename=(Ntrees=Trees));
run;

data fitstats;
   set fitstats;
   label Trees = 'Number of Trees';
   label MiscAll = 'Full Data';
   label Miscoob = 'OOB';
run;

proc sgplot data=fitstats;
   title "OOB vs Training";
   series x=Trees y=MiscAll;
   series x=Trees y=MiscOob/lineattrs=(pattern=shortdash thickness=2);
   yaxis label='Misclassification Rate';
run;
title;

Figure 16: Plot of OOB versus Training Misclassification Rate

Figure 16 shows the misclassification rate is worse (larger) based on the out-of-bag (OOB) data, and more trees are needed for the out-of-bag rates to level off. Both characteristics are typical of a forest.