The GRADBOOST Procedure

Note: Input data must be in a CAS table that is accessible in your CAS session. You must refer to this table by using a two-level name. The first level must be a CAS engine libref, and the second level must be the table name. For more information, see the sections Using CAS Sessions and CAS Engine Librefs and Loading a SAS Data Set onto a CAS Server in Chapter 2, Shared Concepts.

This example illustrates transfer learning. Transfer learning augments the training data with auxiliary data and attempts to down-weight the influence of observations that are not representative of the original training data. The original training data typically come from a target population from which data are hard to get. For simplicity, this example refers to auxiliary observations that are similar to the target population as friends, and the other auxiliary observations as aliens. PROC GRADBOOST tries to identify and down-weight the aliens. This example runs PROC GRADBOOST twice without using transfer learning: once with all the data, and once without the aliens. All models are evaluated with data from the target population that are not part of the training data. The model incorporating transfer learning should provide a fit than one without it, though not as good as the one in which the alien population is removed.

The following DATA step generates three data sets: one for training that includes the friends and the aliens, one without the aliens, and a third data set with test observations held out from training.

data mycas.train mycas.noAlien mycas.test;
     array x(2)  x1 - x2;
     keep datarole y x: constantZero;

     constantZero = 0;
     call streaminit(3331333);
     do datarole = -1 to 2;
        select(datarole);
           when(-1) do;  * test data;
                      y =1;
                      n = 1000;
                    end;
           when(0)  do;  * target population;
                      y =1;
                      n = 1000;
                    end;
           when(1)  do;  * friend population;
                      y  =1;
                      n = 2000;
                    end;
           when(2)  do;  * alien population;
                      y= -1;
                      n = 1000;
                    end;
        end;

        nhalf = n/2;
        mu    = 0.5;

        do i = 1 to n;
           do j = 1 to 2;
              x(j)  = rand('normal', mu);
           end;
           select(datarole);
               when(-1)  output mycas.test;
               when(0,1) output mycas.train mycas.noAlien;
               when(2)   output mycas.train;
           end;
           if i = nhalf then do;
              mu = -mu;
              y  = -y;
           end;
        end;
     end;
run;

The following macro invokes PROC GRADBOOST twice: first to train the model, and then again to apply the model to the test data and output fits statistics. The TRANSFERLEARN statement specifies the variable that identifies the auxiliary observations. When that variable is zero for all observations, transfer learning is not done. This happens when constantZero is passed to the macro.

The BURN=10 option in the TRANSFERLEARN statement delays down-weighting until tree 11.

%macro runProc(data=, auxVar=, outfit=);
  proc gradboost data=&data. outmodel=mycas.model seed=3331333;
     input x: /level=interval;
     target y /level=nominal;
     transferLearn &auxVar. / burn=10;
  run;
  proc gradboost data=mycas.test inmodel=mycas.model;
     output out=mycas.score;
     ods output FitStatistics=&outfit.;
  run;
%mend;

%runProc(data=mycas.train,   auxVar=datarole, outfit=fit_train);
%runProc(data=mycas.train,   auxVar=constantZero, outfit=fit_noTransfer);
%runProc(data=mycas.noAlien, auxVar=constantZero, outfit=fit_noAlien);

The following code combines the average square error from the three models into a single table.

   %macro extractFit(data=, infit=);
      data &data.;
        set &infit.;
        keep trees ase;
        rename ase = &data._ase;
      run;
   %mend;

   %extractFit(data=train, infit=fit_train);
   %extractFit(data=noTransfer, infit=fit_noTransfer);
   %extractFit(data=noAlien, infit=fit_noAlien);

   data result;
     merge train noTransfer noAlien;
     by trees;
   run;

The following code plots the average square error for each model by the number of trees in the model:

proc template;
     define statgraph transferLearning;
         begingraph;
             layout overlay;
             scatterplot y=train_ase
                         x=trees / markerattrs=(color=blue)
                                   name='with'
                                   legendlabel="With Transfer Learning";
             scatterplot y=noTransfer_ase
                         x=trees / markerattrs=(color=red)
                                   name='without'
                                   legendlabel="Without Transfer Learning";
             scatterplot y=noAlien_ase
                         x=trees / markerattrs=(color=brown)
                                   name='noAliens'
                                   legendlabel="Without Aliens";

             discretelegend 'without' 'with' 'noAliens';
             endlayout;
         endgraph;
     end;
 run;

 proc sgrender data=result template=transferLearning;
 run;

Output 11.2.1 shows that the fit with transfer learning is better than without it, though not as good as when all the alien observations are removed from the data. The fit with transfer learning is identical to the fit without it for the first 10 trees because down-weighting does not begin until tree 11 in this example.

Output 11.2.1: ASE versus Number of Trees for Three Models