The QTRSELECT Procedure

When you have sufficient data, you can subdivide your data into three parts called the training, validation, and test data. The selection process fits models to the training data and uses the validation data to find the prediction errors for the models that are obtained in this way. This prediction error on the validation data can be used to decide when to terminate the selection process or to decide what effects to include as the selection process proceeds. Finally, after a selected model has been obtained, the test data can be used to assess how the selected model generalizes on data that played no role in selecting the model.

In some cases you might want to use only training and test data. For example, you might want to use an information criterion to decide what effects to include and when to terminate the selection process. In this case no validation data are required, but test data can still help you assess the predictive performance of the selected model. In other cases you might decide to use validation data during the selection process but forgo assessing the selected model on test data. Hastie, Tibshirani, and Friedman (2001) note that it is difficult to give a general rule for how many observations you should assign to each role. They state that a typical split might be 50% for training and 25% each for validation and testing.

You use a PARTITION statement to logically subdivide the DATA= data table into separate roles. You can name the fractions of the data that you want to reserve as test data and validation data. For example, the following statements randomly subdivide the mycas.inData data table, reserving 50% for training and 25% each for validation and testing:

proc qtrselect data=mycas.inData;
  partition fraction(test=0.25 validate=0.25);
  ...
run;

In some cases you might need to exercise more control over the partitioning of the input data table. You can do this by naming both a variable in the input data table and a formatted value of that variable that correspond to each role. For example, the following statements assign roles to the observations in the mycas.inData data table based on the value of the variable group in that data table. Observations in which the value of group is "group 1" are assigned to testing, and those whose value is "group 2" are assigned to training. All other observations are ignored.

proc qtrselect data=mycas.inData;
  partition roleVar=group(test='group 1' train='group 2')
  ...
run;

After you reserve observations for training, validation, and testing, a model fit of the training data is scored on the validation and test data, and the average check loss (ACL) is computed separately for each of these subsets. The ACL for each data role is the sum of check losses for observations in that role divided by the number of observations in that role.

The following statements illustrate the use of the ROLEVAR= option:

%let seed=321;
%let n=600;
%let p=10;

data mycas.roleExample;
   array x{&p} x1-x&p;
   length r $8;
   drop i j k;

   do i=1 to &n;
      do j=1 to &p;
         x{j} = ranuni(&seed);
      end;

      y  = x1 + x2 + x3 + ranuni(&seed);

      k = mod(i,3);
      if      k=0 then r = 'train';
      else if k=1 then r = 'validate';
      else if k=2 then r = 'test';
      output;
   end;
run;


proc qtrselect data=mycas.roleExample;
   model y = x1-x&p;
   selection method=forward(select=validate stop=sbc);
   partition rolevar=r(train='train' validate='validate' test='test');
run;

The "Number of Observations" table shown in Figure 18.14 displays the number of observations used for training, validation, and testing.

The "Selection Summary" table shown in Figure 18.15 displays the validation ACL values for each step of the selection process.

The "Fit Statistics" table shown in Figure 18.16 displays the training ACL, the validation ACL, and the testing ACL for the final model.