REGSELECT Procedure

(View the complete code for this example.)

This example continues the investigation of the baseball data set that were introduced in the section Getting Started: REGSELECT Procedure. That example uses the default stepwise selection method, which is based on the SBC criterion to select a model. This example explores model selection by using LASSO and elastic net selection methods with validation.

You can load the Sashelp.Baseball data set into your CAS session by specifying your CAS engine libref in the first statement in the following DATA step:

   data mylib.baseball;
      set sashelp.baseball;
   run;

The following statements perform LASSO selection with validation for the baseball data:

ods graphics on;
proc regselect data=mylib.baseball;
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = lasso(choose=validate) plots=all;
run;

The PARTITION statement assigns observations to training and validation roles on the basis of the values of the input variable league. The CHOOSE=VALIDATE option in the SELECTION statement selects the model that yields the smallest average square error (ASE) value for the validation data.

Output 29.2.1 shows that 124 observations are used for model training, 139 observations are used for model validation, and the number of effects is 19.

The "Selection Summary" table in Output 29.2.2 shows the effect that was added or dropped at each step of the selection process together with the SBC and validation ASE for the model at each step. You see that the local minimum of SBC occurs at step 11 and that three more steps are completed before the procedure stops (because STOPHORIZON=3). The model that has lowest SBC is not used as the selected model because CHOOSE=VALIDATE. Instead, the model at step 14 (where the validation ASE achieves a local minimum value) is selected. Details of this selected model are shown in Output 29.2.3.

The coefficient panel in Output 29.2.4 enables you to visualize the selection process. In this plot, standardized coefficients of all the effects that are selected at some step of the LASSO method are plotted as a function of the step number. The panel’s lower plot shows how the criterion that is used to choose the selected model changes as effects enter or leave the model.

The criterion panel in Output 29.2.5 shows how the various fit criteria change as the LASSO selection method proceeds. This plot shows that if you had used the SBC statistic as the CHOOSE= criterion, then the selected model would be the model at step 11. However, the AIC and validation ASE continue to decline for all steps of the selection process.

The "Parameter Estimates" table in Output 29.2.3 shows that the coefficients of the parameters are in different scales. In this case, you can use the adaptive LASSO, in which adaptive weights are used for penalizing different coefficients in the L1 penalty. For more information about this selection method, see the section Adaptive LASSO Selection in Chapter 2, Shared Concepts.

The following statements perform adaptive LASSO selection with validation for the baseball data:

proc regselect data=mylib.baseball;
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = lasso(adaptive choose=validate);
run;

The results of adaptive LASSO selection are shown in Output 29.2.6. You can see that the model that is chosen by adaptive LASSO selection has fewer parameters and a lower validation ASE than the model chosen by LASSO selection.

As shown in the previous example, adaptive LASSO selection can improve model fit by adaptively penalizing regression parameters. Relaxed LASSO can also achieve this goal by mixing least squares estimates with LASSO estimates. A noteworthy feature of relaxed LASSO is that you can use the least square estimates for model evaluation after variables are chosen by LASSO selection. You can use the LSCOEFFS suboption to perform the analysis. The following statements perform relaxed LASSO selection with ordinary least squares (OLS) coefficients for the baseball data:

proc regselect data=mylib.baseball;
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = lasso(lscoeffs choose=validate);
run;

The results of LASSO selection with OLS coefficients are shown in Output 29.2.7. You can see that the selected model has fewer parameters and a lower validation ASE than the model chosen by original LASSO selection.

Instead of directly using the least squares estimates to evaluate candidate models, you can use a weighted combination of LASSO estimates and least squares estimates for this purpose. The following statements use this combination to perform relaxed LASSO selection with validation for the baseball data:

proc regselect data=mylib.baseball;
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = lasso(relaxed choose=validate);
run;

The results are shown in Output 29.2.8. You can see that the model that is chosen by relaxed LASSO selection has fewer parameters and a lower validation ASE than the model chosen by LASSO selection.

The "Relaxed LASSO Selection Summary" table in Output 29.2.9 shows that the selected kappa value is 0.1. It also shows that the minimal validation ASE shown in Output 29.2.8 is lower than both the one in original LASSO selection and the one in LASSO selection with the LSCOEFFS suboption. Note that relaxed LASSO selection uses the mixing parameter kappa to control the balance between LASSO solution and OLS solution. By comparing Output 29.2.3 with Output 29.2.9, you can see that the validation ASE from relaxed LASSO selection with a kappa value of 1 is the same as the validation ASE from original LASSO selection. By comparing Output 29.2.7 with Output 29.2.9, you can see that the validation ASE from relaxed LASSO selection with a kappa value of 0 is the same as the validation ASE from LASSO selection with OLS coefficients. For more information about this selection method, see the section Relaxed LASSO Selection in Chapter 2, Shared Concepts.

You might be able to further improve the selection by specifying a finer grid near the current optimal kappa value 0.1, as shown in the following statements:

proc regselect data=mylib.baseball;
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = lasso(relaxed(kappa=0.05 to 0.15  by 0.001) choose=validate);
run;

The "Relaxed LASSO Selection Summary" table in Output 29.2.10 shows that the new optimal kappa value is 0.117. Output 29.2.11 shows that the minimal validation ASE is further improved.

From the previous examples of relaxed LASSO selection, you can see that relaxed LASSO usually produces sparser models with equal or lower prediction error than ordinary LASSO selection produces.

If you are interested only in the catchers’ salaries, you can use the following statements to perform LASSO selection for the baseball_C data:

proc regselect data = mylib.baseball(where=(Position='C'));
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = lasso(choose=validate);
run;

Output 29.2.12 shows that 15 observations were used for training and the number of effects is 19. This implies that you cannot obtain a good result by using LASSO selection because the number of effects is greater than the number of observations. In this case, you can use elastic net selection instead of LASSO selection. For more information about this selection method, see the section Elastic Net Selection in Chapter 2, Shared Concepts.

The following statements perform elastic net selection for the baseball_C data:

proc regselect data=mylib.baseball(where=(Position='C'));
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = elasticnet(choose=validate);
   ods output ElasticNetSummary=ElasticNetSummary;
run;

You can specify the L2= suboption of the METHOD option in the SELECTION statement if you have a good estimate of the ridge regression parameter. If not, you can leave the L2= suboption unspecified; in this case, PROC REGSELECT estimates this regression parameter according to the criterion that is specified in the CHOOSE= option, by trying a series of candidate values for the ridge regression parameter. The optimal regression parameter is set to the value that achieves the minimum validation ASE.

The "Elastic Net Selection Summary" table in Output 29.2.15 shows that the optimal L2 value is 0.14677993 and the minimal validation ASE is 0.17967. Output 29.2.14 shows the details of the model that is selected by elastic net method with optimal L2 value. If you compare the ASE values for the model in Output 29.2.14 and the model in Output 29.2.13, you see that the elastic net method selects a model that has a lower validation ASE than the LASSO method does in this case.

Similar to the previous analysis, which used the adaptive LASSO selection, you can also perform adaptive elastic net selection on the data when parameter estimates has different scales. The following statements perform adaptive elastic net selection for the baseball_C data:

proc regselect data=mylib.baseball(where=(Position='C'));
   partition roleVar = league(train='National' validate = 'American');
   class division;
   model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                     yrMajor crAtBat crHits crHome crRuns crRbi
                     crBB nOuts nAssts nError;
   selection method = elasticnet(adaptive choose=validate);
run;

The "Adaptive Elastic Net Selection Summary" table in Output 29.2.16 shows that the optimal L2 value is 0.0215 and the minimal validation ASE is 0.1661. Output 29.2.17 shows that adaptive elastic net selection can produce better results in the sense of a lower validation ASE and a sparser model. For more information about this method, see the section Adaptive Elastic Net Selection in Chapter 2, Shared Concepts.