The REGSELECT Procedure

This example continues the investigation of the baseball data set that was introduced in the section Getting Started: REGSELECT Procedure and in LASSO and Elastic Net Selection. That example uses convex penalized selection—the LASSO and elastic net methods—to select an optimal model. In comparison, this example explores model selection by using folded concave penalized selection—the smoothly clipped absolute deviation (SCAD) and minimax concave penalty (MCP)—to select an optimal model.

The following statements perform SCAD selection for the baseball data:

   proc regselect data=mycas.baseball;
       partition roleVar = league(train='National' validate='American');
       class division;
       model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                         yrMajor crAtBat crHits crHome crRuns crRbi
                         crBB nOuts nAssts nError;
       selection  method = scad(choose=validate) details=all;
   run;

The PARTITION statement assigns observations to training and validation roles on the basis of the values of the input variable league. The METHOD=SCAD option requests SCAD selection, and the CHOOSE=VALIDATE suboption in the SELECTION statement selects the model that yields the lowest average square error (ASE) value for the validation data.

The "Selection Information" table in Output 22.3.1 shows the information that is relevant to SCAD selection, including the solver type (mixed integer linear programming, or MILP) and the ranges of the regularization parameters.

Recall that the SCAD estimator solves the following minimization problem:

where the takes the form

This section explains in detail how PROC REGSELECT computes the reported regularization parameter ranges. The values of the grid points of are by default. In addition, the number of nonintercept parameters is . Based on this information and the standard deviation of the response variable, the maximum and minimum values of are computed in the following way by using the formulas in the section Folded Concave Penalized Selection in Chapter 2, Shared Concepts:

The default pattern of the grid is such that the values of the grid are evenly spaced on the logarithmic scale, based on the computed and values.

The "Selection Summary" table in Output 22.3.2 shows the values of and together with the validation ASE and objective value at each step. The model that has the lowest validation ASE value is chosen as the final model.

Output 22.3.3 shows the details of this selected model, including fit statistics and parameter estimates. Notice that the model selected by the SCAD method has lower training and validation ASE values than the models selected by the LASSO method (Output 22.2.3) and the adaptive LASSO method (Output 22.2.6). This example demonstrates the advantages of nonconvex penalties over convex penalties.

Furthermore, you can find that for each fixed , the number of effects increases as the value of decreases to zero.

You can also directly supply your own choice of regularization parameters to obtain a model fit. The following program produces an equivalent final model by taking the selected and values from the previous analysis:

   proc regselect data=mycas.baseball;
       partition roleVar = league(train='National' validate='American');
       class division;
       model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                         yrMajor crAtBat crHits crHome crRuns crRbi
                         crBB nOuts nAssts nError;
      selection  method = scad(alpha=2.7  lambda=0.0154112628
                 solver=MILP  choose=validate);
   run;

Suppose you have a good estimate of the regularization parameter , and you want to use the nonlinear programming (NLP) solver instead of the default MILP solver to search for the best regularization parameter . You can perform SCAD selection in the following way:

   proc regselect data=mycas.baseball;
       partition roleVar = league(train='National' validate='American');
       class division;
       model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                         yrMajor crAtBat crHits crHome crRuns crRbi
                         crBB nOuts nAssts nError;
       selection  method = scad(alpha=2.7 solver=NLP
                  choose=validate) details=all;
   run;

The "Selection Summary" table in Output 22.3.4 shows the optimal validation ASE as well as the value of the objective function for the SCAD selection with the NLP solver. Both values are larger than the ones that are produced by using the MILP solver, shown in Output 22.3.2. In general, the NLP solver is faster but more prone to return local minima because of possible nonconvexity of the reformulated objective function.

Like SCAD selection, MCP selection also applies a nonconvex penalty to the penalized least squares objective function. The penalty function takes the following form:

The following statements perform MCP selection with a fixed value of 1.7. The default solver is MILP.

   proc regselect data=mycas.baseball;
       partition roleVar = league(train='National' validate='American');
       class division;
       model logSalary = division nAtBat nHits nHome nRuns nRBI nBB
                         yrMajor crAtBat crHits crHome crRuns crRbi
                         crBB nOuts nAssts nError;
       selection  method = mcp(alpha=1.7 choose=validate) details=all;
   run;

Output 22.3.6 shows that the model selected by MCP method also has lower training and validation ASE values than those produced by the LASSO method (Output 22.2.3) and the adaptive LASSO method (Output 22.2.6).

For more information about the SCAD and MCP selection methods, see the section Folded Concave Penalized Selection in Chapter 2, Shared Concepts.