SPARSEML Procedure

This example shows how to use the SPARSEML procedure to create a model and use the model to predict whether an email is a junk email or a regular email in the Sashelp.JunkMail data set.

The Sashelp.JunkMail data set comes from a study that classifies whether an email is junk email (coded as 1) or not (coded as 0). The data were collected by Hewlett-Packard Labs and donated by George Forman. The data set Sashelp.JunkMail contains 4,601 observations, with 2 binary variables and 57 continuous explanatory variables. The response variable, Class, is a binary indicator of whether an email is considered spam or not. The partitioning variable, Test, is a binary indicator that is used to divide the data into training and testing sets. The 57 explanatory variables are continuous variables that represent frequencies of some common words and characters and lengths of uninterrupted sequences of capital letters in emails. For more information about the Sashelp.JunkMail data set, see the partition data example in the documentation of the LOGSELECT procedure in SAS Visual Statistics: Procedures.

The following DATA step converts the Sashelp.JunkMail data to the sparse data format so that you can apply it to PROC SPARSEML. The converted table JunkMail contains three variables: Test, Class, and sparseString.

    data JunkMail;
        length sparseString $600;
        length oneVariable  $60;
        set Sashelp.JunkMail;
        array vars{*} _NUMERIC_;
        if Class=0 then do;
            sparseString = put(-1, 2.);
        end; else do;
            sparseString = put(+1, 2.);
        end;
        do i=2 to 58;
            if vars{i} ne 0 then do;
                oneVariable = strip(put(i-1, 5.))||':'||strip(put(vars{i}, 10.3));
                sparseString = strip(sparseString)||' '||oneVariable;
            end;
        end;
        keep Test Class sparseString;
    run;

The following DATA step loads the training data mylib.mailtrain into the CAS session. The training data contain about two-thirds of the data, which have a Test value of 0.

    data mylib.mailtrain;
        id = _N_;
        set JunkMail;
        where Test=0;
        drop Test;
    run;

The following DATA step loads the testing data mylib.mailtest into the CAS session. The testing data contain about one-third of the data, which have a Test value of 1.

    data mylib.mailtest;
        id = _N_;
        set JunkMail;
        where Test=1;
        drop Test;
    run;

The following statements perform the model training. In the options of the PROC SPARSEML statement, the penalty value C= is 0.1, and the value of the maximum iteration option MAXITERS= is 400. The SAVESTATE statement specifies that the trained model is saved in the analytic store mylib.mystate.

 proc sparseml data=mylib.mailtrain
     C=0.1
     MAXITERS=400
 ;
     input sparseString;
     savestate rstore=mylib.mystate;
 run;

The sparse data information, misclassification matrix, and fit statistics tables are shown in Output 36.2.1 through Output 36.2.3.

To score testing data, you can use PROC ASTORE. The following statements score the testing data table mylib.mailtest and save the results in the data table mylib.scoreout:

 proc astore;
     score data=mylib.mailtest out=mylib.scoreout
           rstore=mylib.mystate copyvars=(id Class);
 run;
 quit;

To calculate the misclassification for the testing data, the following DATA step creates the variable Score to show the predicted value:

    data scoreout;
        set mylib.scoreout;
        Score=_I_;
        if Score=-1 then Score=0;
    run;

You can compare the predicted value Score with the true value Class by using PROC FREQ. The results of the comparison are displayed in Output 36.2.4.

 proc freq data=scoreout;
     table Class*Score;
 run;

The misclassification rate for the testing data is .