The LOGISTIC Procedure

Example 73.8 Comparing Receiver Operating Characteristic Curves

(View the complete code for this example.)

DeLong, DeLong, and Clarke-Pearson (1988) report on 49 patients with ovarian cancer who also suffer from an intestinal obstruction. Three (correlated) screening tests are measured to determine whether a patient will benefit from surgery. The three tests are the K-G score and two measures of nutritional status: total protein and albumin. The data are as follows:

data roc;
   input alb tp totscore popind @@;
   totscore = 10 - totscore;
   datalines;
3.0 5.8 10 0   3.2 6.3  5 1   3.9 6.8  3 1   2.8 4.8  6 0
3.2 5.8  3 1   0.9 4.0  5 0   2.5 5.7  8 0   1.6 5.6  5 1
3.8 5.7  5 1   3.7 6.7  6 1   3.2 5.4  4 1   3.8 6.6  6 1
4.1 6.6  5 1   3.6 5.7  5 1   4.3 7.0  4 1   3.6 6.7  4 0
2.3 4.4  6 1   4.2 7.6  4 0   4.0 6.6  6 0   3.5 5.8  6 1
3.8 6.8  7 1   3.0 4.7  8 0   4.5 7.4  5 1   3.7 7.4  5 1
3.1 6.6  6 1   4.1 8.2  6 1   4.3 7.0  5 1   4.3 6.5  4 1
3.2 5.1  5 1   2.6 4.7  6 1   3.3 6.8  6 0   1.7 4.0  7 0
3.7 6.1  5 1   3.3 6.3  7 1   4.2 7.7  6 1   3.5 6.2  5 1
2.9 5.7  9 0   2.1 4.8  7 1   2.8 6.2  8 0   4.0 7.0  7 1
3.3 5.7  6 1   3.7 6.9  5 1   3.6 6.6  5 1
;

In the following statements, the NOFIT option is specified in the MODEL statement to prevent PROC LOGISTIC from fitting the model with three covariates. Each ROC statement lists one of the covariates, and PROC LOGISTIC then fits the model with that single covariate. Note that the original data set contains six more records with missing values for one of the tests, but PROC LOGISTIC ignores all records with missing values; hence there is a common sample size for each of the three models. The ROCCONTRAST statement implements the nonparametric approach of DeLong, DeLong, and Clarke-Pearson (1988) to compare the three ROC curves, the REFERENCE option specifies that the K-G Score curve is used as the reference curve in the contrast, the E option displays the contrast coefficients, and the ESTIMATE option computes and tests each comparison. With ODS Graphics enabled, the plots=roc(id=prob) specification in the PROC LOGISTIC statement displays several plots, and the plots of individual ROC curves have certain points labeled with their predicted probabilities.

ods graphics on;
proc logistic data=roc plots=roc(id=prob);
   model popind(event='0') = alb tp totscore / nofit;
   roc 'Albumin' alb;
   roc 'K-G Score' totscore;
   roc 'Total Protein' tp;
   roccontrast reference('K-G Score') / estimate e;
run;

The initial model information is displayed in Output 73.8.1.

Output 73.8.1: Initial LOGISTIC Output

The LOGISTIC Procedure

Model Information
Data SetWORK.ROC
Response Variablepopind
Number of Response Levels2
Modelbinary logit
Optimization TechniqueFisher's scoring

Number of Observations Read43
Number of Observations Used43

Response Profile
Ordered
Value
popindTotal
Frequency
1012
2131

Probability modeled is popind=0.


Score Test for Global Null
Hypothesis
Chi-SquareDFPr > ChiSq
10.793930.0129


For each ROC model, the model fitting details in Outputs Output 73.8.2, Output 73.8.4, and Output 73.8.6 can be suppressed with the ROCOPTIONS(NODETAILS) option; however, the convergence status is always displayed.

The ROC curves for the three models are displayed in Outputs Output 73.8.3, Output 73.8.5, and Output 73.8.7. Note that the labels on the ROC curve are produced by specifying the ID=PROB option, and are the predicted probabilities for the cutpoints.

Output 73.8.2: Fit Tables for Popind=Alb

 
 

Model Convergence Status
Convergence criterion (GCONV=1E-8) satisfied.

Model Fit Statistics
CriterionIntercept OnlyIntercept and
Covariates
AIC52.91849.384
SC54.67952.907
-2 Log L50.91845.384

Testing Global Null Hypothesis: BETA=0
TestChi-SquareDFPr > ChiSq
Likelihood Ratio5.533910.0187
Score5.689310.0171
Wald4.686910.0304

Analysis of Maximum Likelihood Estimates
ParameterDFEstimateStandard
Error
Wald
Chi-Square
Pr > ChiSq
Intercept12.46461.59132.39880.1214
alb1-1.05200.48594.68690.0304

Odds Ratio Estimates
EffectPoint Estimate95% Wald
Confidence Limits
alb0.3490.1350.905


Output 73.8.3: ROC Curve for Popind=Alb

ROC Curve for Popind=Alb


Output 73.8.4: Fit Tables for Popind=Totscore

 
 

Model Convergence Status
Convergence criterion (GCONV=1E-8) satisfied.

Model Fit Statistics
CriterionIntercept OnlyIntercept and
Covariates
AIC52.91846.262
SC54.67949.784
-2 Log L50.91842.262

Testing Global Null Hypothesis: BETA=0
TestChi-SquareDFPr > ChiSq
Likelihood Ratio8.656710.0033
Score8.361310.0038
Wald6.384510.0115

Analysis of Maximum Likelihood Estimates
ParameterDFEstimateStandard
Error
Wald
Chi-Square
Pr > ChiSq
Intercept12.15421.24772.98080.0843
totscore1-0.76960.30466.38450.0115

Odds Ratio Estimates
EffectPoint Estimate95% Wald
Confidence Limits
totscore0.4630.2550.841


Output 73.8.5: ROC Curve for Popind=Totscore

ROC Curve for Popind=Totscore


Output 73.8.6: Fit Tables for Popind=Tp

 
 

Model Convergence Status
Convergence criterion (GCONV=1E-8) satisfied.

Model Fit Statistics
CriterionIntercept OnlyIntercept and
Covariates
AIC52.91851.794
SC54.67955.316
-2 Log L50.91847.794

Testing Global Null Hypothesis: BETA=0
TestChi-SquareDFPr > ChiSq
Likelihood Ratio3.124410.0771
Score3.112310.0777
Wald2.905910.0883

Analysis of Maximum Likelihood Estimates
ParameterDFEstimateStandard
Error
Wald
Chi-Square
Pr > ChiSq
Intercept12.82952.20651.64450.1997
tp1-0.62790.36832.90590.0883

Odds Ratio Estimates
EffectPoint Estimate95% Wald
Confidence Limits
tp0.5340.2591.099


Output 73.8.7: ROC Curve for Popind=Tp

ROC Curve for Popind=Tp


All ROC curves being compared are also overlaid on the same plot, as shown in Output 73.8.8.

Output 73.8.8: Overlay of All Models Being Compared

Overlay of All Models Being Compared


Output 73.8.9 displays the association statistics, and displays the area under the ROC curve along with its standard error and a confidence interval for each model in the comparison. The confidence interval for Total Protein contains 0.50; hence it is not significantly different from random guessing, which is represented by the diagonal line in the preceding ROC plots.

Output 73.8.9: ROC Association Table

ROC Association Statistics
ROC Model Mann-Whitney Somers' DGammaTau-a
AreaStandard
Error
95% Wald
Confidence Limits
Albumin0.73660.09270.55490.91820.47310.48090.1949
K-G Score0.72580.10280.52430.92730.45160.52170.1860
Total Protein0.64780.10000.45180.84390.29570.31070.1218


Output 73.8.10 shows that the contrast used ’K-G Score’ as the reference level. This table is produced by specifying the E option in the ROCCONTRAST statement.

Output 73.8.10: ROC Contrast Coefficients

ROC Contrast Coefficients
ROC ModelRow1Row2
Albumin10
K-G Score-1-1
Total Protein01


Output 73.8.11 shows that the 2-degrees-of-freedom test that the ’K-G Score’ is different from at least one other test is not significant at the 0.05 level.

Output 73.8.11: ROC Test Results (2 Degrees of Freedom)

ROC Contrast Test Results
ContrastDFChi-SquarePr > ChiSq
Reference = K-G Score22.53400.2817


Output 73.8.12 is produced by specifying the ESTIMATE option in the ROCCONTRAST statement. Each row shows that the curves are not significantly different.

Output 73.8.12: ROC Contrast Row Estimates (1-Degree-of-Freedom Tests)

ROC Contrast Estimation and Testing Results by Row
ContrastEstimateStandard
Error
95% Wald
Confidence Limits
Chi-SquarePr > ChiSq
Albumin - K-G Score0.01080.0953-0.17610.19760.01270.9102
Total Protein - K-G Score-0.07800.1046-0.28300.12710.55540.4561