The final step in logistic regression analyses is to evaluate model fit. The best thing to do would be to apply the fitted model to new data (external validity). Researchers often report a classification table crosstabulating observed and predicted “hits”. Here is a typical example of a classification table:

 

Observed    Predicted    
    Miss            Hit             Percentage Correct
Target Miss 35  34  50,7
  Hit 23  69  75,0
Overall Percentage       64,6

 

In this example 64.6% of the observations (35 + 69 out of 161) were correctly classified. The cut value is .5 (default in SPSS) so that all cases with a model based probability higher than .5 are considered to be correctly predicted. Thus when age and gender are included in the model and 70% of women younger than 30 years are a hit, than all younger women are classified as correctly predicted. Other cutoff values might be more appropriate. In case the observed sample proportion (or base rate) of “successes” is far from .5, then the cutoff in the classification matrix could be set at .3; so different cutoff values result in different overall percentages correctly classified. In this example:

.20    57,1%
.30    57,1%
.40    64,6%
.50    64,6%
.60    64,6%
.70    59,0%
.80    42,9%

Key question: is 65% more than we would expect just by chance? One way to answer this question is to expect every observation to have come from the larger group. We would have gotten right 57% (23 + 69) in this example. This provides a lower bound for the expected proportion correctly classified. The observed 65.0% correctly classified is only 8 points larger, questioning the fit of the logistic regression model.

Another approach is to assume that the model has no predictive power. So observed and expected groups are independent. P(Actual group miss and Predicted group miss) = P(Actual group miss) × P(Predicted group miss) = 69/161 * 58/161 = 0.43 * 0.36 = 15%. The same calculation for the hit category gives = P(Actual group hit) × P(Predicted group hit) = 92/161 * 133/161 = 0.57 * 0.83 = 47%. That is, we would expect to get 15% + 47% = 62% of the observations correctly classified by chance. This is close to 65%.

However, 62% doesn’t take into account that the data was used to build the model as well as to evaluate its power of prediction. The proportion correctly classified will be biased upwards, probably by 25%. Therefore, the expected proportion predicted correctly by chance could be 1.25 × 62% = 77,5%. The observed 65% is lower, casting doubt on the usefulness of this logistic regression model. 

See Jeffrey S. Simonoff (2012) for more background information >> 

 


Professor Harrel responded to a question by a graduate student:

 

Armida, 

I regret putting CTABLE as an option on the old SAS PROC LOGIS which was a basis for PROC LOGISTIC. Classification tables are arbitrary and misleading so I would stay away from them.

You might build a model with and without the variable of interest and plot the two predicted probabilities against each other for more insight than what is provided by a classification table.

Frank

--

Frank E Harrell Jr

Professor and Chair Department of Biostatistics
School of Medicine, Vanderbilt University