The final step in logistic regression analyses is to evaluate model fit. The best thing to do would be to apply the ﬁtted model to new data (external validity). Researchers often report a classification table crosstabulating observed and predicted “hits”. Here is a typical example of a classification table:

 Observed Predicted Miss Hit Percentage Correct Target Miss 35 34 50,7 Hit 23 69 75,0 Overall Percentage 64,6

In this example 64.6% of the observations (35 + 69 out of 161) were correctly classiﬁed. The cut value is .5 (default in SPSS) so that all cases with a model based probability higher than .5 are considered to be correctly predicted. Thus when age and gender are included in the model and 70% of women younger than 30 years are a hit, than all younger women are classified as correctly predicted. Other cutoﬀ values might be more appropriate. In case the observed sample proportion (or base rate) of “successes” is far from .5, then the cutoﬀ in the classiﬁcation matrix could be set at .3; so different cutoﬀ values result in different overall percentages correctly classiﬁed. In this example:

.20    57,1%
.30    57,1%
.40    64,6%
.50    64,6%
.60    64,6%
.70    59,0%
.80    42,9%

Key question: is 65% more than we would expect just by chance? One way to answer this question is to expect every observation to have come from the larger group. We would have gotten right 57% (23 + 69) in this example. This provides a lower bound for the expected proportion correctly classiﬁed. The observed 65.0% correctly classiﬁed is only 8 points larger, questioning the fit of the logistic regression model.

Another approach is to assume that the model has no predictive power. So observed and expected groups are independent. P(Actual group miss and Predicted group miss) = P(Actual group miss) × P(Predicted group miss) = 69/161 * 58/161 = 0.43 * 0.36 = 15%. The same calculation for the hit category gives = P(Actual group hit) × P(Predicted group hit) = 92/161 * 133/161 = 0.57 * 0.83 = 47%. That is, we would expect to get 15% + 47% = 62% of the observations correctly classiﬁed by chance. This is close to 65%.

However, 62% doesn’t take into account that the data was used to build the model as well as to evaluate its power of prediction. The proportion correctly classiﬁed will be biased upwards, probably by 25%. Therefore, the expected proportion predicted correctly by chance could be 1.25 × 62% = 77,5%. The observed 65% is lower, casting doubt on the usefulness of this logistic regression model.

Professor Harrel responded to a question by a graduate student:

Armida,

I regret putting CTABLE as an option on the old SAS PROC LOGIS which was a basis for PROC LOGISTIC. Classification tables are arbitrary and misleading so I would stay away from them.

You might build a model with and without the variable of interest and plot the two predicted probabilities against each other for more insight than what is provided by a classification table.

Frank

--

Frank E Harrell Jr

Professor and Chair Department of Biostatistics
School of Medicine, Vanderbilt University