Risk prediction models are important tools in clinical decision-making and prognosis often takes the form of multiple categories. We have compared two commonly used methods for modeling multicategory outcomes, dichotomized logistic regression and multinomial logit regression, in an application of predicting mortality and neurodevelopmental impairment among extremely preterm infants. Because the outcome has three ordinal categories, we also used an alternative multinomial logit model, continuation-ratio logit model.
We assessed both discrimination and calibration of the estimated models. Consistent with the findings by Biesheuvel et al. and Roukema et al. [1, 2], our results showed that the logistic models and continuation-ratio logit models had similarly satisfactory discrimination in predicting death and survival without neurodevelopmental impairment. These models also had similar calibration as measured by the average predicted probabilities and by calibration intercepts and slopes. However, the sum of all predicted probabilities from the logistic models for each infant ranged from 87.7–124.0%. We found that the logistic model of neurodevelopmental impairment had slightly smaller C-statistics and among infants whose sum of all predicted probabilities did not equal 100% it had worse calibration.
To overcome potential bias due to death as a competing risk, we applied an extension of logistic regression method, logistic competing risks regression, to develop a prediction model of neurodevelopmental impairment. Because time to diagnosis of NDI was determined only at one fixed time, 22–26 months’ age corrected for prematurity, the estimated odds ratios for predictor variables were close to those in the logistic model of neurodevelopmental impairment. We also estimated a logistic model of composite of neurodevelopmental impairment or death and showed that it could not be used for predicting neurodevelopmental impairment. Competing risks are not only of statistical interest, but also can be of substantive interest. In pediatric research, for example, it is increasingly concerned how the risk and burden of illness among extremely preterm infants are changing with improved survival [19]. Further investigation into statistical methods for modeling competing risks and collection of more detailed data on event time will be needed.
Constraining sum of all predicted probabilities of outcome categories for each patient to 100% and accommodating competing risks are important considerations in the validation of prediction models for multicategory outcomes. Additionally, there are other statistical and practical issues that should be considered. We prepared a list of these issues for comparison between dichotomized logistic and multinomial logit regression in Table 4. In general, simplicity in model interpretation facilitates acceptance and usage of a model by clinicians. Flexibility in model fitting to allow outcome category-specific predictor variables helps avoid statistical overfitting and including random-effect terms to accommodate patient heterogeneity by hospital improves model calibration [3].
Table 4
Comparison of predictive modeling methods on other statistical and practical issues
| Methods for risk prediction of multicategory outcomes |
Issues to consider | Dichotomized logistic regression | Continuation-ratio logit regression | Logistic competing risks regression |
Interpretation of predictor effects | Odds ratio | Conditional odds ratio dependent on ordered outcome category | Odds ratio |
Constrains sum of all predicted probabilities to 100% | No | Yes | No |
Allows inclusion of random-effect terms | Yes | Yes | No/Robust variance estimates |
Allows inclusion of outcome category-specific predictor variables | Yes | Yes | Yes |
Accommodates competing risks | No | Yes | Yes |
Availability in statistical software | SAS, Stata, R | SAS, Stata, R | R |
Both logistic regression and logistic competing risks regression produce odds ratio estimates for predictor variables but have the flaw that sum of all predicted probabilities of outcome categories for each patient is not constrained to 100%. Logistic regression also has the advantages of allowing for outcome-category specific predictor variables and random-effect terms, and wide availability of statistical programs for model estimation. Logistic competing risks regression accounts for competing risks but does not allow the inclusion of random-effect terms for patient heterogeneity and requires time-to-event data.
Multinomial logit regression constrains sum of all predicted probabilities of outcome categories for each patient to 100%. But a standard multinomial logit model has some known limitations, including difficulty to explain the prediction results to clinicians or patients due to the use of a fixed reference category, lack of flexibility to allow for outcome category-specific predictors and complications caused by the inclusion of random-effect terms. As an alternative for ordinal outcome, we estimated a continuous-ratio logit model to predict the probability of death and the probability of neurodevelopmental impairment conditional on surviving. This addressed the need of clinicians and patients for separate information on death and impairment, which could be valued differently in their decision about treatment options. It also afforded us the statistical benefits of including random-effect terms and outcome category-specific predictor variables of neurodevelopmental impairment in the model. The infant outcomes in our data have been found to vary significantly across hospitals after controlling for infant characteristics [15]. To improve the modest model performance in predicting neurodevelopmental impairment, we hope to be able to add more variables predictive of this outcome in the future [20, 21].