Clinical prediction rules: the importance of the validation phase

doi:10.1016/j.spinee.2017.06.002

The Spine Journal

Volume 17, Issue 10, October 2017, Pages 1393-1396

https://doi.org/10.1016/j.spinee.2017.06.002 Get rights and content

Commentary On: Hicks K.E., Zhao Y., Fallah N, Rivers C, Noonan V, Plashkes T, Wai EK, Roffey DM, Tsai E, Paquet J, Attabib N, Marion T, Ahn H, Phan P. A simplified clinical prediction rule for prognosticating independent walking after spinal cord injury: a prospective study from a Canadian multicenter spinal cord injury registry. The Spine Journal. Article in press.

References (15)

National Spinal Cord Injury Statistical Center (NSCSC)
Facts and figures at a glance
C.E. Cook
Potential pitfalls of clinical prediction rules
J Man Manip Ther
(2008)
P.L. Ross et al.
Comparisons of nomograms and urologists' predictions in prostate cancer
Semin Urol Oncol
(2002)
M.W. Kattan et al.
Clinicians versus nomogram: predicting future technetium-99m bone scan positivity in patients with rising prostate-specific antigen after radical prostatectomy for prostate cancer
Urology
(2013)
J.R. Wilson et al.
A clinical prediction model for long-term functional outcome after traumatic spinal cord injury based on acute clinical and imaging factors
J Neurotrauma
(2012)
J.J. van Middendorp et al.
A clinical prediction rule for ambulation outcomes after traumatic spinal cord injury: a longitudinal cohort study
Lancet
(2011)
D.G. Altman et al.
What do we mean by validating a prognostic model?
Stat Med
(2000)

There are more references available in the full text version of this article.

Cited by (5)

Highlighting discrepancies in walking prediction accuracy for patients with traumatic spinal cord injury: an evaluation of validated prediction models using a Canadian Multicenter Spinal Cord Injury Registry
2019, Spine Journal
Citation Excerpt :
Other methods to validate predictive models that are being developed, such as a prediction-recall curve, may be more appropriate in future evaluations [21]. Despite the proposal by Hicks et al. [7] that a three-variable prediction model could provide nearly equivalent predictive accuracy to the validated five-variable model, a notion that was echoed in the recent commentary by Nater and Fehlings [11], our analyses suggest that perhaps generating prediction models with increased variable count and/or using non-LR methodologies could improve tSCI prognostication. Additional variables that could be of interest include spinal MRI findings and serum biomarkers.
Models for predicting recovery in traumatic spinal cord injury (tSCI) patients have been developed to optimize care. Several models predicting tSCI recovery have been previously validated, yet recent findings question their accuracy, particularly in patients whose prognoses are the least predictable.
To compare independent ambulatory outcomes in AIS (ASIA [American Spinal Injury Association] Impairment Scale) A, B, C, and D patients, as well as in AIS B+C and AIS A+D patients by applying two existing logistic regression prediction models.
A prospective cohort study.
Individuals with tSCI enrolled in the pan-Canadian Rick Hansen SCI Registry (RHSCIR) between 2004 and 2016 with complete neurologic examination and Functional Independence Measure (FIM) outcome data.
The FIM locomotor score was used to assess independent walking ability at 1-year follow-up.
Two validated prediction models were evaluated for their ability to predict walking 1-year postinjury. Relative prognostic performance was compared with the area under the receiver operating curve (AUC).
In total, 675 tSCI patients were identified for analysis. In model 1, predictive accuracies for 675 AIS A, B, C, and D patients as measured by AUC were 0.730 (95% confidence interval [CI] 0.622–0.838), 0.691 (0.533–0.849), 0.850 (0.771–0.928), and 0.516 (0.320–0.711), respectively. In 160 AIS B+C patients, model 1 generated an AUC of 0.833 (95% CI 0.771–0.895), whereas model 2 generated an AUC of 0.821 (95% CI 0.754–0.887). The AUC for 515 AIS A+D patients was 0.954 (95% CI 0.933–0.975) with model 1 and 0.950 (0.928–0.971) with model 2. The difference in prediction accuracy between the AIS B+C cohort and the AIS A+D cohort was statistically significant using both models (p=.00034; p=.00038). The models were not statistically different in individual or subgroup analyses.
Previously tested prediction models demonstrated a lower predictive accuracy for AIS B+C than AIS A+D patients. These models were unable to effectively prognosticate AIS A+D patients separately; a failure that was masked when amalgamating the two patient populations. This suggests that former prediction models achieved strong prognostic accuracy by combining AIS classifications coupled with a disproportionately high proportion of AIS A+D patients.
Predicting Outdoor Walking 1 Year After Spinal Cord Injury: A Retrospective, Multisite External Validation Study
2023, Journal of Neurologic Physical Therapy
Utility of transcranial magnetic stimulation in the assessment of spinal cord injury: Current status and future directions
2022, Frontiers in Rehabilitation Sciences
Validation and update of the thoracic surgery scoring system (Thoracoscore) risk model
2020, European Journal of Cardio-thoracic Surgery
Validation of a clinical prediction rule for ambulation outcome after non-traumatic spinal cord injury
2020, Spinal Cord

: FDA device/drug status: Not applicable.

: Author disclosures: AN: Nothing to disclose. MGF: Consulting: Pfizer (B), Zimmer Biomet (B), InVivo Therapeutics (B), outside the submitted work.

: The disclosure key can be found on the Table of Contents and at www.TheSpineJournalOnline.com.

View full text

CommentaryClinical prediction rules: the importance of the validation phase

Facts and figures at a glance

Potential pitfalls of clinical prediction rules

J Man Manip Ther

Comparisons of nomograms and urologists' predictions in prostate cancer

Semin Urol Oncol

Clinicians versus nomogram: predicting future technetium-99m bone scan positivity in patients with rising prostate-specific antigen after radical prostatectomy for prostate cancer

Urology

A clinical prediction model for long-term functional outcome after traumatic spinal cord injury based on acute clinical and imaging factors

J Neurotrauma

A clinical prediction rule for ambulation outcomes after traumatic spinal cord injury: a longitudinal cohort study

Lancet

What do we mean by validating a prognostic model?

Stat Med

Commentary
Clinical prediction rules: the importance of the validation phase