Skip to main content

Advertisement

SpringerLink
Log in

Quantifying the Value of Multidimensional Assessment Models for Acute Concussion: An Analysis of Data from the NCAA-DoD Care Consortium

  • Original Research Article
  • Published:
Sports Medicine Aims and scope Submit manuscript

Abstract

Background

Many concussion assessment methods exist, but few studies quantify the performance of these methods to determine which can best assess acute concussion alone or in combination.

Objectives

The objectives of this study were to evaluate: (1) selected concussion assessments for acute concussion assessment; (2) the utility of change scores for acute concussion assessment; and (3) concussion assessment capabilities when constrained to limited clinical data or objective clinical measures.

Methods

The ‘acute concussion’ group contained assessments from < 6 h post-injury (n = 560) and 24–48 h post-injury (n = 733). The ‘normal performance’ group contained assessments from baseline testing (n = 842) and unrestricted return to play (n = 707) timepoints. Univariate and multivariate logistic regression models were created separately for < 6- and 24- to 48-h timepoints. Models were evaluated on sensitivity, specificity, and area under the receiver operating characteristic curve.

Results

Within the univariate analysis, Sport Concussion Assessment Tool symptom assessments had the highest combination of sensitivity, specificity, and area under the receiver operating characteristic curve, with values up to 0.93, 0.97, and 0.98, respectively. Full models had a sensitivity, specificity, and area under the receiver operating characteristic curve up to 0.94, 0.97, and 0.99, respectively, and outperformed all univariate models, raw score models, and objective models. Objective models were outperformed by all multivariate models and the univariate models containing only Sport Concussion Assessment Tool symptom assessments.

Conclusion

Results support the use of multidimensional assessment batteries over single instruments and suggest the importance of self-reported symptoms in acute concussion assessment. Balance assessments, however, may not provide additional benefit when symptom and neurocognitive assessments are available. Additionally, change scores provide some clinical utility over raw scores, but the difference may not be clinically meaningful.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. McCrory P, Meeuwisse W, Dvorak J, Aubry M, Bailes J, Broglio S, et al. Consensus statement on concussion in sport: the 5th international conference on concussion in sport held in Berlin, October 2016. Br J Sports Med. 2017;51:838–47.

    PubMed  Google Scholar 

  2. Zuckerman SL, Kerr ZY, Yengo-Kahn A, Wasserman E, Covassin T, Solomon GS. Epidemiology of sports-related concussion in NCAA athletes from 2009-2010 to 2013-2014: incidence, recurrence, and mechanisms. Am J Sports Med. 2015;43:2654–62.

    Article  PubMed  Google Scholar 

  3. Giza CC, Kutcher JS, Ashwal S, Barth J, Getchius TSD, Gioia GA, et al. Summary of evidence-based guideline update: evaluation and management of concussion in sports: report of the Guideline Development Subcommittee of the American Academy of Neurology. Neurology. 2013;80:2250–7.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Broglio SP, Cantu RC, Gioia GA, Guskiewicz KM, Kutcher J, Palm M, et al. National Athletic Trainers’ Association position statement: management of sport concussion. J Athl Train. 2014;49:245–65.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Harmon KG, Drezner JA, Gammons M, Guskiewicz KM, Halstead M, Herring SA, et al. American Medical Society for Sports Medicine position statement: concussion in sport. Br J Sports Med. 2013;47:15–26.

    Article  PubMed  Google Scholar 

  6. McCrory P, Meeuwisse WH, Kutcher JS, Jordan BD, Gardner A. What is the evidence for chronic concussion-related changes in retired athletes: behavioural, pathological and clinical outcomes? Br J Sports Med. 2013;47:327–30.

    Article  PubMed  Google Scholar 

  7. Randolph C, Karantzoulis S, Guskiewicz K. Prevalence and characterization of mild cognitive impairment in retired national football league players. J Int Neuropsychol Soc. 2013;19:873–80.

    Article  PubMed  Google Scholar 

  8. Guskiewicz KM, Marshall SW, Bailes J, McCrea M, Cantu RC, Randolph C, et al. Association between recurrent concussion and late-life cognitive impairment in retired professional football players. Neurosurgery. 2005;57:719–26.

    Article  PubMed  Google Scholar 

  9. Daneshvar DH, Riley DO, Nowinski CJ, McKee AC, Stern RA, Cantu RC. Long-term consequences: effects on normal development profile after concussion. Phys Med Rehabil Clin N Am. 2011;22:683–700.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Kerr ZY, Marshall SW, Harding HP, Guskiewicz KM. Nine-year risk of depression diagnosis increases with increasing self-reported concussions in retired professional football players. Am J Sports Med. 2012;40:2206–12.

    Article  PubMed  Google Scholar 

  11. Guskiewicz KM, Marshall SW, Bailes J, Mccrea M, Harding HP, Matthews A, et al. Recurrent concussion and risk of depression in retired professional football players. Med Sci Sports Exerc. 2007;39:903–9.

    Article  PubMed  Google Scholar 

  12. De Beaumont L, Lassonde M, Leclerc S, Théoret H. Long-term and cumulative effects of sports concussion on motor cortex inhibition. Neurosurgery. 2007;61:329–36.

    Article  PubMed  Google Scholar 

  13. Barr WB, McCrea M. Sensitivity and specificity of standardized neurocognitive testing immediately following sports concussion. J Int Neuropsychol Soc. 2001;7:693–702.

    Article  PubMed  CAS  Google Scholar 

  14. Broglio SP, Macciocchi SN, Ferrara MS. Sensitivity of the concussion assessment battery. Neurosurgery. 2007;60:1050–7.

    Article  PubMed  Google Scholar 

  15. Broglio SP, Ferrara MS, Sopiarz K, Kelly MS. Reliable change of the sensory organization test. Clin J Sport Med. 2008;18:148–54.

    Article  PubMed  Google Scholar 

  16. Lovell MR, Iverson GL, Collins MW, Podell K, Johnston KM, Pardini D, et al. Measurement of symptoms following sports-related concussion: reliability and normative data for the post-concussion scale. Appl Neuropsychol. 2006;13:166–74.

    Article  PubMed  Google Scholar 

  17. McCrea M, Barr WB, Guskiewicz K, Randolph C, Marshall SW, Cantu R, et al. Standard regression-based methods for measuring recovery after sport-related concussion. J Int Neuropsychol Soc. 2005;11:58–69.

    Article  PubMed  Google Scholar 

  18. Echemendia RJ, Meeuwisse W, McCrory P, Davis GA, Putukian M, Leddy J, et al. The sport concussion assessment tool 5th Edition (SCAT5). Br J Sports Med. 2017;5:1–3.

    Google Scholar 

  19. Randolph C. Baseline neuropsychological testing in managing sport-related concussion. Curr Sports Med Rep. 2011;10:21–6.

    Article  PubMed  Google Scholar 

  20. Schmidt JD, Register-Mihalik JK, Mihalik JP, Kerr ZY, Guskiewicz KM. Identifying impairments after concussion: normative data versus individualized baselines. Med Sci Sports Exerc. 2012;44:1621–8.

    Article  PubMed  Google Scholar 

  21. Echemendia RJ, Bruce JM, Bailey CM, Sanders JF, Arnett P, Vargas G. The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data. Clin Neuropsychol. 2012;26:1077–91.

    Article  PubMed  Google Scholar 

  22. Valovich McLeod TC, Bay RC, Lam KC, Chhabra A. Representative baseline values on the sport concussion assessment tool 2 (SCAT2) in adolescent athletes vary by gender, grade, and concussion history. Am J Sports Med. 2012;40:927–33.

    Article  PubMed  Google Scholar 

  23. Putukian M, Echemendia R, Dettwiler-Danspeckgruber A, Duliba T, Bruce J, Furtado JL, et al. Prospective clinical assessment using sideline concussion assessment tool-2 testing in the evaluation of sport-related concussion in college athletes. Clin J Sport Med. 2015;25:36–42.

    Article  PubMed  Google Scholar 

  24. Covassin T, Elbin RJ, Harris W, Parker T, Kontos A. The role of age and sex in symptoms, neurocognitive performance, and postural stability in athletes after concussion. Am J Sports Med. 2012;40:1303–12.

    Article  PubMed  Google Scholar 

  25. Covassin T, Buz Swanik C, Sachs ML. Sex differences and the incidence of concussions among collegiate athletes. J Athl Train. 2003;38:238–44.

    PubMed  PubMed Central  Google Scholar 

  26. Covassin T, Schatz P, Swanik CB. Sex differences in neuropsychological function and post-concussion symptoms of concussed collegiate athletes. Neurosurgery. 2007;61:345–50.

    Article  PubMed  Google Scholar 

  27. Shehata N, Wiley JP, Richea S, Benson BW, Duits L, Meeuwisse WH. Sport concussion assessment tool: baseline values for varsity collision sport athletes. Br J Sports Med. 2009;43:730–4.

    Article  PubMed  CAS  Google Scholar 

  28. Bruce JM, Echemendia RJ. Concussion history predicts self-reported symptoms before and following a concussive event. Neurology. 2004;63:1516–8.

    Article  PubMed  Google Scholar 

  29. Covassin T, Stearne D, Elbin R. Concussion history and postconcussion neurocognitive performance and symptoms in collegiate athletes. J Athl Train. 2008;43:119–24.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Guskiewicz KM, Mccrea M, Marshall SW, Cantu RC, Randolph C, Barr W, et al. Cumulative effects associated with recurrent concussion in football players: the NCAA Concussion Study. J Am Med Assoc. 2003;290:2549–55.

    Article  CAS  Google Scholar 

  31. Broglio SP, McCrea M, McAllister T, Harezlak J, Katz B, Hack D, et al. A national study on the effects of concussion in collegiate athletes and US Military Service Academy members: the NCAA–DoD concussion assessment, research and education (CARE) consortium structure and methods. Sport Med. 2017;47:1437–51.

    Article  Google Scholar 

  32. Carney N, Ghajar J, Jagoda A, Bedrick S, Davis-O’Reilly C, Du Coudray H, et al. Concussion guidelines step 1: systematic review of prevalent indicators. Neurosurgery. 2014;75:3–15.

    Article  Google Scholar 

  33. McCrory P, Meeuwisse WH, Aubry M, Cantu RC, Dvorák J, Echemendia RJ, et al. Consensus statement on concussion in sport. In: the 4th international conference on concussion in sport held in Zurich, November 2012. Br J Sports Med. 2013;47:255–79.

  34. Asken BM, McCrea MA, Clugston JR, Snyder AR, Houck ZM, Bauer RM. “Playing through it”: delayed reporting and removal from athletic activity after concussion predicts prolonged recovery. J Athl Train. 2016;51:329–35.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Elbin RJ, Sufrinko AM, Schatz P, French J, Henry L, Burkhart S, et al. Removal from play after concussion and recovery time. Pediatrics. 2016;138:e20160910.

    Article  PubMed  PubMed Central  Google Scholar 

  36. McCrea M, Kelly JP, Randolph C, Kluge J, Bartolic E, Finn G, et al. Standardized assessment of concussion (SAC): on-site mental status evaluation of the athlete. J Head Trauma Rehabil. 1998;13:27–35.

    Article  PubMed  CAS  Google Scholar 

  37. Concussion in Sport Group. Sport concussion assessment tool: 3rd edition. Br J Sports Med. 2013;47(5):259.

    Google Scholar 

  38. Bell DR, Guskiewicz KM, Clark MA, Padua DA. Systematic review of the balance error scoring system. Sports Health. 2011;3:287–95.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Van Buuren S, Boshuizen H, Knook D. Multiple imputation of missing blood pressure covariates in survival analysis. Stat Med. 1999;18:681–94.

    Article  PubMed  Google Scholar 

  40. McCrea M, Guskiewicz KM, Marshall SW, Barr W, Randolph C, Cantu RC, et al. Acute effects and recovery time following concussions in collegiate football players. JAMA. 2004;290:2556–63.

    Article  Google Scholar 

  41. McCrea M, Guskiewicz K, Randolph C, Barr WB, Hammeke TA, Marshall SW, et al. Incidence, clinical course, and predictors of prolonged recovery time following sport-related concussion in high school and college athletes. J Int Neuropsychol Soc. 2013;19:22–33.

    Article  PubMed  Google Scholar 

  42. Bozdogan H. Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika. 1987;52:345–70.

    Article  Google Scholar 

  43. Williamson IJS, Goodman D. Converging evidence for the under-reporting of concussions in youth ice hockey. Br J Sports Med. 2006;40:128–32.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  44. Pepe MS, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. Stata J. 2009;9:1–16.

    PubMed  PubMed Central  Google Scholar 

  45. Meehan WP, Mannix RC, Stracciolini A, Elbin RJ, Collins MW. Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not. J Pediatr. 2013;163:721–5.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Sufrinko A, McAllister-Deitrick J, Womble M, Kontos A. Do sideline concussion assessments predict subsequent neurocognitive impairment after sport-related concussion? J Athl Train. 2017;52:676–81.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Sufrinko AM, Marchetti GF, Cohen PE, Elbin RJ, Re V, Kontos AP. Using acute performance on a comprehensive neurocognitive, vestibular, and ocular motor assessment battery to predict recovery duration after sport-related concussions. Am J Sports Med. 2017;45:1187–94.

    Article  PubMed  Google Scholar 

  48. Register-Mihalik JK, Guskiewicz KM, Mihalik JP, Schmidt JD, Kerr ZY, McCrea MA. Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery. J Head Trauma Rehabil. 2013;28:274–83.

    Article  PubMed  Google Scholar 

  49. Resch JE, Brown CN, Schmidt J, Macciocchi SN, Blueitt D, Cullum CM, et al. The sensitivity and specificity of clinical measures of sport concussion: three tests are better than one. BMJ Open Sport Exerc Med. 2016;2:e000012.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Chin EY, Nelson LD, Barr WB, McCrory P, McCrea MA. Reliability and validity of the sport concussion assessment tool-3 (SCAT3) in high school and collegiate athletes. Am J Sports Med. 2016;44:2276–85.

    Article  PubMed  Google Scholar 

  51. Gessel LM, Fields SK, Collins CL, Dick RW, Comstock RD. Concussions among United States high school and collegiate athletes. J Athl Train. 2007;42:495–503.

    PubMed  PubMed Central  Google Scholar 

  52. Lincoln AE, Caswell SV, Almquist JL, Dunn RE, Norris JB, Hinton RY. Trends in concussion incidence in high school sports. Am J Sports Med. 2011;39:958–63.

    Article  PubMed  Google Scholar 

  53. Dick RW. Is there a gender difference in concussion incidence and outcomes? Br J Sports Med. 2009;43:i46–50.

    Article  PubMed  Google Scholar 

  54. Broshek DK, Kaushik T, Freeman JR, Erlanger D, Webbe F, Barth JT. Sex differences in outcome following sports-related concussion. J Neurosurg. 2005;102:856–63.

    Article  PubMed  Google Scholar 

  55. Valovich McLeod TC, Perrin DH, Guskiewicz KM, Shultz SJ, Diamond R, Gansneder BM. Serial administration of clinical concussion assessments and learning effects in healthy young athletes. Clin J Sport Med. 2004;14:287–95.

    Article  PubMed  Google Scholar 

  56. Moreau MS, Langdon J, Buckley TA. The lived experience of an in-season concussion amongst NCAA Division I student-athletes. Int J Exerc Sci. 2014;7:62–74.

    Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE 1256260. This publication was made possible, in part, with support from the Grand Alliance Concussion Assessment, Research, and Education (CARE) Consortium, funded, in part, by the National Collegiate Athletic Association and the Department of Defense. The US Army Medical Research Acquisition Activity, Fort Detrick, MD, USA is the awarding and administering acquisition office. This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Psychological Health and Traumatic Brain Injury Program under Award No. W81XWH-14-2-0151. Opinions, interpretations, conclusions, and recommendations are those of the author(s) and are not necessarily endorsed by the Department of Defense (Defense Health Program funds).

The CARE Consortium Investigators are listed alphabetically by institution: April Marie (Reed) Hoy, MS, ATC (Azusa Pacific University); Louise A. Kelly, PhD (California Lutheran University); Justus D. Ortega, PhD (Humboldt State University); Nicholas Port, PhD (Indiana University); Margot Putukian MD (Princeton University); T. Dianne Langford, PhD (Temple University); Scott Anderson, ATC and Gerald McGinty, DPT (US Air Force Academy); Patrick O’Donnell, MHA (US Coast Guard Academy); Steven J. Svoboda, MD (US Military Academy); John P. DiFiori (University of California-Los Angeles); Holly J. Benjamin MD (University of Chicago); Thomas Buckley, EdD, ATC, and Thomas W. Kaminski, PhD, ATC (University of Delaware); James R. Clugston, MD, MS (University of Florida); Julianne D. Schmidt, PhD, ATC (University of Georgia); Jason P. Mihalik, PhD, CAT(C), ATC (University of North Carolina at Chapel Hill); Christina L. Master, MD (University of Pennsylvania); Micky Collins, PhD, and Anthony P. Kontos, PhD (University of Pittsburgh Medical Center); Sara P.D. Chrisman, MD, MPH (University of Washington); Christopher M. Miles, MD (Wake Forest University); Brian H. Dykhuizen, MS, ATC (Wilmington College); Alison Brooks MD, MPH (University of Wisconsin).

Author information

Authors and Affiliations

  1. Department of Industrial and Operations Engineering, University of Michigan, 1205 Beal Ave, Ann Arbor, MI, 48109, USA

    Gian-Gabriel P. Garcia & Mariel S. Lavieri

  2. School of Kinesiology, University of Michigan, Ann Arbor, MI, USA

    Steven P. Broglio

  3. Departments of Neurosurgery and Neurology, Medical College of Wisconsin, Milwaukee, WI, USA

    Michael McCrea

  4. Department of Psychiatry, Indiana University School of Medicine, Indianapolis, IN, USA

    Thomas McAllister

Authors
  1. Gian-Gabriel P. Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven P. Broglio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mariel S. Lavieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael McCrea
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thomas McAllister
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

CARE Consortium Investigators

  • April Marie Hoy
  • , Louise A. Kelly
  • , Justus D. Ortega
  • , Nicholas Port
  • , Margot Putukian
  • , T. Dianne Langford
  • , Scott Anderson
  • , Gerald McGinty
  • , Patrick O’Donnell
  • , Steven J. Svoboda
  • , John P. DiFiori
  • , Holly J. Benjamin
  • , Thomas Buckley
  • , Thomas W. Kaminski
  • , James R. Clugston
  • , Julianne D. Schmidt
  • , Jason P. Mihalik
  • , Christina L. Master
  • , Micky Collins
  • , Anthony P. Kontos
  • , Sara P. D. Chrisman
  • , Christopher M. Miles
  • , Brian H. Dykhuizen
  •  & Alison Brooks

Corresponding author

Correspondence to Gian-Gabriel P. Garcia.

Ethics declarations

Conflict of interest

Gian-Gabriel Garcia, Steven Broglio, Mariel Lavieri, Thomas McAllister, and Michael McCrea have no conflicts of interest directly relevant to the content of this study.

Ethics approval

This study was approved by each local institutional review board and the Army Human Research Protection Office.

Additional information

The members of the ‘CARE Consortium Investigators’ are given in Acknowledgements section. This article is part of the Topical Collection on The NCAA-DoD Concussion Assessment, Research and Education (CARE) Consortium.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garcia, GG.P., Broglio, S.P., Lavieri, M.S. et al. Quantifying the Value of Multidimensional Assessment Models for Acute Concussion: An Analysis of Data from the NCAA-DoD Care Consortium. Sports Med 48, 1739–1749 (2018). https://doi.org/10.1007/s40279-018-0880-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40279-018-0880-x

Search

Navigation