Diagnosis of Sports-Related Concussion Using Symptom Report or Standardized Assessment of Concussion

This case-control study analyzes the diagnostic accuracy of various components of a concussion assessment that uses both patient report and standardized assessment among college athletes with and without concussion.


Introduction
The sideline or acute diagnosis of concussion is often challenging, putting an athlete's ability to return to competition on the line.There is currently no objective biomarker for concussion, and diagnosis relies heavily on athlete symptom reporting. 1The Sport Concussion Assessment Tool (SCAT) was introduced in 2004 by the Concussion in Sport Group 2 as a standardized concussion evaluation combining several existing tools, including symptom evaluation (Post-Concussion Symptom Scale [PCSS]), 3 cognitive evaluation (Standardized Assessment of Concussion [SAC]), 4 balance assessment (Modified Balance Error Scoring System), 5 and a neurological screening.Prior to the introduction of the SCAT, the evaluation of concussion was heterogeneous.[8][9] While not billed as a diagnostic tool, clinicians use the SCAT as part of a comprehensive clinical assessment that informs diagnosis.Consequently, it is important to understand the psychometric properties of the individual tests used to inform clinical decision-making.Previous studies 10,11 show reported symptoms to be the most sensitive and specific measure for diagnosing concussion.
However, relying on symptoms alone for concussion diagnosis can be problematic.Symptom reporting requires accurate disclosure by athletes who may be reluctant to report symptoms due to internal or external pressures, an inability to recognize symptoms, or a delay in the development of symptoms. 12The SAC is an objective measure of cognition that is included in the SCAT.The SAC is comprised of 4 subcomponents: (1) orientation, (2) immediate memory, (3) concentration, and (4) delayed recall.The original versions of the SAC used a 5-word list to test immediate memory and delayed recall. 135][16][17][18] The SCAT5 included the option to use a 10-word memory list, with hopes of improving utility and eliminating the ceiling effect. 19Studies examining diagnostic accuracy using the 10-word memory lists are limited.One study of professional hockey players using the 10-word list 10 showed group differences between players with and without concussion, but the authors concluded the 10-word SAC had insufficient sensitivity and specificity to be used as a stand-alone diagnostic tool.
Clinicians interpreting SCAT scores can use either normative data or change scores derived from a previously performed baseline test.1][22][23] Consequently, normative approaches require familiarity with typical scores obtained in the population being evaluated.This approach can result in the misdiagnosis of people who would naturally score either very high or very low on the SCAT, even when not concussed.This issue can be overcome by comparing test scores obtained following a suspected concussion to baseline scores previously obtained.The purpose of this study was to assess the diagnostic accuracy of the PCSS and the 10-word SAC using a baseline-retest approach.

Methods
This case-control study was approved by the University of Washington Human Subjects Division with reliance agreements from other institutions and followed the Strengthening the Reporting of for inclusion if they sustained a concussion from July 13, 2020, to December 31, 2022, and had an acute concussion evaluation within 48 hours of injury.Athletes were excluded if they had a concussion in the previous year and if they did not have a complete baseline and acute injury evaluation using the 10-word memory list for both the athlete with concussion and the control athlete.All athletes completed electronic informed consent.All athletes had complete baseline concussion testing done prior to participating in sport at least once as required by the NCAA.
Baseline frequency was determined by the practice of the university, which often differed by sport.
In most contact sports, baseline testing was performed annually.If an athlete was concussed and agreed to participate in the study, a matched control athlete was selected and invited to participate based on team, sex and gender, concussion comorbidities (clinically diagnosed attention-deficit/ hyperactivity disorder [ADHD], learning disorder, migraine headache disorder, or mood disorder) and baseline total symptom and total SAC scores within a 2-point range.The control athlete was given the SCAT5 within 2 weeks of the incident concussion, so season and school stressors were similar to those of the athlete with concussion.This also meant the athlete with concussion and the control athlete had a similar amount of time between initial baseline testing (which is done at the same time for teams) and repeat testing.The diagnosis of concussion was made by team medical personnel using the definition from the Concussion in Sports Group. 24The PCSS and SAC with the 10-word memory list were performed by team medical staff as described in the SCAT5. 25A total score was calculated for the SAC, which included orientation (0-5 points), immediate memory (0-30 points), concentration (0-5 points), and delayed recall (0-10 points), with a maximum of 50 points possible.Higher symptom scores and symptom severity scores indicate worse symptoms while higher scores on the total SAC and its subcomponents indicate better performance.Time from injury to the SAC evaluation was also noted as immediate (Յ2 hours), between 2 and 8 hours, between 8 and 24 hours, and between 24 and 48 hours.Self-identified race and ethnicity was recorded to ensure the sample was comparable with the race and ethnicity distribution of NCAA athletes.Race and ethnicity groups included African American or Black, Asian, Hispanic or Latinx, Native Hawaiian or Pacific Islander, White, and multiracial.

Statistical Analysis
Normality of all variables was assessed using Shapiro-Wilk test and χ 2 analyses were used to examine between group differences in gender, sport, year in school, and comorbidities.Differences within and between groups and time points were assessed using 2 sample t tests of unequal variances (Welch t test) and the Mann-Whitney U test.Test-retest reliability from baseline to control follow-up was assessed using 2-way intraclass correlations using absolute agreement for the symptom score, symptom severity score, and the total score for the 10-word SAC and subcomponents. 26Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each testing component at various cut points.We examined the role of possible covariates (eg, ADHD diagnosis, migraine disorder diagnosis, mood disorder, and year in school) using logistic regression.
The covariates were identified a priori based on previous literature and clinical expertise. 8,27None were significant and were subsequently excluded from the models.A model was built using logistic regression to examine the association of symptom score, symptom severity score, and total SAC score with the outcome.Logistic regression was used to address any violations of the assumptions of independence, homoscedasticity, and normality.Given the binary nature of the outcome variable (presence or absence of concussion) and a violation of normality in some of the data, we opted for a binomial distribution and specified a logit link function to model the association.Receiver operating characteristic curves (ROC), area under the ROC curve (AUC), and 95% CIs were calculated to assess test validity for all postinjury assessments; excellent diagnostic utility was considered to be an AUC between 0.9 and 1.0; good diagnostic utility, between 0.8 to less than .09;fair diagnostic utitlity, between .07 to less than .08;and poor diagnostic utility, less than .0.6.All analyses were examined at an α level of .05.  ).There were 92 concussions that met inclusion criteria diagnosed during the study period.The distributions of gender and sex, and race and ethnicity closely mirrored that of the larger population of NCAA athletes. 28Of all athletes, 110 (59%) played a sport other than football.Concussions occurred most often in football (37 concussions [40%]) and volleyball (14 concussions [15%]).Of all participants, 6 (3%) had diagnoses of ADHD, 43 (23%) had mood disorders (anxiety or depression), and 14 (8%) had headache or migraine disorder; no participants had a learning disability.Concussions occurred most often in the second year of school.There were no significant differences between athletes with concussion and control athletes (Table 1).

Reliability
The median (IQR) number of days between the baseline and the repeat test was 147 (86-125) days for the symptom score and 198 (117-420) days for the SAC.Median (IQR) number of days between baseline and repeat test for SAC were similar in both athletes with concussion (188 [11-114] days) and control athletes (212 [130-421] days).All tests had poor test-retest reliability, meaning there was significant variation in testing from test to test even in athletes without a concussion.(Table 2).

Baseline Tests
The median (IQR) baseline scores for all athletes using the 10-word memory list was 37. for delayed recall.The total SAC score was normally distributed.There were no significant differences between baseline scores between the concussion and control groups (Table 3).

Discussion
This case-control study highlights several important points.First, the development of symptoms after a suspected concussive blow was the most accurate indicator of concussion with an AUC of 0.93 for symptom score and 0.94 for symptom severity score.Second, the sensitivity and specificity of the 10-word SAC for the diagnosis of concussion reached only fair accuracy (AUC, 0.71), although The figure shows receiver-operator characteristic curves for total SAC, total symptoms, and symptom severity (A); orientation, immediate memory, concentration, and delayed recall (B); immediate evaluation (Յ2 hours), evaluation between 2 and 8 hours, evaluation between 8 and 24 hours, and evaluation between 24 and 48 hours (C); and total SAC, total symptoms, and total SAC + total symptoms (D).AUC indicates area under the receiver-operator characteristic curve; SAC, Standardized Assessment of Concussion.
it was slightly improved compared with previously reported values for the 5-word SAC (AUC, 0.68). 14portantly, almost one-half of concussed athletes (45%) performed at or above their baseline test results (ie, within normal limits) on the SAC, highlighting the limitations of the cognitive examination and the importance of an increase in symptoms, as well as including other objective indicators of concussion such as visual signs and vestibulo-ocular or balance abnormalities.Indeed, the AUC for the symptom score combined with the total SAC score (0.94) was only slightly higher than the AUC for the symptom score alone (0.93), indicating the total SAC adds minimal additional projected variance, which is likely not clinically significant.Third, there was considerable performance variability over time in the 10-word SAC in athletes without concussion.Fourth, the orientation and concentration subcomponents did not add to the diagnostic utility of the SAC, similar to results found in a sample of professional hockey athletes. 10Fifth, the accuracy of the SAC decreased as time from the incident increased, with an AUC of 0.77 at the immediate time frame and AUC of only 0.57 with confidence intervals crossing the 0.5 threshold after 24 hours, although these subsamples were small and the confidence intervals were large and overlapped.
In this sample of athletes with concussion, there was an increase of 9.65 points in mean symptom score from baseline score in athletes with concussion and an increase of 26.14 points in mean symptom severity score from baseline.The cut point data in Table 4 showed an increase of either symptom score or symptom severity score by 8 points had 100% PPV.In contrast, symptom score and symptom severity score did not increase significantly in the control athletes.Although an increase in symptoms is highly suggestive of concussion, this relies on accurate reporting by the athlete who may not report symptoms because of a desire to return to play, a fear of letting teammates down, minimizing the seriousness of concussion, difficulty discerning symptoms, a delay in symptom development, or other reasons.This finding highlights the importance of educating athletes and checking in after the initial evaluation.Although the sensitivity of the SAC was relatively low, when the SAC is below baseline, it can inform concussion diagnosis, primarily by changes in immediate memory and delayed list recall.For instance, while having limited sensitivity (26%), a decline of 3 points on the delayed recall portion of the SAC was highly specific (96%).Likewise, a 6-point decrease on immediate memory had only 13% sensitivity for concussion but was 98% specific.These results show that learning and memory assessment cannot be used in isolation to diagnose concussion; however, a substantial minority of athletes with concussion have memory problems that, when present, are highly suggestive of concussion.
Test-retest reliability is typically measured in weeks or months compared with the 147 days for symptoms and 198 days for the total SAC in this study.However, these longer time frames represent how the test is utilized in actual practice.NCAA athletes are required to have baseline testing performed at least once, but it is not always performed annually, particularly in sports where the risk for concussion is lower. 29Even when annual baseline cognitive testing is done, it may still be many months before an athlete with a suspected concussion has repeat testing.Annual baseline testing for all athletes requires a substantial amount of time and person-power and may not be possible at lower-resourced institutions or for secondary or youth sports.Understanding normative scores is important in these situations.Use of normative scores has been shown to be as accurate as using the baseline and postinjury repeat testing paradigm, but this may be because of suboptimal psychometrics when using both the baseline-retest and normative approaches. 10,18,30,31e accuracy of the SAC decreased as the time from the injury increased.Although the accuracy decreased, the number of athletes in each group was relatively small and the confidence intervals overlapped.A recent systematic review of the SCAT, 32 which included reporting on 14 acceptable or high-quality studies on SAC score and timing of exam, noted that the greatest differences in standard mean difference of total 5-word SAC score was in the first 24 hours after injury, which is supported by our findings.and still be concussed with elevated symptoms and/or ataxia or loss of consciousness.
Strengths of the study include a sample that is almost one-half women, a broad representation of sports, and a racial and ethnic distribution similar to that of NCAA athletes.The use of control athletes matched for ADHD, mood disorders, migraine or headache disorders, and learning disorders for repeat testing at a similar time of the school year to the athlete with concussion also represents a strength.The SCAT5 was used for this study, however a newer version, the SCAT6 was recently published. 33The SAC component of the SCAT6 is identical to the SCAT5 with the exception of the timing the months backward task in the concentration subcomponent. 34This task accounts for 1 point, and we do not believe that this change will significantly change our results.

Limitations
There were limitations to the study.This study included Division I athletes and may not represent other populations such as high school or professional athletes.There are 3 different word lists available in the SAC to decrease the possibility of learning effect, however, a study in professional hockey players has shown that the 10-word lists are not equivalent. 35Nonequivalent lists may account for the SAC improvements observed in our control group upon retest.We did not control for list order in this study and do not believe that it is practical in most acute or sideline settings, especially in the community.Rather than controlling for list order, we would recommend creating equivalent lists in future iterations of the SCAT or development of automated regression-based norms that correct for list differences as part of an application-based delivery of the SCAT.We did not control for concussion history although we excluded athletes or controls with a concussion in the previous year.Studies have shown that those with more than 3 concussions may have lower baseline performance on the SAC. 23

Conclusions
This case-control study affirms that reported symptoms are the most sensitive indicator of concussion and there are limitations to the objective cognitive testing included in the SCAT.An athlete with a clear mechanism of injury who develops symptoms should be considered concussed unless there is a reasonable alternative explanation for these symptoms.The use of the 10-word SAC has eliminated the ceiling effect present with the 5-word SAC but has only minimally increased the accuracy from previously reported values when using the baseline-retest approach. 14The lack of utility of the orientation and concentration subcomponents should be considered in future versions of the SAC.In this study, 45% of athletes with a concussion scored at or above their baseline scores (ie, within normal limits), emphasizing the need for multimodal evaluation, including attention to symptoms, visible signs, and vestibulo-ocular components.In cases where the total SAC or any of the subcomponents are well below baseline, a diagnosis of concussion should be considered.The cut points provided in this study can be used to guide decision making; declines of 3 points or more on immediate memory or delayed recall or a decline of 6 points on the total SAC score should be considered concerning.More research into appropriate time frames for use of the SAC should be conducted.Concussion remains a clinical diagnosis that should be based on a thorough review of signs, symptoms, and clinical findings.
Abbreviation: NA, not applicable.aMann-Whitney U test unless otherwise specified.

Figure
Figure.Receiver-Operator Characteristic Curves

JAMA Network Open | Neurology Diagnosis
of Sports-Related Concussion Using Symptom Report or Standardized Assessment Observational Studies in Epidemiology (STROBE) reporting guideline.English-speaking National Collegiate Athletic Association (NCAA) athletes in any sport from 4 institutions (University of Washington, Oregon State University, Stanford University, and University of Colorado) were eligible Diagnosis of Sports-Related Concussion Using Symptom Report or Standardized Assessment Analyses were conducted in RStudio version 2023.06.1 + 524 (Posit) using the dplyr, ggplot, irr, and pROC packages.Data analysis occurred from August 2023 to October 2023.

Table 1 .
Demographics a

Table 2 .
Test-Retest Reliability Using ICC

Table 3 .
Baseline and Postinjury Time Point Values for Concussion and Control Groups

Table 4 .
Diagnosis of Sports-Related Concussion Using Symptom Report or Standardized Assessment Sensitivity, Specificity, PPV, NPV, and AUC of Symptoms and Total SAC Score and Subcomponent Scores Diagnosis of Sports-Related Concussion Using Symptom Report or Standardized Assessment JAMA Network Open.2024;7(6):e2416223.doi:10.1001/jamanetworkopen.2024.16223(Reprinted)June 11, 2024 6/13 Downloaded from jamanetwork.combyguest on 06/13/2024 b Probability of a positive test in those with a concussion.cProbability of a negative test in those without a concussion.dProbability of having a concussion if the test is positive.eProbability of not having a concusssion if the test is negative.JAMA Network Open | Neurology Diagnosis of Sports-Related Concussion Using Symptom Report or Standardized Assessment from play who does not have a concussion.The sensitivity, specificity, PPV, and NPV can all be used to help make these types of decisions.Clinicians may have different thresholds balancing these competing interests depending on the clinical situation, but Table4can provide data to inform decision-making.The worse an athlete scores when compared with their baseline score, the more likely they are to have a concussion.However, an athlete may test normally (or better than baseline) Larger groups at different time points may help inform at what time point the SAC should no longer be used as an aid to diagnosis.The decision to diagnose an athlete with a concussion is difficult, and medical clinicians must balance the possibilities of letting someone continue to play with concussion vs removing someone JAMA Network Open | Neurology JAMA Network Open.2024;7(6):e2416223.doi:10.1001/jamanetworkopen.2024.16223(Reprinted) June 11, 2024 9/13 Downloaded from jamanetwork.comby guest on 06/13/2024