Population.

The population examined included patients that experienced an external injury of all ages from any country. The ICD-10 codes used in the inclusion criteria to classify external injuries are summarized in Table 1. This includes resulting injury codes and external cause of the injury codes. Only cases that examined and recorded these injuries with the specified ICD-10 injury codes were included in the analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. ICD-10 codes used to classify external cause injuries.

https://doi.org/10.1371/journal.pone.0298411.t001

The ICD-10 codes we selected for our analysis to categorize and present injury data were based on the reliable standards reported by the Association of Public Health Epidemiologists in Ontario (APHEO) [8] and Parachute’s 2022 guidelines for ICD-10 code classifications used to document injury causes [9], The codes included are primarily based on the ICD-10-CA codes, as these are applicable to classify injuries in all countries [10]. These injury codes overlap across all countries, with a few minor discrepancies which are described in the results section. We divided the available results into ICD-10 code categories for external causes of injuries (i.e., self-harm injuries, abuse, transport and pedestrian injuries, and poisoning) and injuries to body regions (i.e., hand and wrist injuries, brain injuries, spinal cord injuries, lower extremities injuries, and multiple (total body) injury types reported).

Intervention.

The intervention evaluated in this review was the validity and/or reliability reported for the specified ICD-10 injury codes.

Comparator.

Studies were included which compared the reported ICD-10 injury codes to chart review and/or physician diagnosis as the gold standard (for validity measures) and/or those that compared ICD-10 injury codes between coders or other healthcare workers (i.e., inter-rater reliability (IRR) for reliability measures).

Outcomes.

The outcome measures included in the analysis to assess the validity and reliability of external injury ICD-10 codes were: (1) sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for validity, and (2) IRR, measured by Krippendorff’s alpha, Cohen’s Kappa, and/or Fleiss’ kappa, for reliability.

Demographic variables.

Of the 27 articles that were included in the final review, 13 (48%) were from the United States of America (U.S.A.), six (22%) were from Canada, four (15%) were from Australia, two (7%) were from Taiwan, one (4%) was from Iran, and one (4%) was from Norway. Characteristics of all included studies are presented in Table 2. Sample sizes of the injury patients’ codes included varied widely between the studies and the codes, ranging from the tens in some studies to the thousands in others (S1 Table). The records that were reviewed cover a 38-year period (1982 to 2020), with two (7%) articles that analyzed records between 1982 and 2000, 10 (37%) articles that analyzed records between 2001 and 2010, and 20 (74%) articles that analyzed records between 2011 and 2020.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of studies included.

https://doi.org/10.1371/journal.pone.0298411.t002

Gold standard.

Chart review was used as the gold standard in 22 articles, and direct physician diagnosis based on patient evaluation was used in two articles. The three remaining articles did not use a gold standard, as they only evaluated the IRR of their respective ICD-10 codes of focus.

Sensitivity and specificity.

Sixteen studies examined sensitivity, with 46 outcome values reported, while 12 studies examined specificity, with 33 outcome values reported for the ICD-10 codes being examined. Across the 9 injury categories, the mean sensitivity was 61.6% (range 35.5%-96.0%), while the mean specificity was 91.6% (range of 85.8%-100%). These values are summarized in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Sensitivity and specificity outcomes of external cause injury ICD-10 codes.

The mean sensitivities (Panel A) and specificities (Panel B), with error bars reflecting the range of values (where reported), from studies that validated ICD-10 codes for injury mechanisms and outcomes in hospitalization data. a. Sensitivity outcomes for all injury categories. b. Specificity outcomes for all injury categories.

https://doi.org/10.1371/journal.pone.0298411.g003

PPV and NPV.

In the context of this study, positive predictive values assess the ratio of true positive cases to the total number of cases identified by the ICD-10 codes. Negative predictive values assess the ratio of true negative cases to the total number of cases identified by the ICD-10 codes as not having the condition. Twenty-three studies examined positive predictive values, with 61 outcome values reported, while 9 studies examined negative predictive values, with 20 outcome values reported for the ICD-10 codes of interest. Across the 9 injury categories, the mean positive predictive value was 74.9%, (range 58.6%-96.5%), while the mean negative predictive value was 80.2%, (range of 44.6%-94.4%). The values for each injury category are summarized in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. PPVs and NPVs of external cause injury ICD-10 codes.

The mean PPVs (Panel A) and NPVs (Panel B), with error bars reflecting the range of values (where reported), from studies that validated ICD-10 codes for injury mechanisms and outcomes in hospitalization data. a. PPV outcomes for all injury categories. b. NPV outcomes for all injury categories.

https://doi.org/10.1371/journal.pone.0298411.g004

Inter-rater reliability.

The inter-rater reliability (IRR) evaluation was conducted using 3 measurement tools: Krippendorff’s alpha, Cohen’s kappa, and Fleiss’ kappa. Nine studies examined Cohen’s kappa, resulting in 16 reported outcome values. One study also examined Krippendorff’s alpha, with 1 outcome value reported. Another study examined reliability using Fleiss’ kappa, reporting 1 outcome. Across the 9 injury categories, the mean Cohen’s kappa value was 0.672, (range of 0.480–0.928). With limited data for Krippendorff’s alpha and Fleiss’ kappa, the values yielded were 0.453 and 0.630, respectively. Fig 5 summarizes the mean and range values for the IRR outcomes in each injury category.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Inter-rater reliability outcomes for external cause injury ICD-10 codes.

The mean inter-rater reliabilities, with error bars reflecting the range of values (where reported), from studies that analyzed the reliability of ICD-10 codes for injury mechanisms and outcomes in hospitalization data.

https://doi.org/10.1371/journal.pone.0298411.g005

Injury category statistical analysis.

The mean and range of each injury outcome are reported in S4 Table.

Highest-quality study outcomes.

The results for the 22 studies considered high-quality from our risk of bias assessment are summarized in Table 3. The overall results were: sensitivity 64.5% (35.5%-96.0%), specificity 88.9% (85.83%-100%), PPV: 71.09% (54.92%-92.0%), NPV: 77.82% (44.6%-92.80%), and IRR values: Krippendorff’s alpha: 0.453, Cohen’s kappa: 0.660 (0.335–0.920), and Fleiss’ kappa (0.630). The calculations for these are listed in S4 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of the validity and reliability outcomes for all high-quality studies included in the analysis.

All values are reported as mean percentage (with ranges) where data was available.

https://doi.org/10.1371/journal.pone.0298411.t003

Transport and pedestrian injuries.

Two studies assessed transport and pedestrian injuries, resulting in 8 relevant outcome values reported (S1 Table). These studies covered bicycle injuries, pedestrian injuries, femur fractures, and transport incident injuries. Sensitivity was examined by both articles and ranged from 33.1%-95.7% (mean 73.8%). Specificity and positive predictive values were examined by one article each, and resulted in values of 100% and 96.5%, respectively. Cohen’s kappa ranged from 0.905–0.945 (mean 0.928). One of the two studies that reported injury mechanism codes for transport and pedestrian injuries was rated as high quality, while the other was rated as medium quality. There was little difference in the inter-rater reliability scores between the studies: inter-rater reliability ranged from 0.91 to 0.98 in the medium-quality study, and from 0.88 to 0.97 in the high-quality study. However, there was an unclear difference in the sensitivity values, which ranged from 87% to 98% in the medium-quality study, but ranged from 25.0% to 45.0% for one half of the values, and from 90.2% to 98.3% for the other half, in the high-quality study.

Self-harm injuries (Intentional).

Four studies examined self-harm injuries, with 22 relevant outcome values reported (S1 Table). These 4 studies investigated poisoning, intentional self-harm, and events of undetermined intent. Intentional self-harm included suicide attempts and self-harm of various types, such as poisoning, asphyxiation, and others. Some of these studies examined multiple outcomes of interest, while others only reported a few. Sensitivity and specificity ranged from 18.7%-44.8% (mean 35.5%) and 92.6%-98.2% (mean 95.4%), respectively. Positive predictive values ranged from 29.1%-96.6% (mean 63.8%), and negative predictive values ranged from 73.3%-95.0% (mean 87.1%). Cohen’s kappa ranged from 0.478 to 0.481 (mean 0.48). All the studies that analyzed self-harm injuries were rated as high-quality.

Abuse.

The overarching injury topic of abuse explored a range of topics, including child physical abuse, assault, sexual abuse, and other forms of maltreatment. Three studies examined injuries as a result of abuse, with 11 relevant outcome values reported (S1 Table). With each study having a different focus area, it led to a complete and well-rounded exploration of this injury type. Sensitivity and specificity ranged from 31.8%-72.6% (mean 55.6%) and 84.6%-90.8% (mean 88.2%), respectively. Positive predictive values ranged from 71.1–76.0% (mean 73.5%). The assessment of negative predictive values was limited to a single study, resulting in a value of only 44.6%. Similarly, Cohen’s kappa and Fleiss’ kappa resulted in values of 0.818 and 0.63, respectively. All the studies that reported injury codes for abuse were rated as high-quality.

Poisoning.

As poisoning is a broad topic, it is important to note that various aspects are included, including toxic effects of carbon monoxide and poisoning from drugs and biological substances. Furthermore, poisoning by opioids, other synthetic narcotics, and psychodysleptics (both intentional and unintentional) were included. Four studies examined injuries caused by poisonings, with 14 relevant outcome values reported (S1 Table). Sensitivity and positive predictive values ranged from 79.5%-89.3% (mean 84.41%) and 32.8%-84.0% (mean 64.3%), respectively. Specificity and negative predictive values were only assessed by one article and gave values of 85.8% and 92.1%, respectively. Cohen’s kappa yielded a value of 0.735. Three of the four studies that reported injury mechanism codes for poisoning (unintentional) were rated as high quality, while the other was rated as medium quality. There was a difference in the sensitivity values, which ranged from 76% to 83% in the medium-quality study, and from 81.2% to 94.9% among the high-quality studies. However, there was an unclear difference in the PPVs, which ranged from 67% to 71% in the medium quality study but ranged from 32.8% to 60.3% for about half of the values, and from 76.6% to 97.9% for the other half, among the high-quality studies (with PPV ≥ 76.6% in two of the three high-quality studies reporting on PPV).

Neurological/spinal cord injuries.

This category covers fractures and nerve injuries, as well as spinal cord injuries. Additionally, it examines outcomes related to injuries of the brain and spinal cord, encompassing concussion, edema, and nerve injuries. Three studies assessed spinal cord injuries, resulting in 47 relevant outcome values reported (S1 Table). Sensitivity and specificity ranged from 0.9%-89.8% (mean 36.4%) and 6.7%-100% (mean 86.3%). Positive predictive values and negative predictive values yielded a range of 30.0%-100% (mean 81.8%) and 10.0%-93.0% (mean 60.8%). Cohen’s kappa ranged from 0.56–0.70 (mean 0.65). Two of the three studies that reported injury outcome codes for spinal cord injuries (unintentional) were rated as high quality, while the other was rated as medium quality. There was little difference in the PPVs, which ranged from 76.2% to 100.0% in the medium-quality study (with the exception of an outlier: 33.3%), and from 76.0% to 97.0% (with the exception of an outlier: 30.0%) among the high-quality studies. There was a difference in the sensitivity values, which ranged from 0.9% to 33.3% in the medium-quality study, and from 50.0% to 89.8% (with the exception of an outlier: 30.0%) among the high-quality studies. However, there was an unclear difference in the specificity values, which ranged from 98.8% to 100.0% in the medium-quality study, but from 6.7% to 25.8% for about half of the values, and from 97% to 98% for the other half, among the high-quality studies.

Hand and wrist injuries.

Hand and wrist injuries encompasses open wounds on the wrists and hands, along with fractures, sprains, and strains of joints and ligaments. One study examined hand and wrist injuries, with 2 relevant outcome values reported (S1 Table). The values for sensitivity and positive predictive values were 96% and 92%, respectively. This study was rated as high-quality.

Brain injuries.

This injury category discusses a range of brain injury outcomes, including skull fractures, concussions, cerebral edema, traumatic brain injuries, hemorrhage, and other intracranial injuries. Additionally, it examines outcomes linked to shaken infant syndrome and unspecified head injuries. Five studies assessed brain injuries, resulting in 13 relevant outcome values reported (S1 Table). Sensitivity and positive predictive values ranged from 6.8%-81% (mean 53.6%) and 33.3%-100% (mean 74.7%), respectively. Specificity and negative predictive values were only assessed by one article and gave values of 88.0% and 92.8%, respectively. Four of the five studies that reported injury outcome codes for brain injuries (unintentional) were rated as high quality, while the other was rated as medium quality. There was not a clear difference in the PPVs, which ranged from 22.7% to 73.7% (with about half of the values ≤ 40.8%, and the other half ≥ 60.3%) in the medium-quality study, and from 60.6% to 100.0% (with the exception of an outlier: 33.3%) among the high-quality studies.

Lower extremities injuries.

As “lower extremities injuries” is a broad topic, the specific factors that are included are ankle fractures, hip fractures (including proximal femur fractures), as well as fractures, sprains and strains of joints and ligaments at the ankle and foot level. Five studies examined injuries in the lower extremities, resulting in 40 relevant outcome values reported (S1 Table). Sensitivity and specificity ranged from 0%-94.5% (mean 53.7%) and 76.0%-98.2% (mean 90.6%), respectively. Positive predictive values and negative predictive values ranged from 0%-100% (mean 58.6%) and 54.0%-99.0% (mean 89.6%), respectively. Cohen’s kappa ranged from 0.26–0.95 (mean 0.54). Krippendorff’s alpha was reported in one study and yielded a value of 0.453. Four of the five studies that reported injury outcome codes for lower extremity injuries (unintentional) were rated as high quality, while the other was rated as medium quality. There was a difference in the PPVs, which ranged from 91.0% to 100.0% in the medium-quality study, and from 0.0% to 100.0% (most ≥ 43.0%) among the high-quality studies. There was a difference in the sensitivity values, which ranged from 94.0% to 95.0% in the medium-quality study, and from 0.0% to 96.0% (most ≥ 50.0%) among the high-quality studies. Inter-reliability scores ranged from 0.93 to 0.97 in the medium-quality study, and from 0.26 to 0.60 among the high-quality studies.

Multiple (total body) injury types reported.

This injury category is broader but includes critical injury types that are essential for a complete analysis from studies that examined multiple injury-types in one analysis. This category discusses external-cause injuries to different body parts, burns, firearm injuries (both accidental and intentional), head and neck injuries, and trauma codes with unspecified details. Four studies examined multiple (total body) injury types, resulting in 17 relevant outcome values reported (S1 Table). Sensitivity and positive predictive values ranged from 5.0%-89.5% (mean 65.8%) and 5.5%-95.5% (mean 69.2%), respectively. Specificity and negative predictive values were only assessed by one article and yielded values of 98.5% and 94.4%, respectively. Cohen’s kappa ranged from 0.15–0.77 (mean 0.557). Two of the four studies that reported injury outcome codes for multiple (total body) injury types (unintentional) were rated as high quality, while the other two were rated as medium quality. There was little difference in the PPVs, which was 92.6% in the medium-quality study, and ranged from 93.3% to 95.5% (with the exception of an outlier: 5.5%) among the high-quality studies. There was little difference in the sensitivity values, which was 76.1% in the medium-quality study, and ranged from 66.3% to 89.5% (with the exception of an outlier: 5.0%) among the high-quality studies. There was, however, a difference in the inter-reliability scores, which was 0.15 in the medium-quality study, and ranged from 0.75 to 0.77 among the high-quality studies.

Sensitivity and specificity.

Mean sensitivity values were generally lower (mean = 61.6%, range = 35.5%-96.0%) while specificity values were high across the studies (mean = 91.6%, range = 85.8%-100%). Importantly, due to the nature of sensitivity and specificity measures, when one of these values increases for a diagnostic accuracy test, naturally the other tends to decrease. Thus, a generally high value for both is better but achieving a very high score on both tests is unlikely. Nevertheless, considering the wide use of ICD-10 codes for injury research, these sensitivity values reported are concerning. Furthermore, no gold-standard “cut-off” values have been widely implemented for what are considered to be high- or low-quality values for sensitivity and specificity of diagnostic accuracy studies. In other research contexts, such as Influenza testing, >90% has been reported as excellent sensitivity/specificity, 80–89% is good, 60–79% is fair, while <60% is considered poor [42].

PPV and NPV.

The overall positive predictive value across the studies was better than the sensitivity values, with a mean of 74.9% (range 58.6%-96.5%), though it was still not very high. The negative predictive value was quite high (mean = 80.0%, range = 44.6%-94.4%). Similarly to sensitivity and specificity, no gold standard “cut-off” has been established for the PPV and NPV of diagnostic accuracy studies. However, in other contexts (e.g., pediatric screening tools), values have been reported for PPV and NPV as: >90%: excellent, 80–89%: good, 60–79%: fair, and <60%: poor [43].

Inter-rater reliability calculations.

Reliability calculations yielded moderate values, as the mean Cohen’s kappa value was 0.672 (range = 0.480–0.928). The limited data for Krippendorff’s alpha and Fleiss’ kappa were agreement values of 0.453 and 0.63, respectively. Previous reports on these reliability statistics have suggested that 0.81–1.00 is excellent agreement/reliability, 0.61–0.80 is substantial agreement/reliability, 0.40–0.60 is moderate agreement, 0.21–0.39 is fair agreement, and 0.00–0.20 is low agreement (or none) [44, 45].

The wide range of values reported within sensitivity, specificity, PPV, NPV, and inter-rater reliability may be attributable to the large scope of injuries included in the study, as the training of medical personnel, as well as the common coding practices and definitions of codes used to represent injury types within different areas of medicine (e.g., brain injuries compared to self-harm injuries) varies widely. Causes of discrepancies between the outcomes within injury categories are unclear, though differences in sample sizes and study design elements may be contributing factors (e.g., one researcher versus multiple reviewing the charts). Furthermore, some studies included had a higher risk of bias, as described, which may have impacted the results. Nevertheless, the studies included in our systematic review did have a uniform gold standard of chart review/physician diagnosis and met strict inclusion criteria for the codes and outcomes included.

Injury outcomes.

Transport and pedestrian injuries, and hand and wrist injuries, had particularly high PPVs (mean = 96.5% and 92.0%, respectively). The rest of the PPVs were moderate to good. The sensitivity values for these categories (transport/pedestrian and hand/wrist injuries) were also quite good (mean = 73.8% and 96.0%, respectively). A common source of misclassification that may have contributed to the good, rather than excellent, sensitivity value, is a lack of training on reporting and classifying pedestrian/transport injuries in neurologist training programs [5, 46] Transport and pedestrian injuries also had particularly high IRR outcomes (mean = 0.93) and specificity values were high for all categories as previously stated. Poisoning codes also had high sensitivity values (mean = 84.4%). The coding discrepancies that may have caused some reduction in this value include that patients with other acute diseases (e.g., burns, and other substance poisoning) may be mistaken for different types of poisoning [21].

Though most sensitivity values were moderate to low, the values for self-harm injuries and spinal cord injuries were particularly low (mean = 36.4% and 35.5%). Some factors that may have contributed to these lower values for self-harm injuries include that a commonly used self-harm code, X84 (intentional self-harm by unspecified means), is more likely to be used in cases of individuals who are Indigenous, those with suicide attempts by cutting, and non-suicidal self-injury in females [32]. Similar bias trends in the reporting of intentional self-harm in different groups, including a bias in reporting of young females self-harm cases in hospital data, have been observed in other studies [47]. The particularly low spinal cord injury sensitivity values may be attributable to the injury characteristics, such as the severity and the level of the spine trauma not being accurately reported in the coding [18]. This was a similar problem found previously in ICD-9 studies that examined the validity and reliability of spinal injuries [48, 49].

The NPVs for abuse were also quite low (mean = 44.6%) compared to the other injury categories, which were all high or relatively high values. This may be attributable to only 5% of the study population receiving the ICD-10-CM code Z04.72 (examination and observation following alleged physical abuse) [33]. The ICD-10-CM guidelines state that all patients should receive this code, though this is not always the case [50]. Furthermore, abuse presents diagnostic challenges as it be inaccurate as proper history is not always taken, and thus may have key omissions [36]. Additionally, abuse may be more difficult for physicians to identify than other conditions, such as traffic injuries, as healthcare professional training in abuse is lacking, especially in certain populations such as elder abuse [51]. Also, there may be reluctances from physicians to diagnose abuse to due to uncertainty and discomfort with these diagnoses [52].

The rest of the outcomes were generally moderate values, including IRR values across the injury categories which were moderate. Of note, there were no results reported for: sensitivity of hand and wrist injuries, NPVs of hand/wrist injuries and transport and pedestrian injuries, and IRR for hand/wrist injuries and brain injuries.

Causes and solutions for external cause injury coding misclassification.

A variety of sources may have contributed to the lower outcome values reported across the injury categories and the discrepancies amongst coders for the external cause injury ICD-10 codes included in our study as a whole.

Health professionals work under time-constraints, which can lead to errors of omission and commission due to inadequate information for coding or unclear documentation [53]. Incomplete medical histories also may contribute to these errors, so training staff how to best report codes in these cases, as well as emphasizing proper history documentation practices, would be beneficial [36].

Furthermore, inadequate training of hospital staff for coding, and a lack of a standardized approach for ICD-10 coding, which are affected further by variations in staff experience may all contribute to these errors [5]. Thus, more emphasis on training programs that teach accurate coding practices for hospital staff, including admissions staff, providers, and hospital coders could substantially improve common coding misclassifications [5, 46].

Future directions.

Although improved physician training programs may have some positive impacts on ICD coding practices, as demonstrated by Paydar & Asadi [55], this alone is likely not enough to significantly improve their validity and reliability. The use of artificial intelligence (AI) for medical record review, such as through natural language processing and deep neural networks to analyze patient files and patient-provided information has been shown to be useful for diagnosis coding practices [56]. Machine learning algorithms can be used to gather chart data and generate codes for diagnoses [57]. For example, Dewaswala and colleagues [58] reported that natural language processing effectively identified and classified hypertrophic cardiomyopathy patients from narrative text reports in cardiac magnetic resonance imaging with high performance compared to manual annotation. Since natural language processing review of medical records can review documents about each patient to reach a diagnostic category, with more development, they may be more efficient and accurate than traditional methods of coding [58].

Thus, implementing these AI algorithms for assigning ICD-10 codes would be beneficial for more accurate coding and to reduce healthcare staff time and energy spent on this. Digital electronic medical records that force clinicians into certain diagnostic categories also hold hope for improving diagnostic and coding accuracy. Some accurate deep learning models have already been created for automatic ICD-10 coding that show promise for the future development of this technology [57]. However, more work is required to integrate these technologies into hospital systems, to train healthcare staff in using them, and to assess the precision of the algorithms’ coding before using them regularly. As well, lower income countries, where the major burden of global injury exists, may not have the capacity to introduce expensive electronic medical records.

Furthermore, an analysis investigating the validity and reliability of ICD-10 codes for individual body regions is important to address in future studies, as these codes are also widely used in research and healthcare contexts.

Consistencies in quality.

Despite quality discrepancies, the overall studies’ quality were good. All 27 studies met the criteria for good quality regarding seven of the 14 questions from the QUADAS tool (Fig 2). The spectra of patients included in all studies were representative of the patients who would receive the test in practice (i.e., injury patients for each particular injury), reducing the overall risk of spectrum bias [12]. The selection criteria of the studies included were clearly described, and the chart review/physician diagnosis were likely to correctly classify injury patients. Furthermore, the whole samples, or a random selection of patients, received verification (i.e., were compared to chart review/physician diagnosis), reducing the overall risk of partial verification bias. The comparison of chart review/physician diagnosis to ICD-10 codes was done independently (blinded) in some studies, however this was unclear in some.