Background/Aims Pneumonia is common and can be devastating in older adults. Health plan data hold promise for studying pneumonia, but ICD-9 codes have poor accuracy for this condition. Natural language processing (NLP) offers potential to accurately and efficiently identify pneumonia from electronic medical records (EMRs). Our aims were to train one NLP tool to identify pneumonia from electronic radiology reports and to assess its validity compared to manual review.

Methods ONYX is an NLP system that identifies clinical conditions (findings, symptoms, diagnoses) in free-text reports using knowledge about language in the reports and the specific medical domain. Building on a knowledge base from a prior NLP system, we trained ONYX using 1,100 chest radiograph reports from among 70,000 that were previously manually reviewed for pneumonia. We trained ONYX to classify reports into one of three mutually-exclusive categories:

1. consistent with pneumonia;

2. not consistent with pneumonia; and

3. requiring manual review (for example, containing conflicting statements about pneumonia).

To assess validity, we ran ONYX on a “test set” of 5,000 randomly selected reports, oversampling reports showing pneumonia (based on manual review) and those from people with comorbidities (cancer, chronic lung disease, or congestive heart failure). We estimated the sensitivity, specificity, and positive predictive value (PPV) of ONYX compared to manual review using multivariable logistic regression, accounting for clustering of reports within people. We examined how ONYX’s performance varied by age and comorbidity.

Results ONYX classified 26% of reports (1,276/5,000; 38% [841/2,200] of true pneumonias and 16% [435/2,800] of non-pneumonias) as “requiring manual review” based on pre-defined criteria. Reports from older people were more likely to require manual review than those from younger people. Among reports that could be classified, ONYX had a sensitivity of 91% (1,242/1,359), specificity of 92% (2,170/2,365), and PPV of 81% (1,242/1,437, modeled based on pneumonia prevalence in the source database). Sensitivity and specificity were similar regardless of comorbidity.

Conclusions NLP offers potential for identifying pneumonia outcomes from EMR data. Next steps include 1) further training to decrease the proportion of reports requiring manual review and 2) evaluating the accuracy of ONYX in other health systems.