Skip to main content
SpringerLink
Log in

Application of computer-aided diagnosis for Lung-RADS categorization in CT screening for lung cancer: effect on inter-reader agreement

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To evaluate the effects of computer-aided diagnosis (CAD) on inter-reader agreement in Lung Imaging Reporting and Data System (Lung-RADS) categorization.

Methods

Two hundred baseline CT scans covering all Lung-RADS categories were randomly selected from the National Lung Cancer Screening Trial. Five radiologists independently reviewed the CT scans and assigned Lung-RADS categories without CAD and with CAD. The CAD system presented up to five of the most risk-dominant nodules with measurements and predicted Lung-RADS category. Inter-reader agreement was analyzed using multirater Fleiss κ statistics.

Results

The five readers reported 139–151 negative screening results without CAD and 126–142 with CAD. With CAD, readers tended to upstage (average, 12.3%) rather than downstage Lung-RADS category (average, 4.4%). Inter-reader agreement of five readers for Lung-RADS categorization was moderate (Fleiss kappa, 0.60 [95% confidence interval, 0.57, 0.63]) without CAD, and slightly improved to substantial (Fleiss kappa, 0.65 [95% CI, 0.63, 0.68]) with CAD. The major cause for disagreement was assignment of different risk-dominant nodules in the reading sessions without and with CAD (54.2% [201/371] vs. 63.6% [232/365]). The proportion of disagreement in nodule size measurement was reduced from 5.1% (102/2000) to 3.1% (62/2000) with the use of CAD (p < 0.001). In 31 cancer-positive cases, substantial management discrepancies (category 1/2 vs. 4A/B) between reader pairs decreased with application of CAD (pooled sensitivity, 85.2% vs. 91.6%; p = 0.004).

Conclusions

Application of CAD demonstrated a minor improvement in inter-reader agreement of Lung-RADS category, while showing the potential to reduce measurement variability and substantial management change in cancer-positive cases.

Key Points

• Inter-reader agreement of five readers for Lung-RADS categorization was minimally improved by application of CAD, with a Fleiss kappa value of 0.60 to 0.65.

• The major cause for disagreement was assignment of different risk-dominant nodules in the reading sessions without and with CAD (54.2% vs. 63.6%).

• In 31 cancer-positive cases, substantial management discrepancies between reader pairs, referring to a difference in follow-up interval of at least 9 months (category 1/2 vs. 4A/B), were reduced in half by application of CAD (32/310 to 16/310) (pooled sensitivity, 85.2% vs. 91.6%; p = 0.004).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

ACR:

American College of Radiology

CAD:

Computer-aided diagnosis

CI:

Confidence interval

Lung-RADS:

Lung Imaging Reporting and Data System

NLST:

National Lung Screening Trial

References

  1. National Lung Screening Trial Research Team, Aberle DR, Adams AM et al (2011) Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 365:395–409

  2. de Koning HJ, van der Aalst CM, de Jong PA et al (2020) Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med 382:503–513

  3. Pastorino U, Silva M, Sestini S et al (2019) Prolonged lung cancer screening reduced 10-year mortality in the MILD trial: new confirmation of lung cancer screening efficacy. Ann Oncol 30:1162–1169

    Article  CAS  Google Scholar 

  4. American College of Radiology (2014) Lung CT screening reporting and data system (Lung-RADS). Available via https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads

  5. Henschke CI, Yip R, Yankelevitz DF, Smith JP, International Early Lung Cancer Action Program Investigators (2013) Definition of a positive test result in computed tomography screening for lung cancer: a cohort study. Ann Intern Med 158:246–252

  6. McKee BJ, Regis SM, McKee AB, Flacke S, Wald C (2015) Performance of ACR Lung-RADS in a clinical CT lung screening program. J Am Coll Radiol 12:273–276

    Article  Google Scholar 

  7. van Riel SJ, Sánchez CI, Bankier AA et al (2015) Observer variability for classification of pulmonary nodules on low-dose CT images and its effect on nodule management. Radiology 277:863–871

    Article  Google Scholar 

  8. Revel MP, Bissery A, Bienvenu M, Aycard L, Lefort C, Frija G (2004) Are two-dimensional CT Measurements of small noncalcified pulmonary nodules reliable? Radiology 231:453–458

  9. Kim H, Park CM, Woo S et al (2013) Pure and part-solid pulmonary ground-glass nodules: measurement variability of volume and mass in nodules with a solid portion less than or equal to 5 mm. Radiology 269:585–593

    Article  Google Scholar 

  10. Ridge CA, Yildirim A, Boiselle PM et al (2015) Differentiating between subsolid and solid pulmonary nodules at CT: inter- and intraobserver agreement between experienced thoracic radiologists. Radiology 278:888–896

    Article  Google Scholar 

  11. van Riel SJ, Jacobs C, Scholten ET et al (2019) Observer variability for Lung-RADS categorisation of lung cancer screening CTs: impact on patient management. Eur Radiol 29:924–931

    Article  Google Scholar 

  12. Zhao Y, de Bock GH, Vliegenthart R et al (2012) Performance of computer-aided detection of pulmonary nodules in low-dose CT: comparison with double reading by nodule volume. Eur Radiol 22:2076–2084

    Article  Google Scholar 

  13. Liang M, Tang W, Xu DM et al (2016) Low-dose CT screening for lung cancer: computer-aided detection of missed lung cancers. Radiology 281:279–288

    Article  Google Scholar 

  14. Rubin GD, Lyo JK, Paik DS et al (2005) Pulmonary nodules on multi-detector row CT scans: performance comparison of radiologists and computer-aided detection. Radiology 234:274–283

    Article  Google Scholar 

  15. Christe A, Leidolt L, Huber A et al (2013) Lung cancer screening with CT: evaluation of radiologists and different computer assisted detection software (CAD) as first and second readers for lung nodule detection at different dose levels. Eur J Radiol 82:e873–e878

    Article  CAS  Google Scholar 

  16. Wang S, Zhou M, Liu Z et al (2017) Central focused convolutional neural networks: developing a data-driven model for lung nodule segmentation. Med Image Anal 40:172–183

    Article  Google Scholar 

  17. Ciompi F, Chung K, van Riel SJ et al (2017) Towards automatic pulmonary nodule management in lung cancer screening with deep learning. Sci Rep 7:46479

    Article  CAS  Google Scholar 

  18. Ardila D, Kiraly AP, Bharadwaj S et al (2019) End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med 25:954–961

    Article  CAS  Google Scholar 

  19. National Lung Screening Trial Research Team, Aberle DR, Berg CD et al (2011) The National Lung Screening Trial: overview and study design. Radiology 258:243–253

  20. Lung-RADS Assessment Categories, Version 1.1. American College of Radiology. Lung CT Screening Reporting and Data System (Lung-RADS™). Available via http://www.acr.org/Quality-Safety/Resources/LungRADS. Accessed September 2019

  21. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  Google Scholar 

  22. Liang M, Tang W, Xu DM et al (2016) Low-dose CT screening for lung cancer: computer-aided detection of missed lung cancers. Radiology 281:279–288

  23. Brown M, Browning P, Wahi-Anwar MW et al (2019) Integration of chest CT CAD into the clinical workflow and impact on radiologist efficiency. Acad Radiol 26:626–631

    Article  Google Scholar 

  24. Armato SG 3rd, McLennan G, Bidaut L et al (2011) The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys 38:915–931

    Article  Google Scholar 

  25. Clark TJ, Flood TF, Maximin ST, Sachs PB (2015) Lung CT screening reporting and data system speed and accuracy are increased with the use of a semiautomated computer application. J Am Coll Radiol 12:1301–1306

    Article  Google Scholar 

  26. Li L, Liu Z, Huang H, Lin M, Luo D (2019) Evaluating the performance of a deep learning-based computer-aided diagnosis (DL-CAD) system for detecting and characterizing lung nodules: comparison with the performance of double reading by radiologists. Thorac Cancer 10:183–192

    Article  CAS  Google Scholar 

  27. Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491

    Article  Google Scholar 

Download references

Funding

This study received funding from the Industrial Strategic Technology Development program (10072064, Development of Novel Artificial Intelligence Technologies to Assist Imaging Diagnosis of Pulmonary, Hepatic, and Cardiac Diseases and Their Integration into Commercial Clinical PACS Platforms) funded by the Ministry of Trade Industry and Energy (MI, Korea) and from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI18C0673).

Author information

Author notes

  1. Sohee Park and Hyunho Park contributed equally to this work.

Authors and Affiliations

  1. Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, 88 Olympic-ro 43 Gil, Songpa-gu, Seoul, 138-736, Korea

    Sohee Park, Sang Min Lee, Yura Ahn, Wooil Kim & Joon Beom Seo

  2. VUNO Inc., Seoul, South Korea

    Hyunho Park & Kyuhwan Jung

  3. Department of Radiology and Medical Imaging, University of Virginia Health System, Charlottesville, VA, USA

    Wooil Kim

Authors
  1. Sohee Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyunho Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sang Min Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yura Ahn
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wooil Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kyuhwan Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Joon Beom Seo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sang Min Lee.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Sang Min Lee.

Conflict of interest

Two of the authors of this manuscript (Hyunho Park and Kyuhwan Jung) belong to VUNO Inc., Seoul, South Korea, but they were not involved in the data analysis, and they did not control the data. Otherwise, none of the authors have conflicts of interest and all authors have read and approved this manuscript.

Statistics and biometry

Seo Young Park, who is a statistician in Asan medical center, provided statistical advice for this manuscript.

Informed consent

All study cases were derived from the National Lung Cancer Screening Trial (NLST) and all participants provided informed consent.

Ethical approval

The NLST was approved by the institutional review board of all participating centers.

Study subjects or cohorts overlap

All of our study patients (200 out of 200) were reported in a previous retrospective study, which built a deep learning-based computer-aided diagnosis system performing lung cancer risk categorization tasks and evaluated its performance for lung cancer screening (reference 14, End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med 25:954-961).

Methodology

• retrospective

• observational

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 23 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, S., Park, H., Lee, S.M. et al. Application of computer-aided diagnosis for Lung-RADS categorization in CT screening for lung cancer: effect on inter-reader agreement. Eur Radiol 32, 1054–1064 (2022). https://doi.org/10.1007/s00330-021-08202-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-021-08202-3

Keywords

Search

Navigation