Skip to main content
SpringerLink
Log in

Computer-assisted detection of pulmonary embolism: performance evaluation in consensus with experienced and inexperienced chest radiologists

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

The value of a computer-aided detection tool (CAD) as second reader in combination with experienced and inexperienced radiologists for the diagnosis of acute pulmonary embolism (PE) was assessed prospectively. Computed tomographic angiography (CTA) scans (64 × 0.6 mm collimation; 61.4 mm/rot table feed) of 56 patients (31 women, 34–89 years, mean = 66 years) with suspected PE were analysed by two experienced (R1, R2) and two inexperienced (R3, R4) radiologists for the presence and distribution of emboli using a five-point confidence rating, and by CAD. Informed consent was obtained from all patients. Results were compared with an independent reference standard. Inter-observer agreement was calculated by kappa, confidence assessed by ROC analysis. A total of 1,116 emboli [within mediastinal (n = 72), lobar (n = 133), segmental (n = 465) and subsegmental arteries (n = 455)] were included. CAD detected 343 emboli (sensitivity = 30.74%, correct-positive rate = 6.13/patient; false-positive rate = 4.1/patient). Inter-observer agreement was good (R1, R2: κ = 0.84, 95% CI = 0.81–0.87; R3, R4: κ = 0.79, 95% CI = 0.76–0.81). Extended inter-observer agreement was higher in mediastinal and lobar than in segmental and subsegmental arteries (κ = 0.84–0.86 and κ = 0.51–0.58 for mediastinal/lobar and segmental/subsegmental arteries, respectively P < 0.05). Agreement between experienced and inexperienced readers was improved by CAD (κ = 0.60–0.62 and κ = 0.69–0.72 before and after CAD consensus, respectively P < 0.05). The experienced outperformed the inexperienced readers (Az = 0.95, 0.93, 0.89 and 0.86 for R1–4, respectively, P < 0.05). CAD significantly improved overall performances of readers 3 and 4 (Az = 0.86 for R3, R4 and Az = 0.89 for R3, R4 with CAD, P < 0.05), by enhancing sensitivities in segmental/subsegmental arteries. CAD improved experienced readers’ sensitivities in segmental/subsegmental arteries (sens. = 0.93 and 0.90 for R1, R2 before and 0.97 and 0.94 for R1, R2 after CAD consensus, P < 0.05), without significant improvement of their overall performances (P > 0.05). Particularly inexperienced readers benefit from consensus with CAD data, greatly improving detection of segmental and subsegmental emboli. This system is advocated as a second reader.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Dalen J, Alpert J (1975) Natural history of pulmonary embolism. Progr Cardiovasc Dis 17:257–270

    Article  Google Scholar 

  2. The PIOPED Investigators (1990) Value of the ventilation/perfusion scan in acute pulmonary embolism. Results of the Prospective Investigation of Pulmonary Embolism Diagnosis (PIOPED). JAMA 263:2753–2759

    Article  Google Scholar 

  3. Price D (1976) Pulmonary embolism. Prophylaxis diagnosis and treatment. Anaesthesia 31:925–932

    Article  PubMed  CAS  Google Scholar 

  4. Engelke C, Rummeny EJ, Marten K (2006) Pulmonary embolism at multi-detector row CT of chest: one-year survival of treated and untreated patients. Radiology 239:563–575

    Article  PubMed  Google Scholar 

  5. Stein PD, Fowler SE, Goodman LR et al (2006) Multidetector computed tomography for acute pulmonary embolism. N Engl J Med 354:2317–2327

    Article  PubMed  CAS  Google Scholar 

  6. Remy-Jardin M, Remy J, Wattinne L, Giraud F (1992) Central pulmonary thromboembolism diagnosis with spiral volumetric CT with the single-breath-hold technique-comparison with pulmonary angiography. Radiology 185:381–387

    PubMed  CAS  Google Scholar 

  7. McCollough C, Zink F (1999) Performance evaluation of a multi-slice CT system. Med Phys 26:2223–2230

    Article  PubMed  CAS  Google Scholar 

  8. Rubin GD, Paik DS, Johnston PC, Napel S (1998) Measurements of the aorta and its branches with helical CT. Radiology 206:823–829

    PubMed  CAS  Google Scholar 

  9. Stein PD (1999) Reassessment of pulmonary angiography for the diagnosis of pulmonary embolism: relation of interpreter agreement to the order of the involved pulmonary arterial branch. Radiology 210:689–691

    PubMed  CAS  Google Scholar 

  10. Ghaye B, Szapiro D, Mastora I et al (2001) Peripheral pulmonary arteries: how far in the lung does multi-detector row spiral CT allow analysis? Radiology 219:629–636

    PubMed  CAS  Google Scholar 

  11. Raptopoulos V, Boiselle P (2001) Multi-detector row spiral CT pulmonary angiography comparison with single-detector row spiral CT. Radiology 221:606–613

    Article  PubMed  CAS  Google Scholar 

  12. Schoepf U, Holzknecht M, Helmberger TK et al (2002) Subsegmental pulmonary emboli improved detection with thin-collimation multi-detector row spiral CT. Radiology 222:482–490

    Article  Google Scholar 

  13. Diffin D, Leyendecker J, Johnson S et al (1998) Effect of anatomic distribution of pulmonary emboli on interobserver agreement in the interpretation of pulmonary angiography, AJR Am J Roentgenol 171:1085–1089

    PubMed  CAS  Google Scholar 

  14. Mayo J, Remy-Jardin M, Muller N (1997) Pulmonary embolism: prospective comparison of spiral CT with ventilation perfusion scintigraphy. Radiology 205:447–452

    PubMed  CAS  Google Scholar 

  15. Qanadli S, Hajjam ME, Mesurille B (2000) Pulmonary embolism detection: prospective evaluation of dual-section helical CT versus selective pulmonary arteriography in 157 patients. Radiology 217:447–455

    PubMed  CAS  Google Scholar 

  16. Goodman LR, Curtin J, Mewissen M (1995) Detection of pulmonary embolism in patients with unsolved clinical and scintigraphic diagnosis. Helical CT versus angiography. AJR Am J Roentgenol 164:1369–1374

    PubMed  CAS  Google Scholar 

  17. Senac J, Verhnet H, Bousquet C (1995) Embolie pulmonaire: apport de la tomodensitometrie helicoidale. J Radiol 76:339–345

    PubMed  CAS  Google Scholar 

  18. van Rossum AB, Pattynama PM, Ton ER, Treurniet FE, Arndt JW, van Eck B, Kieft GJ (1996) Pulmonary embolism Validation of spiral CT angiography in 149 patients. Radiology 201:467–470

    PubMed  Google Scholar 

  19. Remy-Jardin M, Remy J, Petyt L, Duhamel A (1996). Diagnosis of acute pulmonary embolism with spiral CT Comparison with pulmonary angiography and scintigraphy. Radiology 200:699–706

    PubMed  CAS  Google Scholar 

  20. Ferretti G, Bosson J, Buffaz P (1997) Acute pulmonary embolism role of helical CT in 164 patients with intermediate probability at ventilation-perfusion scintigraphy and normal results at duplex US of the legs. Radiology 205:453–458

    PubMed  CAS  Google Scholar 

  21. van Rossum AB, Pattynama PM, Mallens WM, Hermans J, Heijerman HG (1998) Can helical CT replace scintigraphy in the diagnostic process in suspected pulmonary embolism? A retrolective-prolective cohort study focusing on total diagnostic yield. Eur Radiol 8:90–96

    Article  PubMed  Google Scholar 

  22. Drucker E, Rivitz S, Shepard J (1998) Acute pulmonary embolism: assessment of helical CT for diagnosis. Radiology 209:235–241

    PubMed  CAS  Google Scholar 

  23. Herold C, Remy-Jardin M, Grenier P (1998) Prospective evaluation of pulmonary embolism initial results of the European multicenter trial (ESTIPEP) [abstract]. Radiology 209:299

    Google Scholar 

  24. Garg K, Welsh C, Feyerabend A (1998) Pulmonary embolism diagnosis with spiral CT and ventilation-perfusion scanning-correlation with pulmonary angiographic results or clinical outcome. Radiology 208:201–208

    PubMed  CAS  Google Scholar 

  25. Baghaie F, Remy-Jardin M, Remy J, Fribourg M (1998) Diagnosis of peripheral acute pulmonary emboli: optimization of the spiral CT acquisition protocol [abstract]. Radiology 209:299

    Google Scholar 

  26. Kim K, Muller N, Mayo J (1999) Clinically suspected pulmonary embolism: utility of spiral CT. Radiology 210:693–697

    PubMed  CAS  Google Scholar 

  27. Brunot S, Corneloup O, Latrabe V, Montaudon M, Laurent F (2005) Reproducibility of multi-detector spiral computed tomography in detection of sub-segmental acute pulmonary embolism. Eur Radiol 15:2057–2063

    Article  PubMed  CAS  Google Scholar 

  28. Le Gal G, Righini M, Parent F, van Strijen M, Couturaud F (2006) Diagnosis and management of subsegmental pulmonary embolism. J Thromb Haemost 4:724–731

    Article  PubMed  Google Scholar 

  29. Wood KE (2002) Major pulmonary embolism: review of a pathophysiologic approach to the golden hour of hemodynamically significant pulmonary embolism. Chest 121:877–905

    Article  PubMed  Google Scholar 

  30. Egermayer P (2001) Differences in the rate of anticoagulant complications between routine practice and controlled prospective trials [letter]. J Intern Med 250:258–261

    Article  PubMed  CAS  Google Scholar 

  31. Zhou C, Chan HP, Patel S, Cascade PN, Sahiner B, Hadjiiski LM, Kazerooni EA (2005) Preliminary investigation of computer-aided detection of pulmonary embolism in three-dimensional computed tomography pulmonary angiography images. Acad Radiol 12:782–792

    Article  PubMed  Google Scholar 

  32. Masutani Y, MacMahon H, Doi K (2002) Computerized detection of pulmonary embolism in spiral CT angiography based on volumetric image analysis. IEEE Trans Med Imaging 21:1517–1523

    Article  PubMed  Google Scholar 

  33. Liang J, Bi J (2007) Computer aided detection of pulmonary embolism with tobogganing and multiple instance classification in CT pulmonary angiography. Inf Process Med Imaging 20:630–641

    Article  PubMed  Google Scholar 

  34. Armitage P, Berry G, Matthews JNS (2002) Statistical methods in medical research (4th edn). Blackwell, Oxford

    Google Scholar 

  35. Press WH et al (1992) Numerical recipes: the art of scientific computing (2nd edn). Cambridge University Press, Cambridge

    Google Scholar 

  36. Hanley JA, McNeil BJ (1982) The meaning and use of area under a Receiver Operating Characteristic (ROC) curve. Radiology 143:29–36

    PubMed  CAS  Google Scholar 

  37. DeLong ER, DeLong DM, Clark-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating curves: a nonparametric approach. Biometrics 44:837–845

    Article  PubMed  CAS  Google Scholar 

  38. Fleiss JL (1981) Statistical methods for rates and proportions (2nd edn). Wiley, New York

    Google Scholar 

  39. Fleiss JL, Nee CM, Landis JR (1979) Large sample variance of kappa in the case of different sets of raters. Psychol Bull 86:974–977

    Article  Google Scholar 

  40. Marten K, Grillhösl A, Seyfarth T, Obenauer S, Rummeny EJ, Engelke C (2005) Computer-assisted detection of pulmonary nodules: evaluation of diagnostic performance using an expert knowledge-based detection system with variable reconstruction slice thickness settings. Eur Radiol 15:203–212

    Article  PubMed  Google Scholar 

  41. Croisille P, Souto M, Cova M, Wood S, Afework Y, Kuhlman J, Zerhouni E (1995) Pulmonary nodules: improved detection with vascular segmentation and extraction with spiral CT. Radiology 197:397–401

    PubMed  CAS  Google Scholar 

  42. Wormanns D, Fiebich M, Saidi M, Diederich S, Heindel W (2002) Automatic detection of pulmonary nodules at spiral CT: clinical application of a computer-aided diagnosis system. Eur Radiol 12:1052–1057

    Article  PubMed  Google Scholar 

  43. Marten K, Engelke C (2007) Computer-aided detection and automated CT volumetry of pulmonary nodules. Eur Radiol 17:888–901

    Article  PubMed  Google Scholar 

  44. Armato SG III, Giger ML, Moran CJ, Blackburn JT, Doi K, MacMahon H (1999) Computerized detection of pulmonary nodules on CT scans. Radiographics 19:1303–1311

    PubMed  Google Scholar 

  45. Armato SG III, Li F, Giger ML, MacMahon H, Sone S, Doi K (2002) Lung cancer: performance of automated lung nodule detection applied to cancers missed in a CT screening program. Radiology 225:685–692

    Article  PubMed  Google Scholar 

  46. Rubin G, Naidich D, Sherbondy A, Lyo J, Napel S (2003) Inadequacy of lung nodule reference standard based upon standard methods of expert consensus review using cine-paging of transverse thin-section MDCT lung scans [abstract]. Radiology 229:441–442

    Google Scholar 

  47. Marten K, Seyfarth T, Auer F, Wiener E, Grillhösl A, Obenauer S, Rummeny EJ, Engelke C (2004) Computer-assisted detection of pulmonary nodules: performance evaluation of an expert knowledge-based detection system in consensus reading with experienced and inexperienced chest radiologists. Eur Radiol 14:1930–1938

    Article  PubMed  Google Scholar 

  48. Awai K, Murao K, Ozawa A et al (2004) Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists´detection performance. Radiology 230:347–352

    Article  PubMed  Google Scholar 

  49. Schoepf UJ, Das M, Schneider AC et al (2002) Computer-aided detection (CAD) of segmental and subsegmental pulmonary embolism on 1-mm multidetector-row CT (MDCT) studies [abstract]. Radiology 225:384

    Google Scholar 

  50. Zhou C, Hadjiiski LM, Sahiner B et al (2003) Computerized detection of pulmonary embolism in 3D computed tomographic (CT) images: vessel tracking and segmentation techniques. Proc SPIE 5032:1613–1620

    Article  Google Scholar 

  51. Buhmann S, Herzog P, Liang J, Wolf M, Salganicoff M, Kirchhoff C, Reiser M, Becker CH (2007) Clinical evaluation of a computer-aided diagnosis (CAD) prototype for the detection of pulmonary embolism. Acad Radiol 14:651–658

    Article  PubMed  Google Scholar 

  52. Quiroz R, Kucher N, Schoepf UJ et al (2004) Right ventricular enlargement on chest computed tomography: prognostic role in acute pulmonary embolism. Circulation 109:2401–2404

    Article  PubMed  Google Scholar 

  53. Engelke C, Rummeny EJ, Marten K (2006) Acute pulmonary embolism on multidetector-row CT of the chest: prediction of cor pulmonale and short-term patient survival from morphological embolus burden. AJR Am J Roentgenol 186:1265–1271

    Article  PubMed  Google Scholar 

  54. Mastora I, Remy-Jardin M, Masson P et al (2003) Severity of acute pulmonary embolism: evaluation of a new spiral CT angiographic score in correlation with echocardiographic data. Eur Radiol 13:29–35

    PubMed  Google Scholar 

  55. Pech M, Wieners G, Dul P, Fischbach F, Dudeck O, Hänninen EL, Ricke J (2007) Computed tomography pulmonary embolism index for the assessment of survival in patients with pulmonary embolism. Eur Radiol 17:1954–1959

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Radiology, University of Wuerzburg, 97080, Wuerzburg, Germany

    Christoph Engelke & Katharina Marten

  2. Department of Radiology, Klinikum rechts der Isar, Technical University, Munich, Germany

    Stephan Schmidt & Florian Auer

  3. Siemens Medical Solutions, Siemensstr. 1, Forchheim, Germany

    Annemarie Bakai

Authors
  1. Christoph Engelke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephan Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Annemarie Bakai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Florian Auer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Katharina Marten
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christoph Engelke.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Engelke, C., Schmidt, S., Bakai, A. et al. Computer-assisted detection of pulmonary embolism: performance evaluation in consensus with experienced and inexperienced chest radiologists. Eur Radiol 18, 298–307 (2008). https://doi.org/10.1007/s00330-007-0770-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-007-0770-3

Keywords

Search

Navigation