Skip to main content
SpringerLink
Log in

Improving interobserver agreement and performance of deep learning models for segmenting acute ischemic stroke by combining DWI with optimized ADC thresholds

  • Neuro
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objectives

To examine the role of ADC threshold on agreement across observers and deep learning models (DLMs) plus segmentation performance of DLMs for acute ischemic stroke (AIS).

Methods

Twelve DLMs, which were trained on DWI-ADC-ADC combination from 76 patients with AIS using 6 different ADC thresholds with ground truth manually contoured by 2 observers, were tested by additional 67 patients in the same hospital and another 78 patients in another hospital. Agreement between observers and DLMs were evaluated by Bland-Altman plot and intraclass correlation coefficient (ICC). The similarity between ground truth (GT) defined by observers and between automatic segmentation performed by DLMs was evaluated by Dice similarity coefficient (DSC). Group comparison was performed using the Mann-Whitney U test. The relationship between the DSC and ADC threshold as well as AIS lesion size was evaluated by linear regression analysis. A p < .05 was considered statistically significant.

Results

Excellent interobserver agreement and intraobserver repeatability in the manual segmentation (all ICC > 0.98, p < .001) were achieved. The 95% limit of agreement was reduced from 11.23 cm2 for GT on DWI to 0.59 cm2 for prediction at an ADC threshold of 0.6 × 10−3 mm2/s combined with DWI. The segmentation performance of DLMs was improved with an overall DSC from 0.738 ± 0.214 on DWI to 0.971 ± 0.021 on an ADC threshold of 0.6 × 10−3 mm2/s combined with DWI.

Conclusions

Combining an ADC threshold of 0.6 × 10−3 mm2/s with DWI reduces interobserver and inter-DLM difference and achieves best segmentation performance of AIS lesions using DLMs.

Key Points

• Higher Dice similarity coefficient (DSC) in predicting acute ischemic stroke lesions was achieved by ADC thresholds combined with DWI than by DWI alone (all p < .05).

• DSC had a negative association with the ADC threshold in most sizes, both hospitals, and both observers (most p < .05) and a positive association with the stroke size in all ADC thresholds, both hospitals, and both observers (all p < .001).

• An ADC threshold of 0.6 × 10−3 mm2/s eliminated the difference of DSC at any stroke size between observers or between hospitals (p = .07 to > .99).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

ADC:

Apparent diffusion coefficient

aGT:

ADC-gated ground truth

AIS:

Acute ischemic stroke

DLM:

Deep learning model

DSC:

Dice similarity coefficient

DSCg :

Dice similarity coefficient between two manually contoured ground truths

DSCp :

Dice similarity coefficient between the prediction and the ground truth

DWI:

Diffusion-weighted imaging

ESM:

Electrical supplementary material

GT:

Ground truth

H1:

China Medical University Hospital

H2:

China Medical University Hsinchu Hospital

ICC:

Intraclass correlation coefficient

LOA-95:

95% limits of agreement (i.e., 1.96 standard deviation)

mGT:

Manually contoured ground truth

sGT:

Semiautomatically defined ground truth

References

  1. Nag MK, Koley S, China D et al (2017) Computer-assisted delineation of cerebral infarct from diffusion-weighted MRI using Gaussian mixture model. Int J Comput Assist Radiol Surg 12:539–552

    Article  Google Scholar 

  2. Lee H, Jung K, Kang DW, Kim N (2020) Fully automated and real-time volumetric measurement of infarct core and penumbra in diffusion- and perfusion-weighted MRI of patients with hyper-acute stroke. J Digit Imaging 33:262–272

    Article  Google Scholar 

  3. Lee H, Lee EJ, Ham S et al (2020) Machine learning approach to identify stroke within 4.5 hours. Stroke 51:860–866

    Article  Google Scholar 

  4. Chen L, Bentley P, Rueckert D (2017) Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks. Neuroimage Clin 15:633–643

    Article  Google Scholar 

  5. Kim YC, Lee JE, Yu I et al (2019) Evaluation of diffusion lesion volume measurements in acute ischemic stroke using encoder-decoder convolutional network. Stroke 50:1444–1451

    Article  Google Scholar 

  6. Prakash KNB, Gupta V, Bilello M, Beauchamp NJ, Nowinski WL (2006) Identification, segmentation, and image property study of acute infarcts in diffusion-weighted images by using a probabilistic neural network and adaptive Gaussian mixture model. Acad Radiol 13:1474–1484

    Article  Google Scholar 

  7. Tsai JZ, Peng SJ, Chen YW et al (2014) Automatic detection and quantification of acute cerebral infarct by fuzzy clustering and histographic characterization on diffusion weighted MR imaging and apparent diffusion coefficient map. Biomed Res Int 2014:963032

    PubMed  PubMed Central  Google Scholar 

  8. Winzeck S, Mocking SJT, Bezerra R et al (2019) Ensemble of convolutional neural networks improves automated segmentation of acute ischemic lesions using multiparametric diffusion-weighted MRI. AJNR Am J Neuroradiol 40:938–945

    Article  CAS  Google Scholar 

  9. Zhang R, Zhao L, Lou W et al (2018) Automatic segmentation of acute ischemic stroke from DWI using 3-D fully convolutional DenseNets. IEEE Trans Med Imaging 37:2149–2160

    Article  Google Scholar 

  10. Woo I, Lee A, Jung SC et al (2019) Fully automatic segmentation of acute ischemic lesions on diffusion-weighted imaging using convolutional neural networks: comparison with conventional algorithms. Korean J Radiol 20:1275–1284

    Article  Google Scholar 

  11. Boldsen JK, Engedal TS, Pedraza S et al (2018) Better diffusion segmentation in acute ischemic stroke through automatic tree learning anomaly segmentation. Front Neuroinform 12:21

    Article  Google Scholar 

  12. Perez Malla CU, Valdes Hernandez MDC, Rachmadi MF, Komura T (2019) Evaluation of enhanced learning techniques for segmenting ischaemic stroke lesions in brain magnetic resonance perfusion images using a convolutional neural network scheme. Front Neuroinform 13:33

    Article  Google Scholar 

  13. Mah YH, Jager R, Kennard C, Husain M, Nachev P (2014) A new method for automated high-dimensional lesion segmentation evaluated in vascular injury and applied to the human occipital lobe. Cortex 56:51–63

    Article  Google Scholar 

  14. Xiong Y, Huang CC, Fisher M, Hackney DB, Bhadelia RA, Selim MH (2019) Comparison of automated CT perfusion softwares in evaluation of acute ischemic stroke. J Stroke Cerebrovasc Dis 28:104392

    Article  Google Scholar 

  15. Maier O, Wilms M, von der Gablentz J, Kramer UM, Munte TF, Handels H (2015) Extra tree forests for sub-acute ischemic stroke lesion segmentation in MR sequences. J Neurosci Methods 240:89–100

    Article  Google Scholar 

  16. Maier O, Schroder C, Forkert ND, Martinetz T, Handels H (2015) Classifiers for ischemic stroke lesion segmentation: a comparison study. PLoS One 10:e0145118

    Article  Google Scholar 

  17. Kamnitsas K, Ledig C, Newcombe VFJ et al (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 36:61–78

    Article  Google Scholar 

  18. Wilke M, de Haan B, Juenger H, Karnath HO (2011) Manual, semi-automated, and automated delineation of chronic brain lesions: a comparison of methods. Neuroimage 56:2038–2046

    Article  Google Scholar 

  19. Pustina D, Coslett HB, Turkeltaub PE, Tustison N, Schwartz MF, Avants B (2016) Automated segmentation of chronic stroke lesions using LINDA: lesion identification with neighborhood data analysis. Hum Brain Mapp 37:1405–1421

    Article  Google Scholar 

  20. Ito KL, Kim H, Liew SL (2019) A comparison of automated lesion segmentation approaches for chronic stroke T1-weighted MRI data. Hum Brain Mapp 40:4669–4685

    Article  Google Scholar 

  21. Ogata T, Christensen S, Nagakane Y et al (2013) The effects of alteplase 3 to 6 hours after stroke in the EPITHET-DEFUSE combined dataset: post hoc case-control study. Stroke 44:87–93

    Article  CAS  Google Scholar 

  22. Katyal A, Bhaskar S (2021) CTP-guided reperfusion therapy in acute ischemic stroke: a meta-analysis. Acta Neurol Scand 143:355–366

    Article  Google Scholar 

  23. Loubinoux I, Volk A, Borredon J et al (1997) Spreading of vasogenic edema and cytotoxic edema assessed by quantitative diffusion and T2 magnetic resonance imaging. Stroke 28:419–426 discussion 426-417

    Article  CAS  Google Scholar 

  24. Yu Y, Xie Y, Thamm T et al (2021) Tissue at risk and ischemic core estimation using deep learning in acute stroke. AJNR Am J Neuroradiol. https://doi.org/10.3174/ajnr.A7081

  25. Pistocchi S, Strambo D, Bartolini B et al (2021) MRI software for diffusion-perfusion mismatch analysis may impact on patients’ selection and clinical outcome. Eur Radiol. https://doi.org/10.1007/s00330-021-08211-2

  26. Deutschmann H, Hinteregger N, Wiesspeiner U et al (2021) Automated MRI perfusion-diffusion mismatch estimation may be significantly different in individual patients when using different software packages. Eur Radiol 31:658–665

    Article  Google Scholar 

  27. Yu Y, Xie Y, Thamm T et al (2020) Use of deep learning to predict final ischemic stroke lesions from initial magnetic resonance imaging. JAMA Netw Open 3:e200772

    Article  Google Scholar 

  28. Juan CJ, Chang HC, Hsueh CJ et al (2009) Salivary glands: echo-planar versus PROPELLER diffusion-weighted MR imaging for assessment of ADCs. Radiology 253:144–152

    Article  Google Scholar 

  29. Purushotham A, Campbell BC, Straka M et al (2015) Apparent diffusion coefficient threshold for delineation of ischemic core. Int J Stroke 10:348–353

    Article  Google Scholar 

  30. Zhao B, Ding S, Wu H et al (2019) Automatic acute ischemic stroke lesion segmentation using semi-supervised learning. arXiv:1908.03735

  31. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, LNCS 9351:234–241

    Google Scholar 

  32. Koo TK, Li MY (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 15:155–163

    Article  Google Scholar 

  33. Nazari-Farsani S, Nyman M, Karjalainen T, Bucci M, Isojarvi J, Nummenmaa L (2020) Automated segmentation of acute stroke lesions using a data-driven anomaly detection on diffusion weighted MRI. J Neurosci Methods 333:108575

    Article  Google Scholar 

  34. Luby M, Bykowski JL, Schellinger PD, Merino JG, Warach S (2006) Intra- and interrater reliability of ischemic lesion volume measurements on diffusion-weighted, mean transit time and fluid-attenuated inversion recovery MRI. Stroke 37:2951–2956

    Article  Google Scholar 

  35. Zhao B, Liu Z, Liu G et al (2021) Deep learning-based acute ischemic stroke lesion segmentation method on multimodal MR images using a few fully labeled subjects. Comput Math Methods Med 2021:3628179

    PubMed  PubMed Central  Google Scholar 

  36. Wu O, Winzeck S, Giese AK et al (2019) Big data approaches to phenotyping acute ischemic stroke using automated lesion segmentation of multi-center magnetic resonance imaging data. Stroke 50:1734–1741

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful for the financial support from China Medical University Hsinchu Hospital and Taiwan Ministry of Science and Technology (Taiwan).

Funding

C.J.J. received financial support partly from China Medical University Hsinchu Hospital (CMUHCH-DMR-109-017 and CMUHCH-DMR-110-016). Y.J.L. received financial support partly from Taiwan Ministry of Science and Technology (107-2221-E-035 -033 -MY3).

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, Republic of China

    Chun-Jung Juan & Ruey-Feng Chang

  2. Department of Medical Imaging, China Medical University Hsinchu Hospital, Hsinchu, Taiwan, Republic of China

    Chun-Jung Juan, Shao-Chieh Lin, Ya-Hui Li, Chia-Ching Chang, Yi-Hung Jeng & Wu-Chung Shen

  3. Department of Radiology, School of Medicine, College of Medicine, China Medical University, Taichung, Taiwan, Republic of China

    Chun-Jung Juan & Wu-Chung Shen

  4. Department of Medical Imaging, China Medical University Hospital, Taichung, Taiwan, Republic of China

    Chun-Jung Juan

  5. Department of Biomedical Engineering and Environmental Sciences, National Tsing Hua University, Hsinchu, Taiwan, Republic of China

    Chun-Jung Juan, Yi-Hung Jeng & Hsu-Hsia Peng

  6. Ph.D. Program in Electrical and Communication Engineering, Feng Chia University, Taichung, Taiwan, Republic of China

    Shao-Chieh Lin

  7. Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei, Taiwan, Republic of China

    Ya-Hui Li, Hsiao-Wen Chung & Ruey-Feng Chang

  8. Department of Management Science, National Chiao-Tung University, Hsinchu, Taiwan, Republic of China

    Chia-Ching Chang

  9. Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, Republic of China

    Teng-Yi Huang

  10. Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan, Republic of China

    Hsiao-Wen Chung

  11. Department of Neurology, China Medical University Hospital, Taichung, Taiwan, Republic of China

    Chon-Haw Tsai

  12. Department of Automatic Control Engineering, Feng Chia University, No. 100 Wenhwa Rd., Seatwen, 40724, Taichung, Taiwan, Republic of China

    Yi-Jui Liu

Authors
  1. Chun-Jung Juan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shao-Chieh Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ya-Hui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chia-Ching Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yi-Hung Jeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hsu-Hsia Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Teng-Yi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hsiao-Wen Chung
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Wu-Chung Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Chon-Haw Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ruey-Feng Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Yi-Jui Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ruey-Feng Chang or Yi-Jui Liu.

Ethics declarations

Guarantor

The scientific guarantor of this publication is C.J.J.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was waived by the institutional review board.

Ethical approval

Institutional review board approval was obtained.

Methodology

• retrospective

• diagnostic study

• multi-center study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ruey-Feng Chang and Yi-Jui Liu contribute equally to this work.

Supplementary Information

ESM 1

(DOCX 4342 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Juan, CJ., Lin, SC., Li, YH. et al. Improving interobserver agreement and performance of deep learning models for segmenting acute ischemic stroke by combining DWI with optimized ADC thresholds. Eur Radiol 32, 5371–5381 (2022). https://doi.org/10.1007/s00330-022-08633-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-022-08633-6

Keywords

Search

Navigation