Elsevier

NeuroImage

Volume 157, 15 August 2017, Pages 233-249
NeuroImage

Performance comparison of 10 different classification techniques in segmenting white matter hyperintensities in aging

https://doi.org/10.1016/j.neuroimage.2017.06.009Get rights and content

Abstract

Introduction

White matter hyperintensities (WMHs) are areas of abnormal signal on magnetic resonance images (MRIs) that characterize various types of histopathological lesions. The load and location of WMHs are important clinical measures that may indicate the presence of small vessel disease in aging and Alzheimer's disease (AD) patients. Manually segmenting WMHs is time consuming and prone to inter-rater and intra-rater variabilities. Automated tools that can accurately and robustly detect these lesions can be used to measure the vascular burden in individuals with AD or the elderly population in general. Many WMH segmentation techniques use a classifier in combination with a set of intensity and location features to segment WMHs, however, the optimal choice of classifier is unknown.

Methods

We compare 10 different linear and nonlinear classification techniques to identify WMHs from MRI data. Each classifier is trained and optimized based on a set of features obtained from co-registered MR images containing spatial location and intensity information. We further assess the performance of the classifiers using different combinations of MRI contrast information. The performances of the different classifiers were compared on three heterogeneous multi-site datasets, including images acquired with different scanners and different scan-parameters. These included data from the ADC study from University of California Davis, the NACC database and the ADNI study. The classifiers (naïve Bayes, logistic regression, decision trees, random forests, support vector machines, k-nearest neighbors, bagging, and boosting) were evaluated using a variety of voxel-wise and volumetric similarity measures such as Dice Kappa similarity index (SI), Intra-Class Correlation (ICC), and sensitivity as well as computational burden and processing times. These investigations enable meaningful comparisons between the performances of different classifiers to determine the most suitable classifiers for segmentation of WMHs. In the spirit of open-source science, we also make available a fully automated tool for segmentation of WMHs with pre-trained classifiers for all these techniques.

Results

Random Forests yielded the best performance among all classifiers with mean Dice Kappa (SI) of 0.66±0.17 and ICC=0.99 for the ADC dataset (using T1w, T2w, PD, and FLAIR scans), SI=0.72±0.10, ICC=0.93 for the NACC dataset (using T1w and FLAIR scans), SI=0.66±0.23, ICC=0.94 for ADNI1 dataset (using T1w, T2w, and PD scans) and SI=0.72±0.19, ICC=0.96 for ADNI2/GO dataset (using T1w and FLAIR scans). Not using the T2w/PD information did not change the performance of the Random Forest classifier (SI=0.66±0.17, ICC=0.99). However, not using FLAIR information in the ADC dataset significantly decreased the Dice Kappa, but the volumetric correlation did not drastically change (SI=0.47±0.21, ICC=0.95).

Conclusion

Our investigations showed that with appropriate features, most off-the-shelf classifiers are able to accurately detect WMHs in presence of FLAIR scan information, while Random Forests had the best performance across all datasets. However, we observed that the performances of most linear classifiers and some nonlinear classifiers drastically decline in absence of FLAIR information, with Random Forest still retaining the best performance.

Introduction

White matter hyperintensities (WMHs), commonly identified as areas of increased signal in relation with the surrounding white matter regions on T2w, PD and FLAIR MRIs, are one of the non-specific yet typical and constant MRI expressions of cerebral small vessel disease (CSVD), along with lacunar infarcts and microhemorrhages (Conklin et al., 2014, Gouw et al., 2010). They have been shown to be more extensive in patients with Alzheimer's disease compared to age-matched healthy normal populations (Yoshita et al., 2005). WMHs reflect ischemic injury in the elderly and AD populations and the existence and severity of WMHs can lead to or accelerate decline in cognitive as well as executive functions (Dubois et al., 2014). As a result, the location and load of WMHs are important clinical measures, raising substantial need for their accurate quantifications. WMHs are generally detected using fluid attenuated inversion recovery (FLAIR) or T2w/PD scans. Manually labeling WMHs is challenging due to time constraints as well as inter-rater and intra-rater variabilities (Grimaud et al., 1996). As a result, automated tools that can segment WMHs robustly and with high accuracy are extremely useful, particularly in large scale studies such the Alzheimer's Disease Neuroimaging Initiative (http://www.loni.ucla.edu/ADNI/), the National Alzheimer's Coordinating Center (NACC) database (https://www.alz.washington.edu/) and others where it is desired to estimate the contribution of neurovascular disease to cognitive decline.

The heterogeneity in the distribution and patterns of WMHs makes the segmentation task intrinsically complex (Caligiuri et al., 2015). Automated segmentation tools usually integrate information from multiple complementary MRI contrasts including T1w, T2w, PD and FLAIR to reduce uncertainty and improve segmentation accuracy. Most successful fully automated WMH segmentation techniques extract a combination of location and intensity features from these images and use them as inputs to a linear or nonlinear classifier. Here we review the most commonly used linear and nonlinear classifiers in general as well as their application to the task of segmenting lesions in general or WMHs of vascular etiology specifically.

While there have been many studies attempting to segment WMHs using these classification techniques, drawing meaningful comparisons between their performances is not possible since they have been applied to different datasets and results are highly variable across different populations and imaging protocols (García-Lorenzo et al., 2013; Caligiuri et al., 2015). To our knowledge, no studies have compared the performance of these classification techniques for detecting WMHs against one another on the same datasets, especially for cases where classification is attempted without using the optimal FLAIR information. In this paper, we have extensively compared the performance of these different classification techniques in detecting WMHs with and without FLAIR information using 3 different large publicly available datasets with different scanners and acquisition protocols. This enables us to draw more generalizable conclusions regarding the performance of the classifiers. Our contributions include an extensive comparison of 10 widely used classification techniques in detecting WMHs across 4 different datasets, three of which are from multi-site and multi-scanner studies and across different combinations of imaging modalities. In addition, we make publicly available an implementation of the segmentation tool along with all the pre-trained classifiers (http://nist.mni.mcgill.ca/?p=221). The proposed tool is generalizable to data from different scanners since it has been trained on data from multiple scanners.

Section snippets

Subjects

The performances of the different classifiers were assessed based on four datasets of subjects with different ranges of WMH loads. Table 1 shows the demographic information for each dataset.

Segmentation using T1w, T2w, PD, and FLAIR

The performance of each classifier was validated through 10-fold cross validation using T1w, T2w, PD, and FLAIR images for the ADC dataset. All voxels within a brain mask that contained the cerebrum, cerebellum and brain stem were classified. Table 5 shows the average Dice Kappa, detection/outline error rates (DER/OER), ICC, TPR, and PPV values for different classifiers. Fig. 3 shows boxplot diagrams for the same results separately for subjects with small, medium and large WMH loads. Fig. 4

Discussion

In the recent years, there have been many different studies in the literature that address the challenge of automatically segmenting WMHs (Caligiuri et al., 2015, Admiraal-Behloul et al., 2005, Anbeek et al., 2004, Beare et al., 2009, De Boer et al., 2009, Dyrby et al., 2008, Ghafoorian et al., 2016a, Griffanti et al., 2016, Ithapu et al., 2014, Lao et al., 2008, Ong et al., 2012, Schmidt et al., 2012, Simões et al., 2013, Steenwijk et al., 2013, Wu et al., 2006a, Wu et al., 2006b, Yoo et al.,

Acknowledgement

We would like to acknowledge funding from the Famille Louise & André Charron. This work was also supported by grants from the Canadian Institutes of Health Research (MOP-111169), les Fonds de Research Santé Québec Pfizer Innovation fund, an NSERC CREATE grant (4140438 − 2012), the Levesque Foundation, the Douglas Hospital Research Centre and Foundation, the Government of Canada, and the Canada Fund for Innovation. This research was also supported by NIH Grants P30AG010129, K01 AG030514, and the

References (78)

  • G. Erus et al.

    Individualized statistical learning from medical image databases: application to identification of brain lesions

    Med. Image Anal.

    (2014)
  • V. Fonov et al.

    Unbiased average age-appropriate atlases for pediatric studies

    NeuroImage

    (2011)
  • D. García-Lorenzo et al.

    Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging

    Med. Image Anal.

    (2013)
  • E. Geremia et al.

    Spatial decision forests for MS lesion segmentation in multi-channel magnetic resonance images

    NeuroImage

    (2011)
  • L. Griffanti et al.

    BIANCA (Brain Intensity AbNormality Classification Algorithm): a new tool for automated segmentation of white matter hyperintensities

    NeuroImage

    (2016)
  • J. Grimaud et al.

    Quantification of MRI lesion load in multiple sclerosis: a comparison of three computer-assisted techniques

    Magn. Reson. Imaging

    (1996)
  • R. Khayati et al.

    Fully automatic segmentation of multiple sclerosis lesions in brain MR FLAIR images using adaptive mixtures method and Markov random field model

    Comput. Biol. Med.

    (2008)
  • C. Köse et al.

    Simple methods for segmentation and measurement of diabetic retinopathy lesions in retinal fundus images

    Comput. Methods Prog. Biomed.

    (2012)
  • Z. Lao et al.

    Computer-assisted segmentation of white matter lesions in 3D MR images using support vector machine

    Acad. Radiol.

    (2008)
  • O. Maier et al.

    Extra Tree forests for sub-acute ischemic stroke lesion segmentation in MR sequences

    J. Neurosci. Methods

    (2015)
  • J. Mitra et al.

    Lesion segmentation from multimodal MRI using random forest following ischemic stroke

    NeuroImage

    (2014)
  • K.H. Ong et al.

    Automatic white matter lesion segmentation using an adaptive outlier detection method

    Magn. Reson. Imaging

    (2012)
  • P. Schmidt et al.

    An automated tool for detection of FLAIR-hyperintense white-matter lesions in multiple sclerosis

    NeuroImage

    (2012)
  • N. Shiee et al.

    A topology-preserving approach to the segmentation of brain images with multiple sclerosis lesions

    NeuroImage

    (2010)
  • R. Simões et al.

    Automatic segmentation of cerebral white matter hyperintensities using only 3D FLAIR images

    Magn. Reson. Imaging

    (2013)
  • M.D. Steenwijk et al.

    Accurate white matter lesion segmentation by k nearest neighbor classification with tissue type priors (kNN-TTPs)

    NeuroImage Clin.

    (2013)
  • M. Wu et al.

    A fully automated method for quantifying and localizing white matter hyperintensities on MR images

    Psychiatry Res. Neuroimaging

    (2006)
  • Y. Wu et al.

    Automated segmentation of multiple sclerosis lesion subtypes with multichannel MRI

    NeuroImage

    (2006)
  • B.A. Abdullah et al.

    Textural based SVM for MS lesion segmentation in FLAIR MRIs

    Open J. Med. Imaging

    (2011)
  • A. Akselrod-Ballin et al.

    Automatic segmentation and classification of multiple sclerosis in multichannel MRI

    IEEE Trans. Biomed. Eng.

    (2009)
  • J.A. Alexander et al.

    Adult cerebrovascular disease: role of modified rapid fluid-attenuated inversion-recovery sequences

    Am. J. Neuroradiol.

    (1996)
  • N.S. Altman

    An introduction to kernel and nearest-neighbor nonparametric regression

    Am. Stat.

    (1992)
  • R. Bakshi et al.

    Fluid-attenuated inversion recovery magnetic resonance imaging detects cortical and juxtacortical multiple sclerosis lesions

    Arch. Neurol.

    (2001)
  • F. Barkhof et al.

    Imaging of white matter lesions

    Cerebrovasc. Dis.

    (2002)
  • D.L. Beekly et al.

    The National Alzheimer's Coordinating center (NACC) database: an Alzheimer disease database

    Alzheimer Dis. Assoc. Disord.

    (2004)
  • Boser, B.E., Guyon, I.M., Vapnik, V.N., 1992. A training algorithm for optimal margin classifiers. In: Proceedings of...
  • L. Breiman

    Bagging predictors

    Mach. Learn.

    (1996)
  • L. Breiman

    Random Forests

    Mach. Learn.

    (2001)
  • M.E. Caligiuri et al.

    Automatic detection of white matter hyperintensities in healthy aging and pathology using magnetic resonance imaging: a review

    Neuroinformatics

    (2015)
  • Cited by (55)

    • White matter hyperintensities are associated with grey matter atrophy and cognitive decline in Alzheimer's disease and frontotemporal dementia

      2022, Neurobiology of Aging
      Citation Excerpt :

      For each subject, the T2-weighted, PD, and FLAIR scans were then co-registered to the T1-weighted scan of the same visit, using a rigid body registration (6 parameters: 3 translations and 3 rotations) with a mutual information cost function (Dadar et al., 2018a). Using a previously validated WMH segmentation method, the WMHs were automatically segmented for all longitudinal visits, using the preprocessed and co-registered scans (Dadar et al., 2017c). The quality of the registrations and segmentations was visually assessed, and the results that did not pass this quality control were excluded (n = 102 out of 5774 timepoints).

    View all citing articles on Scopus
    1

    Part of the data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at:

    http://adni.loni.usc.edu/wp-ontent/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf

    View full text