Performance of five research-domain automated WM lesion segmentation methods in a multi-center MS study
Introduction
Multiple sclerosis (MS) is an inflammatory and neurodegenerative disease of the central nervous system, with inflammatory white matter (WM) lesions as prominent pathological hallmark (Benedict and Bobholz, 2007, Lucchinetti et al., 2000). In vivo visualization of lesions by means of MRI plays a crucial role in the diagnosis and study of MS. Moreover, several clinical trials have used WM lesion volume as a (secondary) study outcome (Calabresi et al., 2014, Kappos et al., 2010, Polman et al., 2011, Radue et al., 2015).
For clinical and research purposes, delineation of WM lesions in MS is either performed manually or with a semiautomatic tool. These approaches, however, are labor-intensive and suffer from considerable inter- and intra-rater variability (Grimaud et al., 1996, Paty et al., 1986). To overcome these problems, automated WM lesion segmentation methods have been developed in the last decade (Mortazavi et al., 2012). However, these methods are not routinely applied in research, clinical trials or individual patient care. One important hurdle is the lack of comparative data reporting the accuracy and robustness of these methods when using data obtained from different centers. Evidence of their performance in multi-center investigations is lacking.
The aim of this study was, firstly, to evaluate the performance of research-domain automated WM lesion segmentation methods in a multi-center MS dataset with diverging scanners and protocols. And secondly, to investigate how these methods perform on data from a new center (using other centers for training). We selected five algorithms for automated segmentation: Cascade (Damangir et al., 2012, Damangir et al., 2016); Lesion growth algorithm (LGA) (Schmidt et al., 2012) and Lesion prediction algorithm (LPA) (Schmidt, 2017) both from the Lesion Segmentation Toolbox (LST) (Schmidt et al., 2012); Lesion-Topology-preserving Anatomical Segmentation (Lesion-TOADS) (Shiee et al., 2010); and k-Nearest Neighbor with Tissue Type Priors (kNN-TTP) (Steenwijk et al., 2013).
Section snippets
Subjects
The data for this study were drawn from a multi-center MS dataset that was collected by the MAGNIMS Study Group (www.magnims.eu) as described previously (Ropele et al., 2014). For the analyses described in the current paper, we selected the patients with a 2D FLAIR acquisition, and we excluded three patients with co-morbidity (vascular disease, glioblastoma, surgical removal of part of the brain) that could interfere with the automated lesion segmentation and one patients whose data were
Results
An overview of the optimal configurations derived in the training phase of both experiments is provided in Table 4. Fig. 2 displays a typical example of FLAIR image, the corresponding manual reference segmentation and the corresponding automated segmentation results.
The voxelwise intra-rater variability was SI = 0.73 ± 0.11 (mean ± SD) when comparing the first and second segmentations and 0.75 ± 0.11 when comparing the segmentations on the first and second marking of the lesions.
Discussion
In this study we directly compared five research-domain automated WM lesion segmentation methods in a multi-center MS dataset, to obtain quantitative results on their volumetric and spatial performance in a multi-center dataset. Accurate and robust segmentation of WM lesions would be beneficial for clinical trials in which lesion volumes are used as a (secondary) study outcome and studies on accurately measuring the GM atrophy (Amiri et al., 2017, Rocca et al., 2017). Our results show
Acknowledgements:
Aurélie Ruet was supported by an ECTRIMS research fellowship. Iris D. Kilsdonk was supported by a grant provided by the Noaber Foundation (Lunteren, The Netherlands). Adriaan Versteeg, Ronald A. van Schijndel, Keith S. Cover, Soheil Damangir and Giovanni B. Frisoni were partly funded by neuGRID4you (www.neuGRID4you.eu), an European Community FP7 project (grant agreement 283562). Olga Ciccarelli and Frederol Barkhof were supported by the National Institute for Health Research (NIHR) University
References (31)
- et al.
Fully automatic segmentation of white matter hyperintensities in MR images of the elderly
Neuroimage
(2005) - et al.
Probabilistic segmentation of white matter lesions in MR imaging
NeuroImage
(2004) - et al.
Probabilistic segmentation of brain tissue in MR imaging
Neuroimage
(2005) - et al.
Safety and efficacy of fingolimod in patients with relapsing-remitting multiple sclerosis (FREEDOMS II): a double-blind, randomised, placebo-controlled, phase 3 trial
Lancet Neurol.
(2014) - et al.
Multispectral MRI segmentation of age related white matter changes using a cascade of support vector machines
J. Neurol. Sci.
(2012) - et al.
Quantification of MRI lesion load in multiple sclerosis: a comparison of three computer-assisted techniques
Magn. Reson Imaging
(1996) - et al.
Fully automatic segmentation of multiple sclerosis lesions in brain MR FLAIR images using adaptive mixtures method and Markov random field model
Comput. Biol. Med.
(2008) - et al.
An automated tool for detection of FLAIR-hyperintense white-matter lesions in Multiple Sclerosis
Neuroimage
(2012) - et al.
A topology-preserving approach to the segmentation of brain images with multiple sclerosis lesions
Neuroimage
(2010) - et al.
Accurate white matter lesion segmentation by k nearest neighbor classification with tissue type priors (kNN-TTPs)
Neuroimage Clin.
(2013)
Urgent Challenges in Quantification and Interpretation of Grey Matter Atrophy in Multiple Sclerosis
Multiple sclerosis
Semin. Neurol.
Magnetic resonance materials in physics
Biol. Med.
Measures of the amount of ecologic association between species
Ecology
A placebo-controlled trial of oral fingolimod in relapsing multiple sclerosis
N. Engl. J. Med.
Cited by (27)
Segmentation of white matter lesions in multicentre FLAIR MRI
2021, Neuroimage: ReportsIntracranial volume segmentation for neurodegenerative populations using multicentre FLAIR MRI
2021, Neuroimage: ReportsAccuracy and reproducibility of automated white matter hyperintensities segmentation with lesion segmentation tool: A European multi-site 3T study
2021, Magnetic Resonance ImagingCitation Excerpt :In particular, our main results are summarized as follows: (i) LPA and LGA show a good volumetric accuracy, but LPA performed overall better than LGA; (ii) the LGA and LPA's spatial accuracy increases with the amount of WMHs; (iii) volumetric reproducibility reveals that LST longitudinal pipeline steeply reduces the reproducibility error; (iv) spatial reproducibility of the longitudinal pipeline applied to LGA and LPA outputs was optimal. Compared to De Sitter and colleagues [40] in a cohort of 52 MS patients (mean WMH volume = 4.85 mL), we found a slightly better volumetric accuracy comparing both LPA (mean volume difference = 0.45 mL) and LGA (mean volume difference = 2.88 mL) using SPM12, or other tools that they have tested such as Cascade [41,42] (mean volume difference = 0.67 mL), Lesion-Topology preserving Anatomical Segmentation (Lesion-TOADS) [43] (mean volume difference = 2.18 mL) or using k-Nearest Neighbor with Tissue Type Priors (kNN-TTP) [44] (mean volume difference = −1.46 mL). Similar results have been reported by Egger and colleagues [45] for LGA SPM8 (median volume difference = 0.68 mL), LGA SPM12 (median volume difference = 0.93 mL), LPA SPM12 (median volume difference = 0.85 mL).
ExploreASL: An image processing pipeline for multi-center ASL perfusion MRI studies
2020, NeuroImageCitation Excerpt :LST detects outliers in the FLAIR WM intensity distribution and assesses their likelihood of being WMH (Schmidt et al., 2012). While ExploreASL offers the option of both LST lesion growing and lesion prediction algorithms, the default is set to the latter, which has been shown to be more robust (de Sitter et al., 2017a). This WHM correction described here is only performed when FLAIR images are available.
Multimodal Image Analysis for Assessing Multiple Sclerosis and Future Prospects Powered by Artificial Intelligence
2020, Seminars in Ultrasound, CT and MRICitation Excerpt :Specifically, the segmentation is done based on a multichannel input rather than a single imaging modality. In such an intensity-weighing scheme, the assignment of each voxel to WM or lesions is optimized to the lesion boundary.24,25 MIPAV (https://mipav.cit.nih.gov/pubwiki/index.php/Using_MIPAV_Algorithms) is a well-recognized standalone software that includes features of image enhancement, morphologic operations, surface plotter (ie, 3D plot display of intensities in images), region growing, volume rendering, brain extraction, image statistics, and many other processes.