Keywords

1 Introduction

Magnetic Resonance Imaging (MRI) facilitates identifying brain pathologies. However, variations in MRI acquisition protocols and scanner manufacturer’s parameters lead to differences in the appearance of the clinical MRI features making their automatic detection challenging. Although the widespread use of MRI has produced large amount of datasets to be used in machine learning approaches, the lack of expert labelled data limits their applicability.

A new method named Irregularity Age Map (IAM) has been recently proposed for detecting irregular textures in T2-FLAIR MRI without requiring manual labels for training [5]. The IAM indicates the degree in which the texture of the neighbourhood around each pixel/voxel differs from the texture of the tissue considered normal. Differently, most machine learning algorithms generate a map indicating the probability of each pixel/voxel of belonging to a particular class (e.g., normal white and grey matter, cerebrospinal fluid, lesions, etc.). We believe that the unsupervised nature and the concept of IAM itself are useful for: (1) task adaptation learning in assessing MRI abnormalities and (2) generation of progression/regression patterns that can be used to predict the evolution of these abnormalities. These two topics are the main contributions in this study.

2 Task Adaptation Transfer Learning in MRI

2.1 Current Approaches of Transfer Learning in MRI

Deep neural networks (DNN) architectures are considered the state-of-art machine learning models in MRI data classification and segmentation as they exhibit or surpass human-level performance on the task and domain they are trained. However, when the domain changes (e.g. imaging protocol or sequence type differ), or they are asked to perform tasks that are related to but not the same task they were trained for (e.g. lesion segmentation vs. lesion assessment), they suffer a significant loss in performance.

Transfer learning (TL) helps dealing with these novel scenarios, as enables a model trained on one task to be re-purposed on a second related task. In DNN the first few layers learn the general visual building of the image, such as edges and corners, while the deeper layers of network learn more complex task-dependent features [1]. Using TL, domain, task or distribution in training and target processes can differ and be adjusted to fit the final purpose better.

Domain adaptation TL, where data domains in training and testing processes differ, has been proven useful. In one study, TL improved Support Vector Machine’s performance in MRI segmentation using different distribution of training data [9]. Another study pre-trained DNN using natural images for segmentation of neonatal to adult brain images [10], and another study pre-trained a DNN for brain lesion segmentation using MRI data from other protocols [1].

However, task adaptation TL, where tasks in training and testing processes are different, has not been widely explored in medical image analysis. The newly proposed unsupervised method of Limited One Time Sampling IAM (LOTS-IAM) [5] has been reported to serve the purpose of white matter hyperintensities (WMH) segmentation performing at the level of DNN architectures trained for this specific purpose while executing a different task i.e., extracting irregular brain tissue texture in the form of irregularity age map (IAM).

2.2 Weakly-Training CNN in MRI Using Age Map

In this study, we explore the use of adapting the task of WMH segmentation on DNN, by using the IAM produced by LOTS-IAM as target instead of binary mask of WMH manually generated by an expert. We evaluate how the DNN recognition capabilities are preserved during the task adaptation TL process.

For our experiments we selected UNet [8] and UResNet [2] architectures used in various natural/medical image segmentation studies. In this study we made two modifications to allow UNet and UResNet to learn IAM: (1) no non-linear activation function (e.g., sigmoid, softmax or ReLU) is used in the last layer of both architectures and (2) mean squared error loss function is used instead of Dice similarity coefficient or binary cross entropy in both architectures. Detailed flowchart of the proposed method is available on the GitHub pageFootnote 1.

3 Brain Lesion’s Progression and Regression

3.1 Prediction of Brain Lesion’s Progression/Regression

Brain lesion’s evolution over time is very important in medical image analysis because it not only helps estimating the pathology’s level of severity but also selecting the ’best’ treatment for each patient [7]. However, predicting brain lesion’s evolution is challenging because it is influenced by various hidden parameters unique to each individual. Hence, brain lesions can appear and disappear at any point in time [7] and the reasons behind it are still unknown.

Previous studies that have modelled brain lesion progression/regression, use longitudinal (i.e., time-series) data to formulate lesion’s metamorphosis [3, 7] by estimating direction and speed of the lesion evolution over time. Hence, multiple scans are necessary to simulate the evolution of the lesion.

In this study, we propose the use of IAM for simulating brain lesion evolution (i.e., progression and regression) by using one MRI scan at one time point. This is possible thanks to the nature of IAM which retains original T2-FLAIR MRI’s complex textures while indicating WMH’s irregular textures. Compared to manually produced WMH binary mask by experts or automatically produced probability masks by machine learning algorithms, information contained/retained in IAM is much richer (see Fig. 1).

Fig. 1.
figure 1

Information density retained in each domain of the original T2-FLAIR, irregularity age map (IAM), probability mask and binary mask of WMH.

Full size image

3.2 Proposed Brain Lesions’ Regression (Shrinkage) Algorithm

We predict the regression pattern of brain lesions by lowering the threshold value of the IAM. This is possible as each IAM voxel contains different age value. It can be observed in Fig. 1 where age values of brain lesion decrease gradually from the border to the centre of each brain lesion. This is not possible using probability masks produced by most machine learning algorithms or binary masks of WMH produced manually by expert where most lesion voxels have flat value of 1.

The algorithm for predicting brain lesions’ regression is detailed in Algorithm 1. To predict the brain lesions’ regression pattern, we generate pseudo-healthy tissue of T2-FLAIR MRI first calculating the age map (Algorithm 2). In IAM, the nearest neighbour patches of the original patches are decided based on a distance value calculated using the distance function as per Eq. 1.

figure a
figure b

3.3 Proposed Brain Lesions’ Progression (Growth) Algorithm

Compared to the previous algorithm for predicting regression, the algorithm for predicting brain lesions’ progression is more complex as it involves nearest neighbour searching and patch replacement processes. The idea is simple; we need to find similar (i.e., nearest neighbour) IAM patches for each original IAM patch while the nearest IAM patch needs to have slightly higher age values than the original IAM patch. Once the nearest IAM patch is found, the original IAM patch is then replaced. Once all patches are replaced by their nearest IAM patches, a new T2-FLAIR MRI showing brain lesion progression can be produced by blending the new IAM with the pseudo-healthy T2-FLAIR MRI.

The algorithm for predicting brain lesion progression is detailed in Algorithm 3. It uses the pseudo-healthy T2-FLAIR MRI produced by Algorithm 2. The distance function used in Algorithms 2 and 3 is defined below. Let \(\mathbf {s}\) be the original IAM patch and \(\mathbf {t}\) be the candidate of nearest neighbour patch, the distance d between the two patches is:

$$\begin{aligned} d = \alpha \cdot \left| \text {max}(\mathbf {s} - \mathbf {t} ) \right| + (1 - \alpha ) \cdot \left| \text {mean}( \mathbf {s} - \mathbf {t}) \right| . \end{aligned}$$
(1)

where \(\alpha = 0.5\). Whereas, the patch’s size used in this study is \(4 \times 4\).

figure c

4 MRI Data and Experiment Setup

A set of 60 T2-Fluid Attenuation Inversion Recovery (T2-FLAIR) MRI data from 20 subjects was used. Each T2-FLAIR MRI volume has dimension of \(256 \times 256 \times 35\). Data used in this study were obtained from the ADNI [4] public databaseFootnote 2. Training/testing and pre-processing steps are the same as in [6]. The Dice similarity coefficient (DSC) was used to evaluate performance of UNet and UResNet segmenting WMH weakly-trained using IAM.

5 Results

5.1 Weakly-Training of UNet and UResNet Using IAM

Figure 2 shows the performance of the two algorithms evaluated in this study: UNet(1) and UResNet(2) segmenting WMH in our sample. Figure 2A shows distribution of results (DSC) by both algorithms trained without TL(Aa) (i.e., manual WMH labels) and with TL(Ab) where IAM from LOTS-IAM was thresholded\(^3\) at 0.18 (see [5]). Both DNN schemes could yield better results if task-adaptation TL using IAM is performed. However, the IAM’s dependence on pre-processing poses a risk for their use in TL, as it can also worsen DNN’s performance.

Fig. 2.
figure 2

Performance of UNet(A1) and UResNet(A2) in WMH segmentation without transfer learning(Aa) and using transfer learning(Ab, B and C).

Full size image

In another experiment where UNet and UResNet are directly trained using IAM as target\(^3\) (Fig. 2B), the peak mean performances are 0.2888 (0.0990) for IAM-UResNet, 0.4409 (0.1410) for IAM-UNet and 0.4704 (0.1587) for the LOTS-IAM. The UNet performs 15.21% better than the UResNet, which is quite opposite to when TL is not used (see [5] and Fig. 2A). Our guess is that residual blocks in UResNet perform poorly if it has to learn from real values of IAM. Whereas, UNet learned IAM with minimal performance drops (i.e., 2.95% from the LOTS-IAM and 6.21% from manual WMH labels as per [5]). Although the performance of IAM-UResNet and IAM-UNet apparently follow the LOTS-IAM’s performance at different thresholds in terms of DSC, a closer look at the learning process shows these relationships are not linear. Figure 2C shows the ratio between the mean DSC values of these DNN schemes and LOTS-IAM output. The peak DSC performance is not achieved using exactly the same threshold.

Fig. 3.
figure 3

Visualisation of brain lesion’s progression and regression prediction by manipulating age values of IAM.

Full size image

5.2 Results on Prediction of Brain Lesions’ Progression/Regression

Figure 3 shows an example of IAM and T2-FLAIR generated by using Algorithms 1, 2 and 3, from the original IAM and T2-FLAIR (centre with \(t = 1.00\))(also shown). The regression step of IAM and T2-FLAIR (\(2^{nd}\) column with \(t = 0.50\)) was generated by using Algorithm 1 whereas the progression steps of IAM and T2-FLAIR (\(4^{th}\) and \(5^{th}\) column with \(t = 1.25\) and \(t = 1.50\)) were generated by using Algorithm 3. On the other hand, the pseudo-healthy T2-FLAIR (\(1^{st}\) column with \(t = 0.00\)) was generated using Algorithm 2.

As Fig. 3 shows, prediction of brain lesions’ regression works really well for WMH, but prediction of brain lesions’ progression shows a small unmatched tessellation problem. This problem is common in computer graphics and should be easy to fix as there have been many studies that have proposed different solutions to this problem. Nevertheless, this experiment shows the suitability of IAM for predicting brain lesions’ progression/regression.

6 Discussion

In this study, we have presented the use of a publicly available unsupervised method (i.e. IAM produced by LOTS-IAMFootnote 3) as target for weakly-training two DNN schemes, i.e., UResNet and UNet, and predicting brain lesions’ progression/regression. Performance of UNet weakly-trained using IAM was close to the LOTS-IAM and UNet trained by using manual label of WMH and can sometimes be improved. In the future, we will widen our sample and investigate the conditions under which TL improves/worsens the quality of the DNN outputs.

Furthermore, IAM has shown to be very useful for the prediction of brain lesions progression/regression. There are still some problems in the prediction of progression such as unmatched tessellation, T2-FLAIR contrast changes and slightly higher computation time compared to predicting regression. However, it does not change the fact that the use of IAM facilitates the prediction and modelling of brain lesions’ progression/regression. Next steps in this research topic would be fixing unmatched tessellation, avoiding the effect caused by contrast differences and the use of pre-trained DNN (e.g., UNet) for predicting brain lesions’ progression/regression.