The Multiscale Bowler-Hat Transform for Blood Vessel Enhancement in Retinal Images

Enhancement, followed by segmentation, quantification and modelling, of blood vessels in retinal images plays an essential role in computer-aid retinopathy diagnosis. In this paper, we introduce a new vessel enhancement method which is the bowler-hat transform based on mathematical morphology. The proposed method combines different structuring elements to detect innate features of vessel-like structures. We evaluate the proposed method qualitatively and quantitatively, and compare it with the existing, state-of-the-art methods using both synthetic and real datasets. Our results show that the proposed method achieves high-quality vessel-like structure enhancement in both synthetic examples and in clinically relevant retinal images, and is shown to be able to detect fine vessels while remaining robust at junctions.


I. Introduction
Many biomedical images contain vessel-like structures such as blood vessels or cytoskeletal networks. Automated extraction of these structures and their connected network is often a key step in quantitative image analysis and computer-aided diagnostic pipelines. For example, automated retinal vessel extraction is used for diagnosis, screening, and evaluation in a wide range of retinal diseases, including diabetes and arteriosclerosis [1].
However, biomedical imaging modalities may suffer from poor quality due to many reasons, including, but not limited to, noisy image capture, sample/patient variability, and low contrast scenarios. As such standard image segmentation methods are not able to robustly detect vessel-like structures, and therefore some form of vessel-like structure enhancement is required [2].
However, many of these methods still have considerable issues when faced with variations in contrast, high levels of noise, variation in image features (e.g. lines vs junctions; retention of network connectivity), and complexity of method parameter space.

A. Contribution and Organisation
In this paper, we introduce a new enhancement method for vessel-like structures based on mathematical morphology, which exploits a key shape property of the vessellike structures: elongation. The proposed method, called the bowler-hat transform, has been qualitatively and quantitatively validated and compared with the stateof-the-art methods using a range of synthetic data and available retinal image datasets. The obtained results show that the proposed method achieves high-quality vessellike structures enhancement in both synthetic examples and clinically relevant retinal images. The method is suitable for a range of biomedical image types without needing prior training or tuning. Finally, we have made the implementation of our approach available online, along with source code and all test functions.
The rest of this paper is organised as follows. In Section II, we introduce existing vessel-like structures enhancement methods and highlight their known limitations. Section III introduces and explains the proposed bowler-hat transform, Section IV presents validation experiments and results on synthetic and real data. Finally, in Section V, we discuss the results and future work.

A. Hessian-based Methods
In [4], Frangi and colleagues introduce a novel Hessianbased multi-scale concept for 2D/3D curvilinear/tubular structure enhancement in images. They construct the Hessian matrix using second-order Gaussian derivatives. The eigenvectors and eigenvalues of the Hessian matrix then define the principal directions of local image features. These can then be combined to form different measures of vesselness or blobness in biomedical images.
1) Vesselness: The vesselness measure is proportional to the ratio of the eigenvalues [4]. If the magnitude of both eigenvalues is small, i.e. the local image structure is likely to be background, then the vesselness measure is small. If one eigenvalue is small and the other large then the local structure is likely to be vessel-like and the vesselness measure is large. Finally, if both eigenvalues are high then the structure is likely to be blob-like and the vesselness measure is again small. This approach, however, leads to a failure at the intersection of vessels as both eigenvalues have similarly large values leading to a vesselness measure close to zero. Thus, vessel-like structures can be lost at junctions and therefore vessels network connectivity may be lost [15]. Extensions of the vesselness approach can be found in [16], [17].
2) Neuriteness: As an alternative to vesselness, Meijering and colleagues [18] introduce the neuriteness measure to enhance low contrast and highly inhomogeneous neurites in bioimages. Using a modified Hessian, with a tuning parameter, and a different combination of eigenvalues, neuriteness infers a putative neurite in every pixel of the image that has a non-zero value. Background intensity discontinuities that are immune to first order derivatives are suppressed by the use of second order derivatives.
A major failing for the neuriteness measure is that background noise signals are enhanced as if they are curvilinear structures. In the original paper [18] this is solved with a tracing stage; however, as an enhancer only this can cause serious problems for further analysis. The neuriteness measure also leads to a failure at the intersection of vessels as both eigenvalues have similarly large values leading to a neuriteness measure close to zero. A further example of their work is found in [19].
3) Regularized Volume Ratio: Recently, Jerman and colleagues [20] propose a new Hessian-based vessel enhancement method which is able to resolve the drawbacks found in most of the previous Hessian-based methods, 1) eigenvalues are non-uniform throughout an elongated or rounded structure that has uniform intensity, 2) eigenvalues vary with image intensity, and 3) enhancement is not uniform across scales. To address such drawbacks, a modified volume ratio is introduced to ensure method robustness to low magnitude intensity changes in the image.
A major issue of this method is the false vessel affect, as shown in Figure 7k sensitivity.

B. Phase Congruency Tensor-based Methods
A major issue with many image enhancement methods is that they depend, to some extent, on image intensity and, therefore, fine, and usually lower intensity, vessels may be missed. To address this issue a contrast-independent method based on Phase Congruency (PC) was introduced in [21]. This approach builds upon the idea of phase congruency, which looks for image features in the Fourier domain.
The development of a contrast-independent, image enhancement measurement built upon PC has been shown in [6]. The Phase Congruency Tensor (PCT) is built upon PC principles but the tensor is decomposed. The calculated eigenvalues are then used in the same way as Hessian eigenvalues (see sections II-A1 and II-A2), to define PCT vesselness and PCT neuriteness measures. An extension of this method into 3D has recently been shown in [22].
A major drawback of the PC-based concept is the complexity of its parameter space. Moreover, as with Hessian-based measures, the PCT-based measures also lead to a failure at the intersection of vessels as both eigenvalues have similar, large values leading to PCT-based vesselness and neuriteness measures close to zero.

C. Adaptive Histogram Equalisation-based Methods
Contrast Limited Adaptive Histogram Equalisation (CLAHE) [9], originally developed for spiculations en-hancement in mammograms, is widely used for vessel enhancement. In this simple, histogram-based method an image is first divided into small regions, each of which then undergoes a histogram equalisation. To avoid overenhancement of noise, a contrast limiting procedure is applied between regions. Further development of this method is demonstrated in [23], [24]. A major drawback of this method is the noise sensitivity.

D. Wavelet Transform-based Enhancement Methods
Bankhead and colleagues [13] propose the use of wavelets for vessel enhancement and segmentation. They calculate an isotropic, undecimated wavelet transform using the cubic B-spline mother wavelet and employing its coefficients to the threshold steps for enhancement and then segment vessels. Further improvement of this approach is demonstrated in [25]. A major drawback of this method is the complexity of its parameter space.

E. Line Detector-based Enhancement Methods
Vessel-like feature enhancement has also been done using multi-scale line detectors [14]. The basic line response, identified by subtraction of average value and the maximum value of each pixel, is computed at 12 different line directions. A major drawback of this method is at crossover points, where the method produces a 'false vessels' by merging close vessels. Further improvement of this method is demonstrated in [26] where a linear combination of all the line responses at varying scales is proposed to produce the final enhancement and segmentation.

F. Mathematical Morphology-based Enhancement Methods
Zana and Klein [27] proposed a novel method which combines morphological transforms and cross-curvature evaluation for vessel-like structures enhancement and segmentation. This method relies on the assumption that vessels are linear, connected and have smooth variations of curvature along the peak of the feature. First, a sum of top hats is calculated using linear structuring elements at different angles, then a curvature measure is calculated using a Laplacian of Gaussian, and finally, both of them are combined to reduce noise and enhance vessel-like structures in an image. Further improvement of this method is demonstrated in [24].
A major issue with this method is that is quite slow and sensitive to noise.

G. Limitations and Challenges
Many existing vessel-like structures enhancement methods still have substantial issues when faced with variations in contrast (low-accuracy enhancement), high level of noise (introduction of 'false vessels' effect), dealing with junctions/bends (suppression of disk-like structures; vessels network connectivity is lost), large image size (high computing time), and complexity of parameter space.  Figure 1: Flow chart of the bowler-hat.

III. Method
In this section, we introduce our novel, mathematical morphology-based method for vessel-like structures enhancement in images the bowler-hat transform. We highlight the key concepts that allow this method to address the major drawbacks of existing, state-of-the-art methods.

A. Mathematical Morphology
Morphological operations are a set of non-linear filtering methods formed through a combination of two basic operators: dilation and erosion.
Dilation, (⊕), for a given pixel in any grayscale image, I(p), can be described as the maximum of the points in the weighted neighbourhood described by the structuring element b(p), and mathematically: where sup is the supremum and x ∈ E denotes all points in Euclidean space within the image. Likewise, we describe the erosion, ( ), as the minimum of the points in the neighbourhood described by the structuring element and, mathematically:  where inf is the infimum. Dilation is able to expand bright areas and reduce dark areas, whilst erosion expands dark areas reducing bright areas.
From these two operators we can define two commonly used morphological filters: closing : where an opening (•) will preserve dark features and patterns, suppressing bright features, and a closing (•) will preserve bright features whilst suppressing dark patterns. Figure 1 presents a flow diagram of the proposed method which combines the outputs of morphological operations upon an image carried out with two different banks of structural elements: one bank of disk elements with varying radii, and one bank of line elements with varying radii and rotation.

B. Proposed Method
For a given input image, I, we carry out a series of morphological openings with a bank of disk-shaped where d max is the expected maximum vessel size. This produces a stack of images, for all d, such that In each I disk image, vessel segments wider than d remain and those segments smaller than d are removed.
We also produce a similar stack of images using a bank of line-shaped structuring elements, b d,θ ; each line-shaped is of length d ∈ [1, d max ], has a width of 1 pixel, and of orientation θ ∈ [0, 180).
As a result, vessel segments that are longer than d and along the direction defined by θ will remain, and those shorter than d and along the direction defined by θ will be removed. For each line length d we produce a stack of images for all orientations defined by θ ∈ [0, 180). Then, for each d, we calculate a single image, I line as a pixel-wise maximum of the stack such that These two stacks, {I disk } and {I line }, are then combined by taking the stack-wise difference, the difference between the maximum opening with a line of length d across all angles and an image formed of opening with a disk of size d, to form the enhanced image. The final enhanced image is then formed from maximum difference at each pixel across all stacks, Pixels in the background, i.e. dark regions, will have a low value due to the use of openings; pixels in the foreground of blob-like structures will have a low value as the differences will be minimal, i.e. similar values for diskbased and line-based openings; and pixels in the foreground of vessel-like structures will have a high value, i.e. large differences between longer line-based openings and diskbased openings.
The combination of line and disk elements gives the proposed method a key advantage over the existing methods, given an appropriate d max , i.e. larger than any vessels in the image, a junction should appear bright like those vessels joining that junction, something that many other vessel enhancement methods fail to do. This is due to the ability to fit longer line-based structural elements within the junction area. As a result, the vessels network stays connected when enhanced and segmented, especially at junctions.
In Section IV, we demonstrate, qualitatively and quantitatively, the key advantages of the bowler-hat transform over the existing, state-of-the-art vessel-like structures enhancement methods.

C. Implementation
All codes were implemented and written in MATLAB 2016b [28] on Windows 8.1 Pro 64-bit PC running an Intel Core i7-4790 CPU (3.60 GHz) with 16GB RAM. The source code is available in a GitHub repository (TBC).

IV. Results
In this section, the proposed method is qualitatively and quantitatively validated and compared with the existing state-of-the-art methods using synthetic and real image datasets, including retinal image datasets with humanannotated ground truths and other biomedical images.
As with any image processing method, an understanding of how the parameters involved affect the result is essential. In general, we have found the bowler-hat transform to be robust, usually requiring 10-12 θ orientations for line structuring element and the size of the disk/line structuring element d to be greater than the thickest vessel structure in an image. Figure 2, we plot the responses of the proposed and state-of-the-art enhancement methods on a synthetic, simple vessel-like structure. As can be seen, whilst the Hessian-based methods have an enhanced signal at the center of the vessel, i.e. a peak value of one at the vessels centre-line, their value quickly drops off and decreases the perceived thickness of the vessel. Contrariwise, PCT-based methods do not necessarily peak at the vessel centre, but their response does not drop off quickly, and they maintain a higher response through to the edges of the vessel, i.e. the perceived thickness is not decreased. The proposed method has both these benefits: a maximal peak value at the vessel centre-line and an enhanced response to the edges of the vessel. As a result the reliable vessel thicknesses can be captured.

1) Profile Analysis: In
2) Response to Vessels, Intersections, and Blobs: Figure 3 presents a qualitative comparison between the proposed method and the state-of-the-art methods when applied to synthetic images with vessel-like, intersectionlike, and blob-like structures. Key issues that occur across the state-of-the-art methods include defects at junctions (purple arrows), noise enhancement, tip artefacts (orange arrows) and loss of signal (yellow arrows). These issues are all absent with our proposed method.
3) Response to Noise: Figure 4 shows effect of three different noise types on the proposed and state-of-theart methods. Given that the proposed method has no built-in noise suppression, it is unsurprising that the effect of noise on the enhanced image is in-line with the raw image. We note that the method is weakest in response to speckle noise (multiplicative Gaussian) and also weak in response to salt and pepper noise. This follows from the noise-sensitivity in morphological operations and should be taken into consideration when choosing an enhancement method. Figure 5 presents the response of the proposed method to an uneven illumination scenario. Key features such as junctions are preserved and appear unaffected by even severe illumination problems. This ability to preserve junctions under uneven illumination is important for many real applications of vessel enhancement and the proposed method is able to do this, unlike the current state-of-the-art methods.

A. Real Data -Retinal Image Datasets
In this section, we show the quality of the proposed method validated on three publicly available retinal image datasets: the DRIVE, STARE, and HRF databases. These datasets have been chosen because of their availability and their ground truth data. We have used these ground truth segmentations to quantitatively compare the proposed method with the other vessel enhancement methods.
The Digital Retinal Images for Vessel Extraction (DRIVE) [29] dataset is a published database of retinal images for research and educational purposes. The database consists of twenty colour images that are JPEG compressed,          Table I.   Figure 6.
The STructured Analysis of the REtina (STARE) dataset is another publicly available database [30] containing twenty colour images with human-determined vasculature ground truth. We have compared all these images against the AH labelling.
The High-Resolution Fundus (HRF) image dataset [12] consists of 45 retinal images. This dataset has three type of subjects include healthy, diabetic retinopathy, and glaucoma.

1) Quantitative Validation: Whilst a visual inspection
can give some information regarding the effectiveness of the vessel enhancement method, a form of quantitative validation is required. In order to compare the proposed method with the other state-of-the-art methods, we have chosen to calculate the Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC). The sensitivity and specificity for the ROC are defined as: where T P is the true positive count, F P the false positive count, T N the true negative and F N the false negative counts of the segmented pixels after thresholding the enhanced images with different threshold levels. These metrics take in an image, either the original or an enhanced image, and the ground truth image, for each data set. A higher AUC value indicates a better enhancement, with a value of 1 indicating that the enhanced image is identical to the ground truth.
2) Healthy Subjects: Figure 7 shows the results of the proposed and state-of-the-art methods applied to a sample image from the HRF dataset (results for DRIVE and STARE datasets can be found in Supplementary Materials). We can see that the proposed method is able to enhance finer structures as detected by the human observer but not emphasised by many of the other methods (see arrows). We can also see that, whilst the connectivity seems to be maintained (unlike in Figure 7c), 'false vessels' are not introduced (c.f. Figure 7f). Finally, Figure 6 and Table I present ROC curves and mean AUC values for the results of the proposed and state-of-the-art methods applied to a all images across the DRIVE, STARE and HRF datasets.
3) Unhealthy Subjects: Figure 8 presents a visual comparison between the proposed method and the other state-ofthe-art methods for sample images of subjects with diabetic retinopathy and with glaucoma from the DRIVE, STARE and HRF datasets.

B. Other Biomedical Data
Although we have demonstrated above the vessel-like structures enhancement ability of the proposed method on retinal image datasets, the proposed method can also be used with a wide range of biomedical images, see Figure 9.

V. Conclusion and Discussion
A wide range of image processing methods have been proposed for vessel-like structure enhancement in biomedical images, see section Section I. Most of them, however, suffer from issues with low-contrast signals, enhancement of noise or when dealing with junctions.
In this paper, we reviewed the state-of-the-art approaches and highlighted the challenges that image enhancement methods have to face and their weaknesses; then we introduced a new enhancement method for vessel-like structures based on mathematical morphology, which exploits the elongated shape of vessel-like structures. The proposed method, the bowler-hat transform, was qualitatively and quantitatively validated and compared with the stateof-the-art methods using a range of synthetic and real image datasets, including retinal image collections (DRIVE, STARE and HRF). We showed the effectiveness of the bowler-hat transform, and its superior performance on retinal imaging data, see Figure 6 and Table I. Furthermore, experimental results on the unhealthy retinal images have shown that the vessels enhanced by our bowler hat transform are continuous and complete in the problematic regions as illustrated in Figure 8.
As with any image processing technique, our proposed method has limitations. Firstly, morphological operations are renowned for their large computational requirements.
Another limitation of the proposed method is displayed in Figure 3 row 4, which shows a vessel-like structure with an attached 'blob' (green arrow), a perfect vessel enhancement method would enhance all of the linear structure and none of the blob. Whilst none of the comparison methods act in this ideal manner many of them show a clear difference between the blob response and vessel response, our proposed method shows some difference, but this difference impacts the signal of the vessel.
Moreover, as we note in Figure 7, the proposed method is quite sensitive to noise, as is the PCT neuriteness method in Figure 7h. In the future, we will investigate introducing a line-shaped morphological structuring element with varying thickness to address this issue. Whilst one would expect the lack of noise suppression to be a major issue with regard to quantified measurements of vessel enhancement, we find that the proposed method gives the best enhancement of all methods on the DRIVE, STARE and HRF datasets (see Table I and Figure 6).
Future extensions of this work will include the development of a three-dimensional equivalent, exploration of blob-like structures enhancing variants of this method and analysis of parameter sensitivity for different modalities. Nevertheless, our implementation demonstrates an improved and easy to use vessel enhancement alternative that can be used in a wide range of biomedical imaging scenarios [31].