Simultaneous arteriole and venule segmentation with domain-specific loss function on a new public database

The segmentation and classification of retinal arterioles and venules play an important role in the diagnosis of various eye diseases and systemic diseases. The major challenges include complicated vessel structure, inhomogeneous illumination, and large background variation across subjects. In this study, we employ a fully convolutional network to simultaneously segment arterioles and venules directly from the retinal image, rather than using a vessel segmentation-arteriovenous classification strategy as reported in most literature. To simultaneously segment retinal arterioles and venules, we configured the fully convolutional network to allow true color image as input and multiple labels as output. A domain-specific loss function was designed to improve the overall performance. The proposed method was assessed extensively on public data sets and compared with the stateof-the-art methods in literature. The sensitivity and specificity of overall vessel segmentation on DRIVE is 0.944 and 0.955 with a misclassification rate of 10.3% and 9.6% for arteriole and venule, respectively. The proposed method outperformed the state-of-the-art methods and avoided possible error-propagation as in the segmentation-classification strategy. The proposed method was further validated on a new database consisting of retinal images of different qualities and diseases. The proposed method holds great potential for the diagnostics and screening of various eye diseases and systemic diseases. © 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement OCIS codes: (100.0100) Image processing; (150.0150) Machine vision. References and links 1. R. F. Gariano and T. W. Gardner, “Retinal angiogenesis in development and disease,” Nature 438(7070), 960– 966 (2005). 2. M. D. Abràmoff, M. K. Garvin, and M. Sonka, “Retinal imaging and image analysis,” IEEE Rev. Biomed. Eng. 3, 169–208 (2010). 3. S. B. Seidelmann, B. Claggett, P. E. Bravo, A. Gupta, H. Farhad, B. E. Klein, R. Klein, M. Di Carli, and S. D. Solomon, “Retinal vessel calibers in predicting long-term cardiovascular outcomes: the atherosclerosis risk in communities study,” Circulation 134(18), 1328–1338 (2016). 4. K. Rothaus, X. Jiang, and P. Rhiem, “Separation of the retinal vascular graph in arteries and veins based upon structural knowledge,” Image Vis. Comput. 27(7), 864–875 (2009). 5. Q. Hu, M. D. Abràmoff, and M. K. Garvin, “Automated construction of arterial and venous trees in retinal images,” J. Med. Imaging (Bellingham) 2(4), 044001 (2015). 6. B. Dashtbozorg, A. M. Mendonça, and A. Campilho, “An Automatic Graph-Based Approach for Artery/Vein Classification in Retinal Images,” IEEE Trans. Image Process. 23(3), 1073–1083 (2014). Vol. 9, No. 7 | 1 Jul 2018 | BIOMEDICAL OPTICS EXPRESS 3153 #327542 https://doi.org/10.1364/BOE.9.003153 Journal © 2018 Received 6 Apr 2018; revised 7 Jun 2018; accepted 7 Jun 2018; published 15 Jun 2018 7. R. Estrada, M. J. Allingham, P. S. Mettu, S. W. Cousins, C. Tomasi, and S. Farsiu, “Retinal Artery-Vein Classification via Topology Estimation,” IEEE Trans. Med. Imaging 34(12), 2518–2534 (2015). 8. H. Narasimha-Iyer, J. M. Beach, B. Khoobehi, and B. Roysam, “Automatic identification of retinal arteries and veins from dual-wavelength images using structural and functional features,” IEEE Trans. Biomed. Eng. 54(8), 1427–1435 (2007). 9. M. Niemeijer, B. van Ginneken, and M. D. Abràmoff, “Automatic classification of retinal vessels into arteries and veins,” in SPIE Medical Imaging (International Society for Optics and Photonics, 2009), 72601F. 10. M. Niemeijer, X. Xu, A. V. Dumitrescu, P. Gupta, B. van Ginneken, J. C. Folk, and M. D. Abramoff, “Automated measurement of the arteriolar-to-venular width ratio in digital color fundus photographs,” IEEE Trans. Med. Imaging 30(11), 1941–1950 (2011). 11. M. Saez, S. González-Vázquez, M. González-Penedo, M. A. Barceló, M. Pena-Seijo, G. Coll de Tuero, and A. Pose-Reino, “Development of an automated system to classify retinal vessels into arteries and veins,” Comput. Methods Programs Biomed. 108(1), 367–376 (2012). 12. S. Vázquez, B. Cancela, N. Barreira, M. G. Penedo, M. Rodríguez-Blanco, M. P. Seijo, G. C. de Tuero, M. A. Barceló, and M. Saez, “Improving retinal artery and vein classification by means of a minimal path approach,” Mach. Vis. Appl. 24(5), 919–930 (2013). 13. X. Xu, W. Ding, M. D. Abràmoff, and R. Cao, “An improved arteriovenous classification method for the early diagnostics of various diseases in retinal image,” Comput. Methods Programs Biomed. 141, 3–9 (2017). 14. C. Kondermann, D. Kondermann, and M. Yan, “Blood vessel classification into arteries and veins in retinal images,” in Medical Imaging: Image Processing (International Society for Optics and Photonics, 2007), 651247–651247. 15. K. He, X. Zhang, S. Ren, and J. Sun, “Spatial Pyramid pooling in deep convolutional networks for visual recognition,” IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015). 16. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Region-Based Convolutional Networks for Accurate Object Detection and Segmentation,” IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 142–158 (2016). 17. E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2017). 18. K.-K. Maninis, J. Pont-Tuset, P. Arbeláez, and L. Van Gool, “Deep retinal image understanding,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 (Springer International Publishing, 2016), 140–148. 19. Q. Li, B. Feng, L. Xie, P. Liang, H. Zhang, and T. Wang, “A Cross-Modality Learning Approach for Vessel Segmentation in Retinal Images,” IEEE Trans. Med. Imaging 35(1), 109–118 (2016). 20. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015), 234–241. 21. M. Niemeijer, B. V. Ginneken, and M. Loog, “Comparative study of retinal vessel segmentation methods on a new publicly available database,” Proc. SPIE 5370, 648–656 (2004). 22. M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Thomson Learning, 2008). 23. A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, “Discriminative unsupervised feature learning with convolutional neural networks,” in Advances in Neural Information Processing Systems 27 (NIPS 2014), pp. 766–774 (2014). 24. J. Staal, M. D. Abràmoff, M. Niemeijer, M. A. Viergever, and B. van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” IEEE Trans. Med. Imaging 23(4), 501–509 (2004). 25. E. Ricci and R. Perfetti, “Retinal blood vessel segmentation using line operators and support vector classification,” IEEE Trans. Med. Imaging 26(10), 1357–1365 (2007). 26. D. Marin, A. Aquino, M. E. Gegundez-Arias, and J. M. Bravo, “A new supervised method for blood vessel segmentation in retinal images by using gray-level and moment invariants-based features,” IEEE Trans. Med. Imaging 30(1), 146–158 (2011). 27. M. S. Miri and A. Mahloojifar, “Retinal image analysis using curvelet transform and multistructure elements morphology by reconstruction,” IEEE Trans. Biomed. Eng. 58(5), 1183–1192 (2011). 28. S. Roychowdhury, D. D. Koozekanani, and K. K. Parhi, “Iterative Vessel Segmentation of Fundus Images,” IEEE Trans. Biomed. Eng. 62(7), 1738–1749 (2015). 29. S. Wang, Y. Yin, G. Cao, B. Wei, Y. Zheng, and G. Yang, “Hierarchical retinal blood vessel segmentation based on feature and ensemble learning,” Neurocomputing 149, 708–717 (2015). 30. X. Xu, W. Ding, X. Wang, R. Cao, M. Zhang, P. Lv, and F. Xu, “Smartphone-based accurate analysis of retinal vasculature towards point-of-care diagnostics,” Sci. Rep. 6(1), 34603 (2016). 31. J. Zhang, B. Dashtbozorg, E. Bekkers, J. P. W. Pluim, R. Duits, and B. M. Ter Haar Romeny, “Robust retinal vessel segmentation via locally adaptive derivative frames in orientation scores,” IEEE Trans. Med. Imaging 35(12), 2631–2644 (2016). 32. G. Mirsharif, F. Tajeripour, F. Sobhanmanesh, H. Pourreza, and T. Banaee, “Developing an automatic method for separation of arteries from veins in retinal images,” in 1st International Conference on Computer and Knowledge Engineering (2011). Vol. 9, No. 7 | 1 Jul 2018 | BIOMEDICAL OPTICS EXPRESS 3154


Introduction
Retinal arterioles and venules, defined as small blood vessels directly before and after the capillaries, are the only human microcirculation that can be non-invasively observed in vivo with an optical method.Various eye diseases and systemic diseases manifest themselves on the fundus image as retinal arteriole and venule changes [1,2].Specifically, diseases may affect the arterioles and venules differently.For example, the narrowing of retinal arterioles and widening of retinal venules independently predict the risk of mortality and ischemic stroke [3].Therefore, identifying and quantifying the changes of retinal arterioles and venules may serve as potential biomarkers for the diagnosis and long-term monitoring of these diseases.
Manual segmentation is extremely labor-intensive and clinically not feasible.Thus, it is of great importance to automatically segment and analyze the arterioles and venules individually on the retinal images.The main challenges for automatic segmentation are three-fold.First, the vascular morphology varies greatly in an image because of an inherited inhomogeneous illumination partially caused by the image acquisition procedure (i.e., projecting a spherical surface onto a plane).Second, by projecting three-dimensional vascular trees to a twodimensional image, the vessel trees are overlapped with incomplete structures.At last, the retina shows diverse background pigmentation across images because of different biological characteristics (i.e., races and ages).
In the past decades, various methods have been developed to classify the retinal vessels into arteriole and venule, which can be categorized as tree-based methods and pixel classification methods.The tree-based methods target at dividing the retinal vessels into individual biological vessel trees often by a graph-theoretic method [4][5][6][7].Rothaus et al. reported a rule-based method that propagates vessel labels through a pre-segmented vascular graph, in which the labels need to be manually initialized [4].Hu et al. proposed a graphbased, meta-heuristic algorithm to separate vessel trees [5].Dashtbozorg et al. proposed to classify the vessel trees by deciding the type of each crossing point and assigning arteriole or venule label to each vessel segment [6].A more recent work by Estrada et al. also used a graph-theoretic framework incorporated with a likelihood model [7].Tree-based methods usually require manual seeds for initialization.Furthermore, a single mislabel along the propagation may lead to mislabel of the entire vessel tree.On the other hand, the pixel classification methods extract hand-crafted local features around pixels with known true labels and build classifiers or likelihood models for test images [8][9][10][11][12][13].The local features are usually designed based on observable colorimetric and geometric differences between arterioles and venules.For example, arterioles contain oxygenated hemoglobin that shows higher reflectance than deoxygenated hemoglobin in specific wavelengths and are thus brighter than venules.The central light reflex (CLR) phenomenon is another widely used feature, which is more frequently seen in retinal arterioles than venules.Narasimha-Iyer et al. reported an automatic method to classify arterioles and venules in dual-wavelength retinal images with structural and functional features, including the CLR and oxygenation rate difference between arterioles and venules [8].Saez et al. reported a clustering method and the features are RGB/HSL color and gray levels [11].Niemeijer et al. reported a supervised method and the designed features are HSV color features and Gaussian derivatives [9,10].Vazquez et al. proposed a retinex image enhancement method to adjust the uneven illumination inside a retinal image [12].Our group also proposed an intra-image and interimage normalization method to reduce the sample differences in feature space [13].Pixel classification methods often struggle at finding the most representative features.As discussed above, the retinal images have an inherited inhomogeneous illumination and have large variation across images.Though some efforts have been made to alleviate the problem [12,13], the improvement is limited because pixel classification methods show an inability to incorporate context information.At last, there are some common limitations for tree-based methods and pixel classification methods.First, a high-quality vessel segmentation is usually required as input, which is used for graph construction in tree-based methods and for reducing the three-class problem (e.g., arteriole, venule, and background) to a two-class problem (e.g., arteriole and venule) in pixel classification methods.However, vessel segmentation, by itself, is another challenging task in image analysis.In the reported arteriole and venule classification methods, the vessel segmentation is either from manual or semi-automatic methods [7,14], which makes the algorithm pipeline time-consuming, or from other automatic algorithms [6,10,11], which makes an error propagation inevitable.At last, most of the methods reported in literatures are only trained and tested on limited data sets.The performance on data sets with large variations has not been reported yet.
Convolutional neural networks (CNN) have been widely applied in image labeling and segmentation in recent years.Though initially developed for image labeling, various methods have been proposed for semantic segmentation in an image [15][16][17].In the field of retinal image analysis, although several CNN methods have been reported for vessel segmentation [18,19], to our best knowledge, this is the first application for simultaneous vessel segmentation and arteriovenous classification.In this study, we apply and extend a fully convolutional network (FCN) first proposed by Ronneberger et al. [20].Specifically, the architecture is modified to allow true color image as input and multi-label segmentation as output.Second, a domain-specific loss function is designed, which can be used as a general strategy for imbalanced multi-class segmentation problem.The proposed method was compared extensively with literatures on available public data sets.To assess the robustness of our method between images with various qualities, we further assessed our method on a new database consisting of fundus images with diverse qualities and disease signs.

Experimental materials
The DRIVE (Digital Retinal Image for Vessel Extraction) database consists of 40 color fundus photographs from a diabetic retinopathy screening program [21].Seven out of the 40 images contain pathologies such as exudates, hemorrhages and pigment epithelium changes.The image size is 768 × 584 pixels (Canon CR5 non-mydriatic 3-CCD camera).The 20 images from the test set was manually labeled by trained graders and thus is included in this study.Full annotation was given for this data set, meaning each pixel was given a label of arteriole (red), venule (blue), crossing (green), uncertain vessel (white), or background (black).This A/V annotated data set is also known as the RITE data set [5].The INSPIRE data set contains 40 optic-disc centered color fundus images with a resolution of 2392 × 2048 pixels [10].Only selected vessel centerline pixels are labeled in this data set [6].Arteriole centerlines, venule centerlines, and uncertain centerlines are labeled in red, blue, and white, respectively.
The method is further assessed on a new database (REVEAL, REtinal arteriole &VEnule AnLysis), which includes three sets of images of different image quality and disease signs.Specifically, REVEAL1 includes ten images from ten participants with diabetes mellitus (Topcon TRC-NW6S retinal camera, FOV of 45°, image size 3872 × 2592 pixels).This data set contains early signs of diabetic retinopathy, such as microaneurysm and exudates.

Methods
We employ th trees in the r channels (i.e., pilot study, it to be misclas ability of the n

Training the FCN architecture for retinal arteriole and venule classification
The original FCN architecture can be recognized as two parts, the descending part and the ascending part, with a total number of 23 convolutional layers.However, in the original network, the feature map after each convolution loses its boundary pixels, resulting in different input and output image size.We apply same padding in convolution to avoid inconsistent image sizes.The activation function is rectified linear unit (ReLU).The details of the network can be found in Fig. 2. To maintain the same input and output image size, the size of the input image should be a multiple of 16.In our applications, the input images are resized to 576 × 576 pixels.Data augmentation of 60 times is implemented by applying a combination of elastic deformation [23] and random flipping of the original image considering the limited number of training images.Hence, the final number of training image is 50 × 60 = 3,000 color fundus images.
In a pilot study, we use the cross entropy of a pixel-wise soft-max loss between the predicted label and ground truth label over the final feature map as the energy function.This simple energy function result in 1) generally under-segmented fine vessels and 2) arterioles prone to be misclassified as venules.In this study, we apply a weighted cross entropy that penalizes each pixel with different labels with different weights, which is precomputed from each ground truth segmentation, as given in Eq. ( 1).
where l(x) is the true label of each pixel, p k (x) is the softmax output of each pixel, and w c (x) is the weight of each pixel.In the design of w c , more importance is given to arteriole pixels and venule pixels.In our experiment, w c is 10 for arteriole pixels, 5 for venule pixels, and 1 for background pixels.

Testing
Before being input to the pipeline, the testing image is resized to 576 × 576 pixels.We noticed that slightly different information is provided when the images are oriented in different directions.To better capture the fine vessels, the original images are flipped in the x-

Assessm
As shown in considered at directions and rule.For each xel is signed a blood vessel, bilities and th zed to the origin cross validation els for selected ge, crossing po ackground.
. Histogram match images after histo VE data set.After asse (MISC a ) and MISC a is defi MISC v is defin where TP a , TP of venules (1 (3 in Fig. 5), r defined as:

Assessment on public data sets
Visualization of original image, ground truth, and automatic segmentation results on the DRIVE data set are given in Fig. 4 as the first column, second column, and third column, respectively.The Se and Sp of vessel segmentation are assessed and compared with the stateof-the-art methods in Table 1.The proposed method achieves significantly better Se of 0.944 and comparable Sp of 0.955 compared with the state-of-the-art methods.The misclassification rate is 10.3% for arteriole and 9.6% for venule.Visualization of the original image, ground truth, and automatic segmentation on the INSPIRE data set is given in Fig. 6.It should be noted that the original ground truth label only contains centerline pixels for selected vessel segments, which is dilated in Fig. 6 only for the sake of better visualization.It should also be noted that the proposed method was not trained on the INSPIRE data set, meaning INSPIRE is an outside test set.The overall Se is 0.889 and the misclassification rate is 26.1% for arteriole and 15.4% for venule.
After assessing the vessel segmentation result, the accuracy of arteriole and venule classification is compared with methods in literatures (Table 2).Our method achieved comparable results with the literatures.However, it should be noted that the arteriovenous classification methods reported in literature are usually developed and assessed on selected/known vessel segments or centerlines.Our method, on the other hand, is assessed on automatically detected vessel pixels with a ground truth label.The number o d method on R l segmentation etter Se, compa a good perform ms and hemorrh method on R s a low perfor ssification rate, s hard to reco arable Sp, mance on hages was REVEAL3 rmance in , which is gnize the type of vessel even for the human grader.The ROC of proposed method on DRIVE and REVEAL is given in Fig. 10.

Discussion
Applying fully convolutional networks in the segmentation of medical image is challenging because the targets are usually small or have diverse appearances.In this study, retinal blood vessels are fine structures with complicated tree patterns.The simultaneous segmentation of arterioles and venules is challenging as vessel boundary information comes from a finer scale but distinguishing arteriole from venule requires a coarser scale.In our approach, we take advantage of FCN architecture, in which high resolution features from the contracting path are combined with the subsampled features in the expanding path to increase the localization accuracy for fine structures.Another advantage of our method is that it directly classifies arterioles and venules from the original image without need for pre-segmentation of the blood vessels and thus avoids possible error propagation.On the other hand, algorithms reported in the literature that show better result usually require a high-quality vessel segmentation as input.
We tested our method on public data set and more images of different conditions, such as severe DR and fundus images taken using a smartphone at POC settings (Fig. 9).This is of great importance as real clinical images are quite often imperfect with various disease signs and low image quality.To our knowledge, our method is the first method assessed on severe disease cases and on fundus images taken using a smartphone.We made this data set publicly available for the researchers to develop and evaluate their methods (http://bebc.xjtu.edu.cn/retina-reveal).
To provide an overall view of the performance, the true background (GT bkg ), true vein (GT v ), and true artery (GT a ) versus predicted background (P bkg ), predicted vein (P v ), and predicted artery (P a ) are shown in 3 × 3 matrices for the DRIVE and REVEAL data sets in Table 4.For the sake of better comparison, the starred number was normalized to 1 for each data set.For further discussion, the precision and recall of the vessel segmentation was also calculated.Precision is defined as TP/(TP + FP) and recall is defined as TP/(TP + FN).The precisions for DRIVE and REVEAL data sets are 0.65, 0.55, 0.58, 0.53 while the recalls are 0.94, 0.94, 0.90, and 0.79, respectively.We can see the precisions are relatively low comparing with the recalls.The reason is that, as discussed in Fig. 7, graders/automatic methods sometimes have different bias on general vessel width.Table 4 shows that the automatically detected vessels are slightly wider than the manually segmented vessels.Figure 11 shows the color-coded result images, with each color defined as in Table 4.The disagreement on general vessel width can be solved by applying an accurate vessel boundary delineation method [33].The network can be trained in less than 70 minutes in cross validation and it takes about 17 seconds to run a test image.The configuration of the computer is Intel CoreTM i7 6700k CPU and NVIDIA GeForce GTX 1080 GPU (8GB).The project was implemented in Python using Keras.The FCN method has been widely studied and applied in medical image labeling and segmentation in the past few years [15,16,34,35].Though it generally yields better result, its limitation is also obvious.First, it needs large training data.This is particularly difficult for segmentation tasks as the manual creation of training image is extremely labor-intense.Second, the current architecture shows limited ability to incorporate prior or domain knowledge.In the arteriole and venule segmentation, for example, there are some important prior knowledges to improve the performance, including 1) the label within one vessel segment should be consistent; 2) the daughter branches should have the same label as the mother branch; and 3) the arteriole/venule would not cross another arteriole/venule.How these knowledges can be incorporated in the pipeline should be studied in the future.
Simultaneous arteriole and venous segmentation is the basis for other retinal image analysis towards computer-aided diagnosis, such as population-based screening of diabetic retinopathy and the measurement of arteriolar-to-venular ratio (AVR) [36,37].As discussed by Estrada et al., though the significance of AVR has long been appreciated in the research community, the measurement of AVR has been limited to the six largest first level arterioles and venules within a concentric grid (0.5-2.0 disc diameter) centered on the optic disc [7].Yet the subtler and earlier changes in smaller arterioles and venules have not been studied yet.Another future work will focus on a computer-aided labeling of DR signs and on the study of the association between smaller vessels and systemic diseases.

Conclusion
In this paper, we proposed to simultaneously segment retinal arterioles and venules using an improved FCN architecture.This method allowed end-to-end multi-label segmentation of a color fundus image.We assessed our method extensively on publicly data sets and compared with the state-of-the-art methods.The result shows the proposed method outperforms the state-of-the-art methods in vessel segmentation and classification.This method is a potential tool for the computer-aided diagnosis of various eye diseases and systemic diseases.
Fig. 1 REVE data s should pixel image from R first image from the DRIVE data set was selected for convenience.The R, G, B channels of all other images, including other images in the DRIVE data set, are matched respectively to the R, G, B channels of the target image, as shown in Fig.3.
Fig. 3 is the DRIV

Fig
Fig designed a tw a pixel-wise m teriovenous cl els.the general v ity (Sp), overal re defined as fo Fig. 6 fundu groun dilated autom

Fig. 10 .
Fig. 10.The ROC of different methods and manual classification on REVEAL data sets.(a) ROC for arterioles segmentations.The AUC is 0.980 for DRIVE and 0.979, 0.969, and 0.926 for the three REVEAL data sets.(b) ROC for venule segmentations.The AUC is 0.981 for DRIVE and 0.987, 0.975, 0.967 for the three REVEAL data sets.

Fig. 11 .
Fig. 11.Sample color-coded result images for the DRIVE and REVEAL1 data sets.The corresponding colors are defined in Table4.
Fig. 11.Sample color-coded result images for the DRIVE and REVEAL1 data sets.The corresponding colors are defined in Table4.