Deep learning based detection of cone photoreceptors with multimodal adaptive optics scanning light ophthalmoscope images of achromatopsia

Fast and reliable quantification of cone photoreceptors is a bottleneck in the clinical utilization of adaptive optics scanning light ophthalmoscope (AOSLO) systems for the study, diagnosis, and prognosis of retinal diseases. To-date, manual grading has been the sole reliable source of AOSLO quantification, as no automatic method has been reliably utilized for cone detection in real-world low-quality images of diseased retina. We present a novel deep learning based approach that combines information from both the confocal and non-confocal split detector AOSLO modalities to detect cones in subjects with achromatopsia. Our dual-mode deep learning based approach outperforms the state-of-the-art automated techniques and is on a par with human grading. © 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement OCIS codes: (100.2960) Image analysis; (100.4996) Pattern recognition, neural networks; (170.4470) Ophthalmology; (110.1080) Active or adaptive optics. References and links 1. J. Carroll, M. Neitz, H. Hofer, J. Neitz, and D. R. Williams, “Functional photoreceptor loss revealed with adaptive optics: An alternate cause of color blindness,” Proc. Natl. Acad. Sci. U.S.A. 101(22), 8461–8466 (2004). 2. K. M. Litts, R. F. Cooper, J. L. Duncan, and J. Carroll, “Photoreceptor-based biomarkers in AOSLO retinal imaging,” Invest. Ophthalmol. Vis. Sci. 58(6), BIO255 (2017). 3. K. E. Stepien, W. M. Martinez, A. M. Dubis, R. F. Cooper, A. Dubra, and J. Carroll, “Subclinical photoreceptor disruption in response to severe head trauma,” Arch. Ophthalmol. 130(3), 400–402 (2012). 4. A. Roorda and D. R. Williams, “The arrangement of the three cone classes in the living human eye,” Nature 397(6719), 520–522 (1999). 5. A. Roorda, F. Romero-Borja, W. Donnelly Iii, H. Queener, T. Hebert, and M. Campbell, “Adaptive optics scanning laser ophthalmoscopy,” Opt. Express 10(9), 405–412 (2002). 6. R. J. Zawadzki, S. M. Jones, S. S. Olivier, M. Zhao, B. A. Bower, J. A. Izatt, S. Choi, S. Laut, and J. S. Werner, “Adaptive-optics optical coherence tomography for high-resolution and high-speed 3D retinal in vivo imaging,” Opt. Express 13(21), 8532–8546 (2005). 7. D. Merino, C. Dainty, A. Bradu, and A. G. Podoleanu, “Adaptive optics enhanced simultaneous en-face optical coherence tomography and scanning laser ophthalmoscopy,” Opt. Express 14(8), 3345–3353 (2006). 8. C. Torti, B. Považay, B. Hofer, A. Unterhuber, J. Carroll, P. K. Ahnelt, and W. Drexler, “Adaptive optics optical coherence tomography at 120,000 depth scans/s for non-invasive cellular phenotyping of the living human retina,” Opt. Express 17(22), 19382–19400 (2009). 9. R. D. Ferguson, Z. Zhong, D. X. Hammer, M. Mujat, A. H. Patel, C. Deng, W. Zou, and S. A. Burns, “Adaptive optics scanning laser ophthalmoscope with integrated wide-field retinal imaging and tracking,” J. Opt. Soc. Am. A 27(11), A265–A277 (2010). Vol. 9, No. 8 | 1 Aug 2018 | BIOMEDICAL OPTICS EXPRESS 3740 #327228 https://doi.org/10.1364/BOE.9.003740 Journal © 2018 Received 4 Apr 2018; revised 15 Jul 2018; accepted 15 Jul 2018; published 18 Jul 2018 10. A. Dubra and Y. Sulai, “Reflective afocal broadband adaptive optics scanning ophthalmoscope,” Biomed. Opt. Express 2(6), 1757–1768 (2011). 11. R. S. Jonnal, O. P. Kocaoglu, Q. Wang, S. Lee, and D. T. Miller, “Phase-sensitive imaging of the outer retina using optical coherence tomography and adaptive optics,” Biomed. Opt. Express 3(1), 104–124 (2012). 12. M. Laslandes, M. Salas, C. K. Hitzenberger, and M. Pircher, “Increasing the field of view of adaptive optics scanning laser ophthalmoscopy,” Biomed. Opt. Express 8(11), 4811–4826 (2017). 13. A. Roorda and D. R. Williams, “Optical fiber properties of individual human cones,” J. Vis. 2(5), 404–412 (2002). 14. Y. Kitaguchi, K. Bessho, T. Yamaguchi, N. Nakazawa, T. Mihashi, and T. Fujikado, “In vivo measurements of cone photoreceptor spacing in myopic eyes from images obtained by an adaptive optics fundus camera,” Jpn. J. Ophthalmol. 51(6), 456–461 (2007). 15. T. Y. Chui, H. Song, and S. A. Burns, “Adaptive-optics imaging of human cone photoreceptor distribution,” J. Opt. Soc. Am. A 25(12), 3021–3029 (2008). 16. A. Dubra, Y. Sulai, J. L. Norris, R. F. Cooper, A. M. Dubis, D. R. Williams, and J. Carroll, “Noninvasive imaging of the human rod photoreceptor mosaic using a confocal adaptive optics scanning ophthalmoscope,” Biomed. Opt. Express 2(7), 1864–1876 (2011). 17. M. Pircher, J. S. Kroisamer, F. Felberer, H. Sattmann, E. Götzinger, and C. K. Hitzenberger, “Temporal changes of human cone photoreceptors observed in vivo with SLO/OCT,” Biomed. Opt. Express 2(1), 100–112 (2010). 18. O. P. Kocaoglu, S. Lee, R. S. Jonnal, Q. Wang, A. E. Herde, J. C. Derby, W. Gao, and D. T. Miller, “Imaging cone photoreceptors in three dimensions and in time using ultrahigh resolution optical coherence tomography with adaptive optics,” Biomed. Opt. Express 2(4), 748–763 (2011). 19. M. Lombardo, S. Serrao, and G. Lombardo, “Technical factors influencing cone packing density estimates in adaptive optics flood illuminated retinal images,” PLoS One 9(9), e107402 (2014). 20. S. S. Choi, N. Doble, J. L. Hardy, S. M. Jones, J. L. Keltner, S. S. Olivier, and J. S. Werner, “In vivo imaging of the photoreceptor mosaic in retinal dystrophies and correlations with visual function,” Invest. Ophthalmol. Vis. Sci. 47(5), 2080–2092 (2006). 21. J. L. Duncan, Y. Zhang, J. Gandhi, C. Nakanishi, M. Othman, K. E. H. Branham, A. Swaroop, and A. Roorda, “High-resolution imaging with adaptive optics in patients with inherited retinal degeneration,” Invest. Ophthalmol. Vis. Sci. 48(7), 3283–3291 (2007). 22. S. S. Choi, R. J. Zawadzki, M. A. Greiner, J. S. Werner, and J. L. Keltner, “Fourier-domain optical coherence tomography and adaptive optics reveal nerve fiber layer loss and photoreceptor changes in a patient with optic nerve drusen,” J. Neuroophthalmol. 28(2), 120–125 (2008). 23. S. Ooto, M. Hangai, A. Sakamoto, A. Tsujikawa, K. Yamashiro, Y. Ojima, Y. Yamada, H. Mukai, S. Oshima, T. Inoue, and N. Yoshimura, “High-resolution imaging of resolved central serous chorioretinopathy using adaptive optics scanning laser ophthalmoscopy,” Ophthalmology 117(9), 1800–1809 (2010). 24. D. Merino, J. L. Duncan, P. Tiruveedhula, and A. Roorda, “Observation of cone and rod photoreceptors in normal subjects and patients using a new generation adaptive optics scanning laser ophthalmoscope,” Biomed. Opt. Express 2(8), 2189–2201 (2011). 25. Y. Kitaguchi, S. Kusaka, T. Yamaguchi, T. Mihashi, and T. Fujikado, “Detection of photoreceptor disruption by adaptive optics fundus imaging and fourier-domain optical coherence tomography in eyes with occult macular dystrophy,” Clin. Ophthalmol. 5, 345–351 (2011). 26. J. Lammer, S. G. Prager, M. C. Cheney, A. Ahmed, S. H. Radwan, S. A. Burns, P. S. Silva, and J. K. Sun, “Cone photoreceptor irregularity on adaptive optics scanning laser ophthalmoscopy correlates with severity of diabetic retinopathy and macular edemacone mosaic irregularity in diabetic eyes on AOSLO,” Invest. Ophthalmol. Vis. Sci. 57(15), 6624–6632 (2016). 27. T. Y. P. Chui, D. A. Vannasdale, and S. A. Burns, “The use of forward scatter to improve retinal vascular imaging with an adaptive optics scanning laser ophthalmoscope,” Biomed. Opt. Express 3(10), 2537–2549 (2012). 28. D. Scoles, Y. N. Sulai, and A. Dubra, “In vivo dark-field imaging of the retinal pigment epithelium cell mosaic,” Biomed. Opt. Express 4(9), 1710–1723 (2013). 29. D. Scoles, Y. N. Sulai, C. S. Langlo, G. A. Fishman, C. A. Curcio, J. Carroll, and A. Dubra, “In vivo imaging of human cone photoreceptor inner segments,” Invest. Ophthalmol. Vis. Sci. 55(7), 4244–4251 (2014). 30. E. A. Rossi, C. E. Granger, R. Sharma, Q. Yang, K. Saito, C. Schwarz, S. Walters, K. Nozato, J. Zhang, T. Kawakami, W. Fischer, L. R. Latchney, J. J. Hunter, M. M. Chung, and D. R. Williams, “Imaging individual neurons in the retinal ganglion cell layer of the living eye,” Proc. Natl. Acad. Sci. U.S.A. 114(3), 586–591 (2017). 31. K. A. Sapoznik, T. Luo, A. de Castro, L. Sawides, R. L. Warner, and S. A. Burns, “Enhanced retinal vasculature imaging with a rapidly configurable aperture,” Biomed. Opt. Express 9(3), 1323–1333 (2018). 32. A. Roorda and J. L. Duncan, “Adaptive optics ophthalmoscopy,” Annu Rev Vis Sci 1(1), 19–50 (2015). 33. D. Cunefare, R. F. Cooper, B. Higgins, D. F. Katz, A. Dubra, J. Carroll, and S. Farsiu, “Automatic detection of cone photoreceptors in split detector adaptive optics scanning light ophthalmoscope images,” Biomed. Opt. Express 7(5), 2036–2050 (2016). 34. M. A. Abozaid, C. S. Langlo, A. M. Dubis, M. Michaelides, S. Tarima, and J. Carroll, “Reliability and repeatability of cone density measurements in patients with congenital achromatopsia,” in Advances in Vol. 9, No. 8 | 1 Aug 2018 | BIOMEDICAL OPTICS EXPRESS 3741

The most ophthalmosco more recently intact and fun retina -rods developed to provide an al widely used segment mosa Quantitati of each indiv media, there i [29,33], whi identification factors severe Several autom images [33,3 images of dis images from d than those fro accounted fo consensus of approaches to t common o ope (AOSLO), y, also non-con nctional outer s and foveal co utilize the m lternate source non-confocal aic [29].ve analysis of vidual photorec is ambiguity in ch is the all-im of these photo ely limiting the mated method [35][36][37][38][39][40][41][42][43][44][45][46][47][48].Howe seased eyes or diseased eyes om healthy su r in algorithm f the AO resea o identify cones .Dual-mode AOS the fovea of an A the same location er subject with A location as (c).O at can be seen to b m. f these AO , which is ofte nfocal modaliti segment structu ones [16].Non multiple-scatter e of informatio AOSLO detec f the photorece ceptor.Unfortu n the identificat mportant first oreceptors is hi e translation of ds have been d ever, most of t r have shown often exhibit s ubjects, as wel ms designed o arch communi s should be use systems is t en implemente ies [27][28][29][30][31]. Re ure [32] to rev n-confocal AO ed light not d on [27][28][29][30] [33,48] and signal-to-n tures that migh nas.Thus, the "… today's a [32].AOSLO imaging is already being used to select candidates for and predict the effectiveness of gene therapy [32,49] for conditions such as ACHM, a retinal condition characterized by a lack of cone function resulting in color blindness, photophobia, nystagmus, and severely reduced visual acuity [50].Unfortunately, quantification of cone photoreceptors in ACHM AOSLO images is especially challenging, even for human graders [34].In confocal AOSLO images of healthy eyes, cones appear as bright spots in the image, whereas in ACHM they appear as dark spots [51].As the rods appear to waveguide normally, it is sometimes possible to indirectly infer the presence of a cone when seeing a dark spot circumscribed by a ring of reflective rods, however this becomes challenging in images closer to the central fovea, where rod numerosity declines.Non-confocal split detector AOSLO imaging reveals remnant cone inner segment structures in areas that lack reflectivity in confocal AOSLO [29,52] (Figs.1(a) and 1(b)), showing potential for predicting therapeutic outcomes [32,49], and thus making automated detection of these cone structures desirable.Even though visualization of cones is possible with this imaging modality, there is often uncertainty in identifying cone locations due to the relatively poor contrast seen in typical images such as that shown in Fig. 1(c).It has been recently suggested that combining multiple modalities could improve the reliability/accuracy/other for cone identification [32], and it has been shown that multiple AOSLO modalities could improve performance in other image processing tasks such as mosaicking [53].As seen in Fig. 1(d), simultaneously captured confocal AOSLO images can help resolve some ambiguities seen in the matching split detector image, even with cones lacking intensity in ACHM subjects.
As with other computer vision tasks, automated analyses of AOSLO images with deep learning convolutional neural networks (CNNs) that learn features directly from training data are expected to outperform classic machine learning based techniques.CNNs have been utilized in numerous ophthalmic image processing applications [46,[54][55][56][57][58][59][60][61][62][63].In our previous work [46], we developed the first CNN based AOSLO image analysis method for detecting cones, demonstrating superiority to existing state-of-the-art techniques.Here, we expand on this work by combining the complimentary confocal and non-confocal AOSLO information to improve performance in low contrast images of diseased retinas.
The organization of the paper is as follows.We first introduce a novel dual-modality deep learning AOSLO segmentation paradigm for identification of cones.We then demonstrate that our method that incorporates dual-mode information from confocal and split detector AOSLO images outperforms a comparable deep learning method that only uses a single AOSLO imaging modality.Finally, we show that the dual-mode deep learning based method outperforms the state-of-the-art automated techniques and is on a par with human grading.

Methods
Our proposed algorithm for identification of cones, shown in Fig. 2, is comprised of a training and a testing phase.In the training phase, a set of reflectance confocal and split detector AOSLO image pairs was broken into small patches.A subset of all patches was classified (labeled) as cone and non-cone, based on manual markings.These labeled patches were used to train a CNN classifier, which was then utilized to generate probability maps from all overlapping patches in the images, which in turn allowed optimization of the parameters used for detecting cones.The trained CNN was then used to detect cones in previously unseen image pairs without known labels.(size 100 mage pairs , the split tector and ould have al graders, with the ones were ll images 0.5 × 0.5 s change.ages from  e pair, the nferred as h images.extracted mages that hat would he trained cone.We one of the sian filter d-maxima al regions puts these map to use oving any eshold T. to be the mizing the f potential lternative o evaluate our method, which means that for each subject, all images from the other subjects were used for training the network and cone localization parameters, and all images from that subject were used as the validation data set.Thus, there was no overlap between subjects or images used for training and testing of the algorithm.The first set of manual markings by the more experienced grader (DC) was used for training.

Convolut
For comparison to the state-of-the-art cone detection methods, we first evaluated the performance of Bergeles et al. [48], which was designed for detecting cones in split detector images and tested on subjects with Stargardt disease.We validated this algorithm across the entire split detector data set.We horizontally flipped the split detector images to match the orientation used in [48], and flipped the detected cone coordinates back to the original orientation of the images.The parameters for diseased images in their software were used.We also evaluated the software developed in Cunefare et al. [46] using the trained networks and optimization parameters learned from healthy split detector (SD-CNN) and confocal (C-CNN) AOSLO images exactly as reported in [46] across our split detector and confocal data sets, respectively.Additionally, we evaluated the performance of Cunefare et al. [46] after training new networks and parameters on the current ACHM split detector and confocal images (SD-CNN-ACHM and C-CNN-ACHM) using leave-one-subject-out cross validation.
To quantify the performance of the different methods, we first matched the automatically detected cones to the cones marked by the first grader one-to-one for each image pair in a similar fashion to Cunefare et al. [46].To summarize, an automatic cone was considered a true positive if it was located within some distance d of a manually marked cone.The value d was set to the smaller between 0.75 of the median spacing between manually marked cones in the image and 8 μm.The upper limit was used to account for images with sparse cone mosaics due to disease, and was chosen to be smaller than the maximum value of d found in healthy eyes in [46].Automatically detected cones that were not matched to a manually marked cone were considered false positives, and manually marked cones that did not have a matching automatically detected cone were considered false negatives.In the case that a manually marked cone matched to more than one automatically detected cone, only the automatically marked cone with the smallest distance to the manually marked cone was considered a true positive, and the remaining were considered false positives.To remove border artifacts, we did not analyze marked cones within 7 pixels (3.5 μm) of the edges of the images.After matching, for each image pair the number of automatically marked cones (N Automatic ) and manually marked cones (N Manual ) can then be expressed as: where NTP is the number of true positives, NFP is the number of false positives, and NFN is the number of false negatives.For each image pair, we then calculated the true positive rate, false discovery rate, and Dice's coefficient [75,76] as: FP Automatic False discovery rate N / N , = The second set of manual markings (SB) was compared to the first set of manual markings in the same way to assess inter-observer variability.

Results
Figure 7 shows a representative example of each automated method tested as well as the second set of manual markings in comparison to the first set of manual markings.In the marked images, automatically detected cones that were matched to a manually marked cone (true positives) are shown in green, cones missed by the automatic algorithm (false negatives) are shown in cyan, and automatically detected cones with no corresponding manually marked cone (false positive) are shown in red. Figure 8 displays examples of the performance of the single modality Cunefare et al. [46] method with the SD-CNN-ACHM and our proposed method using the dual-mode LF-DM-CNN architecture.Instances where the LF-DM-CNN, which uses multimodal information, correctly marks ambiguous locations in the split detector image where the single mode SD-CNN-ACHM method does not are indicated by orange arrows.Table 1 summarizes the performance of the automated methods in comparison to the first (more experienced) manual grader, as well as the variability between the two graders over the 200 ACHM image pairs in our data set.A large increase in performance can be seen by training Cunefare et al. [46] on ACHM images before testing.Our proposed method using the LF-DM-CNN architecture had the best performance in terms of Dice's coefficient.C-CNN and SD-CNN had higher true positive rates at the cost of substantially worse false discovery rates.
Fig. 1 near t from anothe same (c) tha 20 μm SLO cone imaging ACHM subject.(b n as (a).(c) Split CHM.(d) Simult Orange arrows poin be cones based on Fig. 3 AOSL locatio rando patch positioWe first norm stretched betw training imag al.[46].In br to define the marking in bo individual pat naturally mor so we used V location, in or nearest cone m created the no rounding to t position from extend outsid set of manual 3(b).Note tha differs from generating the

4 .
Fig. 4 the fo kernel outpu (AveP We built upon incorporate d fusion archite images [67, 6 CNN).The la on results acr pooling, recti max layers.T with N kernel the output can CNN adds a p network.Figu split detector on mean and decrease ove operation ove by a factor o inputs by sett the performan ations of the a n groups for our the automated con image.(b) Simult mparison to the fir kings (0.915), (d) 00), (g) C-CNN-A using the LF-DM false negatives, an n all methods i 0-beta23 for PC with an i7 GPU.The avera using the LF efare et al. [46 method of Berg total training or the LF-DM automatically c r method were ne detection algor taneously captured rst manual markin Bergeles et al. [4 ACHM (0.835), (h M-CNN network nd red denotes fals in MATLAB 2 training and 7-5930K CPU age run time o -DM-CNN on ] with the C-C geles et al. [48] time (includin M-CNN was un hosen cone de 1.69 ± 0.25, 0 rithms on an ACH d confocal AOSLO ngs (with Dice's co 8] (0.667), (e) C-C h) SD-CNN-ACHM (0.932).Green p se positives.± 0.03, and HM image pair.(a) O image from the oefficients) for (c) CNN [46] (0.178), M (0.907), and (i) points denote true athWorks, Nat e CNN's.We 64 GB of RAM for cone detec pixel image p N, C-CNN-AC 0.4, 10.2, 10.0, e network and The mean va eters σ, H, and d 0.77 ± 0.07.

8 .
Fig. 8 with t are sh image for th metho true p point ACHM CNN4.DiscussioWe develope confocal split representative dual-mode in