http://orcid.org/0000-0002-7269-8255
Figures
Abstract
Objectives
To evaluate the diagnostic accuracy of a digital fundus photographic system that consists of taking fundus photographs by a trained technician using a RetCam® shuttle and interpreting fundus images by an expert to detect Retinotapthy of Prematurity requiring treatment (ROP-RT) which defined as type I ROP according to the Early Treatment for ROP study (ETROP).
Materials and methods
One hundred infants were examined by (1) an expert ophthalmologist experienced in ROP care using indirect ophthalmoscopy; (2) digital wide-field imaging by a trained technician using a RetCam® shuttle and images were sent remotely for interpretation by two ophthalmologists experienced in ROP care (Reader A, and Reader B); and (3) local ophthalmologists using indirect ophthalmoscopy. The diagnostic acurracy consisting of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (LR+), and negative likelihood ratio (LR-) were calculated. Agreement between all examiners and readers were evaluated.
Results
A total of 100 infants (mean gestational age 31.1 weeks, mean birth weight 1,511.1 grams) participated in the study. Nine infants were classified as ROP-RT. Reader A and B had very good agreement in detection of ROP- RT (Kappa 1.00, 95% CI 1.00, 1.00). For reader A, diagnostic performance parameters (95% confidence intervals) for detecting ROP-RT were; sensitivity 100.0% (66.4, 100.0), specificity 97.8% (92.1, 99.7), PPV 81.8% (48.2, 97.7), NPV 100.0% (95.8, 100.0), LR+ 44.5 (11.3, 175.2), and LR- 0.1 (0.0, 0.8). For reader B these were; sensitivity 100.0% (66.4, 100.0), specificity 95.6% (89.0, 98.8), PPV 69.2% (38.6, 90.9), NPV 100.0% (95.8, 100.0), LR+ 22.5 (8.6, 58.6), LR- 0.1 (0.0, 0.8). No adverse events were reported.
Citation: Wongwai P, Suwannaraj S, Asawaphureekorn S (2018) Diagnostic accuracy of a digital fundus photographic system for detection of retinopathy of prematurity requiring treatment (ROP-RT). PLoS ONE 13(7): e0201544. https://doi.org/10.1371/journal.pone.0201544
Editor: Antonio Palazón-Bru, Universidad Miguel Hernandez de Elche, SPAIN
Received: February 1, 2018; Accepted: July 17, 2018; Published: July 31, 2018
Copyright: © 2018 Wongwai et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper.
Funding: Faculty of medicine, Khon Kaen University. The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Competing interests: The authors have declared that no completing interests exist.
Introduction
Retinopathy of prematurity (ROP) is one of the leading causes of blindness in prematurely born children. Prompt diagnosis and treatment is key in the management of this potentially blinding disease. Carefully timed retinal examinations should be done by an ophthalmologist with experience in the examination of tiny babies, as described in the international recommendations [1]. In practice, not all countries and regions can provide enough ophthalmologists qualified as previously described especially in the developing countries. For these reasons, not all cases with ROP requiring treatment (ROP-RT) are picked up in due time, and even if they are, some babies have to be referred to a center to be examined by an ophthalmologist who expert in ROP care. Often these fragile infants still have an endotracheal tube and many catheters, so that transportation may put them at an additional risk. Another important issue is the cost of transportation across the city or country that may be required repeated times until the retinas are mature.
There are many articles that mentioned the use of wide field digital fundus cameras that could capture the fundus images and send to expert ophthalmologists for making the diagnosis [2–22]. These infants therefore do not have to travel to meet the expert, but there is still some debate about the role of this method for screening purposes.
The objective of this study was to evaluate the diagnostic performance of digital fundus photographic system for detection of ROP requiring treatment (ROP-RT, according to the ETROP study [23]), ROP of any stage (ROP-AS) and the presence of plus disease (Plus) when a trained technician take the photos and an expert ophthalmologist make the diagnosis. In addition, the diagnostic performance of fundus examination with indirect ophthalmoscopy by the local ophthalmologists who were not the expert in ROP care were also evaluated. In provincial hospitals, the local ophthalmologists are those who usually perform the ROP screening.
Materials and methods
This study was approved by the Khon Kaen University Ethics Committee in Human Research before beginning of the research procedures.
Study population
The participants were: (1) preterm infants who were referred from other hospitals to Srinagarind Hospital, Khon Kaen for ROP diagnosis and treatment, (2) infants for whom ROP screening was requested according to their own criteria from 4 provincial hospitals in the northeast of Thailand including Khon Kaen Hospital, Kalasin Hospital, Roi-et Hospital, and Mahasarakham Hospital. The exclusion criteria were: (1) an infant with an unstable condition as determined by the pediatrician, (2) an infant with a history of hyperglycemia, (3) an infant with ocular infection.
Study procedure
After written informed consent was obtained from the parents, the infant’s pupil was dilated with 1% Mydriacyl eye drops every 5 minutes for 3 times alternated with 2.5% phenynephrine eye drops every 5 minutes for 3 times. The examination started 30 minutes after the last eye drop. The infant was swaddled, fed with 24% sucrose solution through a pacifier for reducing pain and stress. The examination was started by putting in the anesthetic eye drop, inserting lid speculum and performing the retinal examination using indirect ophthalmoscopy with 28 diopters lens and scleral indentation by the expert ophthalmologist who had experience in ROP care. This is supposed to be the practical reference standard. The infant was then allowed to rest for 10 minutes. The second examination was the fundus photography by using a RetCam® shuttle (Clarity Medical Systems, Inc., Pleasanton, CA). Four images were captured for each eye including; the anterior segment, posterior pole, nasal and temporal retina. The fundus images were sent to two ophthalmologists who were experienced in ROP care (reader A and B) for making the diagnosis via a CD without other identifiable information. The infant was then allowed to rest for another 10 minutes. The last examination was indirect ophthalmoscopy with 20 or 28 diopters lens and scleral indentation by the local ophthalmologists of each hospital who were general ophthalmologists and did not have much experience in ROP care. For infants who were referred to Srinagarind Hospital, all the above procedures were done, but the diagnosis of the local ophthalmologists were identified from the referral document. The duration from the examination at the referral hospitals until the examination at Srinagarind Hospital should not be more than 48 hours. Each ophthalmologist and the photographer were masked to the diagnosis and the identifiable information of the infants. ROP-RT was classified according to the ETROP study as type I ROP (zone I, any stage, with plus disease; zone I, stage 3 with or without plus disease; or zone II, stage 2 or 3, with plus disease) [23].
Statistical analysis
The unit of analysis for ROP-RT was at the level of individuals; an infant would be classified as ROP-RT if one or both eyes were classified as ROP-RT. For ROP-AS and Plus, to avoid for correlation between eyes, one eye was randomly selected from each infant for statistical analysis.
All data were analyzed using R software version 3.4.1 [24]. Demographic data were described as mean, standard deviation (SD), median, minimum and maximum values. Diagnostic performance parameters were calculated using epiR package in R [25]. The following diagnostic performance parameters and their 95% confidence intervals (95% CI) were calculated; sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (LR+), and negative likelihood ratio (LR-). The 95% CI for sensitivity, specificity, PPV, and NPV were calculated with the exact confidence intervals method [26]. Confidence intervals for LR+ and LR- were based on formulas provided by Simel et al [27]. Since there were cells with zero value in two tables, 0.05 was added to all cells in those tables to solve the mathematical calculation of LR-. This method had been proposed by Haldane [28] and Anscombe [29].
Likelihood ratio (LR) was included in the analysis because it was a powerful tool for clinical decision making [30]. LR is the ratio of the probability of a given test result in patients with the disease to the probability of the same test result in patients without the disease. It incorporates both sensitivity and specificity into a single measure. For LR+, it is sensitivity/(1 –specificity). For LR-, it is (1 –sensitivity)/specificity.
LR is stable to changes in prevalence of disease as opposed to PPV and NPV. PPV and NPV calculated in a setting with a certain prevalence of the outcome cannot be applied to other settings with different prevalence. Therefore LR+ and LR- are recommended to use for calculation of PPV and NPV respectively based on the specific prevalence or pretest probability. This can be easily done with a standard Fagan’s nonogram [31] or a modified one [32]. LR for a given test result indicates how much that result will raise (if LR > 1) or lower (if LR < 1) the pretest probability. If LR = 1, the test is of no utility. For a rough guide, LR of > 10 or < 0.1 generate large and often conclusive changes from pretest to posttest probability [33].
The level of agreement between each method of examination was reported as Cohen’s kappa using psych package in R [34]. Confidence intervals for kappa were calculated using methods suggested by Fleiss [35].
Results
From May to September 2013, 126 infants met the eligible criteria. The parents of 26 infants were not available to provide their informed consent. A study flow diagram was shown in Fig 1.
Ret-cam = trained technician took fundus photograph and sent to expert ophthalmologist (reader A and reader B) for interpretation; IO by expert ophthalmologist = Indirect ophthalmoscopy by expert ophthalmologist (reference standard); IO by local ophthalmologist = Indirect ophthalmoscopy by local ophthalmologist; ROP-RT = ROP requiring treatment; ROP-AS = ROP of any stage; Plus = presence of plus disease.
The characteristics of all enrolled 100 infants and that were diagnosed as ROP were shown in Table 1. There were 18 infants who were referred to Srinagarind Hospital while 82 were infants from the 4 provincial hospitals. According to the expert ophthalmologist (reference standard), 9 of 100 infants were diagnosed as ROP-RT (2 infants were those referred to Srinagarind Hospital, and 7 infants were those from the 4 provincial hospitals). In addition, 31 infants were diagnosed as ROP-AS, and 8 infants as Plus. For ROP-AS, 9 infants (29%) were diagnosed as stage 1 ROP, 20 infants (65%) as stage 2 ROP, 1 infant (3%) as stage 3 ROP, and 1 infant (3%) as stage 4 ROP. There was no stage 5 ROP.
Based on the diagnosis of the expert ophthalmologist as a reference standard, the number of infants in each method of examination were shown in Table 2. For ROP-RT, poor image quality precluded the interpretation in 2 infants for reader A and 1 infants for reader B. For ROP-AS, poor image quality precluded the interpretation in 2 infants for reader A. For Plus disease, poor image quality precluded the interpretation in 2 infants for reader A and 1 infant for reader B. Two infants had media opacity that precluded the evaluation of Plus disease by the local ophthalmologist.
The diagnostic performance of the 3 methods of examination for the 3 categories of classification including sensitivity, specificity, PPV, NPV, LR+, and LR- were calculated from the numbers in Table 2 and were shown in Table 3.
The agreement of classifications among reader A, reader B, local ophthalmologist, and expert ophthalmologist for ROP-RT were shown in Table 4.
All infants successfully went through all 3 methods of examinations without any complications.
Discussion
There were many reports from all regions around the world that mentioned about the diagnostic accuracy of digital fundus photography for the diagnosis of ROP-RT or high risk ROP [2–18,20–22,36]. This study aimed to evaluate the accuracy of this technique with the addition of examination by the local ophthalmologists who were usually performed the ROP screening in the provincial hospitals. If the accuracy of ROP examination by digital fundus photography is accurate and clinically acceptable, it will be a useful alternative ROP screening method in the area where there are work overload and insufficient expert ophthalmologists experienced in ROP care.
The infants in this study had older gestational age and higher birth weight which were different from previous studies [2–4,6,7,9–14,16,17,19–22,37–44]. This might reflect the neonatal care system in this region and might suggest the need for ROP screening in the older gestational age and higher birth weight infants.
Regarding the diagnostic accuracy of digital fundus photographic system when ROPT-RT was the target outcome, reader A and B performed better than the local ophthalmologist which was reflected by very high values of sensitivity, specificity, PPV, and NPV (Table 3). This indicates that digital fundus photographic system is a good and accurate method for ROP screening. The local ophthalmologist missed 1 of 9 cases of ROP-RT while reader A and B did not miss any case (Table 2). This could probably be explained by the inexperience of the local ophthalmologist in examination of very small infants.
This study had similar results to the previous studies that supported the benefit of digital fundus photographic system to detect clinically significant ROP or ROP-RT [2,3,6,9,11–13,15–17,19–22,36,38–40,42,44] but our results had higher sensitivity, specificity, and LR+. The difference in instrumentation and images quality may contribute to these differences.
When the target outcome was ROP-AS, the sensitivity values of digital fundus photographic system were decreased for reader A (38.7%) and reader B (41.9%). However, the specificity values were still very good for reader A (98.5%) and reader B (97.1%) (Table 3). One explanation could be the limitation of digital fundus camera that could not capture the fundus images beyond anterior zone II and preclude the diagnosis of ROP in zone III, while the local ophthalmologist could use scleral indentation technique to visualize more peripheral retinal area. This would not be a problem because, according to the ETROP study, type 2 ROP including ROP in zone III was low risk ROP and did not require any treatment. Another possible explanation was difficulty of the readers to differentiate three dimensions nature of ridges from lines in some images.
When plus disease was the target outcome, both readers and local ophthalmologist had moderate sensitivity but high specificity with varying values (Table 3). The diagnosis of plus disease was subjective in nature even among the expert ophthalmologists, which might be the reason for variation of the diagnostic performance in our study and also previous published studies [45–47].
All the LR+ of any methods of examination (images reading remotely by reader A and B, and indirect ophthalmoscopy by the local ophthalmologist) according to any categories (ROP-RT, ROP-AS and Plus) were very high (Table 3). That is, almost all of them had the LR+ of greater than 10 (only reader B for Plus disease had LR+ of 8.5). In other words, the true positive results given by any of these methods of examination were about more than ten times the false positive. Hence, these methods were very useful in clinical decision making. For ROP-RT, the LR+ for reader A and B were much higher than the local ophthalmologist, which highlighted the usefulness of the digital fundus photographic system.
There was very good agreement among reader A, reader B and expert ophthalmologist in detection of ROP- RT from interpreting the fundus photographs compared with other methods (Table 4). This was anticipated because all of them were experts in ROP care. The agreement between local and expert ophthalmologist (kappa 0.66) was less than the agreement between reader A and expert ophthalmologist (kappa 0.89) and between reader B and expert ophthalmologist (kappa 0.80) (Table 4).
Many articles also reported similar results with this study that no difference in overall accuracy between ophthalmoscopy and telemedicine for detection of clinically significant ROP. But our study had added another aspect regarding the accuracy of currently ROP screening practice done by local ophthalmologists. Our results showed that digital fundus photographic system had better diagnostic accuracy than the local ophthalmologist for ROP-RT. However, the diagnostic performance of this method greatly affected by the quality of the fundus images. The diameter of pupils was an important factor that influenced the quality of the fundus images. There were 2 infants with poor quality images for reader A and 1 infant for reader B that precluded the categorization of ROP-RT. The size of the pupils were about 2 and 4 millimeters in these cases. We recommended that the pupil size for obtaining an interpretable image should be more than 5 millimeters. Another factor that might affect the quality of fundus images was the training of fundus imaging. Standardization of training and protocols for the photographers is the key to get a good diagnostic performance of the digital fundus photographic system. Morrison et al used 5 fundus images for telemedicine in ROP [48] while Quinn et al used 6 images [22]. Both reports found higher sensitivity to detect clinically significant ROP. Our study submitted only 4 images to the readers and still had high sensitivity and specificity in detection of ROP-RT. This could be explained by the difference in the experience of the readers. In our study, the readers were expert ophthalmologists in ROP care while Morrison and Quinn used non-physician as readers.
One of the limitation in our study was that all hospitals did not use the same standard criteria for ROP screening. The Royal College of Ophthalmologists of Thailand had developed a guideline for ROP screening but each hospital adapted the guideline according to their situation of neonatal care. Our study allowed each hospital to use their own ROP screening guideline because they might have different level of neonatal care.
Another limitation in our study was that the order of examination might affect the outcome of the last examiner (indirect ophthalmoscopy by the local ophthalmologist). The sequence of our study started with fundus examination by the expert ophthalmologist, followed by RetCam® imaging, and fundus examination by the local ophthalmologist. The long process of examination might alter the clarity of the cornea causing difficulty to visualize the fundus by the local ophthalmologist.
In conclusion, the diagnosis of ROP-RT from RetCam® images taken by trained technicians and evaluated remotely by an expert ophthalmologist had good diagnostic accuracy for screening purposes.
The authors listed in this study do not have any financial conflict of interest in any items mentioned in this study.
References
- 1. Fierson WM, American Academy of Pediatrics Section on Ophthalmology, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus, American Association of Certified Orthoptists. Screening examination of premature infants for retinopathy of prematurity. Pediatrics. 2013;131:189–95. pmid:23277315
- 2. Chiang MF, Keenan JD, Starren J, Du YE, Schiff WM, Barile GR, et al. Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124:322–7. pmid:16534051
- 3. Chiang MF, Wang L, Kim D, Scott K, Richter G, Kane S, et al. Diagnostic performance of a telemedicine system for ophthalmology: advantages in accuracy and speed compared to standard care. AMIA Annu Symp Proc. 2010;2010:111–5. pmid:21346951
- 4. Dai S, Chow K, Vincent A. Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clin Experiment Ophthalmol. 2011;39:23–9. pmid:20796264
- 5. Chiang MF, Gelman R, Martinez-Perez ME, Du YE, Casper DS, Currie LM, et al. Image analysis for retinopathy of prematurity diagnosis. J AAPOS. 2009;13:438–45. pmid:19840720
- 6. Vinekar A. IT-enabled innovation to prevent infant blindness in rural India: the KIDROP experience. J Indian Bus Res. 2011;3:98–102.
- 7. Chiang MF, Starren J, Du YE, Keenan JD, Schiff WM, Barile GR, et al. Remote image based retinopathy of prematurity diagnosis: a receiver operating characteristic analysis of accuracy. Br J Ophthalmol. 2006;90:1292–6. pmid:16613919
- 8. Wu C, Petersen RA, VanderVeen DK. RetCam imaging for retinopathy of prematurity screening. J AAPOS. 2006;10:107–11. pmid:16678743
- 9. Salcone EM, Johnston S, VanderVeen D. Review of the use of digital imaging in retinopathy of prematurity screening. Semin Ophthalmol. 2010;25:214–7. pmid:21091002
- 10. Shah PK, Narendran V, Saravanan VR, Raghuram A, Chattopadhyay A, Kashyap M. Screening for retinopathy of prematurity—a comparison between binocular indirect ophthalmoscopy and RetCam 120. Indian J Ophthalmol. 2006;54:35–8. pmid:16531668
- 11. Murakami Y, Jain A, Silva RA, Lad EM, Gandhi J, Moshfeghi DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 12-month experience with telemedicine screening. Br J Ophthalmol. 2008;92:1456–60. pmid:18703553
- 12. Murakami Y, Silva RA, Jain A, Lad EM, Gandhi J, Moshfeghi DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 24-month experience with telemedicine screening. Acta Ophthalmol. 2010;88:317–22. pmid:19930212
- 13. Fielder AR, Hildebrand PL, Ells A, Lorenz B, Trese MT, Capone A Jr, et al. Systematic review of digital imaging screening strategies for retinopathy of prematurity. Pediatrics. 2009;123:e360–361. pmid:19171594
- 14. Kemper AR, Wallace DK, Quinn GE. Systematic review of digital imaging screening strategies for retinopathy of prematurity. Pediatrics. 2008;122:825–30. pmid:18829807
- 15. Williams SL, Wang L, Kane SA, Lee TC, Weissgold DJ, Berrocal AM, et al. Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus non-expert graders. Br J Ophthalmol. 2010;94:351–6. pmid:19955195
- 16. Chiang MF, Wang L, Busuioc M, Du YE, Chan P, Kane SA, et al. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125:1531–8. pmid:17998515
- 17. Dhaliwal C, Wright E, Graham C, McIntosh N, Fleck BW. Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol. 2009;93:355–9. pmid:19028742
- 18. Richter GM, Williams SL, Starren J, Flynn JT, Chiang MF. Telemedicine for retinopathy of prematurity diagnosis: evaluation and challenges. Surv Ophthalmol. 2009;54:671–85. pmid:19665742
- 19. Quinn GE, Ying G-S, Pan W, Baumritter A, Daniel E, Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity Cooperative Group. Detection of Potentially Severe Retinopathy of Prematurity by Remote Image Grading. JAMA Ophthalmol. 2017;135:982–6. pmid:28796856
- 20. Vinekar A, Mangalesh S, Jayadev C, Gilbert C, Dogra M, Shetty B. Impact of expansion of telemedicine screening for retinopathy of prematurity in India. Indian J Ophthalmol. 2017;65:390–5. pmid:28573995
- 21. Thanos A, Yonekawa Y, Todorich B, Moshfeghi DM, Trese MT. Screening and treatments using telemedicine in retinopathy of prematurity. Eye Brain. 2016;8:147–51. pmid:28539810
- 22. Quinn GE, Ying G, Daniel E, Hildebrand PL, Ells A, Baumritter A, et al. Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity. JAMA Ophthalmol. 2014;132:1178–84. pmid:24970095
- 23. Early Treatment For Retinopathy Of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121:1684–94. pmid:14662586
- 24.
R Core Team. R: A Language and Environment for Statistical Computing [Internet]. Vienna, Austria: R Foundation for Statistical Computing; 2017. Available from: https://www.r-project.org/
- 25. Stevenson M, Nunes T, Heuer C, Marshall J, Sanchez J, Thornton R, et al. epiR: Tools for the Analysis of Epidemiological Data. R package version 0.9–93 [Internet]. 2017. Available from: https://CRAN.R-project.org/package=epiR
- 26. Clopper CJ, Pearson ES. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika. 1934;26:404–13.
- 27. Simel DL, Samsa GP, Matchar DB. Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J Clin Epidemiol. 1991;44:763–70. pmid:1941027
- 28. Haldane JBS. The Mean and Variance of χ2, When Used as a Test of Homogeneity, When Expectations are Small. Biometrika. 1940;31:346–55.
- 29. Anscombe FJ. On Estimating Binomial Response Relations. Biometrika. 1956;43:461–4.
- 30. Hayden SR, Brown MD. Likelihood ratio: A powerful tool for incorporating the results of a diagnostic test into clinical decisionmaking. Ann Emerg Med. 1999;33:575–80. pmid:10216335
- 31. Fagan TJ. Letter: Nomogram for Bayes theorem. N Engl J Med. 1975;293:257. pmid:1143310
- 32. Caraguel CGB, Vanderstichel R. The two-step Fagan’s nomogram: ad hoc interpretation of a diagnostic test result without calculation. Evid Based Med. 2013;18:125–8. pmid:23468201
- 33.
Furukawa TA, Strauss S, Bucher HC, Guyatt G. Diagnostic Tests. In: Guyatt G, Rennie D, Meade MO, Cook DJ, editors. Users' guides to the medical literature: a manual for evidence-based clinical practice. 2nd ed. New York: McGraw Hill Medical; 2008. p. 428.
- 34.
Revelle William. psych: Procedures for Psychological, Psychometric, and Personality Research. R package version 1.7.3 [Internet]. Evanston, Illinois: Northwestern University; 2017. Available from: https://CRAN.R-project.org/package=psych
- 35. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72:323–7.
- 36. Ossandón D, Zanolli M, Stevenson R, Agurto R, Ortiz P, Dotan G. A national telemedicine network for retinopathy of prematurity screening. J AAPOS. 2018;22:124–7. pmid:29535051
- 37. Chiang MF, Keenan JD, Du YE, Schiff W, Barile G, Li J, et al. Assessment of image-based technology: impact of referral cutoff on accuracy and reliability of remote retinopathy of prematurity diagnosis. AMIA Annu Symp Proc. 2005;126–30. pmid:16779015
- 38. Chiang MF, Melia M, Buffenn AN, Lambert SR, Recchia FM, Simpson JL, et al. Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology. 2012;119:1272–80. pmid:22541632
- 39. Biten H, Redd TK, Moleta C, Campbell JP, Ostmo S, Jonas K, et al. Diagnostic Accuracy of Ophthalmoscopy vs Telemedicine in Examinations for Retinopathy of Prematurity. JAMA Ophthalmol. 2018;136:498–504. pmid:29621387
- 40. Silva RA, Murakami Y, Jain A, Gandhi J, Lad EM, Moshfeghi DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 18-month experience with telemedicine screening. Graefes Arch Clin Exp Ophthalmol. 2009;247:129–36. pmid:18784936
- 41. Shah PK, Ramya A, Narendran V. Telemedicine for ROP. Asia-Pac J Ophthalmol Phila Pa. 2018;7:52–5.
- 42. Skalet AH, Quinn GE, Ying G-S, Gordillo L, Dodobara L, Cocker K, et al. Telemedicine screening for retinopathy of prematurity in developing countries using digital retinal images: a feasibility project. J AAPOS. 2008;12:252–8. pmid:18289897
- 43. The photographic screening for retinopathy of prematurity study (photo-ROP). Primary outcomes. Retina Phila Pa. 2008;28:S47–54.
- 44. Lorenz B, Spasovska K, Elflein H, Schneider N. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP). Six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247:1251–62. pmid:19462177
- 45. Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12:352–6. pmid:18329925
- 46. Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125:875–80. pmid:17620564
- 47. Chiang MF, Gelman R, Jiang L, Martinez-Perez ME, Du YE, Flynn JT. Plus disease in retinopathy of prematurity: an analysis of diagnostic performance. Trans Am Ophthalmol Soc. 2007;105:73–84; discussion 84–85. pmid:18427596
- 48. Morrison D, Bothun ED, Ying G-S, Daniel E, Baumritter A, Quinn G, et al. Impact of number and quality of retinal images in a telemedicine screening program for ROP: results from the e-ROP study. J AAPOS Off Publ Am Assoc Pediatr Ophthalmol Strabismus. 2016;20:481–5.