Comparison of the Repeatability of Macular Vascular Density Measurements Using Four Optical Coherence Tomography Angiography Systems

The aim of this study was to compare the repeatability of optical coherence tomography angiography (OCT-A) measurements of macular vessel density using four OCT-A systems, including Heidelberg Spectralis HRA, Optovue RTVue XR, Zeiss Cirrus HD-OCT 5000, and Topcon DRI OCT Triton. A cross-sectional design was used for this study. The vascular density and vascular length density of the superficial and deep retinal capillary plexuses were imaged with OCT-A using 3 mm and 6 mm scan patterns and were calculated using ImageJ. Comparisons of intraclass correlation coefficients (ICC) were conducted. We found that the OCT-A systems had various levels of repeatability. Zeiss had better repeatability for vessel density than the other systems (overall ICC = 0.936). Optovue had better repeatability for vessel length density when the 6 mm scan pattern was used (ICC = 0.680 and 0.700 for retinal superficial and deep capillary plexus, respectively). We concluded that repeatability varied when different scan patterns of various OCT-A systems were used for imaging the superficial retinal and deep capillary plexuses. Results should be seen as valid only for a given method. The repeatability of various OCT-A systems should be considered in clinical practice and in clinical trials that use OCT-A metrics as outcome measures.


Introduction
Optical coherence tomography angiography (OCT-A) was recently developed for imaging the retinal vasculature without dye injection [1]. OCT-A allows for the measurement of both the superficial and the deep retinal vessels quantitatively, including those in the macular region. Vessel density, which is quantified by calculating the percentage of the OCT image occupied by blood flow information as a fraction of the total image area, and vessel length density, which is quantified by calculating the percentage of skeletonized images of the retinal vasculature occupied by the vascular skeleton as a fraction of the total image area, have gained increasing popularity and represent promising quantitative metrics for future studies.
With OCT-A quantification becoming increasingly common, there is a pressing need to understand whether OCT-A systems can provide reliable and stable quantitative results and to determine which OCT-A system has better repeatability when using a specific scan pattern in clinical practice. Several studies have evaluated the reproducibility of various OCT-A systems; the results of these studies indicate that the outcomes of these instruments are generally uninterchangeable [2][3][4]. erefore, the repeatability of one OCT-A system should be taken into consideration when that OCT-A system is used. However, none of these aforementioned studies compared the repeatability of macular vessel density measurements taken with different OCT-A systems in detail.
Furthermore, assessment of various retinal diseases requires different scan areas and different OCT-A image resolutions. In the available literature that examined the validity and repeatability of these systems, emphasis was placed on using one pattern for a scan area, ignoring the use of multiple patterns for the scan area. Given the popularity of OCT-A, a more recent trend is to perform OCT-A examination with a larger scan area, as this provides a more relevant assessment of traditional fluorescence angiography. Evaluation of the repeatability of various scan areas is needed for future application of OCT-A. erefore, there is a need to investigate the repeatability of OCT-A systems using various scan patterns. If differences in the repeatability of macular vascular measurements exist between various OCT-A systems, this would be consequential for any clinical trial or research that uses a specific OCT-A system. To the best of our knowledge, no published study has compared the repeatability of various OCT-A systems for various scan areas and different retinal capillary layers.
is study examines the repeatability of macular vascular density measurements calculated with four commercially available OCT-A devices.

Participants.
is cross-sectional study was approved by the Peking Union Medical College Hospital Institutional Review Board, Beijing, China (reference number: ZS-1976). All procedures followed the tenets of the Declaration of Helsinki. e 48 adult subjects (13 males and 35 females) included in this study were outpatients recruited from the Peking Union Medical College Hospital between April 2019 and May 2019; individual participants could not be identified using the collected data. e inclusion criteria required each subject to be a consenting adult aged 18 and over, with healthy eyes. e exclusion criteria included a history of retinal and choroidal diseases, glaucoma, any other ocular diseases, and previous ocular surgery. e subjects underwent a complete ophthalmic examination, including measurement of best-corrected visual acuity and refractive errors, slit lamp biomicroscopy, fundoscopy, and OCT-A, along with documentation of a complete medical history.

Optical Coherence Tomography Angiography Acquisition
and Processing. All subjects underwent OCT-A imaging with Heidelberg Spectralis HRA (Heidelberg Engineering GmbH, Germany), Optovue RTVue XR (Optovue, Inc., USA), Zeiss Cirrus HD-OCT 5000 (Carl Zeiss Meditec, Inc., USA), and Topcon DRI OCT Triton (Topcon, Corp., Japan) in the same clinical setting. e four OCT-A examinations were performed twice by one experienced ophthalmologist in dark rooms on the same day, and subjects were blinded to personal information during and after the study. ere was sufficient interval for rest between the OCT-A examinations carried out with the four systems, but the interval between the two examinations for each system was as short as possible to minimize the influence of diurnal physiological conditions [5]. e pupil was not dilated before examination, and an internal fixation target in the OCT device was used.
e image quality scores of the included OCT-A images acquired with the Optovue and Zeiss devices were not less than nine, and these scores were generated by the devices automatically. e quality of the OCT-A images acquired with the Heidelberg and Topcon devices was evaluated by two retinal specialists independently (Jingyuan Yang and Mingzhen Yuan,). In cases of disagreement, a third retinal specialist (Youxin Chen) made the final decision. Unacceptable images, such as those with poor scan quality or those not centered on the fovea, were excluded [6]. e included OCT-A images were exported using Heidelberg Eye Version 1. e 3 × 3 mm (3 mm scan pattern) and 6 × 6 mm (6 mm scan pattern) scan patterns were used with maximal resolution. e scan protocol for the 3 × 3 mm scan pattern composed of 512 × 512 scans for Heidelberg, 304 × 304 scans for Optovue, 429 × 429 scans for Zeiss, and 320 × 320 scans for Topcon; the scan protocol for the 6 × 6 mm scan pattern was composed of 512 × 512 scans for Heidelberg, 400 × 400 scans for Optovue, 429 × 429 scans for Zeiss, and 512 × 512 scans for Topcon. Default segmentation was not adjusted and manual correction of segmentation was not applied. e retinal capillary plexus was divided into the superficial capillary plexus (SCP) and the deep capillary plexus (DCP). Default brightness and contrast were not adjusted to avoid manual bias. Considering that only two of these OCT-A systems provided personalized quantitative measures and to ensure that the OCT-A images were compared in the same setting, we used the third-party software ImageJ v2.0.0 (National Institutes of Health, available at https://imagej.net/Fiji/ Downloads) to analyze quantitative metrics. To investigate the reliability of the third-party software, a comparison of the repeatability of the quantitative metrics generated by the inbuilt software of the OCT-A systems and the third-party software was conducted first, in which the original vessel length density was converted into a decimal format. e macular OCT-A images were binarized with the Phansalkar local binarization thresholding method, which has been used on en face OCT-A images in recent studies [7][8][9][10]. e default parameter of a radius of 15 pixels in ImageJ was used. Once each pixel in the en face OCT-A image was determined to represent flow or nonflow using the thresholding method, binarized images were generated for further analysis, including analysis of vessel density and generation of skeletonized images for analyzing vessel length density. Binarized images were then skeletonized using ImageJ. Vessel density [11,12] and vessel length density [13,14] were subsequently calculated.

Statistical Analysis.
All statistical analyses were performed using SPSS version 25.0 (IBM Corp., USA). Data are presented as mean ± standard deviation (SD) or 95% confident interval (CI). e repeatability of the two consecutive measurements was assessed using the intraclass correlation coefficient (ICC). e ICC was used to determine the repeatability for each system by employing a two-way random effects model. e degree of repeatability was classified according to the ICC as follows: slight (0-0.2), fair (0.21-0.4), moderate (0.41-0.6), substantial (0.61-0.8), and almost perfect (0.8-1) [15]. To determine the intended sample size, the minimum acceptable repeatability (ICC) was set at 0.6, and the significance level and power was set at 0.05 and 0.8, respectively. ere was no dropout subject. e values of expected repeatability (ICC) for vessel density and vessel length density were set according to the overall results generated by the inbuilt software of the Optovue (0.808 for vessel density) and Zeiss (0.858 for vessel density and 0.973 for vessel length density) systems, which were the only two OCT-A systems that provided quantification functions. e minimum sample size should be 44 for vessel density and 5 for vessel length density, and the sample size of the present study met this requirement [16]. To compare the ICC values for intradevice repeatability between two OCT-A systems, we used the statistical method of Diedenhofen and Musch [17]. Statistical significance was defined as P < 0.05.

Results
Ninety-six eyes from 48 subjects (13 males and 35 females) were included. e mean (SD) age was 30.69 ± 4.67 years. e mean (SD) vessel density and vessel length density values measured with the four systems are summarized in Table S1; there were no missing data to report. No obvious segmentation errors were noticed.
e ICC values of the measurements derived with the built-in software of the Optovue and Zeiss systems and the third-party software of each system are summarized in Tables 1 and 2, respectively. e comparison of repeatability of the measurements derived with the built-in software and the third-party software is shown in Table 3. No significant difference in ICC values was noticed except for the ICC of the vessel density in the SCP imaged with the 6 mm scan pattern using Zeiss; this suggests that the third-party software showed comparable or even better repeatability than the built-in software during analysis.
When using the third-party software, the mean ICC for the measurements taken by each OCT-A system was 0.652, 0.881, 0.658, and 0.936 for vessel density, and 0.577, 0.925, 0.626, and 0.979 when using Heidelberg, Optovue, Topcon, and Zeiss, respectively. e comparison of the repeatability of each system is summarized in Table 4.

Vessel Density.
e ICCs of the vessel density measurements generated using both the 3 mm and 6 mm scan patterns in Zeiss exceeded 0.6, and the ICCs for the measurements derived using the 6 mm scan pattern in Optovue exceeded 0.6. e ICCs of the vessel density measurements of the SCP layer obtained using the 3 mm scan pattern in Heidelberg and Topcon exceeded 0.6 as well. e ICC for the vessel density of the DCP layer obtained using the 3 mm scan pattern in Zeiss was higher than that generated using Optovue (P � 0.0065). e ICCs for the vessel density of the DCP layer generated using the 3 mm scan pattern and the ICCs for the vessel density measurements of both the SCP and DCP layers derived using the 6 mm scan pattern in Zeiss were significantly higher than the ICCs of the measurements generated by both Heidelberg and Topcon (all P values <0.05). e ICCs of the vessel density measurements generated using the 6 mm scan pattern in Optovue were higher than the ICC of the vessel density measurement of the SCP layer derived using the 6 mm scan pattern in Heidelberg and the ICC of the vessel density of the DCP layer derived using the 6 mm scan pattern in Topcon (P � 0.0023 and 0.0009, respectively). No significant differences in ICCs were noticed between Heidelberg and Topcon (all P values >0.05).

Vessel Length Density.
e ICCs for vessel length density measurements for both the SCP and DCP layers obtained using the 6 mm scan pattern in Optovue exceeded 0.6. e ICC of the vessel length density measurement of the DCP layer derived using the 3 mm scan pattern in Zeiss exceeded 0.6 but was close to 0.6 for the vessel length density measurements of the SCP layer. However, no significant differences were observed between the ICCs of the vessel length density measurements derived with Optovue and Zeiss (all P values >0.05). e ICCs of the vessel length density of the SCP layer obtained using the 3 mm scan pattern in both Heidelberg and Topcon exceeded 0.6. When using Optovue, the ICCs of the vessel length densities of both the SCP and DCP layers derived using the 6 mm scan pattern were higher than the ICCs of those generated using Heidelberg, and the ICCs of the vessel length density of the SCP layer obtained with using 3 mm scan pattern and the DCP layer measurements derived with the 6 mm scan pattern in Optovue were higher than the ICCs of those generated using Topcon (all P values <0.05). When using Zeiss, the ICC of the vessel length density measurement of the SCP layer derived using the 3 mm scan pattern was higher than the ICC of that generated using Topcon, and the ICC of the vessel length density of the DCP layer derived using the 6 mm scan pattern was higher than the ICC of that generated using both Heidelberg and Topcon (all P values <0.05).

Discussion
In the present study, we compared the repeatability of four OCT-A systems. Each system was evaluated using the ICCs for the macular vessel density and vessel length density measurements of both the SCP and DCP layers derived using 3 mm and 6 mm scan patterns. We found that various OCT-A systems showed different levels of repeatability, and no OCT-A system showed practical repeatability in all conditions. Zeiss had better repeatability for measurement of vessel density than the other systems. Optovue had better repeatability for measurement of vessel length density using the 6 mm scan pattern. e repeatability of OCT-A systems should be considered in clinical practice, research, and clinical trial.
To investigate whether the third-party software could provide acceptable results, we compared the third-party software and the commercial built-in software of the OCT-A systems that provided quantification functions first; we found that the third-party software showed comparable repeatability with the built-in software in most cases when Journal of Ophthalmology 3 quantifying OCT-A images (Table 3). Since the OCT-A images of one eye are supposed to have similar vessel density and vessel length density values, the third-party software showed better repeatability when analyzing vessel density in the SCP layer with the 6 mm scan pattern in Zeiss. erefore, we believed that using the third-party software could provide reliable results.
e repeatability between scans on individual commercialized OCT-A instruments, including Optovue, Topcon, and Zeiss has been reported to be high [5,[18][19][20][21][22][23]. Nevertheless, the different evaluation approaches used in these studies and in the present study (especially the thresholding methods and the evaluated regions) make it almost impossible to conduct comparisons among various     studies, although these aforementioned studies reported that these devices showed good or even excellent repeatability. Additionally, a few previous studies had demonstrated that the quantitative measurements were not directly interchangeable among different OCT-A systems [2][3][4].
However, only few studies compared their repeatability on the same cohort. It is essential to perform OCT-A examinations using a reliable and repeatable system for clinical diagnosis and follow-up. e results of our study provide real-world data for improved examination in clinical practice.
In the present study, we used methods that previous studies had applied to evaluate OCT-A images. e metrics of vessel density and vessel length density were widely used in previous studies to evaluate retinal microvasculature quantitatively [11][12][13][14], and they were also calculated by the inbuilt software of Optovue and Zeiss, respectively. In the present study, the Heidelberg and Topcon systems did not provide inbuilt software for quantitative analysis. To calculate the metrics, an appropriate thresholding method for image binarization is essential. Rabiolo et al. [24] compared seven thresholding methods, and only one local thresholding method was included. However, in clinical practice, local threshold methods produce a binarized image that appears more uniform because these methods account for local variations in image appearance. e Phansalkar local thresholding method was used on OCT-A images in a recent study [7][8][9][10].
is thresholding method was proposed to address the nonuniform appearance of images [25]. In our study, the Phansalkar thresholding method enabled us to distinguish subtle capillary networks, which is believed to generate more accurate binarized images compared with ground truth. Nevertheless, given the lack of a standard method of assessing the quality of binarization thresholding methods, it is difficult to conclude that any one method is superior to the others.
Furthermore, we evaluated the SCP and DCP layers using 3 mm and 6 mm scan patterns. e OCT-A images of the DCP were obtained using different OCT-A systems with different projection artifact removal algorithms and light sources of different wavelengths [26,27]. erefore, the repeatability of OCT-A images of the SCP and DCP were supposed to be evaluated separately. On the other hand, the OCT-A images of 3 mm and 6 mm scan patterns had different resolutions and scan areas, which suggested that OCT-A images of different scan areas should be evaluated separately.
None of these OCT-A systems had the best repeatability in all circumstances (Tables 2 and 4). Heidelberg had a relatively poor repeatability in our study. e main explanation may be the long acquisition time and bright fixation target, which could lead to more motion artifacts and eye blinks [28]. Topcon had a poor repeatability in the measurement of the vascular density of the DCP; notably, the Topcon system was the only swept-source OCT used in the present study and should have had better performance on deep tissue than spectral-domain OCT. We speculate that its projection artifact removal algorithm and motion artifact correction algorithm may need further development.
Among the four OCT-A systems, Optovue and Zeiss had inbuilt software for quantitative analysis in sequence; not surprisingly, they had better repeatability for quantitative analysis. Interestingly, Optovue had better repeatability when the 6 mm scan pattern was used. One of the possible reasons is that the 6 mm scan pattern covers a larger area, so that local differences in OCT-A images had less influence on the vessel density and vessel length density values. e ICC values in the present study may differ from those of previous studies; this may be because we recruited subjects from a real clinical practice rather than from experienced staff, and the methods we used were not completely similar to those of these previous studies.
However, there were several limitations in our study. Firstly, we enrolled only healthy eyes; further studies including larger numbers of patients are needed to confirm our present findings. On the other hand, this study was not influenced by obvious segmentation errors because no eyes with retinal diseases were enrolled [29]. Further investigation is needed to evaluate the influence of segmentation approaches on the repeatability of these OCT-A systems. Secondly, the majority of the subjects were young adults hence we could not evaluate the influence of age on the measurements of each device; on the contrary, elderly subjects, especially those with macular diseases, who may be not be compliant and cooperative enough during the examination and may lead to poor OCTA image quality and motion artifacts, were not enrolled in the present study [30]. Although evaluation of the repeatability of OCT-A devices with elderly eyes may lead to unstandardized evaluation methods, OCT-A examination in clinical practice is usually performed on elderly patients. erefore, further studies focusing on subjects of less compliance are needed to evaluate the repeatability of OCT-A systems. Additionally, whether age has an influence on the repeatability of quantification of the macular capillary is unknown; this needs further investigation as well [31]. More importantly, although our results were valid for the given methodology, updates of OCT-A software and alterations of methods may affect the results.

Conclusions
All the OCT-A systems used in the present study may provide images with sufficient quality for qualitative analysis, but not all of the systems showed good repeatability for quantitative analysis of macular vessels. e present study provided detailed comparisons of the repeatability of various OCT-A systems with different scan patterns.
e repeatability of various OCT-A systems should be considered when using a specific scan pattern for quantitative assessment in clinical practice. Perhaps most importantly, this study suggests that ICC values should be seen as valid only for a given methodology. is has significant consequences for clinical trials involving the use of vessel density and vessel length density values as outcome measures. Future studies comparing the repeatability of each OCT-A system in eyes with retinal diseases would be worthwhile.

Journal of Ophthalmology
Data Availability e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.