External validation of cut-off points for foveal thickness taking into account the intraretinal fluid using optical coherence tomography to diagnose diabetic macular oedema

Study population

The study population comprised diabetic patients who were referred from primary health care centres to the ophthalmology services of the General University Hospital of Elche. Patients are primarily referred when they present a visual acuity deficit or there is suspected diabetic retinopathy or diabetic macular oedema in the screening tests (Generalitat Valenciana: Conselleria de Sanitat, 2006). This hospital has a catchment population of 169,555 inhabitants in the region of Bajo Vinalopó, which is located in the province of Alicante (southeast Spain). The health system is free and universal, both at the primary and specialised care levels.

Study design and participants

The study design was cross-sectional observational with the objective of externally validating the diagnostic test published by Hernández-Martínez et al. (2015) in our geographical area. A random sample of diabetic patients who attended hospital ophthalmology services between October 2012 and June 2013 was selected. A random day was selected each week and all patients who attended that day and wanted to participate in the study were recruited. Patients were excluded if they had any of the following (Azrak et al., 2015a; Azrak et al., 2015b): cataract surgery in the last three months, laser treatment in the macular area or panphotocoagulation, dementia, treatment with intravitreal anti-angiogenic drugs, vitreoretinal surgery, high myopia or other macular conditions. The justification for these criteria is that they are conditions which can alter the anatomy of the macula.

Variables and measurements

The main study variable (Clinical Standard) was the diagnosis of macular oedema through indirect ophthalmoscopy with a 28-diopter lens (28D aspheric lens, Volk Optical Incorporated, Mentor, OH, USA) and the stereoscopic exploration of the fundus with a Topcon SL-8Z Slit Lamp (Topcon Corporation, Tokyo, Japan) and a 78-diopter magnifying lens (Volk Optical Incorporated, Mentor, OH, USA) with the pupil in mydriasis. These examinations were carried out by an expert retinal ophthalmologist (Anonymous, 1985; Kinyoun et al., 1989). Through the analysis of the images obtained using these techniques, the ophthalmologist was able to confirm the presence of diabetic macular oedema, according to the following definition: the presence of hard exudates or retinal thickening located 500 µm from the fovea (Azrak et al., 2015a; Azrak et al., 2015b).

As validation variables, both the foveal thickness and the presence of intraretinal fluid were used (Fig. 1). These parameters were obtained through OCT (Topcon 3D OCT-2000; Topcon Corporation, Tokyo, Japan) and interpreted by a different expert retinal ophthalmologist. The foveal thickness is given automatically by the device software, which obtains images of the macular area through 512 horizontal and 128 vertical line scans, which focus on the fixation point, resulting in a 3D 6 mm × 6 mm volume cube pattern. To perform the indicated measurement, an area of 6 mm in diameter was used, where the centre was the fovea, to thus obtain an area of 1,000 central microns for the assessment. To measure the presence of intraretinal fluid (cysts or hyporeflective spaces), the ophthalmologist examined the horizontal OCT image of the retina (B-Scan) (Azrak et al., 2015a; Azrak et al., 2015b; Hernández-Martínez et al., 2015).

At the descriptive level, to compare our population with that of the original study determining cut-off points (Hernández-Martínez et al., 2015), we used the electronic medical history, ophthalmological assessment and clinical interview with the patient, recording gender, type 2 diabetes mellitus, arterial hypertension, dyslipidaemia, smoking, age, HbA1c (%), years since diabetes diagnosis and visual acuity.

Sample size calculation

Given that data were collected over a specific period of time, the sample size was calculated a posteriori, that is, determining whether the collected sample was sufficient to answer our research question (to validate a diagnostic test by calculating the sensitivity, specificity and AUC). Our sample size was 134 eyes. To estimate whether this sample was sufficient to validate the diagnostic test, we computed the specificity estimation using the following parameters: expected specificity value of 89% (Hernández-Martínez et al., 2015), prevalence of macular oedema in referred patients of 30.6% (Azrak et al., 2015a; Azrak et al., 2015b) and Type I error of 5%. With these parameters, a precision of 6.36% was obtained, which was satisfactory for our objective.

Statistical methods

The quantitative variables were described by calculating the mean and standard deviation, and the qualitative variables with the absolute and relative frequencies. The descriptive characteristics of our population were compared with those of Hernández-Martínez et al. (2015), through tests based on the χ² distribution (Pearson or Fisher) and the Student’s t-test. To determine the discrimination of the diagnostic test, the AUC was calculated using the formula proposed by Hernández-Martínez for the probability of diabetic macular oedema (Supplemental Material by Hernández-Martínez et al. (2015)). Sensitivity, specificity and positive likelihood ratio (PLR) and negative likelihood ratio (NLR) were calculated to measure the calibration (observed similar to expected). All these calculations were performed by bootstrapping (1,000 samples). A bootstrap sample is a random sample with replacement taken from the original sample and with the same number of elements as this. In each of these samples a statistical estimate is made, which allows its distribution to be constructed as 1,000 values are available for this distribution. This technique is the most accurate when validating predictive models (Steyerberg et al., 2001). All analyses were performed with a significance of 5% and the associated confidence interval (CI) was obtained for each relevant parameter. The statistical packages used were IBM SPSS Statistics 19 y R 2.13.2.

Ethical issues

All patients gave their written informed consent. The study was approved by the Ethics Committee of the General University Hospital of Elche and complies with the World Medical Association Declaration of Helsinki and with the standards described by the European Union Guidelines for Good Clinical Practice.

Summary

This study externally validated the cut-off points for the diagnosis of diabetic macular oedema considering the presence of intraretinal fluid (90 µm when there is fluid and 310 µm otherwise) (Hernández-Martínez et al., 2015). This validation was performed on 1,000 bootstrap samples, which gives greater accuracy to the results obtained (Steyerberg et al., 2001).

Similarly, we could say that the presence of intraretinal fluid confirms the diagnosis of macular oedema with almost any macular retinal thickness, whereas in order to confirm macular oedema conclusively in the absence of intraretinal fluid, a macular retinal thickness of 310 µm is necessary.

Strengths and limitations of the study

The main strength of our work is the novel way in which the clinical question was addressed by externally validating the cut-off points for foveal thickness taking into consideration the presence of intraretinal fluid. This is the key point for being able to use a diagnostic test in routine clinical practice. In addition, this validation was not performed in just one sample (split samples), rather it was corroborated in 1,000 bootstrap samples with very accurate and satisfactory results.

When examining the possible limitations of our study, one might consider that we introduced selection bias, since the characteristics of our patients are different from those of the original study (Hernández-Martínez et al., 2015). However, we must bear in mind that we are validating a diagnostic test through the calculation of sensitivity and specificity, independent parameters of disease prevalence (Lalkhen & McCluskey, 2008). In other words, the use of a sample obtained in specialised care consultations is not a selection bias. In addition, our sample was selected completely at random. Furthermore, all the fundamental variables of this study (diagnosis of diabetic macular oedema, foveal thickness and presence of intraretinal fluid) were measured by specialised professionals with validated devices. Finally, the precision for our sample size was around 6% based on the test specificity. Thus, validation studies involving larger sample sizes should be done to reduce this error. Nonetheless, we performed the analysis with the bootstrap technique in order to obtain stable results (Steyerberg et al., 2001), using 1,000 random samples with replacement taken from the original sample.

Comparison with the existing literature

In the absence of previous validation studies of the cut-off points indicated by the article published in late 2015 (Hernández-Martínez et al., 2015), we could only compare our results with those of the original study. In the AUC we obtained nearly the same value (0.88 versus 0.89), which indicates that the discrimination is practically the same. Very similar results were also obtained in terms of sensitivity and specificity, with an 8% decrease in sensitivity, while specificity increased by 6%. Clearly, as the likelihood ratios depend on sensitivity and specificity, our values varied accordingly, with the PLR increasing from 7.28 to 14.31 and the NLR increasing from 0.19 to 0.26 (Hernández-Martínez et al., 2015). Now, if we look at the PLR, when we obtain a value greater than 10, we can confirm the diagnosis of macular oedema in an epidemiologically conclusive way, since it is 14.31 times more likely that the patient has the disease (Guyatt & Drummond, 2002).

Implications for clinical practice and research

We have externally validated the cut-off points for foveal thickness to diagnose diabetic macular oedema resulting from the original study (90 µm in the presence of intraretinal fluid and 310 µm otherwise) in a population referred to ophthalmology services. The original study was validated in primary care (Hernández-Martínez et al., 2015). Thus we now have a decision algorithm to diagnose diabetic macular oedema in these two areas (primary and specialised care).

Given that the validation was performed in the health area covered by our hospital, we encourage other researchers to perform validation in their own communities. If similar results are found, these cut-off points for foveal thickness can be used in routine clinical practice (Hernández-Martínez et al., 2015).

The results of our study have great clinical relevance in the screening for diabetic macular oedema, as we are validating a diagnostic test with a high discriminative capacity and good calibration (sensitivity and specificity). This results in a much reduced rate of false positives and false negatives for every 1,000 patients referred. Out of this total of 1,000 patients, 888 are diagnosed correctly. In addition, there are 37 patients whom we are diagnosing with diabetic macular oedema but who do not really have it and 75 patients who we consider not to have the condition but who do in fact have it. Considering that we are assessing a screening test for diabetic macular oedema, these error rates are low (<15%) and allow us to free up ophthalmology consultation offices as this test can be performed in primary care offices by health care personnel not necessarily specialised in ophthalmology. This is mainly because latest generation OCT devices focus the image of the retina automatically (Hernández-Martínez et al., 2015). In addition, the protocol dictates that the false negatives will return in a year for another screening test and at this time they will be detected, whilst the false positives will have their initial diagnosis of diabetic macular oedema ruled out in the specialist ophthalmology service.