Introduction

Dry eye disease (DED) is a prevalent ocular disease involving tear film instability and/or insufficiency that substantially affects the quality of life due to ocular discomfort and blurred vision [1]. There are mainly two types of DED, evaporative and aqueous-deficient DED, and these two types of DED could coexist in one eye. From anterior to posterior, the lid margin is about 2 mm comprised of the cutaneous outer lid, ciliary zone, occlusal skin of the appositional surface, the mucocutaneous junction (MCJ) and marginal conjunctiva [2]. Normal lid margin structure is essential for tear film function, and its abnormalities are often observed in DED, especially in meibomian gland dysfunction (MGD) [3, 4]. The meibomian gland orifice (MGO) locates on the lid margin that expels meibum to maintain the tear film stability [5], and its plugging indicates the obstruction of terminal ducts [4]. The relative location between MGO and MCJ ensures the meibum to be appropriately delivered to the skin and tear meniscus [6]. For some patients, anteroplacement or retroplacement of MCJ or MGO retroplacement causes abnormal relative location between them that renders corresponding MG functionally ineffective [7, 8]. Thus, accurate and comprehensive identification of lid margin signs is crucial for DED classification, severity assessment, and precise treatment.

Traditionally, the lid margin signs are observed and evaluated manually with a slit lamp biomicroscope. Because the lid margin structure from anterior to posterior is complicated and there are many signs required to be identified, manual assessment is a significant burden for doctors in a clinical setting. In addition, manual evaluation might cause inter-and intra-doctor variability that reduces evaluation accuracy. Though it is necessary for DED evaluation, many dry eye clinics tend to avoid comprehensive lid margin assessment due to limited time and learning curve. Automatic identification of lid margin signs from anterior segment photography with an advanced algorithm could solve the problem in DED patients. In recent years, deep learning has been wildly adopted in the field of ophthalmology [9]. The neural network is a commonly constructed deep learning model with an input layer and multiple layers of nonlinear transformations to obtain an output that can perform image classification directly. The algorithmic model is trained with input images previously labelled manually (e.g., colour anterior segment photography categorized for the lid margin signs), and then the model can identify new images with similar labels following the training processes. The algorithmic model has been successfully applied to identify fundus photographs, optical coherence tomography and visual fields to evaluate age-related macular degeneration, diabetic retinopathy, and glaucoma [10,11,12,13,14,15]. Moreover, our previous research has constructed a convolution neural network to segment sub-basal corneal nerve based on in vivo confocal microscopy [16]. To the best of our knowledge, few studies have focused on evaluating lid margin signs in DED patients with a deep learning model.

The present research aimed to evaluate the feasibility of applying deep learning models to identify lid margin signs from colour anterior segment photography. The research potentially facilitates the rapid and accurate assessment of lid margin for clinical decision-making in DED patients, and the lid margin signs identified with the software might serve as an criterion for the DED evaluation.

Materials and methods

Dataset

The research protocol was approved by the local review board, and the research was performed following the Declaration of Helsinki. The informed consent was not obtained from all of the participants due to the retrospective study design. The individual information for each image was removed for privacy, and the image could not be linked to an individual patient.

We collected the colour anterior segment photographs showing the lid margin from a total of 428 patients diagnosed with DED between Sep 2019 to May 2020 in the Department of Ophthalmology of Peking University Third Hospital. DED was diagnosed according to the criteria established by Tear Film and Ocular Surface Society Dry Eye Workshop II [17]. We excluded patients who previously suffered lid margin trauma or surgeries that severely damaged the lid margin structure. All of the images were captured in a standardized way with a digital slit lamp camera system, consisted of a BX-900 Eyecap system (Haag-Streit, Koeniz, Switzerland) and a Canon EOS 40 D (Canon Inc., Tokyo, Japan). The photography was taken with 40× magnification and focused on the central upper or lower lid margin. Approximately a central 10–15 mm lid margin was included in the middle part of the image. The upper and lower eyelid was slightly opened manually during the photography for better exposure. Eight hundred fifty-six images from 428 patients were included initially, including 231 males and 197 females with an average age of 37.1 ± 8.2 years old. We performed a preliminary review of the included images and excluded images that did not focus on the lid margin. Finally, 832 images were eligible in the subsequent analysis.

The ground truth applied in the deep learning model was marked by three senior ophthalmologists. Initially, two graders independently annotated eight lid margin signs using binary classification for each enroled image. Subsequently, for the disagreement between two graders, the third senior ophthalmologist would perform the final grade. Graders were masked from the individual information during the classification processes, and the order of the images was disrupted for two graders. Finally, one image had eight labels.

The annotated lid margin signs included rounding of the posterior lid margin, lid margin irregularity, vascularization, hyperkeratinization, retroplacement and plugging of MGO, and anteroplacement and retroplacement of MCJ identified according to The International Workshop on Meibomian Gland Dysfunction [4]. Multiple signs could coexist in one image. In detail, rounding defines the thickening of the posterior lid margin that loses its sharp curve contacting the globe. The irregularity is the notching of lid margin frequently observed in the region of MGO. Vascularity is the apparent telangiectasis of lid margin vascular around the MGO. Hyperkeratinization is observed at the skin zone to depict the eczematous changes. MCJ observed to spread forward to cover the MGO or movement posterior with squamous metaplasia is regarded as anteroplacement or retroplacement. The retroplacement of MGO describes focal cicatricial processes that tracts orifices to move posteriorly. The representative of lid margin signs is shown in Fig. 1.

Fig. 1: The representative of lid margin signs.
figure 1

A Rounding of the posterior lid margin; B lid margin irregularity; C lid margin vascularization; D lid margin hyperkeratinization; E retroplacement of The meibomian glands orifices (MGO); F plugging of MGO; G anteroplacement of the mucocutaneous junction (MCJ); H retroplacement of MCJ.

Full size image

Model

The deep learning model identifying the lid margin signs was constructed based on VGGNet-13. Eight independent models were established to detect each sign. VGGNet is one of the convolution neural networks [18]. In detail, the convolutional layers are divided into five units. Each unit contains two convolutional layers and a max-pooling layer deployed after each unit to reduce the size of the output of the convolutional layer for better generalization. Also, compared to the original structure of VGGNet-13, we added a batch normalization (BN) layer after each convolutional layer to avoid gradient vanishing and gradient explosion so that better performance of gradient propagation could be obtained. Three fully connected (FC) layers are followed after five units. The last FC layer has only one artificial neuro whose value represents the probability of suffering from a particular lid margin sign.

The whole dataset is divided into three parts as training dataset (60%), validation dataset (20%) and test dataset (20%) with the same images for each lid margin sign task. For better accuracy of the output result, this division was stringent so that one image data must not show up in two datasets. The training dataset was used to train the weight parameters of the ConvNet to make it fit the labelled image data better. The validation dataset is used to avoid over-fitting problems, which can be discovered when the loss function decreases on the training dataset but increases continuously on the validation dataset. Finally, the test dataset is used to check whether the ConvNet gradually gains the ability to detect disease characteristics.

The original image was colour anterior segment photography with 3456 × 2304 pixels. The image was compressed smoothly to a square and scaled to 256 × 256 pixels with Image ANTIALIAS in the Pillow library before training. The loss function we applied in this paper was mean square error shown as followed:

$$J = \frac{1}{{N_b}}\mathop {\sum }\limits_{i = 1}^{N_b} \left( {\hat y_i - y_i} \right)^2$$

where Nb is the batch size, \(\hat y_i\) is the output of ConvNet and \(y_i\) is the label marked manually.

The optimization algorithm in the present research was Adam optimizer with an initial learning rate of 0.0001. During the ConvNet training process, the loss function decreased rapidly in the very beginning 100 episodes, and then it became slow. Around 150 episodes, the best performance on the test dataset could be achieved. After about 200 episodes, the over-fitting problem was dominant for accuracy deterioration on the validation dataset and test dataset. Given this, stopping the training process by around 150 episodes in time was necessary. The experiments were performed with Anaconda Python 3.7, TensorFlow 2.1, CUDA Toolkit 10.1 (NVIDIA GPU acceleration toolbox), CUDNN 7.6.5 (NVIDIA GPU deep learning acceleration toolbox) on a computer with intel I7-10750H, 6C12T CPU, using RTX2060 GDDR6 6G GPU for training and testing with memory.

Statistical analysis

The statistical analysis was performed with Anaconda Python 3.7. The result of each model in the testing set was evaluated against the label. The following metrics were obtained: sensitivity and specificity. The outcomes were evaluated with receiver operative characteristic (ROC) curves and the area under the curve (AUC).

Results

The ROC curves from deep learning models for each lid margin sign are shown in Fig. 2. The AUC value for rounding of posterior lid margin was 0.979, and the AUC value was 0.977 and 0.980 for lid margin irregularity and vascularization respectively. For hyperkeratinization, the AUC value was 0.964. The AUC value for MGO retroplacement and plugging were 0.963 and 0.968 respectively. Finally, for the model identifying anteroplacement and retroplacement of MCJ, the AUC value was 0.950 and 0.978, respectively.

Fig. 2: The receiver operative characteristic (ROC) curves for each lid margin sign model.
figure 2

A Rounding of the posterior lid margin; B lid margin irregularity; C lid margin vascularization; D lid margin hyperkeratinization; E plugging of MGO; F retroplacement of The meibomian glands orifices (MGO); G anteroplacement of the mucocutaneous junction (MCJ); H retroplacement of MCJ.

Full size image

The sensitivity and specificity of the eight models are shown in Table 1. The sensitivity and specificity for rounding of the posterior lid margin model were 0.974 and 0.921 respectively. The lid margin irregularity model achieved a sensitivity of 0.930 and specificity of 0.938. High sensitivity (0.923) and specificity (0.961) were also demonstrated in the model for lid margin vascularization. For hyperkeratinization, the sensitivity and specificity were 0.889 and 0.948. The model to identify MGO plugging and retroplacement demonstrated high sensitivity (0.979; 0.909) and specificity (0.867; 0.967). The model for anteroplacement and retroplacement of MCJ also manifested a sensitivity of 0.875 and 0.969 and specificity of 0.969 and 0.888. The analysis speed of the deep learning model can reach 64 images per second.

Table 1 Sensitivity, specificity and AUC for each lid margin sign.
Full size table

Discussion

Lid margin abnormality identification is crucial in dry eye assessment. The deep learning model in the present research demonstrated outstanding performance in detecting lid margin signs from anterior segment photography. Both sensitivity and specificity were high for identifying the rounding of posterior lid margin, lid margin irregularity, vascularization, hyperkeratinization, retroplacement and plugging of MGO, and anteroplacement and retroplacement of MCJ. The model could be potentially applied to facilitate DED evaluation in various clinical and research scenarios.

To further probe the performance of the model identifying lid margin signs, we conducted an error analysis in the false-negative and -positive cases, and the results are summarized in Table 2. Rounding of the posterior lid margin defines the thickening of the interface between lid margin and globe and loses its original abrupt curve [4]. The judgement is comparable subjective that depends on experience and angle of photography. Most of the false-negative cases occurred when the lid was not slightly opened during photography. Then, the contact curve was not sufficiently presented, so that it is difficult to identify the rounding of the lid margin. Standardizing the degree of lid margin opening during photography might solve the problem. Irregularity often occurs in the region of obliterated MGO due to tissue absorption. The false-negative cases mostly appear when the indentation was less noticeable. Some small notches at the lid margin might be difficult to identify and easily confused with MGO plugging. For the lid margin vascularization, it was the apparent telangiectasis of vascular around the MGO [4, 19]. False-positive cases mainly occur when the lid margin is dark, with pigmentation and excessive demonstration of conjunctival vessels, which might be falsely identified as vascularization. For the extraordinarily bright images at the lid margin or blocked with reflection, the model was less sensitive to the signs. Further training the model with more images with similar labels might facilitate the identification of these delicate features.

Table 2 Error analysis of the false-negative and -positive cases.
Full size table

The MGO plugging pertains to exhibiting elevated orifice due to obstruction of terminal ducts and extrusion of lipid and cell debris [20]. For most false-positive cases for MGO plugging, the MGO manifests an elevation due to clear meibum or obstructed without elevation. The retroplacement of MGO occurs in cicatricial processes that tract orifices to move posteriorly. The lid margin was slightly open during photography. The lid margin corresponding to the manual force point opens more compared with the less opened lid margin at other areas. In most false-positive cases, the disparity in the relative distance between MGO and the opened edge of the conjunctiva might be falsely identified as a movement of MGO. The anteroplacement also causes an unequal distance between MGO and lid margin that could be judged as retroplacement by the model. MCJ is the line between the occlusal skin and conjunctiva that might spread forward to cover the MGO or movement posterior with squamous metaplasia as anteroplacement or retroplacement [2, 21, 22]. For the model detecting MCJ retroplacement, high specificity was also achieved with robust sensitivity. The false-negative identification occurs mostly accompanied by MGO retroplacement or severe lid margin vascularization. MGO retroplacement would shorten its distance to MCJ that might cause a false judgement of MCJ location. Severe lid margin vascularization could interfere with the clear identification of MCJ that could cause false-negative cases. Additionally, the disparity in the proportion of positive and negative cases in the dataset might affect the model performance. The negative case takes the majority in images identifying rounding of the posterior lid margin and MCJ anteroplacement. Thus, the model is inclined to identify that the image did not have the sign.

In future research, potential approaches to improve the accuracy of the model to detect lid margin signs include applying additional datasets, advanced models and label with more details. Specifically, further training could be performed with more images from more patients. The photography processes should be standardized considering the brightness, centration, magnification and the pressure to open the lid margin. The construction of the model, including the amount of the convolutional and pooling layer could be further optimized in identifying lid margin signs. Detailed labels could be applied to facilitate the training. For example, the MCJ or the posterior lid margin could be marked on the image.

The present model is a preliminary model. With further improvement and validation, the advanced automatic lid margin signs identification algorithm could be applied in various scenarios, including clinical practice, trial enrolment, and population-based screening. Lid margin abnormalities could be frequently observed in DED patients, especially patients with MGD [3, 4]. Identifying lid margin abnormality is crucial in dry eye assessments and treatment decision-making [7, 23,24,25]. In traditional clinical settings, ophthalmic doctors applied a slit lamp biomicroscope with or without anterior segment photography to detect lid margin signs. However, conducting a comprehensive evaluation on each lid margin sign is a time-consuming process due to the complicated lid margin structure, so that many doctors and specialists would ignore lid margin assessments. The improved software would rapidly identify various lid margin signs accurately with anterior segment photography that facilitates a better evaluation of DED. Nowadays, for clinical trials targeting dry eye, we could rapidly enrol certain patients with particular lid margin abnormalities more conveniently with the software and continue to follow specific lid margin abnormality changes after the treatment. The application of the software to assess lid margin signs during clinical trials guarantees the consistency and repeatability of the assessment that improves the reliability of the trial. With further improvements, the model might be presented on mobile terminals. The software would identify the lid margin signs with photographs captured with a camera by users themselves on mobile terminals. The outcome could serve as a referral recommendation for seeking medical care.

Certain limitations exist in the present research. Firstly, the amount of enrolment images in the dataset and the proportion of positive cases (i.e., rounding of the posterior lid margin and MCJ anteroplacement) is relatively small. The model would tend to identify the image as positive in the testing dataset. Secondly, the present research only included images from a single machine and centre. The generalization of the model could not be guaranteed when test on another external dataset. Thirdly, eight models were created in the experiment to detect each lid margin sign. These models might combine into one model that identifies all signs with one input with further improvement. Fourthly, heatmaps were not generated due to technical difficulties, and future research should attempt to demonstrate the features the model detected.

In conclusion, the present research applied a deep learning model to detect lid margin signs automatically. The model achieved robust sensitivity and specificity identifying rounding of posterior lid margin, lid margin irregularity, vascularization, and hyperkeratinization, retroplacement and plugging of MGO, anteroplacement and retroplacement of MCJ. The study provided the potentiality of applying the deep learning model in automatic lid margin signs identification that eliminates the significant burden on comprehensive DED assessment in a clinical setting. The rapid, accurate and potential objective lid margin evaluation with the software facilitates DED evaluation and treatment decision-making.

Summary

What was known before

  • Accurate and comprehensive identification of lid margin signs is crucial for dry eye classification, severity assessment, and precise treatment.

  • Traditionally, the lid margin signs are observed and evaluated manually, which is a significant burden for doctors and causes inter-and intra-doctor variability that reduces evaluation accuracy.

  • Deep learning-based model could be trained to identify disease-related features in images automatically; however, few studies have focused on evaluating lid margin signs with it.

What this study adds

  • The deep learning model could identify lid margin signs with high sensitivity and specificity, including rounding of the posterior lid margin, lid margin irregularity, vascularization, hyperkeratinization, retroplacement and plugging of meibomian gland orifice, and anteroplacement and retroplacement of mucocutaneous junction.

  • The study provided the potentiality of applying artificial intelligence in lid margin evaluation to assist dry eye decision-making.