Eye-tracking-aided digital system for strabismus diagnosis

Strabismus is one of the most common vision disorders in preschool children. It can cause amblyopia and even permanent vision loss. In addition to a vision problem, strabismus brings to both children and adults serious negative impacts in their daily life, education, employment etc. Timely diagnosis of strabismus is thus crucial. However, traditional diagnosis methods conducted by ophthalmologists rely significantly on their experiences, making the diagnosis results subjective. It is also inconvenient for those methods being used for strabismus examination in large communities such as schools. In light of that, in this Letter, the authors develop an objective, digital and automatic system based on eye-tracking technique for diagnosing strabismus. The system exploits eye-tracking technique to acquire a person's eye gaze data while he or she is looking at some targets. A group of features are proposed to characterise the gaze data. The person's strabismus condition can be diagnosed according to the features. A strabismus gaze dataset is built using the system. Experimental results on the dataset demonstrate the effectiveness of the proposed system for strabismus diagnosis.

1. Introduction: Strabismus, termed squint and heterotropia as well, is that the two eyes do not point to the same direction. It is a common ophthalmic disorder with a prevalence of around 4% of the adult population [1]. If strabismus is not well treated, it would result in amblyopia or weak three-dimensional (3D) perception [2]. In addition to a vision problem, strabismus has been shown to have adverse psychosocial consequences in both children and adults [3]. For many young patients, their strabismus problems could be well treated or significantly alleviated if diagnosis and treatment are taken early. Therefore, timely diagnosis of strabismus is essential.
Traditional strabismus diagnosis methods include cover test, Hirschberg test, Maddox rod etc. These methods are conducted manually by the ophthalmologists according to their experiences. The diagnosis correctness is remarkably influenced by the ophthalmologist's professional qualities. Furthermore, these methods are manual, which makes it inconvenient to use for large communities (e.g. schools). In view of that, we propose and develop a digital strabismus diagnosis system based on eye-tracking technique in this Letter. The system is non-invasive and objective, and the diagnosis is generated automatically without any ophthalmologist's instruction. This makes it easy to carry out strabismus examination in large communities. For example, we can place the system in a primary school and the students can take their examinations at any time.
In the proposed system, we first show the subject nine target points ordered in different positions on a screen, and record the subject's gaze data. Then, a group of features are proposed to characterise the gaze data. A diagnosis result is finally made by analysing the features. We build a gaze dataset using our eye-tracking system for performance evaluation. The effectiveness of our method is demonstrated by comparing the diagnosis result with professional ophthalmologist's diagnosis result.
The rest of this Letter is organised as follows. We first review Letters on the eye-tracking methodology for solving various problems, especially disease detection and analysis, and then describe the proposed system and elaborate the features that are used to characterise gaze data. After that, we introduce our dataset and report experimental results. A conclusion is drawn at the end of this Letter.

Related work:
Approaches have been developed to detect eye movements since the last century. For example, Jolson et al. [4] proposed an apparatus for evaluating the alignment of both eyes. Knapp et al. [5] developed a method and apparatus for determining the position of each eye of a human with respect to a fixed reference point or with respect to each other by using electro-oculogram signals produced by eye movement, so as to measure strabismus. Compared to modern eye trackers, however, the accuracy of this apparatus is pretty low. In [6], Schaeffel developed an automated procedure to measure kappa and the Hirschberg ratio for immediate use in a video gaze tracker. Yang et al. [7] used a 3D strabismus photo analyser to estimate binocular alignment using photographs, and found that the 3D Strabismus Photo Analyser is a simple and reliable tool for measuring ocular deviation.
There are some attempts that leverage eye-tracking methodology for strabismus examination [14][15][16][17][18]. In the last century, people have attempted to build automatic eye-tracking systems for measuring strabismic deviation based on television cameras [14,15]. In the past several years, people continued to refine strabismus examination by developing more advanced eye-tracking methods. For example, Pulido [16] used Tobii eye tracker to acquire gaze data, and calculated the deviation of gaze data for ophthalmic diagnostics including strabismus. However, Pulido proposed a method prototype only. The author had no real strabismic gaze data to demonstrate the prototype's performance. Model and Eizenman [17] proposed an automatic method based on eye-tracking for performing the Hirschberg test, a classical method to measure binocular ocular misalignment. To precisely locate the pupil centres and corneal reflexes, they built a remote binocular gaze-tracking system using two video cameras and three infrared light sources. However, the performance was studied with five healthy infants only. Thus, the method's effectiveness for strabismus examination had not been tested. In Model's doctoral dissertation, he proposed two personal calibration procedures that do not require active user participation for estimation of gaze point and measurement of ocular alignment for adults and infants [18], but his methods were not been tested with sufficient strabismic data. Bakker et al. [19] developed a gaze direction measurement instrument to estimate strabismus angle. The instrument allows for unrestrained head movement by means of a triple camera vision system that simultaneously estimates the head rotation and the eye pose. Only three subjects participated in the experiment.
In recent year, virtual reality technique is also applied to eye health. For instance, Blaha and Gupta [20] designed a virtual reality game to help people with amblyopia restore vision, but they focused on amblyopia training rather than diagnosis. Fig. 1 shows the framework of the proposed system, in which the diagnosing procedure for a subject is carried out as follows:

Methodology:
(i) The subject sits at a fixed near distance from a laptop with an eye tracker mounted below the laptop's screen. (ii) The subject fixates on a number of calibration points to perform calibration. (iii) The subject fixates on a number of target points for eye gaze data collection. (iv) A group of features are extracted from all the gaze data.
(v) A diagnosis result is made according to the features.
The above procedures will be detailed in the subsequent sections.
3.1. Gaze data acquisition 3.1.1 Eye-tracking system: We use eye tracker Tobii X2-60 to build our system. Its sampling rate is 60 Hz and tracking accuracy is 0.4°. To make the system portable, we mount the eye tracker below the monitor of a laptop, as shown in Fig. 1. The laptop is Lenovo Thinkpad T540p with a 1920 × 1080 screen resolution. We employ Tobii MATLAB SDK to design our experiments for gaze data acquisition. The coordinate system of the screen is defined as follows. The upper-left corner of the screen is set as the origin, the coordinate value of which is (0, 0), with horizontal denoting x-coordinate and vertical denoting y-coordinate. The coordinate value of the lower-right corner is (1, 1), and the values of the upper-right corner and lower-left corner are (1, 0) and (0, 1), respectively. Both the x-value and y-value of any position on the screen are between 0 and 1, as shown in Fig. 2. 3.1.2 Calibration: The purpose of calibration is to teach the eye-tracking system the characteristics of the subject, such that the eye tracker can well detect the subject's eye movements in a subsequent test. In the calibration process, we adjust the subject's chair and the eye-tracking screen to make sure that the subject is at a fixed distance (50 cm in our experiments) from the screen and his or her eye level is horizontally pointing to the screen centre, i.e. point (0.5, 0.5). We choose a distance of 50 cm as it is an optimal distance for eye tracker Tobii X2-60 to track the subject's eye movements. We use nine-point calibration scheme. If the calibration result indicates that the fixation accuracy of either eye is acceptable (average accuracy higher than a predefined value, e.g. 0.05 in this Letter), we can then start the subsequent test. Otherwise, we should recalibrate.
3.1.3 Data acquisition: A classical way for the ophthalmologist to examine strabismus is a nine-point method. That is, the ophthalmologist instructs the patient to fixate on nine points at a certain distance in front, and then observes the patient's eye movements. The nine-point method can comprehensively examine one's eye movements with rotations at different angles. We adopt the same method to design our gaze data acquisition interface. The coordinate values of the nine points that we used are the same for the nine calibration points. The points are displayed one by one orderly. Fig. 2 shows the nine-point gaze data acquisition interface. The white arrows point out the display order. A black background of the interface helps the subject to concentrate on the target points. Data acquisition programme runs as follows. Each time one target point is displayed. Meanwhile, the subject's gaze points are recorded by eye tracker. If the number of effective gaze pairs acquired exceeds 100, current target point is removed and the next target point would be displayed. A gaze pair is defined as two gaze points of the two eyes captured by eye tracker at one sampling moment. 'Effective' here means that at least one gaze point of a gaze pair locates close enough to the target point. That is, the distance between the target point and either gaze point of a gaze pair must be smaller than a threshold (0.05 in this Letter) predefined empirically. For those who suffer serious strabismus, it is sometimes hard to capture effective gaze points at some target points, e.g. points at the four corners of the screen, because these points need strabismic subjects to rotate eyeballs to their extreme. To handle this case, we let eye tracker record gaze points at each point for at most 10 s. The next target point would be displayed after 10 s no matter whether or not the system has collected 100 pairs of gaze points. The sampling rate of our eye tracker is 60 Hz. It takes only 2 s to collect 100 gaze pairs for normal people. Thus, 10 s are long enough to capture gaze data for each point.

3.2.
Feature extraction for diagnosis: The next step after gaze data acquisition is feature extraction from all the gaze points. On the basis of the features, we should be able to tell whether the subject has strabismus or not, which eye is strabismic, and what type of strabismus it is. The features for answering these questions are constructed and described as follows.
3.2.1 Strabismus occurrence feature: Strabismus occurrence feature is defined to answer the question of whether or not the subject has strabismus. To realise that we propose to exploit two features: fixation deviation and fixation contrast. Fixation deviation is  defined as the Euclidean distance between the centre of a gaze pair and the target point. We can also calculate the Euclidean distance between left or right gaze point and the target point. Fixation contrast is then defined as the difference between the Euclidean distances of left gaze point and right gaze point to the target point. Fig. 3 shows an example of a normal gaze point pair (blue) and a strabismic gaze point pair (red) together with their target point (black). (x nl , y nl ), (x nr , y nr ), and (x nc , y nc ) denote left gaze point, right gaze point, and the centre of a normal gaze pair, and (x sl , y sl ), (x sr , y sr ), and (x sc , y sc ) are corresponding points for a strabismic gaze pair. (x t , y t ) is the target point. d nl , d nr , d n , d sl , d sr , and d s are distances between the target point and corresponding points. The significant difference between a normal gaze pair and a strabismic gaze pair is that the strabismic pair has always a point close to the target point and the other one relatively far from the target point, while the two points of the normal pair locate at a similar distance from the target point. As a result, the strabismic pair usually has a fixation deviation d s larger than that (d n ) of the normal pair, and a fixation contrast |d sl −d sr | larger than that (|d nl −d nr |) of the normal pair. Therefore, we can use fixation deviation and fixation contrast to classify strabismic gaze data and normal gaze data accordingly.
Mathematically, let (x l , y l ) and (x r , y r ) denote the left gaze point and right gaze point for a gaze pair, and (x t , y t ) denote the target point. We can then have the formulation of fixation deviation defined as below: and the formulation of fixation contrast defined as below: where and We calculate f d and f c for all the gaze pairs of nine target points. A simple way to obtain a representative value for all the gaze data is to average f d and f c over all the gaze pairs. However, the average result would be dominated by some extreme gaze points, which cannot represent well the subject's normal fixation state. To alleviate the effect of extreme points, we take into account a statistical measure of percentile for the calculations of f d and f c . A percentile indicates the value below which a given percentage of observations fall. For example, the 60th percentile of f d is the value below which 60% of gaze pairs' f d may be found. It can characterise the major gaze pairs' f d . We tested percentiles for both f d and f c from 70 to 95 and found that the diagnosis results do not vary much. In this Letter, for a convenient comparison, we present the widely used 85th percentile only. We denote the percentiles of f d and f c over all the gaze data of nine target points as f dp and f cp . For some strabismic people, their eyes can fixate precisely at some directions, whereas for some other directions the two eyes do not align well. This case happens a lot in particular to incomitant strabismus, the strabismic eye of which cannot move in some specific directions. That is why the ophthalmologist needs to use the nine-point method to examine the patient's eye movements. Taking this case into account, we add two features called maximum fixation deviation f dm and maximum fixation contrast f cm . We first calculate 85th percentile of f d and f c for each target point. Then, max fixation deviation f dm is defined as the maximum value of 85th percentile of f d over the nine target points, and max fixation contrast f cm is defined as the maximum value of 85th percentile of f c over the nine target points. f dm and f cm can be formulated as follows: and where f (i) dp and f (i) cp represent, respectively, the 85th percentiles of f d and f c for the ith target point.
We can have strabismus occurrence feature f so by combining features f dp , f cp , f dm , and f cm as follows: where k 1 , k 2 , k 3 , and k 4 are coefficients used to weigh the four features. They can be determined according to practical requirements. In this Letter, k 1 = k 2 = k 3 = k 4 = 1. Constant α is used to bias the output, such that f so > 0 represents strabismus and f so < 0 represents normal. Constant α may be calculated by different algorithms. The algorithm we adopt is to compute f dp + f cp + f dm + f cm for both normal subjects and strabismic subjects. Moreover, then the largest value of normal subjects and the smallest value of strabismic subjects are selected. Their average value is assigned to α. We calculate α based on the data attained. This mechanism is reasonable, as many medical and health indicators are decided by analysing a large number of healthy and unhealthy people's data. With strabismus occurrence feature f so defined in (7), the decision function is finally defined as where F so = 1 indicates the subject has strabismus and F so = 0 indicates the subject is normal.

Strabismic eye feature:
After the subject is diagnosed to have strabismus with strabismus occurrence feature, the next step is to work out which eye is strabismic. The feature used to solve the problem is called strabismic eye feature, denoted by f se . Its definition is as follows: where d (i) l and d (i) r are the Euclidean distances between the target point and the left gaze point and right gaze point of the ith gaze pair, respectively. The formulas of d (i) l and d (i) r are the same in (3) and (4). N is the total number of gaze pairs acquired for the subject. The decision function of feature f se is Function F se = 1 means that the right eye fixates better than the left eye, and thus the left eye is a strabismic eye, and F se = 0 indicates the right eye is a strabismic eye.

Strabismus direction feature:
To determine which one of the four types (horizontal exotropia and esotropia and vertical hypertropia and hypotropia) the strabismus is, we should identify its direction (horizontal or vertical) first. The angle between the line connecting the two gaze points of a gaze pair and the horizontal line is utilised to estimate the strabismus direction. We call the angle strabismus direction feature, which is defined as follows: where F sd = 1 denotes horizontal strabismus, F sd = 0 denotes horizontal-prior strabismus, and F sd = −1 denotes vertical strabismus. The decision result here is three-fold rather than two-fold. One big advantage of a three-fold decision is that it can provide more detailed information to ophthalmologists in treating strabismus, e.g. strabismus surgery. For example, if the decision result is F sd = 0, the ophthalmologist knows that the patient has both horizontal and vertical strabismuses. In real diagnosis, f sd will be also used as an important indicator of strabismus evaluation.

Horizontal strabismus feature:
The horizontal strabismus feature used to distinguish exotropia and esotropia is defined as where x (i) l and x (i) r are the x values of the left and right gaze points of the ith gaze pair. N is the total number of gaze pairs. Bias β is calculated by averaging (x (i) r − x (i) l ) over all the normal subjects. The decision function is where F hs = 1 is exotropia and F hs = 0 is esotropia. The magnitude of f hs is a measurement of severity. A smaller magnitude of f hs indicates a smaller severity of strabismus.
3.2.5 Vertical strabismus feature: The vertical strabismus feature used to classify hypertropia and hypotropia is defined as where y (i) l and y (i) r are the y values of the left and right gaze points of the ith gaze pair. N is again the total number of gaze pairs. f vs has a similar form with f hs , except for bias β. Bias β is used to eliminate the horizontal distance produced by pupil distance. f vs tells the relative position of the two eyes horizontally. However, if we want to determine hypertropia or hypotropia, we need to find out which eye is a strabismic eye. In other words, we have to take into account both f vs and f se in order to make the decision. Mathematically, the decision function of vertical strabismus is defined as follows: where F vs = 1 is hypertropia and F vs = 0 is hypotropia. Similar to f hs , the magnitude of f vs measures the severity of vertical strabismus, with a smaller value denoting a smaller severity.
3.2.6 Diagnostic procedure: Fig. 5 depicts the whole diagnostic procedure using the proposed features extracted from the gaze data of a subject. First of all, strabismus occurrence feature f so is used to estimate whether strabismus occurs or not. If strabismus occurs, strabismic eye feature f se is exploited to figure out which eye is a strabismic eye. After that, strabismus direction feature f sd is employed to determine the direction of strabismus, i.e. horizontal or vertical. If the result is horizontal, horizontal strabismus feature f hs is used to work out the strabismus type, i.e. exotropia or esotropia. Otherwise, vertical strabismus feature f vs is used to work out the vertical strabismus type, i.e. exotropia or esotropia. Our system can also provide relative severity of strabismus. The severity of strabismus can be measured by horizontal strabismus feature f hs and vertical strabismus feature f vs . A larger value of these features indicates a more severe strabismus. In particular, f hs and f vs can be used to compare the severity of the same types of strabismuses.

Experiments: After ethics approval from Human Research
Ethics Committee of Chu Hai College of Higher Education and informed consent, we collected gaze data from 15 normal subjects and 10 strabismic subjects. The age range of normal subjects is from 7 to 40, whereas the age range of strabismic subjects is from 3 to 63. The mean ages are 30 and 40 for normal subjects and strabismic subjects, respectively. Both male and female persons are involved. The numbers of females are 9 and 6 for normal subjects and strabismic subjects, respectively. We denote 15 normal subjects as N 1 -N 15 and 10 strabismic subjects as S 1 -S 10 . The strabismic subjects have been diagnosed by professional ophthalmologists, and the results are used as ground truths. The ground truths of ten strabismic subjects are tabulated in Table 1. Table 1 reports strabismic eye, strabismus direction, type, and degree, such that we can have a clear comparison between the results generated by our system and the ground truths. The severity of strabismus for ten strabismic subjects can be roughly ranked as S 1 > S 8 > S 4 > S 2 > S 5 = S 7 > S 3 = S 9 > S 6 = S 10 .

Experimental results:
Strabismus occurrence feature f so is shown in Fig. 6. Bias α = 0.428. All the strabismic subjects have positive values, and all the normal subjects have negative values. Evidently, the strabismus occurrence feature f so defined in (7) separates the normal data and strabismic data well. This indicates that feature f so can effectively characterise the difference between gaze data of normal subjects and strabismic subjects. With a simple calculation, we can know that the mean values of f so are −0.2 and 0.67 for normal subjects and strabismic subjects, and the standard deviations are 0.06 and 0.37 for normal subjects and strabismic subjects, respectively. This signifies that the intradifference of normal subjects is much smaller than that of strabismic subjects, and the values of f so are pretty diverse for strabismic subjects. To evaluate the difference between normal data and strabismic data statistically, we perform a t-test for the two groups of data. The p-value is 4.7 × 10 −9 , much smaller than 5% significant level, which means that the f so values of normal subjects and strabismic subjects are significantly different. This indicates that feature f so is an effective indicator to separate normal subjects and strabismic subjects. After strabismus is confirmed, strabismic eye feature f se is adopted to find out the strabismic eyes for ten strabismic subjects. Fig. 7 depicts the f se values for ten strabismic subjects. Feature f se > 0 indicates left eye is a strabismic eye. Therefore, we can see that for subjects S 1 , S 2 , S 6 , S 9 , and S 10 , their left eyes are strabismic eyes, and for subjects S 3 , S 4 , S 5 , S 7 , and S 8 , their right eyes are strabismic eyes. The results are consistent with the ground truths as shown in Table 1. We perform a t-test for the two groups (left strabismic eye and right strabismic eye). The p-value is 0.0088, demonstrating that the feature f se of two groups are significantly different. Note that the magnitude of f se signifies the fixation difference between the left eye and right eye. A smaller |f se | stands for more similar fixation accuracies that the two eyes have. To further investigate that we show |f se | for 15 normal subjects in Fig. 8. One important finding is that all the normal subjects have a small magnitude of f se . The largest one is not larger than 0.025. This is because those normal subjects possess good fixation capabilities for both eyes.
The next step is to find out strabismus direction, e.g. horizontal or vertical, using feature f sd . Fig. 9 depicts the values of f sd for ten strabismic subjects. The three intervals of f sd are differentiated by three colours in the sector. We can estimate that the strabismus directions of S 1 and S 9 are vertical, S 8 is horizontal, S 2 , S 3 , S 4 , S 5 , S 6 , S 7 , and S 10 are horizontal prior, which could be finally classified to horizontal. In real diagnosis, the value of f sd will be outputted to the doctors for reference. The p-value of t-test for the vertical and horizontal (including horizontal prior) data is 0.008, which demonstrates the significant difference between vertical and horizontal data, and the effectiveness of feature f sd in representing strabismus direction.
We can then use horizontal strabismus feature f hs or vertical strabismus feature f vs to work out strabismus type. Let us investigate S 2 , S 3 , S 4 , S 5 , S 6 , S 7 , S 8 , and S 10 that have been classified to horizontal strabismus first. Fig. 10 shows their f hs . Obviously, S 3 , S 4 , and S 10 are exotropia, and S 2 , S 5 , S 6 , S 7 , and S 8 are esotropia. The results are consistent with Table 1. Their p-value is 0.08, larger than 5% significant level. In other words, the difference between f hs value of exotropia and that of esotropia is not significant enough. The  6 Strabismus occurrence feature f so of 15 normal subjects and 10 strabismic subjects Fig. 7 Strabismic eye feature f se of ten strabismic subjects magnitude of f hs measures strabismus severity. A larger magnitude of f hs represents a more serious strabismus. The subjects' strabismus severity can be thus ranked as S 8 > S 4 > S 2 > S 5 > S 7 > S 3 > S 10 > S 6 . As shown in Table 1, the ground truth order is S 8 > S 4 > S 2 > S 5 = S 7 > S 3 > S 10 = S 6 . The difference between the order generated by our system and the ground truth is that S 5 and S 7 and S 10 and S 6 are diagnosed by ophthalmologists to have the same degrees, while our system diagnoses that S 5 is more severe than S 7 and S 10 is severe than S 6 . Fig. 11 shows vertical strabismus feature f vs of S 1 and S 9 . Note that vertical strabismus type is determined by the product of f se and f vs in (16). It can be calculated from Figs. 7 and 11 that f se × f vs > 0 for both S 1 and S 9 . According to (16), S 1 and S 9 are hypertropias. Moreover, the magnitude of f vs signifies that the severity ranking is S 1 > S 9 . This is consistent with the ground truth.
Overall, the experimental results demonstrate that our method is able to diagnose out all the normal and strabismic subjects using eye-tracking data. The results are consistent with ophthalmologist's diagnosis reports.

Conclusion:
In this Letter, we propose and develop an intelligent digital system based on eye-tracking technique for objective and automatic strabismus diagnosis. An eye-tracking system is first developed to acquire people's gaze data. A group of features are then proposed to characterise the gaze data. On the basis of these features, the proposed system is able to tell whether or not the subject has strabismus, which eye is strabismic, what type of strabismus it is, and the severity of strabismus. We build a gaze dataset that contains both normal and strabismic gaze data for experiments. The results generated by our system are consistent with those diagnosed by ophthalmologists. All experimental results have demonstrated the effectiveness of our method. More importantly, the proposed system is objective, non-invasive, and automatic. It could be a potential powerful alternative for strabismus diagnosis.
Compared to other automatic strabismus diagnosis systems, one major advantage of our system is its portability and convenience. We can easily bring the system to different places to conduct strabismus examination for people. The setup is very simple; meanwhile, high accuracy can be still achieved for diagnosing strabismus occurrence and types. The limitation of our system is that it cannot yet precisely measure the strabismus angle as the cover test with a prism, though the relative severity is quite accurate. In the future work, we will develop more effective features for precise strabismus degree evaluation.
6. Acknowledgments: The authors express their sincere gratitude to the Hong Kong Association of Squint and Double Vision Sufferers and Department of Optometry of Shanghai Tenth People's Hospital, for helping collect gaze data. This work was fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Fig. 11 Vertical strabismus feature f vs of two subjects that have vertical strabismuses