Application of ImageJ program for the Analysis of Pupil Reaction in Security Systems

One of the tools of eye tracking technology is the eyetracker, which recognizes and records the position of the pupils and eye movements. The existing interconnection of saccades, microsaccades, the center of attention with the diameter of the pupils and the psychophysical state of a person gives prerequisites for the use of pupillography in security systems controlled by artificial intelligence. It is important to solve the problem of accurately distinguishing pupil boundaries. An algorithm for pupil extraction using the ImageJ program has been developed.


Introduction
Today, an important aspect of modernity is security [1]. Flight safety is especially important. Of particular note is air transport terrorism. In recent years, the number of air attacks has increased [2]. To ensure security, the latest technologies are introduced. For example, iris authentication will soon become ubiquitous technology. To register eye movements, electrooculography, photo-optical and electromagnetic methods related to contact methods are used, as well as photoelectric and recently received widespread video recording methods related to non-contact methods. One of the most important indicators of oculomotor functions is eye micromotion (microaccades) -fast eye movements lasting 10-20 ms, amplitude range 2′ -50 ′. Previously it was believed that micro-saccades only complicate the analysis of oculograms, and are a source of artifacts in electroencephalograms. However, recent studies have shown the relationship of microaccades with cognitively modulated brain potentials, and may serve as non-intrusive attention probes [3]. Later it was found that microsaccades restore the visibility of a stationary target in a wide range of spatial frequencies, with larger micro-saccades increasing the visibility of the target more efficiently than smaller micro-saccades [4]. When studying the relationship of microaccades in macaques with brain potentials that regulate the flow and processing of visual information, a narrow-band spatially specific response to the stimulus in the gamma range  and microcacades associated with visual fixation (3-4 Hz) were found. [5]. This suggests that the way in which information is transmitted and integrated between the early visual cortical layers depends on the time of microaccades. Further study of microsaccades is important not only from a clinical point of view, but also for the study of visual processing of information [6]. Thus, the task becomes to accurately track pupil size and eye movement. One of the tools of eye tracking technology is the eyetracker, which recognizes and records the position of the pupils and eye movements. Most of the works devoted to the use of eye-tracking technologies analyze the focus of attention when performing technical actions (including ignoring visual stimuli), visual search strategies in the process of activity, measuring the diameter of the pupil (as an indicator of cognitive load), the number of saccadic eye movements, and also fixations, blinks and other parameters [7]. All these parameters can be tracked using the method of pupillometry. This method, aimed at determining the size of the pupil, is widely used in toxicological and ophthalmic practice, reflects both quantitative and qualitative changes that occur in the body. The study of pupilograms as an indicator of emotional response showed that the pupil size was significantly larger with emotionally negative and positive stimuli than with neutral stimuli [8]. Later, a connection was established between the increase in the amplitude of low-frequency fluctuations in the diameter of the pupil and the drowsiness, the effect of the depressive state on the pupil reaction was proved, the conditions for using the pupil reaction as correlates were found [9][10][11], and the characteristics of the main emotional stimuli were determined, descriptive statistical indicators of pupil dilatation during emotional load and the fact of correlation with breathing and heart rate has been established [11]. At this stage of the study, the presence of a relationship between microsaccades and attention with cognitively modulated brain potentials, pupil diameter and the psychophysical state of a person was established. The range of uses for eyetracker and pupillometry is quite wide. In the future, the established relationship can be used in artificial intelligence systems to detect drowsiness, diagnose various clinical conditions or recognize the iris, cognitive and behavioral therapy, visual search, advertising, analysis in security systems [12,13]. The aim of the work is to develop a pupil contouring algorithm using the ImageJ program.

Overview
The problem of accurate pupil allocation is associated with contouring. In many cases, it is very important to highlight the pupil, track its size and track. The complexity of the contouring of the pupil and its analysis arises, first of all, when identifying a person. To solve it, a large number of different methods have been created. We can call the morphological selection of the circle [14], the projection of brightness or its gradient [15], and the construction of the optimal contour [16]. In the morphological approach and methods based on brightness projections, binarization of the image with a certain threshold is used, after which the largest connected set of points with brightness below the threshold is considered, based on the assumption that the pupil is the darkest region in the image of the eye [17]. The morphological approach allows you to remove glare and extraneous noise in the image, and the Canny operator can help with the selection of boundary points corresponding to the boundaries of the pupil and iris. The Canny operator [18] smooths the image to suppress noise and marks the boundaries at those points at which the brightness gradient takes its maximum value. The advantage of the morphological method is the speed of execution, since the boundaries allocated during binarization consist of a small number of points. The imperfection of this method is a significant number of errors associated with distortion of the shape of objects. There is also a Hough transform based method. Its use allows you to correctly select the center, radius and contour of the pupil, even with partial visibility of the contour of the pupil in the image [19]. The imperfection of the method lies in the fact that a large amount of time is required when iterating through many connected components and several binarization thresholds. This defect is partially offset by the introduction of a quality attribute of the connected component. J. Daugman [20] uses the Daugman operator to find the coordinates of the center and radius of the pupil and iris of the eye, as well as the upper and lower eyelids. The operator searches in a circular path, changing the radius and coordinates of the center of the circular path, determining the maximum change in the brightness of pixels in the image. The imperfection of the algorithm is that it can lead to an erroneous determination of the boundaries of the pupil due to noise in the image. The active contour model was proposed by N. Ritter [21] for localization of the iris. Pre-processing included creating a dispersion image from the original image, and then detecting a ring of a given size. The active circuit responds to internal and external forces, deforming and moving around the image. To localize the pupil, the internal force is adjusted so that the contour forms a circle, and the external force takes into account information about the boundaries of the image. Tests are carried out until a balance is established between these forces. This is the imperfection of this model.  3 Kharitonov A.V. offers pupil localization using contour analysis [22]. The method is based on determining the jump in image brightness. Using the filter, smoothing is performed and the place of the largest jump is determined, which will correspond to the external radius. Eyelashes are cleaned on the image of the iris by analyzing the histogram of brightness. Too light or too dark places are unsuitable for analysis and are entered into the mask. The method imperfection is that places that are unsuitable for parameterization must be entered into the mask manually. The specificity of the optimal circular path method [23] is that this method detects a closed loop enclosing inside itself some given point, which is considered to be its approximate center. The optimal circular path method is useful in the task of refining the shape and position of the pupil already approximated by a circle. Then this method processes most of the noise: glare, eye shadow, blurriness of the image, partial shadowing over the centuries, partial overflow of the image. However, this method is not reliable enough even if the exact coordinate of the center of the pupil is known. Artificial neural networks are increasingly being used, which are the most effective tool for intelligent search and classification, as they have the ability to identify significant signs and hidden patterns in the analyzed data [24]. In addition, neural networks do not require significant computing resources and are quite resistant to errors and noise arising at the stages of data collection and processing. Such networks have the ability to identify patterns in the data. The high performance of ANNs during training and the formation of results when making decisions is also noted as a merit [25].

Methodology
A special helmet was used to calibrate the optoelectronic system, creating a rigid coordinate connection between the video camera and the head. For video shooting, we used a video camera with a video mode of 30 fps. The experiment was attended by 13 people. The age of all the subjects was 18-19 years, males -9, females -4. Eye diseases in the respondents were absent. All participants were warned in advance and voluntarily decided to participate in the experiment. Four slides depicting points that were located in different places on the screen (in the center, near the center, in the upper left corner and in the lower right corner) were used as a calibration object. The point size was 7.3 mm * 8.8 mm. The distance between the calibration object and the eye was at least 2.5 m. The total time of the entire study procedure takes no more than 5 minutes, which does not affect the respondent's fatigue, but is sufficient to track the track of the gaze. As a result, 10 of the best video files were selected. Processing and analysis of the results was carried out in two stages. First, the preparation of images, the processing and contouring of the pupils in ImageJ. This was necessary for further analysis of the obtained data: the coordinates of the trajectory of the pupil and its relative sizes. For statistical analysis, we used the StatPlus program to visualize OriginLab2019 data. For the obtained images, the following algorithm can be successfully applied. First of all, you need to trim the files, since they have a high resolution, and the computer will not be able to process them whole. It is necessary to select a portion of the image containing the pupil. The selection should be of sufficient size so that the pupil does not leave the selected area in any of the frames. Next, execute the Subtract Background command, which removes smooth solid backgrounds. This function can be used to correct uneven background lighting. We improve contrast in order to get rid of defects: glare from the light source or shadow from the eyelashes. After that, set the threshold Threshold. Using this tool, you can adjust the upper and lower threshold values to segment the region of interest and background image. The Remove Outliers tool replaces a pixel with the median of pixels in the environment if it deviates from the median by more than a certain value (threshold). The processing process is presented in Fig. 1. The images are then binarized and measured. In order to visualize the data, it is necessary to select the parameters that need to be measured in the Set Measurements tool: Area, Center of mass, Stack position. The Analyze Particles command allows you to measure selected parameters on the principle of scanning the entire image or selection until it finds the edges of the object. To do this, the following should be selected: Display results, Add to Manager, Include holes. The measurement results are presented in the text file Results. One of the advantages of the ImageJ program [26] is that the results can be presented in the form of a MS Excel spreadsheet for further calculations. The program also has a simple interface that allows researchers not to seek help from programming experts in analyzing the data. There is still the opportunity to automate the image processing. Often you have to contour images that have some defects. For example, with a strong darkening of the image around the pupil after binarization of the image, it is not possible to select the pupil, and therefore it is impossible to correctly determine its position. With strong noise in the image after the selection of boundary points, it is often not possible to correctly determine the radius of the pupil. To solve this problem, you can use the Noise tool, which can remove noise in the image. If the image of the pupil contains glare or reflection from a light source, then contouring of the pupil becomes almost impossible. To get rid of the shadow of eyelashes, you can change the intensity of the pixel: saturation, brightness, lightness. The use of various color settings will also help to solve this problem. For example, Split Channels splits an RGB image into three 8-bit grayscale images containing the red, green, and blue components of the original. However, the proposed processes take a large amount of time, which is why automation of the pupil contouring process is an urgent aspect. Having learned the parameter center of mass, you can determine the coordinates of the "center of attention". To do this, it is necessary to find the boundaries of the pupil at each frame, indicating the threshold (minimum brightness) by which the separation is carried out. We get a black and white image (Fig. 2).

Relativ pupil size
Pearson's correlation coefficient of relative pupil size and center of attention displacement due to the large size of the table is presented as a surface in Fig. 3. Thus, we can conclude that when the concentration of the eye on the calibration point without an emotional response, the change in the size of the pupils is caused by a shift in the center of attention and an error in contouring. When there is an emotional response to the test object, then the pupillogram should contain areas in which the change in the size of the pupils cannot be explained by a shift in the center of attention. We hypothesized that, provided there is no change in the emotional state, focusing the eye on the calibration object, the eye makes movements (saccades and microsaccades), leading to a change in the size of the pupil. Since test objects with different emotional colors are shown on the monitor, it is first necessary to find out the value of the maximum change in the size of the pupils, which can result in the movement of the gaze (shift of the center of attention) without experiencing emotions. This is necessary in order to understand whether the change in the size of the pupils was caused precisely by emotion. Thus, it is necessary to study the variation in the size of the pupils depending on the change in the position of the center of attention. A statistical analysis of the research results is presented in Fig.  4   Description and comparison of the relative size of the pupils without an emotional response and with an emotional response showed that they correspond to theoretical information about the reaction of the pupil. File processing in ImageJ allows you to distinguish between emotions that a person experience. At the first stage of the experiment, it is sufficient to apply contouring "manually", which occurs in a semi-automatic mode. Let Si be the ideal pupil highlight obtained by stretching the "circle" tool so that it covers the entire surface of the pupil without going beyond it. Such measurements can be done not so much, so they were used only to select the contouring method and verify the results. Then let S be the area of the pupil allocated in the semi-automatic mode (first, the optimal method is selected and applied to one of the frames of the video sequence, then it is automatically applied to all slides by pressing the "ok" button). I-average brightness (in grayscale) of the pixels entering the pupil area in the image of each frame. S, Si, I are normalized to the most common value. As you know, the size of a person's pupil is constantly changing even in the absence of emotions. Experiments have shown that the semi-automatic method is the most optimal, since it gives a small error and partially automates the work. In this case, the relative pupil size (without emotion) S \ ⟨S⟩ = 1.02 with a standard deviation of σ = 0.03, due to a shift in the center of attention, was [27]; with emotion "fear" S \ ⟨S⟩ = 1.24 with a standard deviation of σ = 0.30 due to emotion. It must be said that a steady feeling of hatred (relative pupil size S \ ⟨S⟩ = 1.01) leads to a deviation of σ = 0.12. The fraction of distortions introduced by contouring in the semi-automatic mode is an order of magnitude smaller [27]. Based on this, we can say that the method is quite applicable for determining the emotional state of a person using the pupillographic method.

Main conclusions
The paper considers the existing methods and algorithms for the contouring of the pupil. During the research, an algorithm was developed that allows one to distinguish the boundaries of the pupils and to monitor changes in their relative sizes. It was found that when you look at the calibration point without an emotional response, changes in the size of the pupils are caused by a shift in the center of attention and the contouring error; Calibration showed that a shift of the center of attention by 1 ° 38' causes the standard deviation of the pupil size by no more than σ = 0.09. An additional advantage of our proposed algorithm is that it can be used regardless of the amount of knowledge in programming.