Computer vision for system protection of elevators

With rapid development of computer vision (CV), traditional elevator control and protection technology have begun to integrate with CV technology in order to improve safety of elevators. By introducing CV technology, the real-time state monitoring of elevator components is realized, the elevator door control system based on CV is proposed, and the abnormal behavior of passengers in the elevator car is identified and alarmed. Based on summarizing the basic principles of CV technology, this paper summarized the application of CV technology in elevator safety from two aspects: equipment safety and passenger safety. Finally, the future trends of CV-based system protection of elevators are discussed.


Introduction
Elevators have become indispensable transportation equipment nowadays, and their safety is closely related to people's life safety. How to improve elevator safety has become one of the issues that people care about. From the perspective of system safety, studying and improving the safety of elevators has been a topic of long-term research by many researchers. Traditional system protection methods of elevators focus on electronic and mechanic aspects [1]. For example, by setting the speed limiter, the elevator car can be braked urgently when the traction rope is broken. The elevator control circuit allows the elevator car to move only when all doors are closed. This type of method has disadvantages such as serious failure, difficult maintenance, and high maintenance cost.
With development of computer vision (CV) technology, target detection, feature extraction and abnormal behavior recognition have been applied in elevator intelligent detection [2] and elevator group control systems [3], which provides a new idea for system protection of elevators. Moreover, CV-based elevator robot automatic navigation technology [4] and environmental detection technology [5] have also become the key technologies of elevator assistant protection system. The application of CV technology for elevator protection enables elevators to operate in a more efficient and intelligent way, which not only reduces the operation and maintenance costs of elevators, but also solves the problems of elevator out-of-control during interaction with people [6], avoiding the elevator accidents. Although the application of CV technology has improved the safety of elevators, as special equipment closely related to people's lives, elevators still have potential risks. Therefore, this article aims to summarize the application status of CV technology in elevator protection, discuss the future trends of CV-based system protection technology for elevators, and provide some references and suggestions for related research.
Herein, we aim to present a research summary of current application of CV technology in system protection of elevators from both theoretical methods and practical applications. The rest of this paper is organized as follows. Section 2 introduces the main CV technology principles currently applied in 2 system protection of elevators. Section 3 reviews the application of these CV technology among different systems and different objects of elevators. The development trends of CV technology in system protection of elevators is proposed in Section 4. The conclusion is drawn in Section 5.

Theoretical background of CV technology
2.1. Image preprocessing 2.1.1. Grayscale methods. Grayscale is the basis of image processing, and the original RGB image is often converted to a grayscale image during actual processing.

Histogram equalization.
Histogram equalization is to nonlinearly stretch the image and redistribute the number of pixels in each gray level of the image, so that the number of pixels in a certain gray range is approximately equal, thereby improving the contrast of the image [10]. This technology is simple in the calculation and excellent in performance, and has been widely used in real-time image processing algorithms. The variable r is used to represent the gray level in the original image, and the gray level is normalized, that is, 0 ≤ ≤ 1, where = 0 represents black and = 1 represents white. For a given image, the total number of pixels is n, and the function for the image histogram equalization is [11]: The calculation result, , is the converted original image gray level.

Linear filtering noise reduction.
For various reasons, various types of noise are always generated during the image acquisition process, so it is extremely important to select an appropriate filtering algorithm in the image preprocessing technique. In image processing, when the calculation of pixels in the neighborhood is linear, if the window function is used for smooth weighted summation or some kinds of convolution operation, the processing is called linear filtering.
(1) Mean filtering algorithm Mean filtering is generally a convolution operation using a coefficient template and an image. The formula is [12]: where ( , ) is the gray value of the point after the mean filtering, n is related to the size of the coefficient template. The coefficient template is generally a matrix with 3*3 elements being 1, and is 9.
(2) Gaussian filtering algorithm Gaussian filtering is generally aimed at Gaussian noise, which can suppress the noise randomly introduced during image input and regards pixel points and neighborhood pixels as a Gaussian  = * (6) Where is a two-dimensional Gaussian kernel with a standard deviation of : 2.1.4. Nonlinear filtering noise reduction. Nonlinear filtering uses the logical relationship between the original image and the template to get the results of filtering, mainly including median filters and bilateral filters.
(1) Median filtering algorithm The median filter sorts all the pixels in the window function to obtain the median to represent the pixel value in the center of the window [14]. It suppresses the salt-and-pepper noise and the impulse noise particularly well while retaining the edge details. The formula is [14]: ( , ) = ( ( , )) , ∈ ℎ (8) (2) Bilateral filtering algorithm Bilateral filtering is also a nonlinear filtering method, which is a compromise processing between spatial proximity and pixel similarity of the image [15]. While considering the spatial information and gray similarity, it can achieve the purpose of edge preserving and denoising, which is simple, noniterative and local. It has one more Gaussian variance than Gaussian filtering, and the formula is as follows: Where ( , ) is the weighting coefficient, which depends on the product the kernel of the function definition domain and the kernel of the range.

Sharpening filtering noise reduction.
After the image is collected, blurring may occur due to shaking, light irradiation, etc., and local details cannot be meticulously reflected. Therefore, the introduction of sharpening filtering can efficiently restore image details and simplify the difficulty of further analysis and research. The commonly used two-dimensional image sharpening algorithms is Laplacian sharpening [16].
The expression of Laplacian sharpening is: ( , ) = { ( , ) − ∇ 2 ( , ); Laplace template center value is negative ( , ) + ∇ 2 ( , ); Laplace template center value is positive Where ∇ 2 is the Laplace operator of second order function: The characteristics of the above filtering algorithms are shown in Table 1. The effects of the various image preprocessing techniques mentioned above are shown in Figure 1.

Feature extraction
Usually, it is necessary to extract the features of the specific part of the object displayed in the image after obtaining the image of the specific position of the elevator system. Separating the parts to be studied that may pose a safety hazard from the background information in the image, thereby improving the analysis efficiency, and then specifically determining the feature information of the component be studied and discovering the safety hazard of the elevator system and responding in an emergency. The methods of feature extraction in system protection of elevators include edge detection, threshold segmentation and corner detection.

Edge detection.
Edge detection is a common method for determining image boundaries based on gray-scale mutations to achieve the target and background segmentation. The principle of edge detection is based on the discontinuity of gray-scale changes in the image. The steps of edge detection are shown in Figure 2. The gradient operators commonly used in edge detection are Roberts operator [17], Prewitt operator [18], and Sobel operator [19].

Threshold segmentation. (1) Grayscale based threshold segmentation
Threshold processing is simple and efficient, and the original image can be segmented effectively by the distribution of gray elements in the image, and then the target object in the image can be extracted.
Let the grayscale distribution of the elements of the original image be ( , ), then the visible image can be divided into two parts according to the threshold T. The divided image ( , ) is defined as follows [20]: (2) Best global threshold processing based on Otsu The above-mentioned methods are capable of characterizing significant differences between objects and the background. In practical applications, it is often necessary to process a series of images, and the same object can automatically perform threshold estimation, that is, a global threshold processing algorithm for each image, where the same object may have slight differences and the background may also be different [8]. The processing flow of this method is shown in Figure 3. Threshold processing can be considered as a statistical decision problem, the purpose is to minimize the average error as an important basis for grouping elements. The Otsu algorithm is an excellent solution [21]. It only needs the gray histogram of the image and utilizes the huge difference of the gray values of different targets, so that the final classification class difference is the largest, so the threshold of the best class separation will be the optimal threshold. The principle of the algorithm is shown in Figure 4.

Set value as global threshold
Set ∆ as predefined parameter 1 : set of all pixels larger than T 2 : set of all pixels smaller than T Start 1 : the average gray value of 1 2 : the average gray value of 2 :  In the last step in Figure 4, the value of that maximizes the variance 2 ( ) between the classes is the Ostu threshold * . When the maximum value is not unique, all the obtained maximum values are averaged to obtain * .
Threshold processing is simple to implement and works well, but even small noise can make threshold segmentation a problem that is extremely difficult to solve. In practical applications, image smoothing techniques, image edges, etc. are often used to improve the effectiveness of global threshold processing.

Corner detection.
A corner point is the intersection of two sides, usually above the boundary of two different areas. When the window moves through the image, the corners have large grayscale changes in any direction. Corner detection is a more common method of acquiring image features and can also be called feature point detection. The commonly used corner detection algorithm is Harris corner detection [22]. In practical applications, corner detection often combines with the kinetic energy method to judge the abnormal behavior of people in the elevator car.

Convolutional neural network
Convolutional neural network (CNN) is a deep feedforward neural network that includes convolution calculations. The deep convolutional neural networks can effectively classify images and achieve excellent results [23]. Convolutional neural networks generally include an input layer, convolutional layers, pooling layers, fully connected layers (also known as dense layers) and an output layer. The general structure of a convolutional neural network is shown in Figure 5.

Application of CV in system protection of elevators
The application of CV in elevator protection mainly includes two aspects: protection of elevator equipment and protection of elevator passengers. The protection of elevator equipment is concentrated in power system and door control system, including defect identification and diagnosis of elevator components, state detection of elevator components, and elevator door control technology based on CV. The protection of elevator passengers is mainly conducted by identifying and warning their abnormal behaviors, and stopping their dangerous behavior in time, so as to ensure the safety of elevator passengers.

CV for elevator protection
3.1.1. Power system. In this paper, the elevator power system refers to the system that provides kinetic energy for the elevator to make the elevator run normally. It includes the traction system of the traction elevator, the pumping station and hydraulic system of the hydraulic elevator. The elevator traction system is mainly composed of a traction motor, traction wire rope, guide wheel, and an electromagnetic brake, etc. Among them, the traction wire rope is the key to make the elevator car reciprocate up and down, which has high-frequently use and tends to wear. The traditional detection method requires the staff to perform visual identification, determine the wear degree of the wire rope, and decide whether it should be replaced [24]. This method has high labor intensity and strong subjectivity, and its accuracy is largely affected by the experience of the staff. Meanwhile, the traditional method can only detect the traction wire rope regularly but can't monitor it in real-time in a continuous period. However, CV technology can effectively realize the real-time monitoring of the wire rope to prevent the occurrence of unexpected situations.
O. Yaman et al. [25] summarizes the common four damage states (shown in Figure 6) of hoisting ropes: deformity, knitting in the reverse direction, broken wire, and inner wire out. The Sobel edge detection algorithm is used to extract the edge of the traction rope converted from a grayscale image, and the video is extracted between frames to construct the autocorrelation value of the time series. The results show that when the autocorrelation value is abrupt, it represents the failure of the traction rope. Dai et al. [26] expands the types of wire rope defects to eight categories, including cage distortion, strand extrusion, wire extrusion, local increase in rope diameter and kink. The rope diameter is locally reduced, flattened and bent. Meanwhile, an image feature fusion discrimination method based on image sharpening filtering and binary threshold segmentation is proposed. The result shows that the detection accuracy of their method is 99.99%. Z.N. Hou et al. [27] uses the maximum inter-class difference method to obtain the image after binarization of the image, and then uses the corner detection method to identify the faulty part of the wire rope and utilize it. This technology builds a real-time monitoring system for elevator traction ropes. In the power system, the CV technology is not only used to monitor the defects of the traction wire rope, but also used by some researchers [28] to detect the motion state of the elevator hydraulic buffer, as shown in Figure 7. This system captures the movement process of the hydraulic buffer by the camera and exchanges the information inside and outside the well by the wireless transmission technology. After obtaining the video and image of the hydraulic buffer moving process, the image is processed, including image sharpening, color threshold filtering, effective interval calculation, edge feature extraction, and compression calculation. In this way, the reset time and reset distance of the elevator hydraulic buffer can be detected, and the dynamic compression and reset process of the elevator hydraulic buffer can be grasped. In addition, this system also has many advantages, such as wide versatility, convenient installation, a high degree of intelligence, little influence of human factors, and no risk for inspectors in the measurement process.
3.1.2. Door control system. The elevator door system is one of the most frequently used systems with the most frequent failures. If the door system is not well controlled, it is easy to catch people and fail to open the door in time. To improve the safety of elevator door system, the CV-based elevator door system control technology is being popularized gradually.
Elevator door edge detection technology is the front part of a CV-based elevator security system. Li et al. [29] detected the edge of an elevator door by the edge detection technology, and compared the differences of Roberts operator, Sobel operator, Prewitt operator, and Canny operator in elevator door To improve the safety of the elevator door system, another important feature of the elevator door control system based on CV technology is the recognition of moving objects in the elevator door area. Zeng et al. [30] developed an elevator door control system based on image processing technology. This system uses Semi-Neighborhood Averaging Algorithm to deal with the smooth noise in the image to get a clear edge of the object. Then the Background Difference (BD) method is used to identify the target object between the background image and the current image. The next year, Zeng et al. [31] improved the method mentioned in [30]. They used the Otsu threshold segmentation method to binarize the image, and then used the Frame Difference (FD) method to identify the moving objects between the doors. Zheng et al. [32] also proposed an elevator door security protection method based on moving image recognition. In this method, the median filtering method is used to ensure the sharpness of the image, and then the BD method and the Time Difference (TD) method are used to establish the dynamic target recognition model. Finally, the Otsu method is used to successfully detect the movement of the dynamic target, thus realizing the construction of the door protection system. In addition, Zhang [33] proposed an anti-pinch detection algorithm to detect the objects in the elevator door area. The algorithm uses an improved bilateral filtering method to remove the noise of the video frame and the background image, then calculates the difference image between the video frame and the background image, and uses the maximum entropy particle swarm algorithm to obtain the binary image of the difference image. Finally, the processed image is filtered to achieve target recognition, and the Surendra algorithm is used to update the background image.
Zhou et al. [34] applied CV technology to the detection of the meshing depth of the elevator door lock and established the detection system of the meshing depth of the elevator door lock. In this system, the image segmentation technology of HSV (Hue Saturation Value) color space is used to detect the morphological features of the image, and then the hierarchical Hough transform is used to recognize and pinpoint the tick marks, which are accumulated to get the meshing depth of the door lock. The simulation results show that the accuracy of the system in identifying the meshing depth of the elevator door lock is 93.8%.
The application of CV technology in various functions of the elevator door system is summarized in Table 2. Histogram normalization Otsu segmentation and FD [31] Median filtering BD, TD and Otsu [32] Improved bilateral filtering Maximal entropy particle swarm algorithm for improving algorithm and Surendra algorithm [33]

CV for elevator passenger protection
According to [35], most elevator accidents are related to people, and the elevator door system has the most interaction with people. Therefore, it is very important to detect the abnormal conditions related to people in elevators and make corresponding safety protection measures. The application of CV in elevator passenger safety protection mainly includes passenger flow monitoring and abnormal behavior detection.  [36] and combined with the characteristics of SVM combined with the image acquisition angle of the crowd to achieve the number of waiting for people and get an accurate judgment. Zhao et al. [37] mainly used YOLOv3 (You Only Look Once) to count passengers with SVM detecting the current floor. Liu et al. [38] used Adaboost to train a classifier based on Haar-like features. Tsuji et al. [39] used BD method to extract moving objects, FD method to detect the human head by Hough transform, optical flow method to track human body, and count the number of people in elevator car. Wang et al. [40] put forward an algorithm that combines the BD method and FD method to extract the human body and contour accumulative is used to track the human body. Fan et al. [41] and Narayanan et al. [42] mainly used Canny edge detection to determine the number of people waiting for the elevator and the former construct an intelligent elevator management system. Adaboost Harr-like feature [38] BD and FD Optical flow [39] BD and FD Contour accumulative [40] BD Edge detection by Canny operator [41] RGB image to Gray image Edge detection by Canny operator [42] The abnormal behavior detection of elevator passengers based on CV has great significance to ensure the personal safety of elevator passengers. Jin et al. [43] proposed anomalous behavior detection in elevator cars based on the corner kinetic energy method, especially applying Hausdorff distance to edge detection and combining the optical flow method with the corner detection. When the movement amplitude of the person suddenly becomes larger, the angular kinetic energy will also change abruptly, and then the alarm will be issued. While Tang et al. [44] use K-Means clustering to get objects and Hidden Markov Model (HMMs) for detecting people's normal activity. Y. Lee et al. use the optical flow method to identify the violence in the elevator [45]. The framework and the logic flow chart of the violent event detection in the elevator are shown in Figure 8. To get a better target extraction effect, Ma [46] introduced the background update rule into the background difference method. The elevator passenger abnormal behavior detection method proposed analyzed the corners that can reflect the image features, and used the Lucas-Kanade (L-K) optical flow algorithm based on the Shi-Tomasi corner to judge whether there are violent behaviors. For the behavior of opening the elevator door with bare hands, the two-dimensional attitude evaluation method based on PAFs has achieved good results. Shu et al. [47] also use L-K optical flow to extract velocity information and then establish the feature vector, including the information of corner kinetic energy and feature of targets. Zhu et al. [48] use the background difference method based on the color background model to detect moving targets in the elevator car. At the same time, a fall detection method based on the detection window and a violence detection method based on the Motion History Image (MHI) were proposed. The energy function detection algorithm was introduced into elevator passenger abnormal behavior detection in [49].
The main algorithms applied in abnormal behavior detection are summarized in Table 4. K-Means clustering Hidden Markov Model (HMMs) [44] Foreground segmentation Optical flow [45] BD with background update rule L-K optical flow based on Shi-Tomasi corner [46] Gaussian mixture background model L-K optical flow and SVM for classification [47] BD with color background model Motion History Image (MHI) method [48] BD with color background model Motion History Image (MHI) method and energy function detection algorithm [49] 4. Future trends CV technologies based on traditional image recognition technology have been widely used in elevator power protection systems, door control systems and passenger protection. However, the traditional CV technology, which takes image preprocessing, feature extraction and target detection as the basic process, has disadvantages such as high probability of misjudgment and limited application scenarios. For example, (i) the fineness of the image data in the traction system directly affects the judgment results of the corresponding components; (ii) the research of CV technology in the door control system mainly focuses on the response speed of the control system, and the algorithm accuracy is low; (iii) CV technology still faces problems such as background influence and occlusion in the task of judging the number of elevator passengers; (iv) in the task of judging the abnormal behavior of passengers in the elevator car, this technology still stays at the identification of a few behaviors, and cannot make more accurate judgment of abnormal behaviors. With continuous development of artificial intelligence technology, CV technology will also be constantly innovating. Some future trends in system protection of elevators are as follows: (1) Smarter CV technology Most of the existing CV technologies applied in system protection of elevators still require artificial feature extraction design, and the commonly used feature extraction methods have been introduced in section 2. The deep learning technology in artificial intelligence (AI) technology changes the traditional feature extraction mode, enabling the model to automatically perform feature extraction. Integrating image processing technology with AI technology to develop smarter CV technology can not only be applied to system protection of elevators, but also to other fields, such as medical image inspection.
(2) Machine vision technology combining software and hardware The landing of the algorithm needs to be combined with the hardware. In system protection of elevators, combining CV technology with the video monitor in the elevator car to develop intelligent security equipment suitable for elevators is an important direction for future system protection of elevators. At the same time, combining CV technology with robot technology and developing intelligent elevator service robots is also an important development direction for system protection of elevators. In fact, many researchers [50] have applied CV technology to elevator button recognition. Elevator intelligent service robots not only needs to have the ability to recognize elevator buttons, but also needs to have visual capabilities such as judgment of the number of passengers and passenger behavior recognition. At the same time, the hardware design of the elevator intelligent robot needs to meet the principles of safety, practicality, interaction, intelligence and low cost. In addition, the research and development of intelligent elevators based on CV technology to make elevators have vision is also a major technical field that can be explored in the future.

Conclusion
System protection of elevator is of great significance to improve elevator safety performance and ensure people's life safety. The traditional protection ways for elevators mainly focus on electronic and mechanic aspects, but have some problems such as failure and high maintenance cost. CV technology brings a new scheme for elevator protection. This paper introduces theoretical background of CV technology and summarizes the application of CV technology in elevator protection from two aspects: equipment safety and passenger safety. Finally, the future trends of elevator protection are discussed.
The existing CV-based elevator protection technology has disadvantages such as a high possibility of misjudgment and limited application scenarios. With the development of CV technology, it is believed that smarter CV technology will be applied in system protection of elevators. Furthermore, machine vision technology that combines CV algorithm with hardware technology will be the main research trend of system protection methods for elevators.