Integration of intelligent diagnosis system and augmented reality for electric motors

With the rapid development of industrial automation technology, electric motor control has also become an important technology. However, motor failure has many possible causes, and it is not easy to detect in advance. Therefore the fault diagnosis is an important issue for motors. The augmented reality (AR) technology is also developing towards various industrial and educational application in recent years. This research is focused on developing the intelligent diagnosis system and AR application for electric motors. The intelligent diagnosis system applies cloud data management and machine learning methods to predict the health status of electric motors. And the time-recurrent Long Short-Term Memory (LSTM) neural network algorithm is applied to establish a motor health diagnostic model. The experimental results show that the motor diagnosis method can predict the health status of motor effectively. The proposed system will also provide more industrial application services in the future.


Introduction
Many devices in our life are related to the electric motors, such as the engine of a car, or electric fan and so on. It is also widely used in the machinery industry, electrical machinery industry, medical industry, etc., and also used in the field of automation control and electromechanical integration. The motor can convert electrical energy into mechanical energy to drive for vibration, rotation, and linear motion. There are many different types of motors, which are mainly composed of stators and rotors. The stator is stationary in the space, while the rotor can rotate around the shaft and is supported by bearings. One of the stator and the rotor can also be a permanent magnet.
When the motor is running, it will be accompanied by vibration. When the motor runs with abnormal vibration, it indicates that the faults may occur. For the motor, when it faults, it can be diagnosed through various vibration analysis methods. There are some researches discussed about motor fault diagnosis methods [1][2], and the authors proved that the sequential vibration signals are useful for the motor faults detection. The permanent magnet motor is another type of motor which commonly used in our daily life. Demagnetization fault is one of the faults existing in permanent magnet motors. There are also some demagnetization diagnosis methods have been proposed [3][4], such as finite element analysis (FEA) and time-frequency analysis (TFA), but these methods are difficult to use for real-time analysis. In the meantime, the rapid development of smart phones and various applications that can be used for life, work, and entertainment are constantly being developed and improved. With the massive developments of software and hardware, there are more Augmented Reality (AR) and Virtual Reality (VR) devices and applications, which have generated the motivation for users to enjoy new interactive experiences. At the same time, driven by this wave of AR, it is changing the landscape of operating modes in the fields of education, industry, medical, manufacturing and others [5]. In addition, due to the rising trend of the Industry 4.x, artificial intelligence (AI) technology will rapidly boost applications through AR, which will promptly increase the various types of services. The key technologies of AR are developing towards various applications as well. How to integrate AR services in cloud computing is also a hot research topic.
In addition, machine learning is the key to the next industrial revolution, and it can be used for motor fault diagnosis effectively. Machine learning can replace routine operations and the use of machine learning technology can instantly monitor equipment and predict maintenance needs [6][7]. Massive data systems can collect and analyze various possible variables and build a model to predict the behavior of the entire system. Using a large amount of sensor data in this way has a good help on the device's operating status and maintenance costs. The machine learning technologies can help to analyze and predict the possible faults of electric motors accurately.
This research is mainly based on the development of a machine learning based diagnosis system and AR technology for electric motors. The proposed method is applied to prevent the fault occur for higher power or permanent magnet motors, and combines the AR technology to provide the innovative application for users. The intelligent diagnosis system uses the LSTM neural network model [8] to establish an electric motor health diagnostic model. Based on the differences between the parameters such as the vibration, voltage, temperature, and flux linkage collected during the electric motor operating and the predicted response of the diagnostic model to determine whether the current system is healthy or not. The intelligent diagnosis system also uses cloud data server to monitor and predict the health and potential faults of motors, which provides a more convenient and efficient management and maintenance mechanism, and presents the combination of reality and virtualization operating experiences.
The rest of the paper is organized as follows: the related technologies of augmented reality and machine learning are described in section 2, and the electric motor diagnosis system design is presented in section 3, and the development of augmented reality application is presented in section 4. Finally, the conclusion and future works are presented in section 5.

Augmented reality
Industry and academia in various countries around the world have invested in AR research and development for many years [9]. How industry experts turning requirements into specifications, designs, and manufacturing industry-compliant applications remains an innovative challenge.
Augmented Reality is a technology that augments virtual information into real space. It does not replace real space, but adds a virtual object to the real space to augment the virtual information into reality. In space, emphasis is placed on the interpretation and interaction of AR space with virtual information. Through the combination of camera recognition technology and computer programs, when a set graphic appears in the lens, a virtual object will appears on the screen. Recently, the booming development of mobile technology has also promoted the increased availability of AR applications.
AR consists of technologies that combine the real world and virtual goods, and allows users to use actual applications and system interaction in the real environment. This will increase user focus and motivation. AR is mainly used to detect the user's 3D spatial position in terms of visual presentation, and then to make a corresponding visual representation of virtual and real objects [10]. One of the most important technologies is spatial 3D positioning. Some researches proposed the 3D graphic IOP Publishing doi:10.1088/1757-899X/1113/1/012023 3 display technologies, and to combine the virtual objects with 2D or 3D images. Precise spatial 3D positioning displays the pre-completed AR information on the device, and generates a virtual and real picture through visual overlay effects. The AR technology provides a 3D virtual object with oblique viewing angles and the real-world scene, which can experience the combination of reality and virtualization during the interactive operation.

Machine learning and LSTM neural network
Machine learning is a multi-disciplinary domain that has emerged in recent years. It can automatically analyze from a large amount of data to find the regularity and use the regularity of the data to predict the future. Deep learning is a part or subset of machine learning, and it is an algorithm that attempts to do high-level abstraction of data by using multiple processing layers. Deep neural networks are currently the most common deep learning models, such as the Convolutional Neural Networks (CNN) is suitable for image recognition; and RNN is suitable for sequence to sequence translation, including converting speech to text or translating different languages. RNN is a very powerful dynamic system, but due to the problem of gradient disappearance/explosion, it is restricted to solve the problem of long-term dependence effectively. To solve this problem, the LSTM model with special implicit units is proposed.
LSTM is a kind of time-recurrent neural network, and its unique design structure is suitable for processing and predicting longer and more delayed events in time series. The LSTM contains a special unit called a memory cell that is used to memorize values of varying lengths of time, similar to accumulators and control neurons. LSTM has been proven to perform better than traditional RNN in speech recognition and machine translation applications [11].
LSTM network has three control gates: forget gate, input gate and output gate respectively. These three control gates confirm the feasibility of using gradient-based optimization methods and avoid the diffusion and explosion of gradient. The structure of LSTM neural network is shown as Figure 1. X t represents the input at the current moment, C t-1 represents the long-term memory vector before this moment, h t-1 represents the output of the previous unit, C t represents the long-term memory vector at the current moment, and h t represents the output at the current moment. The operation processes of LSTM are as follows: (1) Combine the output vector at the previous moment and the input vector at the current moment into a vector σ; (2) Use the activation function to matrix multiply the vector σ with the parameters of different states; (3) f t and C t-1 at the previous moment are combined to realize the memory or forgetting effect of the state at the previous moment; (4) Use tanh activation function to operate C t , and combine with the O t vector to control the output vector and get h t ; (5) Input Ct and h t as the current state vector and output vector into the next LSTM unit model.

Design of electric motor diagnosis system
The intelligent diagnosis system uses the LSTM model to establish an electric motor health diagnostic model. Based on the differences of motor parameters between the current motor operating status and the predicted response of the diagnostic model to determine whether the motor is healthy or not. The These data such as vibration, voltage, and flux linkage are sent to the motor diagnosis model to analyze and predict the healthy status of the motor. Two hidden layers with LSTM are chosen in our diagnosis model. The algorithm of the proposed diagnostic model has three main steps: data augmentation, model training, and evaluation. In the first step, the purpose of data augmentation is to reduce computation time and improve the convergence speed. The second step is training the diagnosis model with training dataset. The third step is validating the model with testing dataset, and the accuracy of prediction is the important evaluation index.
When the motor diagnostic model is chosen, it is important to decide the parameters of input layers and output layers for LSTM model. The variable speed transmission system can be referred to the motor diagnostic model. The transmission medium of variable speed transmission system is power, and the system has an obvious vibration reaction. In order to discuss the vibration behavior effectively, the limited main degrees of freedom is supposed to descript the dynamic behavior of structural system. The motion equation can be defined as eq. (1).
the parameter M, C and K represents as mass matrix, damping matrix and stiffness matrix respectively; x is displacement vector, and f is external force vector. Eq. (1) is a description model of mechanical vibration system, and the model can be descripted according to the parameters such as: displacement, velocity and acceleration. Unfortunately, the displacement, velocity and acceleration parameters are unable captured at the same measurement node at the same time. To solve the problem, the centered difference scheme is applied as eq. (2).
and then we can get eq. (3) according to the eqs. (1) and (2): To measure the displacement response, it requires a motionless reference point, and one observation parameter is needed to describe the function. The acceleration parameter is applied to describe this system. Therefore, find the second derivative of time for centered difference eq. (2), and we can get the eq. (4).
by simplifying the eq. (4), it will get the time series model as eq. (5). The values of 1 , 2 , 0 and 2 are calculated as eq. (6).
The eq. (5) is the autoregressive exogenous (ARX) time series model. In this model, the current time responses are outputs, and the external interference at current time and previous time are inputs. The inputs and outputs of the time series model can be applied as the input layer and output layer of LSTM neural network. The prediction model for acceleration response can be built up according to the neural network architecture [12]. By comparing the actual measured response with the predicted response, the degree of difference between the current state of the system and the initial health state can be obtained. If the degree of difference exceeds the alert value, determine that the system is in an unhealthy state. We can use roots mean square error (RMSE) to quantify the degree of difference between the current state and the initial state of the system, the estimation function is represented as eq. (7).
The variables between the stator circuit and the rotor circuit of the permanent magnet AC motor are coupled with each other and have a time-varying nonlinear equation relationship. If the mathematical model is represented by the three-phase axis, the analysis of dynamic characteristics is complicated and difficult to control. Therefore, the two-axis coordinate conversion method is used to convert the stator and rotor rated voltage, current and flux linkage from the three-phase reference coordinate axis with a phase angle of 120 degrees to the two-phase reference coordinate axis with a 90-degree phase difference. The motor mathematical model of the three-phase coordinate system can be transformed into the two-axis static coordinate system of the stator through Clarke Transform, and then transformed into the synchronous rotating coordinate system of the rotor through Park Transform [13]. The motor electrical formula can be represented as eqs. (8) and (9): where, V d and V q are the stator voltage of d-q axis; I d and I q are the stator current of d-q axis; R s is the stator phase resistance. Ed and Eq are the counter electromotive force generated by the stator coil of dq axis, which can be represented as eqs.
where is the electrical speed. and are the flux linkage of d-q axis, which can be represented as equation (12) and (13) where is the permanent-magnet flux linkage. The electrical formula of the permanent magnet motor can be simplified as eq. (14): and then the torque equation of permanent magnet motor can be represented as eq. (15): Since the current variation speed cannot be measured in the motor electrical formula, the central difference method is introduced as eq. (16): and then the motor electrical formula can be discretized and represented as eq. (17) According to this TV-ARX formula, a model for predicting motor current response based on the data of current, voltage, speed, etc. can be established. The LSTM neural network of electrical response can be constructed by substituting the input and output values determined by this model. And then compare the current response obtained from this model with the actual current data, it can determine whether the magnetic characteristics of the motor have made a difference or not. Figure 2 is the LSTM neural network for permanent magnet motor diagnosis system which is determined by the electrical equation of the permanent magnet motor.
Since the demagnetization of the motor is an irreversible chronic deterioration phenomenon, it is not easy to verify the actual data collected by the actual motor. If the data is generated by finite element analysis software, the results are not easy to accept. Therefore, through the hardware-in-theloop system (HIL), the working condition of the real motor is simulated online, and the magnetic There are some hyper-parameters that have to be setup and adjusted while building the LSTM model. We can find the best value for hyper-parameters by evaluating the LSTM model with the test data to. The batch size and learning rate are taken as hyper-parameters of our LSTM model. In the process of back propagation, the value of weight and bias are adjusted continuously until the number of iterations reaches the specific value. To accelerate the training process and prevent the local optima, the mini-batch gradient descent algorithm is applied, in which the batch size of training samples is selected for iteration. The parameters of last iteration of training process was updated and used for testing datasets. The learning rate defines the updating speed of some parameters such as weight and bias. The results of accuracy rate in the testing dataset for different hyper-parameters are shown in Table 1. The best configuration with the batch size of 80% and the learning rate of 0.003 presents a correct prediction. Table 1. Accuracy rate in testing dataset for different hyper-parameters All samples for the experiment will be represented the health conditions with length 256 and each condition has 1680 samples, in which 1344 samples are randomly selected for the training dataset, and the rest 336 samples are for the testing dataset. There are 256 units for the input layer of LSTM model, and 64 units for the first hidden layer of LSTM, and 32 units for the second hidden layer of LSTM. A max iteration value is set as 80, but if the loss of training dataset is still minor changing after a few iterations, the training phase will be ended. To evaluate the effectiveness of prediction model, the root mean square error (RMSE) is employed for evaluation. The average RMSE of the diagnosis model is 1.6253, and the average R-squared value of the testing model is 0.86. The accuracy and loss rate in the training dataset of the model are plotted in Figure 3. The accuracy rate represents the effectiveness of the diagnosis model, and the accuracy rate is the number of correct prediction divided by total number of dataset. The training phase is convergence after about 48 iterations, which proves it can reduce the training time and get a good result. Compared with traditional techniques of motors fault diagnosis, the advantages of using the deep learning networks are comparably fewer iterations and the computing time is acceptable. The proposed health diagnosis method provides the prediction of abnormal vibration and demagnetization for electric motors. The diagnosis model can be applied to the higher power and permanent magnet motors. According to the system response diagnosis model, the accuracy of fault prediction is up to 98%. The permanent magnet motors not only shows the difference caused by motor demagnetization, but it also demonstrates that the degree of demagnetization is positively correlated with the amount of difference. Therefore, this motor demagnetization diagnosis method can indeed present the health status of the permanent magnet motors.

Development of augmented reality application
The motor 3D AR display application is developed with Unity [14] software on mobile device platform. Unity is a cross-platform 2D/3D game engine developed by Unity Technologies. The platform supported by Unity also extends to the HTML5 web platform based on WebGL technology. The graphical expression of the augmented objects became more precise by using Unity's built in shades.
Vuforia [15] AR development tools have been added as a unified workflow across AR devices to provide customized resources that turn immersive vision into reality. Vuforia is a set of software development tool from Qualcomm for mobile device augmented reality applications, which provide good augmented reality development tools for mobile devices. It allows developers to place virtual objects through the camera viewfinder and adjust the position of objects on the solid background in front of the camera.
The augmented reality object display and user interactive operation combines with the motor design platform at the back-end cloud server. The back-end server controls the animation display of the rotation speed of the 3D electric motor model at the mobile device, allowing users to experience the 3D models and animations of various electric motor designs during the interactive operation.
The mobile device application is set to scan specific graphics to load the electric motor 3D module. This 3D electric motor model consists of several components, including a rotating shaft, a rotor, a stator, etc. These 3D motor components can be displayed or hidden by selecting from the program menu. The various components of the electric motor can be exploded or closed to see details of the electric motor model, which is shown in Figure 4. The user not only can control the operation modes of the electric motor model to zoom in, zoom out, rotate up, down, left and right, etc. from the application interface, but also read the speed, voltage, temperature, and magnetic data from the cloud computing server to control the rotation speed of the electric motor model or display possible abnormal conditions. The operating interface provides a convenient management function for users. If an abnormality is detected or a possible abnormal condition is predicted by the diagnosis system, the components of motor unit will appear different color in the application. The diagnosis system can also be applied for the automotive system. The virtual components of automotive model can be displayed by scanning a car model. The motors of the car can be monitored and diagnosed by the intelligent diagnosis system, and displayed the fault components of the car on the screen of mobile device.
The schematics of electric motor AR diagnosis system is illustrated as Figure 5. In addition to maintaining the necessary parameters generated by the electric motor design, the cloud AI computing server writes relevant data generated by the operation of the electric motor into the database, including the speed, voltage, vibration, temperature, and flux linkage, etc. of the electric motor. Then, AI computing module executes the fault prediction for electric motors, and the results are transmitted to the application program on the mobile device. The AR program on the mobile device displays the operating status of the electric motor. After obtaining the electric motor-related parameters on the server, the operating status of the AR electric motor model is displayed on the screen. And the information such as the operation, execution of the motor module, or setting of the motor parameters is synchronously returned to the server. By updating the database, and sending information back to the electric motor controller to set the operating parameters of the motor. The application on the mobile device has the ability to identify different modules of the electric motor and switch to the electric motor model to be simulated or monitored. The switching method can be used to load specific motor modules without image recognition, or use the screen menu to manually switch between different electric motor modules. The operating parameters of the electric motor model can also be set and controlled by the application, and then sent back to the database of the cloud server IOP Publishing doi:10.1088/1757-899X/1113/1/012023 10 to update the electric motor status synchronously. It also allows users to experience 3D digital electric motor models and animations of various electric motor designs during the interactive process.
In terms of the electric motor design and industrial applications, using AR software to control the parameter design and operating status of motor equipment can improve the work flow and efficiency of staff inspection equipment. Based on accumulated data on the cloud server and machine learning technology, the system can determine the status of this equipment is abnormal or not, and display early warning to reduce the risk of equipment failure.

Conclusions and future works
This research provides an intelligent diagnosis system which integrates the augmented reality technology for the electric motors. The intelligent diagnosis mechanism of electric motors applies LSTM machine learning algorithms to establish the motor health diagnostic model. By combing with cloud data management and machine learning technologies, an AI cloud platform is developed to predict the health status and detect possible abnormalities of the electric motor. The AR technology can also displays the alarm signals of the digital electric motor model on the mobile devices immediately to provide a combination of reality and virtualization experience during the interactive operation. In the future, we will enhance the diagnosis system for electric motors, and compare with various machine learning algorithms to get the better diagnosis model and improve the accuracy of prediction results. Besides, the AR display system will scan the real solid motors and vehicles to display virtual information on a real object, and provides real intuitive interactions for users.