Research on recognition of human lying posture based on Kinect sensor

: As a high-tech intelligent product, the intelligent nursing bed largely meets the needs of the disabled for self-care, and saves a lot of manpower and material resources. In order to improve the safety and reliability of the intelligent nursing bed and ensure the safety of the user, this paper adds the recognition of the human body's lying posture as a part of the movement signal of the nursing bed. The Kinect sensor is used to track human bones, record human bone data, and preprocess human bone data. Use the pattern recognition toolbox in Matlab to classify the processed data to realize the recognition of the human body's lying posture. The average recognition rate of the five postures is 98.1%. The results show that the model used in this experiment has a high degree of recognition and can greatly improve the safety and reliability of the intelligent nursing bed.

The nursing of long-term bedridden patients is an inevitable and very laborintensive task for families and nursing institutions and hospitals, especially for critically ill patients who have lost part of their body functions. It's getting more and more prominent. Therefore, people are focusing on smart nursing beds to solve the basic nursing problems of the elderly who cannot take care of themselves, the patients with mobility problems and the disabled due to diseases, and reduce the nursing pressure of medical staff and patients' families. As a medical auxiliary device, it has basic functions such as raising the back, bending the legs, and turning left and right. However, the automatic control of most nursing beds is controlled by a fixed time, and there are certain hidden safety hazards. May cause more harm to the user. Therefore, improving the safety and reliability of nursing beds is particularly important for the development of intelligent nursing beds.
Aiming at the hidden safety hazards of the intelligent nursing bed, this paper designs a human bone data preprocessing algorithm combined with the recognition system model of the pattern recognition toolbox in Matlab according to the bone data of the human body. The system can recognize the lying posture of the human body, and output the signal as a part of the motion signal of the intelligent nursing bed, which greatly improves the safety and reliability of the intelligent nursing bed.

Recognition model design
The work flow of the human body gesture recognition model designed in this paper is shown in Figure 1. The work flow is as follows: first collect the subject's human bone data information through the Kinect sensor, and then use the designed algorithm to select and compare the bone feature points. The coordinates of the bone points are transformed and the joint angle features are calculated. Finally, the Matlab pattern recognition toolbox is used to identify the processed features to determine the current posture of the human body. The Kinect sensor includes a color camera, infrared transmitter and receiver, and four microphone arrays. In Kinect sensor, the RGB camera captures a color image with a resolution of 1920×1080 pixels at 30 frames per second, while an infrared camera also obtains infrared data and a depth map with a resolution of 512×424 pixels in real time at 30 frames per second. Kinect sensor has a field of view of 70 degrees horizontally and 60 degrees vertically, and the measured depth distance is 0.5 m-4.5 m.
It can obtain information on 25 joint points of the human body, and can detect 6 people at the same time. The user can also adjust the motor base The direction and angle to obtain the ideal image data.

Data collection
In order to obtain the data set required for training the neural network, this paper collected the human bones of 10 normal subjects (3 women and 7 men, aged between 25-60 years and height between 1.55-1.90 m) data information. When collecting data in this experiment, the subjects lay on the intelligent nursing bed and presented five postures: lying flat, raising the back, bent legs, and lying on the left and right sides. The Kinect sensor collects the human bone data information, and each posture is maintained 10 seconds. As shown in Figure 2. Part of the output data after the collection is completed is shown in Figure 3.

Selection of bone points
By using the Kinect sensor and the corresponding development software, the spatial coordinates (x, y, z) of the standardized bone joints of 25 human bodies can be obtained. However, since some of the joint points have no effect on gesture recognition, based on the reliability of the purpose of this research and the consideration of recognition speed, this paper screens the number of bone joints and reduces the number of bone joint points from 25 to 11. As shown in Figure 5, the selected 11 joint points are: head(1),spine shoulder (2),shoulder left (3),shoulder right(4),spine base (5),hip  (6),hip right (7),knee left (8),knee right (9),ankle left(10),ankle right(11).

Coordinate transformation
In order to determine the position of the subjects on the intelligent nursing bed, coordinate transformation of the 11 selected bone joint coordinates is needed to transform them into the absolute reference system (x, y, z) with the intelligent nursing bed as the reference. According to the imaging theory [2], the world coordinate system − can be converted to the imaging coordinate system − through = + , where T is the transition matrix and R is the rotation matrix. In Kinect sensor, the coordinate transformation model can be represented by Figure 6.

Fig. 6 Coordinate transformation model
Converting the world coordinate system to the depth system and the RGB system can use equation (1) and equation (2).
In the imaging system, the RGB image coordinate system[ , , 1] can be described by equation (3).
The depth imaging system can be described by equation (4).
Among them, and are the original parameters of the sensor. Through the above four formulas, the transformation from the form [ , , 1] to [ , , 1] .Since the distance between the color camera and the depth camera of the Kinect sensor is very close, it can be assumed that ≈ , so: Since the Kinect sensor is a standardized production equipment, solving the spatial mapping problem can be solved with the calibration parameters of Kinect sensor [1].
When solving the joint point J( , ) in the color image, it can be used to obtain the depth image map J( , ) In the depth camera coordinate system, the depth value of ′ is .So far, the three-dimensional coordinates are based on the world The coordinate system J ( , , ) is obtained with respect to J ( , ) according to equations (2) and (3).

Node angle calculation
For any two joint points ( , , ), ( , , ), the coordinates of the two joint points that need to be calculated can be obtained through the Kinect sensor. After the coordinate transformation, the formula (6) Calculate the direct distance between two joint points. By continuing to process these distance features, the angle features between multiple joint points can be calculated.
There are two main methods for calculating the angle characteristics of each group of joint points: the first is the three-point method, select the three human body joint points ， and that need to be calculated, and use the formula to calculate the relationship between the three joint points. The distances , and , and then the law of cosines (7) can be used to find the angle between each joint point.
The second method is the two-point method. Only two joint points and are used for each group of angles. Using as the reference point, the connection between the two key points and and the reference point X are calculated by formula (8) The included angle of the axis.  This paper defines a set of characteristics that can not be affected by the body size of each subject to represent the kinematic characteristics of different human postures, that is, from the above 10 joint angles, 5 angles are selected As features for identification, they are: the angle µ between the head and the left shoulder, the angle α between the head and the right shoulder, the angle β between the thigh and the calf, and the angle between the shoulder and the leg. The included angle γ is the angle θ between the head and the back.

Data Classification
Mark each frame of pictures with the following 5 poses: The Recognition is based on the selected angle features, where the left side is recognized by the angle µ, the right side is recognized by the angle α, the bent leg is recognized by the angle β, the back is recognized by the angle γ, and the lying down is recognized by the angle θ.

Fig. 8 Neural network structure diagram
The pose classification problem can be regarded as a static mapping problem, and the pattern recognition toolbox in the Matlab neural network toolbox can be used to realize the network required by the experiment in this paper. The Matlab pattern recognition toolbox is a two-layer feedforward network with a sigmoid hidden layer and softmax output neurons. As long as there are enough neurons in its hidden layer, the vector can be classified well. The network will be trained using the proportional conjugate gradient backpropagation algorithm, and the training function is trainscg. It is found through experiments that when the number of neurons in the hidden layer is Fig. 9 Training process diagram 30, the experimental results are more accurate, and the training time is shorter, which only takes 10 seconds, as shown in Figure 9. Therefore, 30 neurons are set in the hidden layer, which are connected to 5 input features. In the training process, the data set is classified, of which 70% is the training set, 15% is the validation set, and 15% is the test set.

Discussion and experimental results
When observing the experimental results, the accuracy of the experiment can be observed through the ROC curve and the confusion matrix.
Through the ROC curve, it is easy to find out a classifier's ability to recognize samples at a certain threshold. The best diagnostic limit value of a certain diagnostic method can be selected with the help of ROC curve. The closer the ROC curve is to the upper left corner, the higher the FPR and the lower the FPR of the test, that is, the higher. As shown in Figure 10 and Figure 11, the ROC curve obtained from the experiment shows that the ROC curves of the five types of postures are very close to the upper left corner, and it can be judged that the model adopted in this experiment has good performance. In addition, the confusion matrix can also be used to analyze the experimental results in more detail. Confusion matrix is the most basic, most intuitive, and easiest method to measure the accuracy of sub-type models. Through the confusion matrix, the accuracy of each classified data set and the average accuracy of all data can be seen, and the prediction.
accuracy of the whole model can be judged by the accuracy.
As shown in Figure 12, When lying flat, the angle characteristics of the head and shoulders will also affect the recognition of the left and right side lying. Figure 1 Model design ow chart Part of the collected data diagram  Partial angle feature data map Training process diagram