On the Design of a Wearable Multi-sensor System for Recognizing Motion Modes and Sit-to-stand Transition

Locomotion mode recognition is one of the key aspects of control of intelligent prostheses. This paper presents a wireless wearable multi-sensor system for locomotion mode recognition. The sensor suit of the system includes three inertial measurement units (IMUs) and eight force sensors. The system was built to measure both kinematic (tilt angles) and dynamic (ground contact forces) signals of human gaits. To evaluate the recognition performance of the system, seven motion modes and sit-to-stand transition were monitored. With a linear discriminant analysis (LDA) classifier, the proposed system can accurately classify the current states. The overall motion mode recognition accuracy was 99.9% during the stance phase and 98.5% during the swing phase. For sit-to-stand transition recognition, the average accuracy was 99.9%. These promising results show the potential of the designed system for the control of intelligent prostheses.


Introduction
Recent microcomputer-controlled powered prostheses have provided less power consumption and more functionality [1,2,3,4]. The movement of the human body differs for different locomotion tasks. Without knowing a user's movement intent, prostheses cannot adjust joint impedance or drive the powered joint motion. Although some prostheses can detect limited locomotion modes [1,4], control approaches cannot achieve smooth transitions between different motion modes. Therefore, the development of an interface for automatic motion mode recognition has been drawing increasing attention.
Previous studies on detecting human movements in different motion modes were mostly realized with the following sensors: EMG sensors [5,6,7], inertial sensors [8,9,10], foot pressure sensors [11] and capacitance sensors [12,13,14]. For EMG sensors, studies on motion mode recognition for lower limbs are limited. In [5], the authors measured EMG signals from 16 muscles of the lower limbs while the subjects were walking on different terrains or paths. The recognition of a total of seven motion modes was tested on eight able-bodied subjects and two subjects with transfemoral (TF) amputations, and promising recognition results were shown. In [8], the authors used a wireless era-worn activity recognition (e-AR) sensor incorporating a three-axis accelerometer to categorize seven kinds of motion transitions. In this study, ten subjects were recruited. The recognition performance was promising: the average precision of the seven transitions was more than 90% when the data was collected in a controlled laboratory environment. Recently, in [11], the authors developed pressure insoles to measure foot pressure. With a phase-dependent classifier, the average error accuracies of five locomotion modes for four phases were 19.6%, 12.6%, 5.2%, and 6.3%, respectively. In [12], the authors designed a capacitance-based sensing system to detect muscle contractions during motion. Nine locomotion modes were tested. With a phase-dependent classification method, the overall recognition accuracy was 95.05%, 95.21%, 95.77%, 96.58% for the four phases, respectively.
Though the signals of the EMG sensors and the capacitance sensors record the muscle contractions which directly reflect the user's intent, the signals are non-stable and time-varying because of muscle fatigue. With the development of Micro-electromechanical Systems (MEMs), a single IMU can now be built in a small size, incorporating accelerometer, gyroscope and magnetometer. On the other hand, the signals are full of noise because of shifting of the sensors. More sensor nodes that are independent sources of noise have to be added to provide complementary information. The foot pressure sensors can be fixed on insoles and measure the ground contact forces (GCFs) unobtrusively. However, the signals lag behind the gait event with the processing time taken into account. Furthermore, information on motion cannot be detected during the swing period.
Recently, some researchers have focused on a multisensor fusion method to synergize the performance of different sensors [15,16,17]. In [15], the authors devised an algorithm based on neuromuscular-mechanical fusion to continuously recognize a variety of locomotion modes. EMG sensors and contact forces/moments measured from prosthetic pylons were fused together as the input to the classifier. An experiment on five subjects with transfemoral amputations was carried out to validate the algorithm. In the experiment, five static motion modes and five mode transitions were measured. With a support vector machine (SVM), the classification accuracies were 99% in the stance phase and 95% in the swing phase during motion mode recognition. However, there are some limitations in this method. The EMG signals are non-stable, which will decrease the recognition performance during long time measurement because of muscle fatigue. Furthermore, the SVM classifier is memory-consuming and needs lots of memory space to store the intermediate results.
In this paper, we propose a new method to recognize locomotion modes. To acquire enough information on human motion, we develop a wearable multi-sensor system, which includes three IMU modules and two foot pressure insoles. The system is designed to measure the kinematic (IMU modules) and dynamic (foot pressure) parameters of the human motion. Previous studies on wearable sensors equipped with IMU modules or foot pressure sensors have focused on gait analysis [18,19] or the detection of postural transitions [20]. Few studies have applied the wearable sensor system to locomotion mode recognition. In this work, linear discriminant analysis (LDA) was used, which is low-cost in terms of calculation. The recognition accuracies obtained are higher than those in [5] and [15].
The paper is organized as follows. The architecture of the designed system is described in detail in Section 2. In Section 3, we interpret the experiment protocol and the classification method. Section 4 shows the experimental results. We conclude in Section 5.

Placement on human body
The sensors used in the multi-sensor system were selected with the goal of measuring as many locomotion characterizations. The distribution of the sensors on the human body is shown in Figure 1. For measuring human kinematic parameters, three inertial measurement units (IMUs) were fixed on the thigh, the shank, and the forefoot of the measured leg, respectively. To detect the gait events and record the dynamic characteristics of the gait during stance period, pressure sensors were fabricated on the insoles of both feet. There is a control circuit (ConMod) on the waist to control the data sequence. The data was wirelessly transmitted to the receiving circuit.

Inertial measurement unit (IMU)
The nine degrees of freedom IMU board is built with an accelerometer, a gyroscope and a magnetometer chip, which is shown in Figure 2. The accelerometer is an ADXL345 digital microchip (Analog Devices Inc.) with three-axis measurement. The ADXL345 is a low-power, three-axis accelerometer with high-resolution (13-bit) measurement up to ±16 g. The gyroscope we used is the ITG-3200 (InvenSense Inc.), which is a three-axis MEMS IC. For the magnetometer, we used the HMC5883L (Honeywell International Inc.), which is a surface-mount, multi-chip module designed for low-field magnetic sensing with a digital interface for applications such as low-cost compassing and magnetometry. We used the ATMEGA328 as the micro control unit (MCU) of the IMU board, which is a low-cost 8-bit microcontroller with 20 MHz frequency. The MCU controls the data stream through an Inter-Integrated Circuit (IIC) bus. We use the data of the gyroscope to estimate the orientation of the IMU board. The magnetometer and the accelerometer were used to complement the errors of the orientation. The output data of the IMU board includes Euler's angle (pitch, roll and yaw), three-axis angular velocity and three-axis acceleration. The data format is Universal Synchronous Asynchronous Receiver Transmitter (USART). The sensor node FS1 includes a pressure insole and an IMU module. The FS2 contains a pressure insole. The sensor data was wirelessly transmitted to the receiving circuit.

Foot pressure insoles
To accurately measure the gait parameters, three aspects have to be taken into account. Firstly, the resolution of the sensors should be high enough to record the changes of GCFs during the stance period. Secondly, the sensors should be thin and small for convenient wear in shoes. Thirdly, the distribution of the sensor should ensure that as much information as possible is collected with the fewest sensors. In this paper, we used FlexiForce A401 (Tekscan, Inc.) as the force sensor. The FlexiForce is a kind of tiny thin forcesensitive resistors (FSRs) whose resistors vary with pressure. The integration of the foot pressure insole in the shoe and the distribution of the sensors are shown in Figure 3. The function of the sampling circuit is to receive the data of the IMU module (see Section 2.2) and to sample the foot pressures. The conductance of the FlexiScan A401 sensor is proportional to the force applied on it. Thus, we designed an inverting amplifier using LMV324 (STMicroelectronics Inc.) to convert the conductance to voltage. Then, the four channels of signals were fed to STM32F103C8T6 (STMicroelectronics Inc.) for the analogue-to-digital conversion (ADC). The STM32 integrates ADC controllers and is capable of measuring up to ten channels of input.
The positions of the sensors on the insole were selected with the goal of measuring the most prominent force changes during the stance period. As shown in the right half of Figure 1, the four key positions are the hallux toe, the first metatarsal bone, the spot between the fourth and fifth metatarsal bone and the calcaneus tuberosity. The positions were determined by the simulation of human locomotion with an artificial foot skeleton [11], similarly to previous studies [21,22,23].

Data transmission
We used an STM32F103C8T6 as the MCU on each sensor nodes to process the sensor data. As mentioned above, the control module of the multi-sensor system is placed on the waist. The function of the control circuit is to control the data sequence of all the sensor nodes on the human body and communicate with the receiving circuit to send the result to the host computer. As the locomotion modes we studied in this paper were placed in the sagittal plane, the kinematic parameters of two axes were enough. Table 1 shows the data format of the system.  As shown in Table 1, the sampling rate of all the sensor nodes is 100 Hz. Therefore, the control circuit on the waist has to collect all the sensor data and transmit to the receiving circuit in 10 milliseconds. The communication between the control circuit and the sensor nodes was implemented with a recommended standard 485 (RS485) bus. The RS485 bus is a kind of half-duplex serial communication protocol, which uses differential voltage representing the data. In our system, the data sequence was controlled with the polling method. In this method, each sensor node was assigned a device number (ID) and the sampling interval (10 ms) was segmented into several time slices. During each slice, the control circuit broadcast the command data packet with an ID and waited for the response. Then, the node with the same ID sent out the result. The polling method guaranteed the unobstructed communication on the bus.
The control circuit communicated with the receiving circuit via a wireless module (see Figure 4). The wireless module was built based on an nRF24L01P (Nordic semiconductor Inc.), which is a single-chip 2.4 GHz transceiver with an embedded baseband protocol engine, which supports various operation modes. In this paper, we use the autonomous operation mode (Enhanced ShockBurst), in which the data is handled on the chip without the cost of MCU. The nRF24L01P communicates with MCU via serial peripheral interface (SPI). The maximum air rate is 2 MHz.

Experiment protocol
Five able-bodied male subjects were recruited in our study and provided written and informed consent. The positions of sensor nodes should be carefully chosen to obtain more useful information on human motion. As shown in Figure 4, the IMU2 was placed on the posterior of the shank and the IMU1 was placed on the anterior of the thigh.  We carried out two kinds of experiments to validate our multi-sensor system. The first was locomotion mode recognition. In the experiment, seven locomotion modes were investigated: standing (ST), sitting (SI), stair ascending (SA), stair descending (SD), ramp ascending (RA), ramp descending (RD) and normal walking on level ground (NW). For the task of standing, the subject was asked to stand still for every trial. For the sitting experiment, the subjects were required to sit on a 42-cmhigh chair for every set of experiments. Stair ascending and stair descending were tested on a four-step staircase. The stairs were 75 cm in width, 40 cm in depth and 15 cm in height. In normal walking, the subjects were encouraged to walk at their most comfortable speed for normal walking. The number of trials and gait cycles in each trial is shown in Table�. There was a baselinemeasuring procedure for each subject. During this procedure, we measured the foot pressures with the foot rested freely and standing, respectively. Then we were able to manually tune the thresholds based on the foot pressures.
The second experiment was to test the system's performance on motion transition. The recognition of motion transitions is also important for application in prosthesis control. Sit-to-stand transition is a crucial state for prosthesis control because a significant amount of power is needed to complete this transition. In this experiment, five able-bodied subjects were recruited. The subjects were asked to perform sit-to-stand transition for ten trials. In each trial, the subjects sat on a chair at first and then stood up in a way that was comfortable for them.

Data processing method
In this work, we used phase-dependent classification method to classify the locomotion modes. Unlike upper limb motions, lower-limb locomotion demonstrates different dynamic and kinetic characteristics in different periods of one gait cycle. We therefore used a phasedependent method to estimate an optimized classifier for each gait phase. The foot pressure insoles detect the pressure changes during ambulation. We summed the pressure of one foot and tuned a threshold to determine the swing period and stance period. By combining the foot pressures of both feet, we divided one gait cycle into four phases ( Figure 5): double stance 1 (DS1), single stance (SS), swing (SW) and double stance 2 (DS2). In order to make full use of the sensor data, sliding analysis windows were used for data processing in this paper, by which method the analysis window slides from the beginning to the end with an increment length. Each window was labelled with a gait phase. The training and testing procedures were performed separately for every gait phase. Similarly to previous studies on EMG-based locomotion mode recognition, time domain features were calculated for all the channels of foot pressure signals as well as the IMU signals to generate a feature set. Maximum, minimum, and mean value, standard deviation and waveform length were chosen as the features. The analysis window length was 150 ms and the increment was 10 ms, i.e., the features were calculated for each sample.
For the recognition of sit-to-stand transition, we regarded sit-to-stand transition as an independent mode comparable with sitting and standing. Sit-to-stand transition is different from ambulation modes (walking, ascending stairs, etc.) because there are no gait phases during the transition procedure. Thus, independent classifier training is needed. We segmented the data into three classes: sitting, sit-to-stand transition, and standing. Two thresholds were determined for each subject based on the roll angle of the thigh and the sum of the foot pressures to label the data. The threshold between sitting and sit-to-stand was determined by the foot pressures in the initial transition. The other threshold was determined by the roll angle of the thigh, i.e., the termination of the transition. The analysis window method and feature set were also used for sit-to-stand transition recognition. denotes the probability of the feature vector x. In the LDA classifier, the likelihood is presumed to be multivariate normal distribution. In our study, the a priori probabilities were presumed to be the same for all the locomotion modes. All the classes share the same covariance in calculation. Thus, the maximum posterior probability can be expressed as: where μ is the mean vector of the data and Σ is the covariance matrix. The training procedure aims to calculate the covariance matrix and the mean vectors. In real-time classification the feature vector of each sample was calculated according to Eq(2).

Evaluation method
For locomotion mode recognition, we used 15-fold leaveone-out cross validation to evaluate the average recognition accuracies. While for sit-to-stand transition recognition there were only three classes (sitting, standing, sit-to-stand transition), for classification we used cross validation to test the performance.
The overall recognition accuracy (RA) is calculated by: where corr N is the number of correctly recognized testing data and total N is the total amount of testing data.
To illustrate the recognition performance of certain locomotion modes in detail, a confusion matrix was used, defined thus: , where each element is defined as: n is the amount of testing data in mode i, recognized as mode j, and  i n is the total amount of testing data in mode i. A higher value of ( ) ≠ ij c i j denotes that it is easier for mode i to be misclassified as mode j.

Gait phase detection method
As mentioned above, a gait cycle was segmented into four phases. In our study, the foot pressures were used to detect the gait phases. In different locomotion modes, the sequence of the sensor contact with the ground is different. We therefore compared the summation of the forces (SumF) of the insole with the predefined threshold. To accurately detect the gait phases with this method, the noise of the signals should be filtered. A first-order lag filter was used, defined by: FData is the result of the time t , t RData is the raw signal of the time t , and a represents the lag factor. The general performance can be adjusted by a . All gait phases were correctly labelled.

Locomotion mode recognition performance
The overall recognition accuracies of DS1, DS2, SS and SW were 99.89%, 99.90%, 99.77% and 98.50%, respectively. The detailed results are shown in Figure 6. During the swing period, the recognition accuracy was 98.50%, which is lower than the other three periods, but still good performance. The lowest recognition accuracy was 95.49%, in the descending-stairs phase of SW. Note that most misclassification took place during ramp descending. For these two modes, the leg swings downwards towards the next step, showing similarity in motion patterns.

Transition recognition performance
The feature set and classifier were the same as those mentioned above. We used cross validation to evaluate the recognition performance. The confusion matrix of the average recognition accuracies for all the subjects is shown below: The accuracies were all higher than 99%. The results show that our system can accurately recognize the sit-tostand transitions with only tiny misclassification rates.

Conclusion
In this paper, we have proposed a multi-sensor system for locomotion mode recognition. With the phasedependent method and LDA classifier, we obtained satisfying results. The accuracies of sit and stand were 100% in our study. Accuracies remained as high as 99.9% during the stance periods and 98.5% during the swing periods -higher than the EMG-based system [5]. Moreover, our system shows excellent performance in sitto-stand transition recognition. In future research, the multi-sensor system should be integrated with intelligent prostheses. The sensors are easy to fix onto prostheses and the classification method is cost-effective in terms of calculation. In addition, the stability of long time measurement and a real-time recognition method should be studied.

Acknowledgements
This work has been funded by the National Natural