A Real-time Patient Monitoring Framework for Fall Detection

Fall detection is a major problem in the healthcare department. Elderly people are more prone to falling than others. There are more than 50% of injury-related hospitalizations in people aged over 65. Commercial fall detection devices are expensive and charge a monthly fee to use their service. A more aﬀordable and adaptable system is necessary for retirement homes and clinics. An eﬀective fall detection system would detect fall and send an alarm to the appropriate authority. We propose a framework which uses edge computing i.e. instead of wearable devices sending data to the cloud they will send it to a nearby edge device like a laptop or mobile for real-time analysis. We use cheap wearable sensor devices from MbientLab, an open source streaming engine called Apache Flink for streaming data analytics and a Long Short Term Memory network model (LSTM) for fall classiﬁcation. The model is trained using a published dataset called MobiAct. Using the trained model we analyze optimal sampling rates, sensor placement and multi stream data correction. Our edge computing framework can perform real-time streaming data analytics to detect falls with an accuracy of 95.8%.


Introduction
The past few years have seen a rapid increase in people using wearable devices. The use of connected wearable sensor devices is predicted to increase from 325 million in 2015 to 929 million in 2021 [1]. Wrist watch is one of the most popular form of wearable devices. Sensors inside the device measure metrics like the steps taken, stairs climbed, sleep, heartbeat and oxygen levels. Typically the data from a wearable sensor device is sent to a cloud service for analysis and displayed on the dashboard of a connected mobile device. On the cloud the data is synchronized with a data management environment, which is typically provided as a service by the manufacturer of the wearable device. A single accelerometer sensor at 200Hz generates about 2.3GB of data per day. The more sensors or monitoring metrics are added, the more data are generated by the sensors. There is a steady progression towards the era of Internet of Things (IoT) where many such devices will be connected generating tera bytes of data that must be analyzed possibly in real time to provide effective decision support.
Usually this data is uploaded and stored in the cloud to do some machine learning analysis. But that leads to wastage of the bandwidth and a decrease in response time.  [5] built an innovative system where a wearable air bag inflates automatically when the system predicts a fall 300ms before the fall occurs. Although systems such as these show great progress towards fall prevention, challenges still exist in handling situations like sideward and forward falls. More work is needed for expediting the inflation process of the air bag and in avoiding false alarms.

Proposed Solution
We propose an edge computing framework which is deployed in close proximity i.e., a maximum of 100ft range from the wireless sensor devices, which collects and performs preprocessing on the data to only transfer the important data to the cloud. We develop machine learning models to analyze the data and generate notifications in real time to enable call for assistance. We validate our framework for real time fall detection use case scenario by analyzing the accelerometer data collected from the wearable sensor devices. Fig 1.1 shows the general architecture of sensor data processing frameworks. BigMine'18 workshop, one is in progress on the complete work and another journal paper has been submitted to TKDE that used part of this study.
• Study of existing sensor devices and their functionalities.
• Study of state-of-the-art data processing tools • Development of an artificial neural network model for fall detection, and • Finally, development of the edge computing framework for real time monitoring of patients for fall detection.

Organization
The rest of the report is organized as follows. Chapter 2 describes the related work.
The fall detection model details is explained in Section 3. Section 4 presents the overview of the framework proposed. Section 5 discusses the results and conclusions are drawn in Section 6.

Background
The advancement in computational power and big data processing had led to progress in modern day data analytics. There has been great progress in the usage of wearable devices, resulting in research on the acquisition and analysis of data from the IoT or connected devices. Accelerometer and gyroscope are the most commonly used sensors for fall detection with mainly threshold based monitoring and machine learning based predictive analytics approach. In threshold based techniques, a fall is detected when monitored data values exceed pre-defined threshold values. In contrast, machine learning techniques analyzes the data and tries to learn hidden patterns to classify data as fall.
Bourke et al [6] presented a fall detection mechanism where sensors were placed on thigh and trunk. By analyzing the values from these sensors upper and lower fall thresholds were determined. If the resultant value exceeded the upper fall threshold at the trunk, a fall was said to be detected. It was able to detect a fall with 100% specificity. But when tested against the real time data a lot of false positives were observed. In a subsequent study Bourke et al [7] improved the algorithm by monitoring Quang Viet Vo et al [12] detected fall using a smartphone placed in different places like in hand while messaging or calling, in chest pocket and in pant pocket.
A fall was divided into three steps where the first step was when the fall took place, second step when the person hit the ground and third step if the person returned to normal activity or continued to lie down. When a signal exceeded the accelerometer thresholds in the first two steps the orientation data were analyzed for the time between the two steps. If a fall was detected and the magnitudes of acceleration values were constant, orientation data were analyzed for any movement in the third step to decide whether the subject was lying down or continuing his work. This mechanism was tested on five young people, which resulted in 85% accuracy.
Kwapisz et al [13] proposed WSDM (Wireless Sensor Data Mining) to detect fall using machine-learning models with sensors present in an Android phone. The phone was placed in the front pocket of the subject. The data were collected every 50ms that is 20 samples per second. For feature generation, the data were divided into 10 second segments, each containing 200 readings. Forty-three features were extracted from these raw values. The authors used J48, logistic regression and multilayer perceptron on the features extracted. Among these, multilayer perceptron performed consistently better when compared to others. Ozdemir et al [14] found the multi-layer perceptron neural network model to be most effective for fall detection resulting in around 95% accuracy.
Abbate et al [15] proposed a mechanism to use both threshold and machine learning algorithms to detect falls. The system was implemented using a smartphone and a wearable sensor placed at the waist with a sampling frequency of 50Hz. When a fall was detected using a threshold based technique, it was sent into a classification engine for further analysis. The mechanism also had a notification center using which the user could turn off the false alarms. The false positive data were sent into the classification model again for training. A two layer feed-forward model was used for classification where features were generated from the input signals and fed into the neural network model.The model performed with an accuracy, specificity and sensitivity of 100%.
Khan et al [16] surveyed different machine learning algorithms for fall detection.
The authors concluded that Recurrent Neural Networks (RNN) were very well-suited for this problem as the data is usually a sequential time series data. Theodoridis et al [17] built LSTM models using a published dataset called the UR Fall Detection. Two LSTM models, one with simple accelerometer data and another with accelerometer data rotated at an angle, were compared against a Support Vector Machine model  [19]. Similar to Vavoulas et al [19], we generated two sets of features, combined them, and then extracted the optimal feature set from them. Different experiments were done to find the best scoring model to incorporate into our framework.

Fall Detection using Machine Learning Model
We developed a machine learning models based on a published dataset "MobiAct" to detect fall as explained in this chapter. Later we incorporated best performing model in our edge computing framework which is described in the next chapter.

MobiAct Data
We used MobiAct, a publicly available dataset developed by the Biomedical Informatics and Health Laboratory of the Technological Educational Institute of Crete for training the neural network model [19]. MobiAct is based on a previously published MobiFall dataset [20] created for fall detection. The data was collected using accelerometer, gyroscope and orientation sensors present in a smartphone by placing the device in the pocket of the experimental subject. The orientation sensor was software based, which used data from accelerometer and geomagnetic field sensor.
By enabling the parameter SENSOR-DELAY-FASTEST, data was collected at the highest sampling rate possible. The MobiAct dataset contains labeled information about four kinds of simulated falls from fifty-four subjects, and nine kinds of daily activities from fifty subjects. Each activity in the dataset contains time stamp, raw accelerometer values, raw gyroscope values and orientation data. In our experiments, we used a subset of the data containing only two kinds of daily activities and four kinds of falls. The daily activities denote the two non-fall classes i.e., standing and lying. The four kinds of falls are as follows: • FOL: forward-lying (fall forward from standing, use of hands to dampen fall) • FKL: front-knees-lying (fall forward from standing, first impact on knees) • SDL: sideward-lying (fall sideward from standing, bending legs) • BSC: back-sitting-chair (fall backward while trying to sit on a chair)

Data Preprocessing
We preprocessed both the datasets using the following operations in the order listed: Scaling, Feature Extraction, Normalization of feature values, Balancing dataset and Feature Selection.

Scaling
The model generated from the existing data is used to classify real-time streaming data. So to maintain compatibility between the two data sources MobiAct dataset and collected MbientLab dataset are rescaled to the range of (+1g,-1g) before extracting the features. It is done using the rescale function in Matlab.

Feature Extraction
The raw accelerometer values in the dataset are used to extract features for fall detection. Vavoulas et al. [19] stated the importance of optimal features to obtain better performance. They created two feature sets, A and B. Feature set A is generated from their previous work [21] and feature set B is generated from the WSDM study [13]. We investigated the feature generation methods used in these papers, and based on that, devised an algorithm to generate feature set A and B for our work. Since we are not focusing on the classification of different types of non-fall activities, some of the features in the datasets were ignored to reduce the complexity. Based on the procedure followed by Aziz et al [22], we segmented the raw data into 200 blocks and then generated the feature sets for each block. After feature generation, we combined feature sets A and B, and derived an optimal feature set using a feature selection algorithm. Feature set A and B were calculated using Matlab R2017b.
Four other features were calculated using mean, STD, skew, and kurtosis of the tilt angle (TAi) between the gravitational vector and the y-axis using Eq 3.2.
where i denotes the sequence of sample.
Using the magnitude of the acceleration vector, 6 features were calculated from the mean, standard deviation, minimum, maximum, difference between maximum and minimum, and zero crossing rate. Magnitude was calculated using Eq 3.3.
where i denotes the sequence of samples.
Feature set B. A total of 10 features were generated in this set. For each of the three axes (x, y, z), mean, standard deviation, and the average absolute difference were calculated making a total of 9 features. Also, the average resultant acceleration of all the three axes was generated using Eq. 3.4.
where i denotes the sequence of samples.

Normalization
Next the extracted features were normalized using Matlab R2017b with the min-max scaling formula given in Eq. 3.5.

Balancing the dataset
The dataset containing the normalized feature values was very unbalanced containing 5,830 non-fall and 1,840 fall data. We balanced the dataset using Matlab R2017b as follows. The different categories of fall data were merged into a single classification of fall data while the two categories of non-fall data were merged into a single classification of non-fall data. Then the fall data was oversampled to create 2,000 samples and the non-fall data was under-sampled to create 2,000 samples to create a combined balanced dataset containing 4,000 samples.

Feature Selection
ReliefF algorithm is one of the filter based feature selection techniques which achieves superior performance in many applications. So, the Relief-F algorithm of the Matlab R2017b feature selection library was used to select the top ten most important features from the combined feature set [23].

System Requirements
All the experiments were run on a system with the following minimum hardware and software requirements:

Experiements and Validation
To validate our models, we performed six experiments with different datasets as listed below. The experimental setup is shown in Fig. 3.1.
Experiment 1: Combined features and multiclass labels.
The combined feature set of 58 features was used as input and 6 classes were identified as the output. The six classes are 0 and 1 for standing and lying, and 2, 3, 4, and 5 for the four fall types FOL, FKL, SDL and BSC as explained in Section 3.1.

Experiment 2: Combined features and 2 class labels.
A combined set of 58 features were used with 2 class labels which are standing and lying as no-fall, and the second one indicating a fall.
Experiment 3: Selected features and multiclass labels.
Here we used a subset of selected features instead of the full feature set with the multiclass label as we did for experiment 1.

Experiment 4: Selected features and 2 class labels.
We used a subset of selected features with the 2-class labels like experiment 2.
Experiment 5: Balanced dataset, combined features and binary labels.
We used the balanced dataset and the combined features in these experiments but with a binary class label as output indicating fall or no-fall.
Experiment 6: Balanced dataset, selected features and binary labels.
In these experiments we used the balanced dataset with the subset of selected features and a binary class label as output indicating fall or no-fall. Accuracy = numberof correctpredictions totalnumberof predictions (3.8)

Model Specification
The LSTM is a variant of the RNN that is capable of learning by remembering order dependence in sequence prediction problems [24]. Typically, a LSTM unit consists of a cell, input gate, forget gate and output gate which enable it to model memory and the models. Each activity in the given dataset was recorded for 10 seconds. So the data was fed into the LSTM accordingly i.e. we used a time step of 10 for all the models. All the output labels were one-hot encoded. One-hot encoding converts the categorical data to a binary sequence for training purposes.

Model Performance
We built LSTM models for the experiments listed in Section 4.4.4 and validated them. The results are shown in  was lost in the process of creating the balanced dataset. LSTM models perform better than other models for time series data due to the ability to remember the sequence.
In creating the balanced dataset, all the fall and non-fall data were randomly resampled. As a result, the sequence information was lost causing LSTM models to perform poorly in the experiments.

Edge Computing Framework: An Overview
We present an edge computing framework for fall detection. Real time data from wearable sensor devices are retrieved by an edge device and sent to Apache Flink for pre-processing and Tensorflow for advanced analysis.

Overview of Framework
We developed a real-time fall detection system using cheap wearable sensor devices.
The system applies our fall detection models to real-time streaming data collected from wearable sensors using an edge computing platform as shown in Fig. 4.1. Our system can be used by the healthcare professionals at the clinics, hospitals and retirement homes and can be extended with other machine learning models to monitor activity, sleep, or tracking dementia patients in real-time. The framework consists of: • Sensors: Multiple sensors can be used to retrieve data from the patients.
• Edge computing node: A node at the edge computing platform retrieves data from the sensors and enables basic data pre-processing and analytics in order to reduce the size of data to be uploaded to a remote server or on the cloud.
• Stream engine: A streaming data processing engine at the edge computing node retrieves the data and performs real-time analysis such as filtering, noise removal, feature extraction, and summarization as necessary.
• Data store: Cleaned and preprocessed data coming out of the stream engine are stored temporarily on the edge platform.

Components
Our framework consists of five main components: Wearable device, Edge device, Apache Flink, Data store and Tensorflow. We developed this pipeline guided by our study of fall detection system model in Chapter 3. It was built so that the streaming data is analyzed as it comes and is stored as a csv file temporarily on the edge device.

Wearable Device
The number of units of wearable device shipments were forecasted to be nearly 400 million in 2018 alone [1]. A huge variety of wearable sensor devices are available today.

Edge Device
An edge computing mechanism brings the computation to the edge of the network so that real time decisions can be taken. It can be done through edge devices like laptops or mobile phones which are closer to the IoT devices. In our framework we use a laptop as it is non-obstructive, mobile and had good computing power to run neural network based analytics. MbientLab provides SDKs to develop applications on the edge device so that sensor data can be collected at our desired rate. We developed a Universal Windows Platform application(UWP app) in C sharp, using the template provided by MbientLab, to stream raw X,Y,Z-axis of the accelerometer data to the streaming engine. Using the SDK the sampling frequency of the accelerometer was set to 50Hz with values ranging from (-2g,+2g). The UWP application can be deployed on any windows operated machine. Fig 4.3 shows the screenshot of an universal windows application displaying all the available wearable devices. When a desired device is clicked, the application initializes the board and its configuration. After completing the initialization user has the option to start or stop the accelerometer.
When the start button is clicked a socket connection is opened and the application starts sending raw accelerometer data to Apache Flink via a socket connection. It provides three levels of abstraction through its API: Process function, Datastream level and SQL/Table level. For our application we use scala API at the datastream level to pre-process the data.

Tensorflow
Tensorflow is an open source library which offers customizable training and layers for the machine learning models. It is one of the libraries that offers advanced machine learning techniques. In our pipeline we used this library to build a long short term memory network which classified whether the activity is a fall or not. We performed six different experiments on the model among which the best one was used for scoring in our data processing pipeline.

Implementation Details
The components in our proposed framework were integrated by using Bluetooth LE or sockets. The data was sent from the wearable devices to the edge device through a Bluetooth LE connection. The data would arrive at the edge device at a specified sampling rate and value range. When the accelerometer was started a socket connection was opened and the raw accelerometer values were streamed to Apache

Data Collection using MbientLab Sensors
The data were collected using an accelerometer present in the MbientLab device, MetaMotionR. Experimental subjects helped in collecting the data to validate the data processing pipeline. The subjects were explained about the four types of falls.
A yoga mat was provided to avoid any unnecessary injuries. All the data collected went through the same pre-processing steps mentioned in Section 3.1 before they were fed into the model.
The data collected and code for execution of the pipeline are available in Github 1 .

Summary
In this chapter we present an overview of the proposed edge computing framework where data is collected from wearable sensor devices using an application deployed on edge device like laptop. The application sends the data to Apache Flink for data pre-processing. When magnitude of the axes exceeds a pre-defined threshold value, a 10 second window of data is sent to Tensorflow which contains a pre-saved LSTM model for fall classification. In Chapter 5 the validation tests for the functionality of the framework are discussed.

Experimentation and Validation of the Framework
We collected data with different configurations to learn about the best possible parameters. The details of the experimental subjects is given in    to 0.31GB. Nevertheless the data reduction also had a disadvantage of losing information so a combination system should be used as Ren et al [27] suggested. They introduced an energy efficient mechanism where data was usually collected at 50Hz and changed to 200Hz when a possible fall was detected.  Thigh had higher rate of noise in the data. We took the first three best performances and formed a sensor combination of waist + leg(calf) and waist + wrist. Data was collected from 3 subjects wearing the first combination of sensors and other 3 wearing the second combination of sensors. The performance of the sensor combination models are shown in Table 5.2. Since the model was trained using MobiAct dataset, from the table, we observe that the

Chapter 6 Conclusion
Fall detection is one of the crucial mechanisms that can enable better care particularly for the elderly people who are more prone to fall to get emergency assistance. We have built an edge computing framework to detect fall automatically using real time monitoring with cheap wearable sensors. Our LSTM model performed the best when correct predictions one second before or after are considered as true positives and when only fall or no-fall are detected. A medium sampling rate of 50Hz and wearable device placed at waist were found to be the optimal settings for fall detection using our selected device. Additionally the use of multi-stream data lead to better performance as well. By training models on existing published datasets and then refining the models proves efficient and can be used more as transfer learning for other use case scenarios. Our future work is focussed on incorporating this framework at the clinics, hospitals and retirement homes, and to test its performance for real patients. In future our system can be expanded by incorporating models to analyze other biometrics like heartbeat of the subject before and after falling, and for tracking patient's activities.