Limitations of prior research

Machine learning techniques have been widely used to recognize users’ activities either using shallow learning or deep learning methods. Shallow learning techniques i.e., K Nearest Neighbour, Naive Bayes, and Random Forest are best-suited algorithms for accurately recognizing different human activities. Moreover, Deep learning methods [15, 16] i.e., Convolutional Networks [17, 18] and Long Short Term Memory(LSTM) [19] due to automatic feature extraction and accuracy are more popular. However, both these methods require a large pool of labeled sensor data that is time-consuming, onerous, and not always feasible or reliable [12]. To gather labeled sensor data in real environments, various annotation techniques have been employed, such as self-reported time-use diaries [20] and ecological momentary assessment [21]. However, these techniques have a history of being disruptive and biased, which can have a direct impact on the activities recorded for context recognition tasks. Other techniques for gathering ground truth, such as recording and later audio or video analysis, are becoming more common, but they may pose ethical and privacy concerns when recording people’s activities. Hence, there is a need for a human behavioral context recognition system that employs less labeled data.

Despite being able to recognise human behavioral context with vast amount of sensors data available in smart devices, the previous researches have still limitations that need to be addressed. Most of the activity recognitions methods are applied on small set of activities [22–24]. Moreover, the prior research is performed in the controlled environment where users are bound to perform specific list of activities in a limited area [25, 26]. Similarly, ubiquitous devices usage is the part of users daily living rather than body worn devices in which users feel burdensome. All the issues discussed with the previous researches cannot truely depict the human behaviour in real life where users are free to do any activity by using smart devices. Therefore, research has to be done in natural and realistic settings, satisfying four natural environmental setup: natural way of device usage, freedom of device placement, natural environment, natural behavioral of user.

Motivation

The availability of an array of built-in sensors in daily used devices has potential to capture human behaviour that is applicable in many fields i.e., Aged care [1], disease prevention [4] etc. For instance, in Australia, it is projected that by 2051, people aged 85 and above will account for 9.1% of the overall population, with the number of individuals aged 100 and above increasing from 2, 503 in 2001 to 38, 000 in 2051 [27]. The growing number of elderly individuals has led to a significant increase in demand for care services. The urge for independent life in the elderly has stimulated the development of ubiquitous systems that monitor them without being burdensome, remind them to take their medications or eat their meals, and notify authorities or family members if something goes wrong [3, 28]. Moreover, preventive interventions for drug addiction have become increasingly important in light of the pandemic, family issues, peer pressure, depression, and financial difficulties faced by young people. Reducing drug usage is one of the main goals of these interventions, which can be made possible through the behavior control of drug users [29]. Thus, accurate identification systems are needed for successful elderly care and behavior control of addicts.

Contribution

In this paper, we propose a novel Active Learning(AL) based query strategy that overcomes the gap of label requirement in supervised learning. AL is a semi-supervised machine learning approach that requires a small set of labeled data and queries informative samples from a large pool of unlabeled data. Using a small set of labeled data, a model is trained that helps in the query strategy. The purpose of query strategy is to measure the informativeness of unlabeled samples. AL is an iterative process in which query strategy works by selecting only informative samples using uncertainty measure from the pool in each stage, thus reducing the annotation load. We also investigated the impact of temporal data, such as time of day (morning, evening, etc.), on recognizing user behavior in natural settings. To evaluate our proposed methodology, we used a publicly available dataset of users performing diverse activities in a natural environment.

Key contributions of our research work is as follows:

• We proposed and implemented a novel technique i.e., DBQS that aims to reduce the user’s annotation effort.
• We exploited the use of time-series analysis to find the diverse and unique samples in the data. Empirically, our technique outperformed when selected sample pool is selected using temporal information.
• Proposed conditional DBQS that assigns higher weight to the unique samples. Intuitively, conditional DBQS reduces the annotation load by recognizing users’ diverse behavior.
• Investigate the mutual influence of temporal information and conditional DBQS on sensor data to build an accurate model with less annotation load.

The subsequent sections of the paper includes related work, proposed methodology, experimental evaluation, discussion and conclusion.

Supervised approaches

Most of the previous research in the domain of HAR [24, 37–39] is based on recognizing basic physical activities like sitting, standing, running, walking, etc. To improve the performance of these recognition systems, context information is then combined with these activities i.e., Sitting at Home/Office, Sitting in a Bus, Sitting in Train, Standing at Home/Office, etc. In this regard, Otebolaku et. al. [35] collected data from 6 individuals in the controlled setting using accelerometer, orientation, and rotation vector sensors of user wearable mobile devices. Different machine learning techniques are applied to analyze the performance and demonstrate an overall performance of 98% using cross-validation. Similarly to get the advantage of contextual data, a unique activity-aware human context recognition(AAHCR) technique is proposed by [36]. Their proposed approach learns human activity routines in various behavioral contexts and predicts user contexts. Daily living physical activities, e.g. sitting, lying, walking, etc., are inferred along with variety of possible contexts, i.e., meeting, in a car, surfing the internet, or lying on bed. They evaluated their proposed approach using a list of machine learning models, of which Random Forest achieved the best recognition rate in recognizing various behavioral contexts, showing their proposed method’s effectiveness. Moreover, Niemann et. al. [22] proposed a method to identify activities for industrial processes, and their experiments on a laboratory setup showed that incorporating context information significantly improves activity detection performance. Bhandari et. al. [23] created a lightweight and precise system on smartphones to detect human activity like standing, sitting, laying, walking, walking upstairs, and walking downstairs.

Moreover, a sensor fusion technique is applied by Vaizman et. al. [9] that concatenates data from multiple sensors to improve the overall context recognition accuracy. The data was collected in natural setting from various smartphone and smartwatch sensors. They have used logistic regression, a linear classifier, and the results demonstrated that fusion of multi-modal sensors helps in improving the recognition of user behavioral context in the wild. Zhong et. al. [34] investigated the contribution of mobile phones or smart watches sensors towards machine learning based models for HAR. In real-time environments, it can be challenging to obtain labeled data necessary for supervised learning techniques, particularly in the context of HAR. Consequently, it becomes necessary to explore unsupervised learning techniques using unlabeled data [40].

Deep learning has achieved some success in recent years, by modeling high-level abstractions from complex data. These techniques have demonstrated success in many areas such as computer vision [41], image classification [42], natural language processing [43], and speech recognition [44]. These mostly depend on feature extraction techniques [45] such as statistical methods [46], time-frequency transformation [47], and symbolic representation [48]. Next, we are listing some related articles on HAR using deep learning methods and then AL integrated with deep learning.

The majority of deep learning research aimed at tackling issues associated with HAR data, such as accuracy, class imbalance, hyper-parameter configuration, automated feature extraction, computational and memory costs, employs Convolutional Neural Network(CNN). In a work proposed by Lee et. al. [49], inertial and ambient sensors data are fused to address class imbalance problem using Deep Convolutional Neural Network (DCNN). Evaluation and analysis of their proposed system showed an improvement of 5.3% in accurate recognition. Similarly, Hammerla et. al. [50] analyzed different deep learning models i.e. Deep neural network, convolutional network and recurrent neural network with the perspective of hyper parameters. In more than 4, 000 experiments they investigated the suitability of each model for different tasks in HAR. Further, they also explored the impact of each model’s hyper parameters on performance, and proposed guidelines for the application scenario.

Furthermore, Yang et al. [17], automated the feature extraction through CNN. The multichannel time series data, which primarily uses convolution and pooling techniques to capture the key patterns of the sensor signals at various time scales, is also investigated using CNN. They showed that the proposed CNN method outperforms other state-of-the-art methods and can be used as a competitive feature learning and classification tool for the HAR challenges. Similarly, Khan et. al. [51] created a hybrid model for activity recognition by combining CNN and LSTM. LSTM network is used to learn temporal information, whereas CNN is used to extract spatial features. They conducted a thorough ablation study, developed a new dataset and created a novel hybrid model powered by deep learning to track and recognize human physical activity in an indoor environment. The CNN-LSTM approach yields an accuracy of 90.89%, demonstrating the suggested model’s suitability for HAR applications. Another research [25] was conducted in the similar manner by hybridizing CNN with a bidirectional (1D-CNN-BiLSTM) model for wearable sensor-based HAR and reported good performance as compared to existing methods. Recently, [52] designed an accurate and efficient MLP architecture with a stack of dense layers to recognize six daily human activities. Their study used small and large feature sets of gyroscope and accelerometer sensors for HAR and their results outperformed LSTM, CNN, and other machine-learning models.

One of the major challenges of HAR model is to develop a highly accurate model with low cost sensors [19] or less sensor units [53]. A Bidirectional-Gated Recurrent Unit-Inception (Bi-GRU-I) model was created by Tong et al. [53] to increase accuracy while minimizing the number of parameters. Their testing using datasets from the University of California, Irvine (UCI-HAR), Wireless Sensor Data Mining (WISDM), and CATP, which they self-collected, demonstrates improved performance and robustness. Moreover, the sensor configuration optimization was examined, and it was shown that this approach may be used with smaller sensor units. Similarly, in [19], LSTM Networks are employed to enable the model to understand and recognise the various activities that the user is carrying out. The UCI-HAR [54] dataset is used to train the model to recognise and identify six different activities, including sitting, walking, lying down, standing up, going up stairs, and going down stairs. The complexity of updating each weight is lowered to O(1) using LSTM. Training becomes simpler and produces great accuracy as weights are updated more quickly.

All the discussed literature on shallow and deep models demand a wide range of annotated data. Besides the advantage of good performance and automatic feature extraction, the main drawback of deep models is computational cost than the conventional shallow learning approaches. Despite being able to collect sensors data ubiquitously, limited availability of quality labeled data remains one of the main challenge [55] in HAR. The issue arises primary due to the fact that data annotation is to be done by the user. Due to this issue, researchers are actively seek solutions that require partial or no labeled data. For this purpose, multiple semi-supervised [37, 56–58] and unsupervised [57, 59, 60] approaches are proposed.

Semi-supervised approaches

Semi-supervised approaches are the broad category of machine learning techniques between supervised and unsupervised learning i.e., self-training, co-training and active learning etc. Semi-supervised learning is a strategy for training models that uses both labeled and unlabeled input and is employed when just a small amount of labeled data is available. For example, in AL [61], model is trained using small portion of labeled data and gradually quality of unlabeled samples are measured so that to be added in the labeled set. The primary idea underlying semi-supervised learning is the labeling of only a small sample of data that may result in the same or greater accuracy than completely labeled training data. The only challenge is finding what that sample is in gaining good accuracy. AL is all about labeling data dynamically and gradually during the training phase so that the algorithm can determine which label is the most informative to learn from it. The use of AL in HAR has been studied extensively either in a pool-based setting(offline) or a stream-based(online) setting. With a pool-based approach, the model has access to all unlabeled data and picks the best from it. Stream-based approach, on the other hand, progressively reads unlabeled data and selects the informative one to be queried by the oracle or user. To enhance the performance of an initial classifier, Longstaff et. al. [37] investigated various active and semi-supervised learning approaches to label the activity data stream. For this, they compared AL and three different semi-supervised learning methods i.e., self-learning, democratic co-learning and En-Co-Training. Although democratic co-learning was nearly as effective and did not require user interaction, but AL showed the greatest improvement. In [62], a semi-supervised auto labeling approach is proposed with the key intuition that labeling, being burdensome on user, could lead to a high misclassification rate. Multiple factors like hectic working/living conditions, misleading emotional fluctuations etc. can possibly result in incorrect labeling. For experimentation, they used dataset collected in the natural setting i.e., [30] and [63]. To improve accuracy, lessen overfitting, and solve imbalanced class problem, a special combination of augmentation, scaling, and boosting is applied.

AL is used by System Maestro [26], Liu et al. [64] and Shahmohammadi et. al [24] to reduce annotation load. System Maestro [26] is a data collection and labeling framework that was developed to work in a smart ambient environment with 5 different ambient sensors. AL was employed to achieve accuracy more than 95%, with less than 10% labeled examples. Moreover, Liu et al. [64] presented early investigation of pool-based AL using multi-sensor data of daily physical activities in a controlled setting and demonstrated the effectiveness of AL approach as compared to supervised approaches. Shahmohammadi et. al. [24] described a smartwatch-based AL method in stream-based setting to identify 5 commonly performed daily activities and revealed that smartwatches has significant advantages over smartphones and other devices for activity recognition tasks. NOHAR(NOvelty discrete data stream for Human Activity Recognition) [65] is an attempt to address the issue of computational resources using AL. Their experiments in the controlled setting proved that NOHAR can cut memory usage by an average of 99.97% and 13 times efficient than the state of the art.

AL with the advantage of less input labeled data requirement has been used with deep models to gain the strength of deep learning. The research combined AL with deep learning e.g., DeActive [66, 67], ActiveHARNet [68] proposed a deep and AL-enabled activity recognition model, and their experimentation on real-world data showed optimum accuracy using less labeled instances. Bettini et. al. [69] used AL with federated learning to address the issue of data scarcity, proactively annotation of unlabeled sensor data and construction of personalised models.

The above discussed literature on deep learning and AL integrated deep learning still have high computation cost when compared with shallow models. Moreover, they are being experimented on controlled setting with few activity set and number of participants that may fail while experiencing in the real world setup. To avoid complexity, simple AL technique is found in many articles. Comparative analysis of related articles under controlled and natural setting are shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparative analysis of literature on Human Activity Recognition under controlled and natural setting using different machine learning approaches.

https://doi.org/10.1371/journal.pone.0286919.t001

Adaimi et. al. [12] investigated AL to reduce annotation load in predicting users’ activities in natural and controlled settings. The intention of using AL is to generate a HAR model with high accuracy and less labeled data requirement. Using four separate publicly available HAR datasets collected in different environments, they tested existing AL techniques in pool-based and stream-based settings. They proposed a cluster based diverse query strategy that considers informative samples from the pool. This maintains the data class ratio in each batch for each iteration. It also ensures diversity in each batch and takes few most informative samples thus inducing promising results. However, this approach guarantees diversity in each batch but introduces stagnation problem that may arise when similar samples might be a part of a batch, thus not improving recognition performance.

In recent studies, the query strategies like maximum entropy, Query by Committee(QBC), random sampling, and cluster-based diversity [12] have not been shown to perform well for human context recognition w.r.t. balanced accuracy because of users’ diverse behavior. Naturally, users’ behavior is diverse from each other but their actions are similar in several situations [62]. Activities of different people exhibit a certain level of similarity because of biological features (e.g., gender, height, weight, etc.), the physical environment (e.g., where the subjects move about), or even sensor biases [77]. This behavior of users inspired us to formulate the query strategy that helps in finding the most informative samples that reduce the annotation effort detailed in section DBQS: Dissimilarity-Based Query Strategy.

All these researches on HAR using AL are based on a controlled setting with few activities and a small number of participants. Compared to previous work, in this paper, we present a more thorough investigation of pool-based AL on a larger dataset of user contextual activities from multi-sensor devices i.e., smartphone and smartwatch in the natural setting.

Data collection

Human behavior is influenced by multiple factors like environmental, physiological, and psychological factors. One of the recent trends in understanding human behavior is the use of smart devices like smartphones and smartwatches. In this work, we acquired the data collected from the sensors of the smart devices like phone accelerometer (pAcc), phone gyroscope (pGyro), and watch accelerometer (wAcc). Eq 1 summarizes the data collected from the smart devices. (1) where I is the input instance, X represents the raw input sensor data, and Y represents the ground truth information. Specifically, (2) and (3) where in Eq 2, pAcc, pGyro, pAud, pLoc, pPS, wAcc represent raw sensor readings from the smartphone’s Acc, gyroscope, audio, GPS, state, and smart watch’s accelerometer respectively, collected at a certain time stamp pTS.

Whereas, in Eq 3, actLabel refers to physical activity like running, walking, standing, etc. and actEnv refers to the environmental information like: outdoors, at the workplace, at home, in gym, etc.

Our proposed system takes N × d dimensional data, where N represent number of samples and d is sum of all sensors axes representing the columns of data X as shown in Eq 4: (4) while Y represents the corresponding ground truth information, i.e.: (5)

In Eq 5, N is the number of samples and L is the label in the form of binary representation where 1 and 0 means if a particular activity being performed or not performed respectively.

In this work, we intend to take into account several actions occurring at once. For instance, “Walking, Outdoor, Shopping, Phone on desk” or “Sitting, in car, Phone in hand, Talking, With friends”. Thus, input to our proposed system will be a single instance of all sensor readings along with multiple activities or labels.

However, there are some challenges associated with the acquisition of data in a real-time environment from multiple sensors e.g.,

• Each sensor’s raw data acquired from smart devices is in a different format e.g. Acc, Gyro, WAcc [x,y,z], Loc [longitude, latitude], and Audio is in the form of msec frames and PS is of discrete type. All these types of raw data are not useful.
• The raw data generated by different sensors may contain various types of noise. Therefore, signals are pre-processed before classification to eliminate the unwanted noise from raw data. [78].
• All the sensors have different sampling rates i.e., the rate at which the samples are taken, may lead us to inaccurate human behavior recognition due to biasness in sensing frequency.
• All the sensors are not available at the same time, e.g., the user might remove the watch while sleeping, turned off the location service, or during a phone call audio was not available (to preserve privacy) [8].

These challenges lead us to preprocess the data so that accurate human behavior recognition is possible. Next subsection will outline the methods used.

Data pre-processing

As data were collected from real world stream of users performing multiple activities in the wild, there is a clear chance of noise that requires preprocessing [79], as a good processing mechanism guarantees a good recognition. All data preprocessing steps are described in the next subsections.

Active Learning framework

One of the major issues in real-world studies using supervised methods is the demand of enormous amount of labeled training data to reliably learn human behaviour. It has become clear that providing thorough and reliable ground truth labeling for the enormous amount of data takes time and is not always practical or reliable.

Thus in order to get the advantage of only available labeled data we are focusing on using the AL approach. AL’s central premise is that, with less training data, a learning algorithm may be able to perform as well as or even better than standard supervised approaches provided it can select the most informative data to learn. Thus, AL model only queries for the data that can iteratively increase knowledge and performance. There are two frameworks proposed in research for AL: (1) pool-based and (2) stream-based. We have only employed pool based approach in this research and every stage of the framework is detailed in the next subsection.

Pool-based Active Learning.

This type of approach is based on two types of data, (1) Seed: which is a tiny subset of labeled data, and (2) Pool: which is a vast pool of unlabeled data. This AL approach, as shown in Fig 2 starts with a seed of the data for training. The size of the seed is a user-defined hyperparameter. The Machine Learning model is trained on the seed batch, and the AL pool-based selective sampling method is used after the model has been fitted. In a pool-based selective sampling strategy, the pool of unlabeled data is available to the model on which it applies the query strategy. The query strategy chooses a new sampled batch consisting of the best informative samples and add these to the already labeled data. Then model is updated, and the process repeats with the updated data. The sampled batch size is also user-defined hyperparameter. The selection of the best samples is decided according to a proposed DBQS query strategy as detailed in the following subsection.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. DBQS (Dissimilarity-Based Query Strategy): Pool based Active Learning framework.

https://doi.org/10.1371/journal.pone.0286919.g002

Model evaluation

To quantify the performance of AL model, we used BA [83] as this suits the dataset because of the imbalanced nature of user activities. BA is the arithmetic mean of the sensitivity and specificity: (10) (11) (12) whereas in Eq 10, BA is balanced accuracy, in Eq 11, TP is a true positive, FN is a false negative, in Eq 12, TN is a true negative and FP is the false-positive case.

Experimental setup

We have applied our proposed technique on ExtraSensory DataSet [30] collected by Yonatan and Ellis that contains over 300k examples(minutes) of 60 users performing Daily Living Activities (DLAs) in the natural setting. The collection includes data from five smartphone sensors (34 iPhone and 26 Android): Accelerometer (Acc), Gyroscope (Gyro), Location (Loc), Audio (Aud), and Phone State sensors (Phone State), as well as accelerometer readings from a wristwatch (WAcc). Raw data collected from smart devices is then segmented into 20 seconds non-overlapping windows that is further processed to create feature vector of each window against each sensor. Further for evaluation purpose, we repeated the experiment with 40 seconds window. Few of the features from ExtraSensory Dataset [30] are listed in the Table 2. Along with these feature sets, we also extracted Temporal features from the time stamp. For this, we generated two combinations: (1) Time Variation 3: A day is divided into three parts i.e., morning, evening, and night with hour wise division as 4 : 00 to 12 : 00, 12 : 00 to 18 : 00, and 18 : 00 to 4 : 00 respectively (2) Time Variation 6: A day is divided into six parts like early morning, morning, noon, evening and night and late night with hour wise division as 4 : 00 to 6 : 00, 6 : 00 to 12 : 00, 12 : 00 to 15 : 00, 15 : 00 to 18 : 00, 18 : 00 to 23 : 00 and 23 : 00 to 4 : 00 respectively. This division is in line of keeping in view of users’ diverse behavior during different parts of the day. The dataset contains numerous binary context labels for each occurrence of the selected DLAs, which offer information on the human behavioural context (such as the user’s activity, circumstance, location, social context, and phone position) when performing a physical activity in a realistic setting. As a result, we chose this publicly available Extrasensory dataset [30] for developing and verifying the suggested strategy since it fits into the proposed method’s process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Feature set of ExtraSensory data.

https://doi.org/10.1371/journal.pone.0286919.t002

The total context labels available in the dataset are 51 and for implementation, we selected 22 frequent occurring labels that include “sitting, lying-down, walking, running, bicycling, sleeping, lab-work, in-class, in-a-meeting, loc-main-workplace, indoors, outside, in-a-car, drive-i-m-the-driver, drive-i-m-a-passenger, at home, restaurant, phone in pocket, exercise, cooking, strolling, bathing-shower”. To extend the experiment performed by a cluster-based sampling scheme [12], we applied DBQS. In DBQS, for uncertainty, we used a margin-based uncertainty measure, and dissimilarity is calculated using the inverse of similarity measurement of each new batch with the previously selected data.

More details of the dataset and experimental setup are described below.

The experimental setup comprises two parts: (1) the Single-Sensor classifier and (2) the Early Fusion(EF) approach. In Single-Sensor classifier for each of the 5 sensors—accelerometer (Acc), gyroscope (Gyro), watch accelerometer (WAcc), location (Loc), and audio (Aud), phone state (PS) for a given context label, classification is done based on each sensor independently. In the EF approach features from multiple sensors are concatenated before classification. In our experimentation, we used all the features provided in the ExtraSensory Dataset [30] for the mentioned sensors. Both variations of temporal data are concatenated with Single Sensor approach and EF approach separately. For evaluation purpose, we implemented a cluster-based sampling scheme [12] as a base-line method. We used logistic regression with balanced class weights. Further for evaluation we used BA by partitioning the data into the ratio of 80 : 20 thus, 60 users data into train (48 users) and test (12 users) sets. For validation testing, we used In the following experiments, the initial batch size and the pool-based queried batch size were empirically set to 2%.

Impact of proposed query strategy on balanced accuracy and training data requirement

In the case of Acc, BA gains 59% to 68% with upto 24% training set using 2% batch size and then stabilizes after that. For Gyro, BA of 56% to 66% is gained with 40% training set. Proposed query strategy has no impact on WAcc data due to too much variation in users’ behavior in hand motion. Aud sensor data gains BA of 55% to 67%, with upto 24% training set for proposed query strategy. With Loc data, only 1% of increment is observed at the start of few iterations. PS sensor data stabilizes at 73% BA with upto 60% training set using proposed query strategy. In the case of EF, with the use of almost 10% training data, BA stabilizes to 75%.

Impact of proposed TV on balanced accuracy and training data requirement

In the case of the Acc sensor, for TV-3 and TV-6, BA stabilizes to 68% with only 16% training data and 71% with only 26% training set respectively. For Gyro, for TV-3 and TV-6, 68% and 71% BA is observed with 54% and 60% training set respectively. WAcc with TV-3 and TV-6, BA increases from 57% to 68% with 40% training set and 62% to 72% with 62% training data. For Aud TV-3 and TV-6, BA stabilizes to 69% and 72% with 60% training set. PS sensor showing the same curve of baseline method. Loc sensor shows a prominent increase of 3% and 7% in BA with only 20% training set with TV-3 and TV-6 respectively. EF approach shows the same curve but at some points, BA goes down than baseline method.

Impact of proposed query strategy and TV on balanced accuracy and training data requirement

Given that TV-6 yields better results as compared to TV-3, we have only considered TV-6 when combined with the proposed query strategy. When both schemes are applied, the Acc sensor stabilizes after 18% training data with 70% BA. Aud and PS sensor stabilizes after 28% training data with 69% and 73% BA respectively. Loc and Gyro stabilize after 22% training data with 67% and 69% BA respectively. WAcc stabilizes after 14% training data with 68% BA.

Impact of increased window size on proposed methods results

We have repeated the experimentation for a greater window length of 40 seconds(40s) to analyze the final performance. In Fig 4, the method TV-6 and DBQS with TV-6 showing the same recognition pattern against all the sensors. Here we can clearly see the overlapping pattern. However in the case of PS sensor, DBQS is performing better.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Results of single sensors approach for the proposed methods (Dissimilarity-Based Query Strategy,Time Variation-6 and combined) with window length of 40s.

https://doi.org/10.1371/journal.pone.0286919.g004

Comparative performance of DBQS and TV

The bar chart plotted in Fig 5 shows the Best Performance(BP) and Comparable Performance(CP) of single sensor classifiers when setting T = 4, 5, 6 as the stabilization wait time (consecutive number of times when BA stabilizes at a point). As T increases, CP gets closer to BP because more training data is required to achieve the BP. There is a significant difference in the percentage of training data between CP and BP as shown in the horizontal axis of Fig 5. This significant difference is the main achievement of our proposed methodology that shows how much annotation load is reduced when less amount of more informative and dissimilar samples are selected.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Comparative performance of Dissimilarity-Based Query Strategy, Time Variation-3, Time Variation-6, Dissimilarity-Based Query Strategy with Time Variation-6 and base line when setting stabilization wait Time T = 4, 5, 6.

The x-axis labels include the percentage of the training data used to achieve the corresponding Comparable Performance when Balanced Accuracy stabilizes and the Best Performance.

https://doi.org/10.1371/journal.pone.0286919.g005

From the bar plots, we concluded that TV-6 has achieved highest BA. To further visualize the categorization results of the proposed methods, we repeated the experiment with 80% and 20% data distribution. For this, we made different setting by imputing all missing labels to 0 for extracting training and testing data from the whole dataset. The purpose behind this setting is to get same number of instances for all labels to compute confusion matrix. We plotted the normalized multi-label confusion matrices(MLCM) [84] as shown in Figs 6–8, of our TV-6 method against Acc, WAcc, Aud, Gyro, Loc, PS sensor only at the stabilization time (T = 5). When setting T = 5, all the labels in each sensor converged at different percentage of training data, i.e., in the case of Acc, label sitting converged at 30% BA with 20% training data. Due to the highly imbalanced nature of users’ activities data diagonal entries not showing good accuracies. Further, a few activity labels i.e., strolling and bathing shower, have not been recognized due to the availability of a single class in this partitioning.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Confusion matrix for accelerometer and watch accelerometer sensors at the stabilization time, T = 5.

https://doi.org/10.1371/journal.pone.0286919.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Confusion matrix for audio and gyroscope sensors at the stabilization time, T = 5.

https://doi.org/10.1371/journal.pone.0286919.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Confusion matrix for location and Phone State sensors at the stabilization time, T = 5.

https://doi.org/10.1371/journal.pone.0286919.g008

We present a detailed comparative analysis on the annotation load of all sensors when stabilization wait time is set to 6. As compared to base line’s BP, DBQS along with TV-6 reduces the annotation load for accelerometer, audio, location and gyroscope by 6%, 18%, 8% and 46% respectively. No annotation reduction is found in the case of WAcc. When compared to base line’s CP, DBQS along with TV-6 reduces the annotation load for Acc, Aud, PS and Loc by 6%, 8%, 18% and 4% respectively. No annotation reduction is found in the case of watch accelerometer and gyroscope. For T = 4 and 5, almost similar annotation reduction can be observed.

Among all sensors, except the PS, TV-6 has high BA that shows more details of temporal data significantly impacts accurate recognition of users’ behavior in the natural setting. For example, accelerometer recognises all activities with 73% BA using TV-6 that is 4% better than cluster-based sampling scheme [12].