Skip to main content

Advertisement

SpringerLink
Log in

A survey on the use of machine learning methods in context-aware middlewares for human activity recognition

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Human activity recognition (HAR) essentially uses (past) sensor data or complex context information for inferring the activities a user performs in his daily tasks. HAR has been extensively studied using different paradigms, such as different reasoning mechanisms, including probabilistic, rule-based, statistical, logical reasoning, or the machine learning (ML) paradigm, to construct inference models to recognize or predict user activities. ML for HAR allows that activities can be recognized and even anticipated through the analysis of collected data from different sensors, with greater accuracy than the other paradigms. On the other hand, context-aware middlewares (CAMs) can efficiently integrate a large number of different devices and sensors. Moreover, they provide a programmable and auto-configurable infrastructure for streamline the design and construction of software solutions in scenarios where lots of sensors and data are their bases, such as ambient intelligence, smart cities, and e-health domains. In this way, the full integration of ML capabilities as services in CAMs can advance the development of software solutions in these domains when ML is necessary, specially for HAR, which is the basis for many scenarios in these domains. In this work, we present a survey for identifying the state-of-the-art in using ML for HAR in CAMs through a systematic literature review (SLR). In our SLR, we worked to answer four research questions: (i) what are the different types of context reasoners available in CAMs; (ii) what are the ML algorithms and methods used for generating models for context reasoning; (iii) which CAMs support data processing in real time; and (iv) what are the HAR scenarios usually tackled by the research works. In our analysis, we observed that, although ML offers viable approaches to construct inference models for HAR using different ML approaches, including batch learning, adaptive learning and data stream learning, there are yet some gaps and research challenges to be tackled, specially on the use of data stream learning considering concept drift on data, mechanisms for adapting the inference models, and further considering all of this as services in CAMs, specially for HAR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://www.engineeringvillage.com/home.url.

  2. https://www.webofknowledge.com/.

  3. https://www.sciencedirect.com/.

  4. https://www.scopus.com.

  5. https://link.springer.com/.

References

  • Abdallah ZS, Gaber MM, Srinivasan B, Krishnaswamy S (2018) Activity recognition with evolving data streams: a review. ACM Comput Surv 51(4):71:1-71:36

    Article  Google Scholar 

  • Abowd GD, Dey AK, Brown PJ, Davies N, Smith M, Steggles P (1999) Towards a better understanding of context and context-awareness. In: International symposium on handheld and ubiquitous computing. Springer, pp 304–307

  • Adelsberger R, Tröster G (2013) Pimu: a wireless pressure-sensing imu. In: Intelligent sensors, sensor networks and information processing, 2013 IEEE 8th international conference on. IEEE, pp 271–276

  • Alam MGR, Haw R, Kim SS, Azad MAK, Abedin SF, Hong CS (2016) Em-psychiatry: an ambient intelligent system for psychiatric emergency. IEEE Trans Ind Inf 12(6):2321–2330

    Article  Google Scholar 

  • Alti A, Laborie S, Roose P (2015) Cloud semantic-based dynamic multimodal platform for building mhealth context-aware services. In: 2015 IEEE 11th international conference on wireless and mobile computing, networking and communications (WiMob), pp 357–364. https://doi.org/10.1109/WiMOB.2015.7347984

  • Anjomshoa F, Aloqaily M, Kantarci B, Erol-Kantarci M, Schuckers S (2017) Social behaviometrics for personalized devices in the internet of things era. IEEE Access 5:12199–12213

    Article  Google Scholar 

  • Archetti F, Djordjevic D, Giordani I, Sormani R, Tisato F (2014) A reasoning approach for modelling and predicting terroristic attacks in urban environments. In: Security technology (ICCST), 2014 international Carnahan conference on. IEEE, pp 1–6

  • Bellavista P, Corradi A (2016) The handbook of mobile middleware. Auerbach Publications

  • Bi Y, Guan J, Bell D (2008) The combination of multiple classifiers using an evidential reasoning approach. Artif Intell 172(15):1731–1751

    Article  Google Scholar 

  • Bifet A, Gavalda R, Holmes G, Pfahringer B (2018) Machine learning for data streams with practical examples in MOA. MIT Press

  • Bikakis A, Patkos T, Antoniou G, Plexousakis D (2007) A survey of semantics-based approaches for context reasoning in ambient intelligence. In: European conference on ambient intelligence. Springer, pp 14–23

  • Bobek S, Porzycki K, Nalepa GJ (2013) Learning sensors usage patterns in mobile context-aware systems. In: FedCSIS, pp 993–998

  • Celdrán AH, Pérez MG, Clemente FG, Pérez GM (2014) Precise: privacy-aware recommender based on context information for cloud service environments. IEEE Commun Maga 52(8):90–96

    Article  Google Scholar 

  • Chen H, Finin T, Joshi A et al (2003) An intelligent broker for context-aware systems. Adjun Proc Ubicomp 2003:183–184

    Google Scholar 

  • Chen M, Gonzalez S, Vasilakos A, Cao H, Leung VC (2011) Body area networks: a survey. Mob Netw Appl 16(2):171–193

    Article  Google Scholar 

  • de la Concepción MÁÁ, Morillo LMS, García JAÁ, González-Abril L (2017) Mobile activity recognition and fall detection system for elderly people using Ameva algorithm. Pervas Mob Comput 34:3–13

    Article  Google Scholar 

  • de Sousa Pinto A, Bernardini F, Miranda L, Viterbo J, Meza EM (2020) An exploratory study using stream learning algorithms to predict duration time of vehicle routes. In: 2020 international conference on systems. Signals and image processing (IWSSIP). IEEE, pp 299–304

  • Dey AK, Häkkilä J (2008) Context-awareness and mobile devices. In: Handbook of research on user interface design and evaluation for mobile technology. IGD-Global, chap 13

  • Dimitrievski A, Zdravevski E, Lameski P, Trajkovik V (2016) A survey of ambient assisted living systems: challenges and opportunities. In: Intelligent computer communication and processing (ICCP), 2016 IEEE 12th international conference on. IEEE, pp 49–53

  • Dimitrov T, Pauli J, Naroska E (2007) A probabilistic reasoning framework for smart homes. In: Proceedings of the 5th international workshop on middleware for pervasive and Ad-Hoc computing: held at the ACM/IFIP/USENIX 8th international middleware conference, association for computing machinery, New York. MPAC ’07, p 1-6, https://doi.org/10.1145/1376866.1376867

  • Doherty AR, Caprani N, Conaire CÓ, Kalnikaite V, Gurrin C, Smeaton AF, O’Connor NE (2011) Passively recognising human activities through lifelogging. Comput Hum Behav 27(5):1948–1958

  • Dragone M, Amato G, Bacciu D, Chessa S, Coleman S, Di Rocco M, Gallicchio C, Gennaro C, Lozano H, Maguire L et al (2015) A cognitive robotic ecology approach to self-configuring and evolving AAL systems. Eng Appl Artif Intell 45:269–280

    Article  Google Scholar 

  • Eddy SR (1996) Hidden Markov models. Current Opin Struct Biol 6(3):361–365

    Article  Google Scholar 

  • European Comission (2019) Ethics guidelines for trustworthy AI. https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai

  • Faial D, Bernardini F, Miranda L, Viterbo J (2019) Anomaly detection in vehicle traffic data using batch and stream supervised learning. In: EPIA conference on artificial intelligence. Springer, pp 675–684

  • Fong J, Indulska J, Robinson R (2013) A framework to support intelligibility in pervasive applications. In: Pervasive computing and communications workshops (PERCOM workshops), 2013 IEEE international conference on. IEEE, pp 37–42

  • Forkan A, Khalil I, Tari Z (2014) Cocamaal: a cloud-oriented context-aware middleware in ambient assisted living. Fut Gener Comput Syst 35:114–127. https://doi.org/10.1016/j.future.2013.07.009. Special Section: Integration of Cloud Computing and Body Sensor Networks; Guest Editors: Giancarlo Fortino and Mukaddim Pathan

  • Fortino G, Galzarano S, Gravina R, Li W (2015) A framework for collaborative computing and multi-sensor data fusion in body sensor networks. Inf Fus 22:50–70

    Article  Google Scholar 

  • Frank E, Hall MA, Witten IH (2016) Data mining: practical machine learning tools and techniques, 4th edn. Morgan Kaufmann

  • Frey J (2013) Adapt—a dynamic approach for activity prediction and tracking for ambient intelligence. In: Intelligent environments (IE), 2013 9th international conference on. IEEE, pp 254–257

  • Gama J, Medas P, Castillo G, Rodrigues P (2004) Learning with drift detection. In: Brazilian symposium on artificial intelligence. Springer, pp 286–295

  • Gani MO, Saha AK, Ahsan GMT, Ahamed SI, Smith RO (2017) A novel framework to recognize complex human activity. In: Proc. 41st IEEE annual computer software and applications conference, pp 948–956

  • Gayathri K, Elias S, Shivashankar S (2015) Composite activity recognition in smart homes using Markov logic network. In: Ubiquitous intelligence and computing and 2015 IEEE 12th intl conf on autonomic and trusted computing and 2015 IEEE 15th intl conf on scalable computing and communications and its associated workshops (UIC-ATC-ScalCom), 2015 IEEE 12th intl conf on. IEEE, pp 46–53

  • Goodfellow IJ, Mirza M, Xiao D, Courville A, Bengio Y (2013) An empirical investigation of catastrophic forgetting in gradient-based neural networks. ArXiv preprint arXiv:1312.6211

  • Hanneke S (2014) Now foundations and trends. https://doi.org/10.1561/2200000037

  • Hasse D, De Rolt CR (2017) Collega semantic middleware for collaborative assistance in mobile social networks. In: AEIT international annual conference. IEEE, pp 1–6

  • Hong J, Suh E, Kim SJ (2009) Context-aware systems: a literature review and classification. Expert Syst. Appl. 36(4):8509–8522

    Article  Google Scholar 

  • Hoque MR, Kabir MH, Jang JH, Yang SH (2014) Middleware aided context-aware service for smart home. In: Consumer electronics (ISCE 2014), the 18th IEEE international symposium on. IEEE, pp 1–2

  • Horrocks I, Patel-Schneider PF, Boley H, Tabet S, Grosof B, Dean M, et al. (2004) SWRL: a semantic web rule language combining owl and RuleML. W3C Memb Submiss 21:79

  • Huang Z, Lin KJ, Tsai BL, Yan S, Shih CS (2018) Building edge intelligence for online activity recognition in service-oriented IOT systems. Fut Gener Comput Syst 87:557–567. https://doi.org/10.1016/j.future.2018.03.003

  • Jurek A, Nugent C, Bi Y, Wu S (2014) Clustering-based ensemble learning for activity recognition in smart homes. Sensors 14(7):12285–12304

    Article  Google Scholar 

  • Karakostas A, Lazarou I, Meditskos G, Stavropoulos TG, Kompatsiaris I, Tsolaki M (2016) Intelligent user interfaces to support diagnosis and assessment of people with dementia: an expert evaluation. In: Serino S, Matic A, Giakoumis D, Lopez G, Cipresso P (eds) Pervasive computing paradigms for mental health. Springer, Cham, pp 196–206

  • Kasarla T, Nagendar G, Hegde GM, Balasubramanian V, Jawahar C (2019) Region-based active learning for efficient labeling in semantic segmentation. In: In proc. 2019 IEEE winter conf. app. computer vision (WACV)

  • Keshtkar Langaroudi M, Yamaghani M (2019) Sports result prediction based on machine learning and computational intelligence approaches: a survey. J Adv Comput Eng Technol 5(1):27–36

    Google Scholar 

  • Kifer D, Ben-David S, Gehrke J (2004) Detecting change in data streams. In: Proceedings of the 30th international conference on Very large data bases-Volume 30, VLDB Endowment, pp 180–191

  • Kitchenham B (2004) Procedures for performing systematic reviews. Keele, UK, Keele Univ 33(2004):1–26

    Google Scholar 

  • Lara OD, Labrador MA et al (2013) A survey on human activity recognition using wearable sensors. IEEE Commun Surv Tutor 15(3):1192–1209

    Article  Google Scholar 

  • Liao J, Bi Y, Nugent C (2010) Using the Dempster–Shafer theory of evidence with a revised lattice structure for activity recognition. IEEE Trans Inf Technol Biomed 15(1):74–82

    Article  Google Scholar 

  • Liu P, Zhang J (2008) A context-aware application infrastructure with reasoning mechanism based on Dempster–Shafer evidence theory. In: Vehicular technology conference, 2008. VTC Spring 2008. IEEE pp 2834–2838

  • Madeira R, Nunes L (2016) A machine learning approach for indirect human presence detection using IOT devices. In: 2016 11th international conference on digital information management (ICDIM). IEEE, pp 145–150

  • Mane YV, Surve AR (2016) Capm: context aware provisioning middleware for human activity recognition. In: 2016 international conference on advanced communication control and computing technologies (ICACCCT), pp 661–665

  • Mohamad S, Sayed-Mouchaweh M, Bouchachia A (2019) Online active learning for human activity recognition from sensory data streams. Neurocomputing 390:341–358. https://doi.org/10.1016/j.neucom.2019.08.092

  • Moher D, Liberati A, Tetzlaff J, Altman DG, Group P, et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: the Prisma statement. PLoS Med 6(7):e1000097

  • Mshali H, Lemlouma T, Moloney M, Magoni D (2018) A survey on health monitoring systems for health smart homes. Int J Ind Ergon 66:26–56

    Article  Google Scholar 

  • Neto ADF, de Azevedo BR, Boufleuer R, Lima JCD, Augustin I, Pasin M (2014) An approach based on activity theory and the SRK model for risk and performance evaluation of human activities in a context-aware middleware. In: Proceedings of the 13th international conference on mobile and ubiquitous multimedia, pp 40–47

  • Nguyen CV, Achille A, Lam M, Hassner T, Mahadevan V, Soatto S (2019) Toward understanding catastrophic forgetting in continual learning. ArXiv preprint arXiv:1908.01091

  • Ortega JG, Han L, Whittacker N, Bowring N (2015) A machine-learning based approach to model user occupancy and activity patterns for energy saving in buildings. In: 2015 science and information conference (SAI). IEEE, pp 474–482

  • Palumbo F, Gallicchio C, Pucci R, Micheli A (2016) Human activity recognition using multisensor data fusion based on reservoir computing. J Ambient Intell Smart Environ 8(2):87–107

    Article  Google Scholar 

  • Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  • Perera C, Zaslavsky A, Christen P, Compton M, Georgakopoulos D (2013) Context-aware sensor search, selection and ranking model for internet of things middleware. In: Mobile data management (MDM), 2013 IEEE 14th international conference on, vol 1. IEEE, pp 314–322

  • Perera C, Zaslavsky A, Christen P, Georgakopoulos D (2014) Context aware computing for the internet of things: a survey. IEEE Commun Surv Tutor 16(1):414–454

    Article  Google Scholar 

  • Phiri CC, Ju Z, Kubota N, Liu H (2016) (2016) Enhanced robot learning using fuzzy q-learning & context-aware middleware. In: Micro-nanomechatronics and human science (MHS), international symposium on. IEEE, pp 1–8

  • Punj R, Kumar R (2018) Technological aspects of WBANs for health monitoring: a comprehensive review. Wirel Netw 25(3):1–33

    Google Scholar 

  • Rafferty J, Synnott J, Nugent C, Morrison G, Tamburini E (2017) NFC based dataset annotation within a behavioral alerting platform. In: Pervasive computing and communications workshops (PerCom workshops), 2017 IEEE international conference on. IEEE, pp 146–151

  • Ramamurthy SR, Roy N (2018) Recent trends in machine learning for human activity recognition: a survey. WIREs Data Min Knowl Discov. https://doi.org/10.1002/widm.1254

  • Ramírez JMN, Roose P, Dalmau M (2016) Distributed interfaces and context-oriented broadcast services in a smart-home environment. In: Wireless and mobile computing, networking and communications (WiMob), 2016 IEEE 12th international conference on. IEEE, pp 1–8

  • Ranganathan A, Campbell RH (2003) A middleware for context-aware agents in ubiquitous computing environments. In: ACM/IFIP/USENIX international conference on distributed systems platforms and open distributed processing. Springer, pp 143–161

  • Rasanen O, Kakouros S (2014) Modeling dependencies in multiple parallel data streams with hyperdimensional computing. IEEE Signal Process Lett 21(7):899–903

    Article  Google Scholar 

  • Raychoudhury V, Cao J, Kumar M, Zhang D (2013) Middleware for pervasive computing: a survey. Pervas Mob Comput 9(2):177–200

    Article  Google Scholar 

  • Razzaque MA, Clarke S (2016) Compression-based energy efficient sensor data gathering framework for smartphones. In: 2016 international wireless communications and mobile computing conference (IWCMC). IEEE, pp 126–132

  • Rekha SB, Rao MV (2017) Methodical activity recognition and monitoring of a person through smart phone and wireless sensors. 2017 IEEE international conference on power. Control, signals and instrumentation engineering (ICPCSI). IEEE, pp 1456–1459

  • Rodrigues É, Viterbo J, Conci A, MacHenry T (2015) A context-aware middleware for medical image based reports. In: 2015 IEEE/ACS 12th international conference of computer systems and applications (AICCSA), pp 1–4

  • Romero D (2008) Context-aware middleware: an overview. Paradigma 2(3):1–11

    Google Scholar 

  • Sakr N, Soliman H (2018) Current trends in complex human activity recognition. J Theor Appl Inf Technol 96(14):4564–4583

    Google Scholar 

  • Sansrimahachai W, Toahchoodee M (2016) Mobile-phone based immobility tracking system for elderly care. In: Region 10th conference (TENCON), 2016 IEEE. IEEE, pp 3550–3553

  • Sezer OB, Dogdu E, Ozbayoglu AM (2018) Context-aware computing, learning, and big data in internet of things: a survey. IEEE Internet Things J 5(1):1–27

    Article  Google Scholar 

  • Sharma A (2020) Ensembled machine learning framework for drug sensitivity prediction. IET Syst Biol 14:39–46. https://digital-library.theiet.org/content/journals/10.1049/iet-syb.2018.5094

  • Sharma A, Rani R (2021) Machine learning perspective in cancer research. In: Handbook of research on disease prediction through data analytics and machine learning. IGI Global, pp 142–163

  • Silva FA, Silva TRM, Ruiz LB, Loureiro AA (2013) Conprova: a smart context provisioning middleware for vanet applications. In: Vehicular technology conference (VTC Spring), 2013 IEEE 77th. IEEE, pp 1–5

  • Sorici A, Picard G, Florea AM (2015) Multi-agent based context management in AMI applications. In: Control systems and computer science (CSCS), 2015 20th international conference on. IEEE, pp 727–734

  • Thiebes S, Lins S, Sunyaev A (2020) Trustworthy artificial intelligence. Electron Mark. https://doi.org/10.1007/s12525-020-00441-4

  • Turaga P, Chellappa R, Subrahmanian VS, Udrea O (2008) Machine recognition of human activities: a survey. IEEE Trans Circuits Syst Video Technol 18(11):1473

    Article  Google Scholar 

  • Vyas VV, Walse K, Dharaskar R (2017) A survey on human activity recognition using smartphone. Int J 5. http://www.ijarcsms.com/docs/paper/volume5/issue3/V5I3-0032.pdf

  • Vales-Alonso J, Chaves-Diéguez D, López-Matencio P, Alcaraz JJ, Parrado-García FJ, González-Castaño FJ (2015) Saeta: a smart coaching assistant for professional volleyball training. IEEE Trans Syst, Man, Cybern: Syst 45(8):1138–1150

    Article  Google Scholar 

  • Wu J, Li C, Miao Y, Song S, Li L, Ding Q (2013) Context-aware reasoning middle ware applied in the mobile environment. In: Machine learning and cybernetics (ICMLC), 2013 international conference on, vol 4. IEEE, pp 1829–1835

  • Yang Y, Huang Y, Cao J, Ma X, Lu J (2012) Formal specification and runtime detection of dynamic properties in asynchronous pervasive computing environments. IEEE Trans Parallel Distrib Syst 24(8):1546–1555

    Article  Google Scholar 

  • Yao L, Benatallah B, Wang X, Tran NK, Lu Q (2016) Context as a service: realizing internet of things-aware processes for the independent living of the elderly. In: Sheng QZ, Stroulia E, Tata S, Bhiri S (eds) Service-oriented computing. Springer, Cham, pp 763–779

  • Yao L, Sheng QZ, Benatallah B, Dustdar S, Wang X, Shemshadi A, Kanhere SS (2018) Wits: an IOT-endowed computational framework for activity recognition in personalized smart homes. Computing 100(4):369–385

    Article  MathSciNet  Google Scholar 

  • Ye J, Dasiopoulou S, Stevenson G, Meditskos G, Kontopoulos E, Kompatsiaris I, Dobson S (2015) Semantic web technologies in pervasive computing: a survey and research roadmap. Pervas and Mob Comput 23:1–25

    Article  Google Scholar 

  • Yurur O, Liu CH, Moreno W (2014) A survey of context-aware middleware designs for human activity recognition. IEEE Commun Mag 52(6):24–31

    Article  Google Scholar 

  • Yürür Ö, Liu CH, Sheng Z, Leung VC, Moreno W, Leung KK (2016) Context-awareness for mobile sensing: a survey and future directions. IEEE Commun Surv Tutor 18(1):68–93

    Article  Google Scholar 

  • Zdraveski E, Lameski P, Trajkovik V, KJulakov A, Chorbev I, Goleva R, Pombo N, Garcia N (2007) Improving activity recognition accuracy in ambient-assisted living systems by automated feature engineering. IEEE Access 5:5262–5280

Download references

Author information

Authors and Affiliations

  1. Institute of Computing, Fluminense Federal University (UFF), Niterói, Rio de Janeiro, Brazil

    Leandro Miranda, José Viterbo & Flávia Bernardini

Authors
  1. Leandro Miranda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José Viterbo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Flávia Bernardini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leandro Miranda.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Miranda, L., Viterbo, J. & Bernardini, F. A survey on the use of machine learning methods in context-aware middlewares for human activity recognition. Artif Intell Rev 55, 3369–3400 (2022). https://doi.org/10.1007/s10462-021-10094-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-021-10094-0

Keywords

Search

Navigation