1 Introduction

Intelligent environments (IEs) are becoming an integral part of our everyday lives [1]. IEs are living environments—both indoors and outdoors—fitted with sensors and actuators to interpret our behaviour, intentions, and goals to provide intelligent services.

Slowly and silently, such technology is becoming interwoven in our lives in the form of various devices that are starting to be used by people of all ages and as part of their daily routine [2]. If the designers want this evolution to become an ecosystem that interacts smoothly with humans, the devices must develop a high level of perception of human behaviour. However, reaching this goal depends on the accuracy of activity and behaviour recognition and on the modelling from ambient and wearable sensors. This is necessary for many applications, especially those that involve ambient assisted living in smart homes from health monitoring or behavioural interventions [3] as well as those related to autonomous systems such as robots and autonomous vehicles [4].

In a traditional approach, building robust and meaningful classifiers often relies on getting the right high-quality and carefully annotated datasets and the suitable classifier model to optimise on the dataset before applying it to new data. However, IEs are characterised by a constantly changing environment, whether in terms of materials (for example, when devices appear and disappear in the network), physical environments (in mobile settings), users (for example, new staff or visitors), or task (for example new activities). Current approaches have made substantial contributions to the field of multi-user, real-time, concurrent, and unknown activity recognition, but these tasks remain challenging, especially when a model learned for one environment is applied to another [5, 6].

Another challenge is the lack of high-quality annotated sensor data that most existing machine learning algorithms rely on. These are incredibly costly to acquire (up to several ten thousand euros in smart homes [7]) and are specific for each new task and environment with the risk of biasing the classifiers. Hence, traditional offline machine learning is often not suitable for real-world applications.

In this themed issue, we present novel research contributions on sensor-based human activity recognition (HAR), human behaviour modelling, and human perception of ethical concerns posed by the emergence of autonomous agents in real-world environments.

2 Accepted contributions

Four papers were accepted to be published in this themed issue. The first two were original works, while the last two were extended papers from the IEEE Intelligent Environments 2021 conference [8]. The first two contributions that we describe in this section deal with activity recognition from sensors in the particular case of missing activity labels.

Ken Sadohara presents in the paper “Activity Discovery Using Dirichlet Multinomial Mixture Models from Discrete Sensor Data in Smart Homes” a work focused on activity discovery from sensor data in smart homes. He presents an approach to extract human activity based on privacy-preserving sensors that are not disturbing daily life, such as presence of infrared sensors. He argues that such model extraction must be adapted to the peculiarity of the resident and that due to the annotation cost, one cannot assume the availability of annotated data. Hence, unsupervised machine learning techniques are necessary to tackle such situations. The author thus proposes to use Dirichlet Multinomial Mixture (DMM) models. The advantage of DMM models is that they assume that sensor data within a short period are generated from an activity occurring among unobservable activities. Such a model can also capture the burstiness of sensors, defining the case of sensors that rarely fire suddenly repeatedly. The paper demonstrates this burstiness phenomenon in actual publicly available data composed of passive infrared ray motion sensors. Furthermore, by taking previous activities into account in the model through a Markov assumption, the model’s accuracy is improved. Such an outcome is thus promising to perform human activity recognition in smart homes where a labelling task is known to be very difficult.

Related to the problem of scarcity of labelled data, Presotto et al. propose in the article “Semi-Supervised and Personalized Federated Activity Recognition Based on Active Learning and Label Propagation” a solution to deal with HAR with few labels using semi-supervised and distributed machine learning in a wearable setting (smartphone and smartwatch). They argue that users cannot be asked to provide a great amount of labelled data. Hence, a fully supervised setting is a too constraining approach for the user. Instead, they combine an active learning approach with a federated learning approach in a method called FedAR. This combination dramatically reduces the amount of labelling that the users must provide. The active learning approach enables to ask the user to label only the most salient examples. Then, these labels are propagated to the most similar instances. The federated learning approach allows aggregating models learned from several independent users to obtain a more general model. It also includes a personalisation step that fine-tunes the general model on the user device. Experiments on two publicly available datasets showed that this semi-supervised approach could reach similar results as other federated fully supervised approaches but requires a much lower number of labelled data. Furthermore, the experiment also demonstrated that the number of active learning questions decreases quickly with time. This solution is thus an effective way to deal with the data scarcity in HAR in distributed settings.

The following contribution, by R. A. Saeed et al., deals with the problem of simulating pedestrian crowd behaviour in a public environment. It is entitled “Modelling Group Dynamics for crowd simulations” and is an extension of [9]. The authors surveyed the state of the art in crowd simulations, mostly grouped into macroscopic- and microscopic-level simulations. The macroscopic techniques focus on the aggregate behaviours of crowds, while the microscopic techniques are used to create realistic trajectories for individual pedestrians. This paper contributes to the domain of crowd simulation by introducing a mesoscopic model specifically concerned with people group dynamics. More specifically, their approach uses the multi-group model to generate a trajectory for each pedestrian navigating in the walking area of the virtual environment. The individual pedestrian behaviour in the groups is simulated through an agent-based model. Hence, every single pedestrian in each group can continuously adjust their attributes. Moreover, pedestrians optimise their path independently and in real-time toward the desired goal points while avoiding obstacles and other pedestrians in the scene. The simulation was performed in a virtual shopping mall where pedestrians, either as individuals or in groups, move through the virtual environment in different directions. The obtained results of the experiments showed that the developed method performs well in generating pedestrians’ crowd movement with 200 different small groups of pedestrians. They showed how the pedestrians adapt their dynamics to avoid static obstacles and other pedestrians. As expected, the increase in the number of groups of pedestrians has the reverse effect on the mean pedestrian velocity in the crowd. The proposed method can describe a realistic simulation of dynamic behaviour in a given virtual environment, which can have many applications for building design and IEs.

The last paper, by Muhammad Umair Shah et al., is related to the ethical implications of IEs. It is entitled “Exploring the Human Factors in Moral Dilemmas of Autonomous Vehicles” and is an extension paper of [4]. In their paper, the authors study the ethical implications of autonomous vehicles (AVs), whose automotive industry has spearheaded efforts to build driverless cars that could operate in the real world. Given that AVs aim to evolve in the same world as humans and should make decisions that could impact them, some researchers have supported that AV algorithms should subscribe to a moral framework to support algorithmic decision-making in real-life settings. The authors focus here on the notion of moral dilemmas, which involve situations where an agent chooses between two (or more) contradictory moral possibilities or ethical conflicts. Applying empirical or rationalistic frameworks and using thought experiments involving hypothetical situations and superficial consequences enable investigating such moral dilemmas. For instance, the trolley problem experiment asks participants to imagine a scenario in which an out-of-control trolley is approaching a cross-section, and a bystander has to choose between sacrificing one person tied to the track versus pulling the lever and changing the track where multiple lives are lost. In this research, the trolley is replaced with an AV, the bystander with the AVs decision-making algorithm, and the track with a roadway. Instead of establishing whether human decision choices in AV scenarios reflect utilitarianism or deontological processes, the authors argue that human moral decision-making is ethically and rationally bounded (pragmatic). The study relied on conducting thought experiments with respondents to investigate this relation. Multiple hypothetical scenarios were provided to study participants to pick their preferred choices according to three factors: “level of harm”, “level of affection”, and “fixing the responsibility”. The experimental findings indicate that human moral judgements cannot be wholly deontological or utilitarian but must be due to a dual-process theory and the notion that two distinct decision-making modes alter moral judgements. The current findings are essential for the development of new ethical theories within AI and can also be applied to re-examine existing theories from an empirical view.