Machine Learning and Cognitive Robotics: Opportunities and Challenges

1.1 Architecture and platforms for cognitive robot research

To facilitate the development of mature cognitive robotic systems, several computing platforms including real robots like humanoids (icub), panda and Hobo, simulators, and middleware like ROS and YARP are available. Particularly, to facilitate the development of mature cognitive systems, robots must continuously interact with the environment, know where objects are in the scene and understand the consequences of their generated actions. The icub [35] humanoid robot is a 53-degree-of-freedom humanoid robot of approximately the same size as a three-year-old child. It can crawl on all four limbs and sit up. Its hand allows dexterous manipulation, and its head and eyes are fully articulated. It is an open systems platform available for research under GNU general public license. Its capabilities are built based on an ontogenetic pathway of human development. Figure 1 shows different postures of icub. Robotic simulators are of interest despite not being able to provide a full model of the complexity present in the real environment. For example, the icub simulator [35] has been designed to reproduce as accurately as possible the physics and dynamics of the robot and its environment with the constraint of running approximately in real-time. It is composed of multiple rigid bodies connected via joint structures. It consists of the following components: physics and rendering engines, YARP protocol for simulated icub and body model. All commands sent to and from the robot are based on YARP instructions. More details are provided in ref. [36]. Besides, there are several platforms for humanoids and other robots in studies reported in ref. [37, 38, 39]. Details of Pioner3-AT bender robotic platform are provided in ref. [40]. There are also several European Union funded research projects on cognitive robotics that have resulted in several architectures, system concepts and benchmark datasets [41]. Several simulators for robotic systems are provided in ref. [42]. To build cognitive systems, several computational architectures have been designed and built to realise different cognitive platforms.

The following are representative architectures: The Clarion [43, 44, 45] architecture is a broadly scoped computational psychological model based on the dual theory of the mind, capturing essential structures mechanisms and processes of the mind. It provides a framework, essential structures and computational model for realising processes of the mind. It also facilitates detailed exploration of the mind and psychological theories. Clarion consist of four subsystems, namely, action-centred subsystem (ACS), non–action-centred subsystem (NACS), motivational subsystem (MS) and metacognition subsystem (MC). MS provides the impetus for action and cognition, whilst MC provides for monitoring and regulating other processes. Together, these subsystems address action, skill learning, memory, reasoning, motivation, personality, emotions and their interactions. Figure 2 is a high-level diagram of Clarion. Each subsystem consists of two levels, which is a dual representation structure. The top-level encodes explicit knowledge, potentially corresponding to ‘conscious’, and the bottom layer encodes implicit knowledge corresponding to ‘unconscious’ knowledge and also corresponds to symbolic versus connectionist representation.

Computationally, ACS is realised with multilayer perceptron or reinforcement learning, whilst NACS implements implicit declarative processes with associative memory. Explicit declarative processes are captured as symbolic associative rules. Implicit processes deal with drive activations captured by MLP and explicit processes deal with goals. More details of the architecture are provided in ref. [44]. Other general purpose architectures include Soar [46], which integrate knowledge, intensive reasoning, reactive execution, hierarchical reasoning and learning from experience. It has the goal of creating systems with cognitive capabilities like humans. Several other projects that target specific robotic platforms have produced application-specific cognitive architectures. These include the HAMMER [47, 48] and ArmarX and Xperience architectures on Armar humanoid robot [49]. The HAMMER architecture is for assistive robotic agents cooperating with humans to carry out tasks. It provides for sensing user states and actions, modelling skills and predicting intentions and personalising to maximise assistance effectiveness over extended periods of interactions. ArmarX is a hybrid architecture, proposed for human observation and experience. Interactions with humans occur in natural language. It recognises the need for help and reason about the world. The original architecture proposed has been continuously extended in several projects. It consists of three layers, namely, high-level layer for planning and reasoning, mid-level layer for mediating symbolic knowledge and sensory-motor data; and low-level layer for robotic behaviour focusing on functions and skills, hardware abstraction layer and bridging middleware to other robot software frameworks.

Using virtual environments for simulation is very important to ensure the safety of robots, humans and other objects in the environment; the slow wall clock time makes it a too slow method to generate enough data in a reasonable time frame, and physical trials are slow and costly and the learned behaviours are limited. Increasingly, the use of complex simulation environments is being used for experimentation and research. By training a virtual robot in countless situations, such as low-probability scenarios, it is the objective of the system to learn to generalize from the scenarios and safely handle future yet unseen scenarios. When the physical properties of the environment, such as gravity, friction coefficients and the object’s visual appearance, are used and randomized, it becomes apparent that the learned models transfer successfully to the physical robot using domain randomization [50]. One such platform is the Unity [51, 52] 3-D rendering platform, a cloud scalable infrastructure for generating thousands of frames per second. For video games, Arcade learning environment (ALE) [53] is a standard test bed for deep reinforcement learning (DRL) algorithms, and it supports discrete actions. TORCS car racing simulator [54], on the other hand, supports continuous actions for deep reinforcement algorithms.

1.2 The role of technology

The pervasiveness of information communication technology (ICT) is evident everywhere in our daily lives. In industrial settings, the following are some examples:

• Monitoring and control of all tools of production;

• Collecting data for many sensors for monitoring, control and predictive maintenance of equipment;

• Use of machine intelligence, wireless connectivity, cloud computing to integrate physical systems and processes (Industry 4.0 effort);

• Use of key enabling networking technologies, namely, edge, fog and 5G using AI agents;

• Applications of service robots (non-industrial) include shopping, travelling, home assistance and elderly care.

In our daily lives, examples include the numerous gadgets in our homes to assist in our daily lives and care for the elderly. Robotics network and cloud robotics have evolved to connect robots and allow a central or distributed intelligence to command and control any set of robots. Advantages include flexibility, simplification of hardware and software about the robot, ease of re-planning and task management of complex robots. Several configurations exist for robots, namely, stand-alone robots, networked robots and cloud robotics [55]. Networked robots address the problem associated with stand-alone robotic systems by sharing perceived data with each other and executing tasks in a cooperative and coordinated manner. Cloud computing empowers robots by providing faster and more powerful computational capabilities through massively parallel computation (using CPUs, GPUs, and clusters and data centres) and higher storage facilities, as well as access to open source, big datasets and software cooperative learning capabilities.

Typical applications include human-assisted driving and self-driving vehicles for safe transportation, Industrial 4.0 drives to create cyber-physical systems for industrial processes based on cloud by creating a replica in the cyberspace for closed-loop feedback [56] and support for autonomous and smarter processes. It also caters for the convergence of sensing, computation and communication by providing a common platform for integrating data acquisition, processing, storage and decision making. AI agents for digital twin 4.0 provide movement prediction, tasking learning, risk reduction and predictive maintenance. Fundamental to most of these developments is AI and ML for continuous decision-making.

CR are expected to continuously learn and adapt to their environments and make decisions in real-time when required under conditions of uncertainties in sensor data, processing complexities, privacy and security constraint to arrive at timely and effective decision-making. AI and ML empowered agents is one approach to realising this goal. Current robotics have made significant progress in sensing perception and control problems but find it challenging to provide integrated thinking, feeling and knowing [57]. It is still very challenging for two-legged robots to walk naturally in unconstrained environments. Several challenges exist in using robotic platforms such as the high cost of prototyping, steep learning curve and programming robots to carry out complex tasks like autonomous driving in unconstrained dynamic environments.

1.3 The role of software

Closely related to cognitive robotics is cognitive computing (CC), which is a multidisciplinary field aiming at devising computational models and decision-making mechanisms based on neurobiological processes of the brain, cognitive sciences and psychology. It aims to endow computers with the ability to think, feel and know. Since there is no commonly accepted definition of cognition, there are several definitions of cognitive computing [58, 59]. Wang [59] defines cognitive computing in terms of cognitive informatics that applies how the brain processes information and copes with decision-making to information sciences. CC is defined as an emerging paradigm of intelligent computing methodologies and systems based on cognitive informatics that implements computational intelligence by autonomous inferences and perceptions mimicking the mechanisms of the brain. Research in cognitive computing is focused on three thematic areas, namely, computer systems with a faculty of knowing, thinking and feeling. Applications of CC include education, healthcare, commerce and industry.

When software adds intelligence to information-intensive processes, it is known as robotic process automation. The process uses AI to extend and improve action and saves cost and customer satisfaction. It is typically used in completing a complex business process that uses unstructured data or persists over a long period [57]. Typically a bot (an agent for a user of a program) observes the process to automate the process.

One of the requirements for robust and effective CR is software integration frameworks. This is justified when one considers the following:

1. Cognitive models are derived from a large spectrum of computational paradigms that are not necessarily compatible when considering the underlying software architecture;

2. Changes in application requirements due to hardware interfaces, computational and network latencies and the need for integration;

3. Cognitive research projects utilize robotic systems as demonstrators, and therefore serve as an important proof of concepts and might also require integration;

4. The need to provide common interfaces and functions;

5. Specific software frameworks may be required to take advantage of innovations in hardware (new development of brain-like hardware architecture) and the development of relationships among concepts of a given domain.

Software frameworks enable thinking by taking advantage of brain-like computer machinery or determine causal relationships among concepts of a given domain. There have been several published works on software frameworks [60] prototyping, development of middleware, sustainable software design and architectural paradigms. MARIE [61] is a component-based software architecture for integrating and combing heterogeneous software and computational paradigms. It adapts the mediator design pattern to create a mediator interoperability layer (MIL). MIL is implemented as a virtual space where applications can interact together using a common language. ROS (robot operating system) [62], an open-source robotic middleware suite, is frequently used in robotic projects. ROS provides a set of software frameworks for software development. ROS provides the following services: hardware abstraction, low-level device control, message passing between two processing, package management and other functions; ROS 2 [63] and above provide real-time support and an embedded system. ROS is made up of three components: language and platform independent tools for building and distributing ROS-based systems; ROS client implementations (Roscpp, rospy, roslisp, etc) and packages containing application-related code. Ros typically connects to robots via webSockets and operates on cloud servers. There are several platforms on which ROS runs including ROSbot, Nao Humanoid [64] and Raven II surgical robotic research platforms. Peira et al. [65] provide a framework for using ROS on the cloud. Davinci [66] is another software framework that is cloud-based for service robots exploiting parallelism and scalability. It is based on the Hadoop cluster combined with ROS as the messaging framework. Fast SLAM algorithm, an environmental mapping algorithm for large-scale mapping, was implemented on this platform with significant performance improvement.

A framework for unifying multi-level computing platforms and orchestrating heterogeneous edge, fog and cloud computing resources compliant with MEC [67] was proposed in ref. [68]. It is suitable for integrating different computing, communication and software technologies.

1.4 The role of decision-making

At the core of most ML tasks is decision-making based on information fed to the decision maker, for example steering or breaking a car. Decision-makers used to be either a human or a group of humans; now it can be AI using different combinations of ML and traditional algorithms via agents technology. According to Kahneman [69], there are two modes, namely, system 1 and 2 modes of the human brain, and most ML methods emulate the mode of operation in system 1. ML establishes empirical associations through training and learning. When given scenarios resembling training scenarios, ML yields results in a fast way. However, it struggles when given scenarios not covered during training or the training was inadequate. In human decision-making, when system 2 fails to intervene because it is fooled by an apparent coherent picture created by system 1 tends to result in decision-making. Thus, if ML is to be used in decision-making, the ability to detect difficult and dangerous situation tend to trigger system 2.

In cardiovascular medicine, ML is routinely used to perceive an individual by collecting and interpreting his/her clinical data, and clinicians would reason on them to suggest actions to maintain or improve the individual’s health. Thus, it mimics the clinicians’ approach when examining and treating sick patients [70]. Big data leveraged by ML can provide well-curated information to clinicians so that they can make better informed diagnosis and treatment. ML analyses have demonstrated human-like performance in low-level tasks in robotics and cardiology.

There have been studies reporting on the success of sensing-perception-control/action loop in autonomous vehicles [56].

Higher-level tasks involving reasoning such as patient status interpretation and decision support, and reasoning under uncertainties and dynamic environment in robotics have proven to be challenging. Intention predictions in a dynamic environment are also challenging. Similarly, human-robot cooperation for safe road transportation includes challenges in infrastructure [71] (sensor, communication subsystem, computing and storage) and predicting behaviour when driving, motion prediction and gesture recognition.

1.5 Review of algorithms

From the cognitive architecture descriptions discussed, at the high level, the actions of a robot are goal-directed, with the middle layer responsible for intermediate organisation, planning and execution using some memory hierarchy. The bottom layer is reactive and deals with the environment. For a robot to be able to interact with other objects and its environment, it needs to know how to predict the consequences of its actions using typically a forward model: X=(S, π_θ), where s is the state of the robot, and π_θ: S->A is a parameterized action policy (A) to the space of effect or task space. Similarly, the inverse model computes the action policies that can generate a given effect (S, Y)-> π_θ. Some examples are mapping of movements of the hand in the visual field to the movement of the end point of a tool, and oscillation of the legs to body translation of a robot. There are two main approaches, analytical approach based on control engineering and learning-based approach. The main challenge is to model a prior all the possible interactions between a robot and its environment. Learning is additionally confronted with multimodal sensing perception, high dimensional spaces, continuous and highly non-stationary spatially, and temporary state spaces. Typically, statistical regression is used to guide autonomous exploration and data collection. Alternatively, an approach for learning and constraining the environment is active learning. Several learning paradigms have been used including reinforcement learning and imitation learning. Several machine learning techniques such as deep learning have been used to model robotic agents in the real world. Deep learning networks build a model that produces end-to-end learning and inference system driven purely by data. Most of the approaches reported in the literature make use of neural networks to construct forward and inverse modules. To overcome the problem of catastrophic forgetting (training a model with new information interferes with previously learned knowledge [72]) in neural networks, special memory architectures may be used [34] besides pure algorithmic approaches. Additionally, other cognitive approaches from developmental robotics, neuroscience and other behavioural science approaches have been used. Active learning and inference approaches constrain the search space and allows self-exploration. These methods generally begin using random and sparse exploration, build meta-models of the performances of the motor learning mechanism and concurrently guide the exploration of various subspaces for which the notion of interest is defined [73]. Interest is defined in terms of variants of information gain (variance, entropy or uncertainty). Motivational and goal-driven approaches where exploration and search are goals/curiosity or attention driven [74, 75, 76] to reduce the large search spaces. Cognitive processing techniques can be split into two main approaches, namely, the control theory approach and the free energy-based approach. Although both of them use optimization techniques, the latter approach seeks to minimize free energy prediction error using variational or Bayesian approaches.

1.5.1 Review of reinforcement learning

There are three main classes of algorithms for machine learning, namely, supervised, unsupervised and reinforcement learning. In supervised learning, data defining the input and corresponding output (often called ‘labelled’ data) are available. In unsupervised learning, only the input is available and the structure of the underlying data is typically solicited. It is used to explore the hidden structure of the data. In reinforcement learning (RL), learning takes place by trial and error interactions with the environments. It is goal-directed learning that constructs a learning model specifying output to maximize long-term profit. Deep RL (DRL) uses deep learning methods (multi-layer neural network) to learn models and representations at different levels of abstraction [77] in an unsupervised manner. It leverages deep learning as a function approximator to deal with high-dimensional data. DRL algorithms have been applied to robotics allowing control policies for robots to be learned directly from camera inputs in the real world [11]. The basic model of RL is shown in Figure 3.

At time t, the agent receives state s_t from the environment. The agent uses its policy to choose an action a_t. Once the action is executed, the environment transitions a step providing the next state S_t+1, as well as feedback in the form of reward R_r+1. The agent uses knowledge of state transitions of the form (S_t, A_t, S_t+1, R_t+1) to learn to improve its policy. A policy (π) is a mapping function from any perceived state s to action taken from that state. Alternatively, a policy can be interpreted as a probability distribution of candidate actions that will be selected from state (s) as in Eq. (1):

π=ϕs=pais∀ai∈ΔπΛ∑pais=1E1

∆_π denotes candidate actions on policy π, and p(a_i|s) denotes the probability of taking action a_i given the state s. A policy is deterministic if the probability of choosing an action a from s is p(a/s)=1 for all state s, otherwise, stochastic, i.e, p(a|s)<1. A value function is used to evaluate how good a certain state or state-action pair (s,a) is. For this purpose, a generalized return value R_t, defined by Eq. (2) is used, where γ (0< γ<1) is the discounted factor.

Rt=rt+1+γrt+2+γrt+3…γT−t−1rt=∑i=0T−t−1γirt+i+1E2

The value of a state under policy π is evaluated as the expectation of R_t defined by Eqs. (3) and (4) for the state and state-action pair, respectively. E denotes expectation operation.

Vπ=ERrst=sπE3

Qssa=ERrst=sat=aE4

Underlying RL is dynamic programming [78] and bellman equations for optimality under Markov decision process modelling. RL algorithms have been successfully applied to several real-world problems with limited state spaces to problems in control and navigation. However, it faces the following challenges:

• The optimal policy must be inferred by trial and error interaction with the environment with the only learning signal being the reward.

• Since the observations of the agent depend on its action, it may contain strong temporal dependencies.

• Long-range dependencies may only emerge after many transitions.

• Balancing exploitation versus exploration.

Underlying RL is the Markov property that the current state affects the next state or is conditionally independent of the past given the present state. Partially observable Markov decision processes (POMDP) are Markov decision processes (MDP) in which agent receives an observation p(o_t+1|s_t+1,a_t) where the distribution is dependent on the current state and previous action [79]. An episodic MDP resets after each episode of length T, and the sequence of states, actions and rewards in an episode constitute a trajectory or rollout of the policy. There are three main types of reinforcement algorithms, namely policy-search, value-function based and those that combine both policy and value function approaches. They include actor-critic method, temporal difference and Monte Carlo-based methods [80, 81]. The increasing use of deep reinforcement learning (DRL) algorithms has been attributed to the low-dimensional representation of deep neural network representation and the powerful functional approximation of neural networks. The following significant recent developments in DRL have made it possible to scale to large dimensional state space:

• The combination of duelling DQN architecture with prioritized experience replay in providing better estimates of expected return functions [82, 83].

• The use of an experience replay and target network that initially contains weights of the network enacting the policy, but is kept frozen for a large period [83, 84, 85].

• Introduction of hierarchical reinforcement learning.

• Improvements in guided policy algorithms.

• Asynchronous advantage actor-critic (A3C) algorithm [86] developed for both single and distributed machine settings. A3C combines the advantages of updates with actor-critic formulation and relies on asynchronous update policy and value networks in parallel.

1.5.2 Review of imitation learning

Imitation learning (IL) aims to mimic human behaviour in a given task by facilitating the teaching of complex tasks with minimal knowledge through demonstration. There are three main classes of ML algorithms for imitation, namely, behaviour cloning, inverse reinforcement learning and generative adversarial learning [87]. Behaviour cloning applies supervised learning by learning a mapping between the input observation and the corresponding actions, provided there is enough data. Generative adversarial imitation is inspired by generative adversarial networks [88]. Typically, an agent uses instances of performed action to learn a policy that solves a given task using ML techniques. The agent could learn from trial and error or observe other agents. It has been applied to problems in real-time perception and reaction, such as humanoid robots, self-driving cars, human-computer interfaces and computer games. The assumption is that an expert (teacher) is more efficient than the agent learning from scratch when given a task [89]. Imitation learning is an interdisciplinary field of research, and it is sometimes difficult to define suitable reward function for complex tasks. For example, it is often the case that direct imitation of an expert’s motion does not suffice due variations in the task such as the position of the object, environmental conditions and inadequate demonstrations [90]. Therefore, it is difficult to learn policies given demonstrations that generalized to unseen scenarios. The policy must be able to adapt to variations in the task and surrounding environment. Argall et al [91] address different challenges in the process of IL, such as computational methods used to learn from demonstrated behaviour and the processing pipeline. A typical representation of a sample for IL consists of pairs of action and state, such as position, velocity and geometric information, and modelling the process as MDP. The learning process is with pre-processing, sample creation and direct or indirect imitation.

The following are some of the challenges of IL [90]: Noisy or unreliable sensing, correspondence problem and observability where the kinematics of the teacher is not unknown to the learner. Further, complex behaviour is often viewed as a trajectory of dependent micro-actions, which violates independent and identically distributed assumptions in machine learning. Lastly, safety concerns in human-robot interactions, the ability of the robot to react to human force and adapt to the task. A typical flow chart [90] is shown in Figure 4.

There are different methods from demonstrations, namely, structured predictions [92], dynamic movement primitives [93], inverse optimal control (inverse reinforcement learning [94], active learning [95], transfer learning and other techniques. Active learning needs a dedicated oracle that can be queried for demonstration. Inverse RL techniques use demonstrations to learn cost functions over extracted features. It first recovers a utility function that makes the demonstration near-optimal and searches for the optimal policy using a cost function as an optimization objective. Closely related is apprenticeship learning, which uses demonstrations from an expert or observation to learn a reward function. A policy that optimises the reward function is then learned through experience (trial and error). Transfer learning use experience from old tasks or knowledge from other agents to learn a new policy. The reader is referred to refs. [87, 96] for details of imitation learning and its applications in robotics. Learning a direct mapping between state and action is not enough to achieve the required behaviour in most cases due to cascade errors, insufficient demonstrations and the difficulty in reproducing the conditions and settings. The learner has to learn actions and re-optimise policies with respect to quantifiable reward functions. Figure 4 is a flowchart showing different variants of imitation learning. The following are some recent developments:

• Use of goal-directed (motivation or curiosity-driven) learning to exploit and explore multi-task spaces;

• Use of developmental robotics concept of goal babbling for visuomotor coordination tasks for coordination of multiple subsystems (head and arms) [97].

• Use of predictive processing techniques [3].

• Use of memory systems for storage of knowledge of agents’ beliefs, goals and short and long-term memory, together with efficient integration with other components of cognitive architectures.

• Use of machine learning for integration of perceptual processing, feature extraction, learning and control.

1.6 Use case

For robots acting as human companions, autonomy is fully-oriented towards navigation in a human-centred environment and human-robot interactions. It is facilitated if the robot’s behaviour is as natural as possible. Some requirements are that robot independent movement must appear familiar and predictable to humans and have similar appearance to humans. Human–robot interactions include the following: use of natural language or subset for communication, gesture or activity interpretation that involves tracking and action recognition; gesture imitation that involves tracking and reproduction and the person following which involves 2-D or 3-D based tracking. Acceptable performance at the task level requires real-time processing constraint of 50 milliseconds per second. Safety is also very important as robots are expected to evolve in a dynamic environment, well populate with humans. The main challenge is that robotic systems lack learning representation, and interactions are often limited to pre-programmed actions. One solution strategy is to conceptualize cognitive robots as permanent learners, who evolve and grow their capacities in close interactions with users [86]. Robots must learn new tasks and actions relative to humans by observing and imitating (imitation learning). Thus human detection and tracking, activity recognition and face detection are some basic tasks that must be performed robustly in real-time. A use case is presented next, which deals with daily activity recognition at home and face recognition using publicly available dataset. These typically fit in several robotic studies investigated in human-centred environments [40]. The algorithms are first described, followed by an evaluation.

1.6.1 Activity recognition

Research activities in domestic service robots have increased in recent years. Some of the main drivers are the projected future use of domestic robots for improving elderly people’s quality of life, childcare, entertainment and education. Several benchmark datasets [126, 127, 128, 129] and methodologies for evaluating the capabilities and performance of robotic platforms are available. Action recognition is used in several application domains such as surveillance, patient monitoring systems, human–computer interface, housekeeping activities and human assistance by robots (guiding humans). There are two processing techniques: spatial approach, which allows recognizing activities from images, and spatio-temporal approach for detecting specific activity as space-time volume.

The HMDB51 [130] is an action dataset whose action categories mainly differ in motion rather than static points. It contains 51 distinct action categories, each containing at least 101 video clips. Video clips are extracted from a wide range of sources. The clips have been annotated and validated by at least two human observers. Additionally, meta information tags allow for a precise selection of tags for training, testing and validation. Meta-data tags include information on camera viewpoint, presence or absence of camera motion, video quality and a number of actors involved. The training procedure is also described.

A simulation study on activity recognition based on spatio-temporal analysis of a large video database of human motion recognition [130] is provided. The main processing steps are shown in Figure 5. The algorithm consists of six main processing steps, namely, pre-processing, spatio-temporal analysis in the wavelet domain, class model construction (class dictionary), batch singular value factorization (BSVF), similarity feature computation and classification. The pre-processing step involves filtering for noise removal and optionally contrast enhancement using histogram equalization. The wavelet analysis step applies orthogonal or biorthogonal wavelet (9/7 or 5/3 filter) to produce subband frames. A silhouette feature map is constructed by combining low–low and high–low subbands as described in Tawiah et al [130]. The map is a tiling of rectangular features describing the dominant objects in the frame. Sparse dictionary is constructed for each activity as described in refs. [131, 132]. Spatial frame resizing and temporal frame subsampling by interpolation are applied to construct an action volume of 64 × 32 × 100 pixels for each action volume. It is then reshaped to a vector of size of 51200. Batch singular value prediction (BSVP) is based on the classical singular value decomposition [133] used in signal processing with batch data input (matrix). Each column of the input matrix represents a sample action. The output is a decomposition consisting of left and right-hand singular vectors (or matrices) for vector (or matrix) input and a covariance matrix as the diagonal matrix.

BSVP prediction step consists of two sub-steps: first, apply singular value decomposition to the same batch training sample used in constructing the dictionary, replacing one column (e.g, the first) with an incoming action sample. Then, apply the computation step in Eq. (5). The class dictionary is constructed using a batch sample matrix, with each sample representing an action volume. The prediction for an input action sample is computed using Eq. (5):

Estrj=∑j=1nsampleφr:∑j=1dimsLHSri∗αjj+∑j=1dimsRHSrj∗αjjE5

Ф denotes the class dictionary matrix, N sample denotes the number of samples in the batch dataset, Dim S denotes the dimension of each sample, RHS (r,i) denotes the right-hand singular vector, LHS(r,i) denotes the left-hand singular vector, α denotes the covariance matrix and Est denotes the estimate of the sample. The indices, r and i, are used to identify specific elements in a matrix. The similarity between the input spatio-temporal volume and Est (refer to Eq. 5) is computed using five similarity measures, namely, canonical correlation [134], Bhattacharyya distance [135], modified Bhattacharyya distance, histogram intersection [136] and cityblock. A similarity vector is formed by concatenating all the similarity values. A multi-class feed-forward classifier [137], consisting of 51 all versus one classifier, is constructed. The classifier is able to assign an action volume to multiple classes. Samples of input video frames and the corresponding object outline maps are shown in Figures 6 and 7.

BSVP does not reconstruct a sample using the sparsest representation as is the case in classical sparse coding but instead uses one-time reconstruction from batch sample whose representations are known (represented as LHS and RHS singular matrices with known covariance) and applies BSVP algorithm. This provides a representation for a sample taking into consideration statistical characteristics of all samples in the batch. It is computationally efficient and avoids solving L1-norm optimization, and it is suitable for real-time classification problems. The result on applying the proposed algorithm to all the fifty-one action classes is summarised in Figure 8, using the action categories provided by HMDB51 dataset (Table 2).

Action class	Scene content precision (%)	Meta descriptors precision(%)
Facial expression	75	25
Facial action	78	23
Body movement	86	37
Body movement with object interaction	97	43
Body movement with human interaction	97	43

Table 2.

HMDB 51 action classification.

The confusion matrix is also shown in Figure 8 to illustrate action classes prone to misclassification.

For robotics applications, facial expression recognition and gesture recognitions are also very important. Reference [138] provides a good review of facial expression recognition.

1.7 Trends in cognitive robotics

Early approaches to imitation aim to reproduce reaching or grasping with simple grippers. Imitation learning provides a desired sequencing of basic sub-skills to achieve an observed task behaviour. Later on, more sophisticated system including modules for visual attention, speech recognition, and integration of visual and linguistic inputs for instructing robots to grasp everyday objects [139]. Online learning and machine learning techniques, such neural networks, have been used in low-level and reactive tasks from trajectory learning and adaptive control of multi-DOF robots, and tasks learning from demonstrations. ML provides different paradigms of learning from transfer learning, representation learning curriculum learning, etc, which provides for systematic means of acquiring systematic models for making inferences [140]. The following are some trends that are apparent from the literature review:

• The use of neuroscience and behavioural psychology to synthesize computational models for high-order cognitive skills in artificial agents.

• The use of neural networks as functional approximators.

• Use of motivations or goal-directed mechanisms to balance exploration and exploitation in tasks space rather than in motor space.

• Use of robotic platforms for research in higher order research (social robots).

• Use of predictive coding mechanisms to synthesise higher-order cognitive behaviours.

• Classical control theory is unable to handle complex scenarios with many parameters.

• Use of swarm robotics to study social behaviours in robotic swarms.

• The increasing use of networked and cloud robotics and cyber-physical systems.

The use of artificial intelligence, especially machine learning, wireless connectivity and cloud computing, is increasing to integrate physical systems and processes, including robotics. At the core of most ML tasks, decision-making is based on information fed to the decision maker. The study of decision-making is closely connected with psychology and cognitive sciences.

1.8 Success, challenges and research directions

Several projects involving the use of cognitive robotics have been reported in industrial settings (Industry 4.0), service robots, robotic surgery, cardiovascular surgery [70], assistive technology [141] and several other fields. In ref. 70, ML methods are used in perceiving an individual’s health by collecting and interpreting his/her clinical data and would reason to suggest actions to maintain or improve the individual’s cardiovascular health. ML augmented decisions point to potential to improve the outcome at a lower cost of care and increase satisfaction. In assistive technology [141], vision-based hand wheelchair control using kinect sensor system enables the user to control without wearing or touching.

As cognitive robotics continues to make some remarkable progress in industrial process automation with Industry 4.0 initiative, cloud robotics and service robots, it has resulted in more challenges [142, 143, 144]. For example, standardisation effort [56] has ushered in a new era of robotics linked to cyber–physical system for effective control and monitoring of industrial processes. Classical approaches to robotics have made significant progress in control-based applications in stand-alone robotic applications, but there are challenges in multi-robot and multi-agent systems applied to complex tasks in dynamic environments.

The main goal of integrating thinking, knowing and feeling in an artificial intelligent system as cognitive process has not been realised today despite advances [57]. In particular, integrating feeling into the existing system has proved very challenging.

The trends towards Industry 4.0 of providing cyber–physical framework for unifying industrial processes and cyber–physical system would be extended to service robots domain as well. The need to develop more robust and sophisticated ML algorithms to enable AI agents to carry out complex tasks in a coordinated and cooperative fashion to ensure reliability and cost-effectiveness. The robustness of ML algorithms under adversarial learning would also have to be investigated.

The need is for more research into the decision-making process (using ML) to make it robust, timely and relevant to situation, as well as meet real-time requirements. For multi-robot systems, the need for cooperation and coordination of tasks is very challenging to improve the effectiveness and improved utilisation of resources. Underlying these problems is the need for research into more robust ML algorithms and transparent model interpretation, and guarantees against adversarial attacks [145].

The need is for cost-effective management of resources (computing, network, storage and devices), all interconnected for ambient intelligence. The problem of scheduling, recovering from unexpected events and scalability issues require urgent attention. Similarly, the integration of heterogeneous platforms (software and hardware) into processes is required. Investigations into robust and generic processing architecture for social robotics are another area worthy of investigation.

Investigations into protocols to ensure effective and robust cooperation between humans and robots via human-machine interfaces to ensure trust and autonomy, as well as ethical considerations, ought to be investigated.

To meet privacy and security concerns distributed learning [146, 147] approaches to train models on the cloud keeping data localized and apply privacy-preserving analysis. However, this has raised the issues of network latency and model consistencies, which has been proven very challenging. Approaches to solving the challenges include MEC-based training, federated learning and capsule network for internet of vehicles. Other persistent challenges are latency, security and management of network infrastructure. Autonomic systems [148] seem very attractive for managing problems related to network and computing infrastructure.

1.9 Conclusion

The chapter has presented a review of recent development in ML techniques for cognitive robotic systems in the overall context of artificial intelligence. The main algorithms for learning, namely, reinforcement and imitation learning techniques, have been discussed.

The recent initiative in Industry 4.0 initiative, increasing trend in research in service robots, telemedicine and computer-assisted medical delivery system means that the industry holds lots of promise for research and personal applications.

Several processing architectures, as well as software frameworks for integrating heterogeneous hardware and software components, have also been presented. Towards simulating further research, current trends and research issues have also been highlighted. An example scenario involving action recognition of humans and facial expression has also been presented.