Digital twin-enabled automated anomaly detection and bottleneck identification in complex manufacturing systems using a multi-agent approach

How can an agent-based technique be applied to develop a DT-CPS to automatically detect anomalies, identify bottlenecks and provide control to remove the bottlenecks in complex manufacturing systems?”.


Introduction
• In digital manufacturing, computational models and simulation techniques play key role in capturing the dynamic behaviour of an asset or process through virtual representations of facilities, information, test equipment, spares and people, including the skills, roles, and priorities of personne.
• With the rapid development of digital and information technologies in the Industry 4.0 era, DT in manufacturing sectors and supply chains has grown rapidly, enabling automation, digitalisation and intelligence

Introduction
• CPSs represent an emerging research area due to the increasing importance of the interactions between inter-connected computing systems and the physical world.
• CPSs and DTs can provide manufacturing systems with greater efficiency, resilience and intelligence .
• In the context of CPSs, DTs can be defined as a cyber representat-ion of a real system in real time.
• In the context of manufacturing sector, among other purposes, DTs are intended to improve the manner in which a process or system is designed, manufactured and operated.

Introduction
• Despite the plethora of academic and industrial research, DT has not yet been properly understood and adopted by many industries, as several challenges are identified in the literature in the development of accurate DTs.
• Considering the existing literature, mechanisms to enable automated anomaly detection, bottleneck identification and response in DTs in manufacturing are limited.

Introduction
• In the context of DT-CPS for complex manufacturing systems, the literature on the development of models and architectures for integrating DTs in CPSs is sparse and relatively new.
• The architecture of integrated DT-CPS can be composed of multiple digital representations for different cyber-elements.The agent-based modelling technique can be applied effectively to develop a multi-agent CPS.
• The challenge in DT-CPS is to develop an integrated DT architecture that aggregates multiple cyber elements and allows data communication and integrity within a multi-agent CPS.
• However, the development of advanced computational models and simulation techniques to design modular and comprehensive DTs for complex manufacturing systems is also very limited.

Introduction
• In this paper, the research question considered is: "How can an agent-based technique be applied to develop a DT-CPS to automatically detect anomalies, identify bottlenecks and provide control to remove the bottlenecks in complex manufacturing systems?".

Introduction
• Hierarchically, this is the top-level agent of the global manufacturing system design in which exo, meso and micro agents belong, operate and interact to each other.
• Manufacturing phases are modelled at the exo-level agent as single agents that will always exist within the macro-level agent and communicate with the meso and micro level agents.
• Manufacturing modules, described as a sequence of activities that are repeated frequently such as picking products or quality control procedures, are modelled at the meso-level agent.These modules are modular and can be deployed in multiple manufacturing phases.
• ABM approach is employed to simulate the interactive structure of phases and modules.

Introduction
• Discrete Event Simulation (DES) modelling approach is additionally employed to model the finite dynamical system of manufacturing processes within each manufacturing phase and module.
• A bottom-level agent, called micro, is developed using ABM approach for modelling manufacturing components, which are included in the meso, exo and macro level agents.At this level, population of agents are created for manufacturing components such as human, equipment and material resources.

Introduction
• In digital manufacturing, computational models and simulation techniques play key role in capturing the dynamic behaviour of an asset or process through virtual representations of facilities, information, test equipment, spares and people, including the skills, roles, and priorities of personne.
• With the rapid development of digital and information technologies in the Industry 4.0 era, DT in manufacturing sectors and supply chains has grown rapidly, enabling automation, digitalisation and intelligence

• Proposal
○ The authors propose a DT-CPS approach, employing the ABM technique, for detecting anomalies in sensor data, and identifying and resolving bottlenecks in multi-agent manufacturing systems in an automated way.
○ By adopting a bottom-up ABM approach for the DT-driven approach and a hybrid ABM-DES technique for real-time simulations, the proposed solution can support decision making and control of complex manufacturing systems, contributing to the automated monitoring of such systems in a flexible, interactive and efficient way.• In the area of autonomous monitoring and control in DTs, Tomiyama and Moyen [49] in their work proposed an architecture for cyber physical production systems (CPPS) to autonomously detect faults and respond to them by maintaining the system's performance at an acceptable level.• Stark et al. [46] proposed an approach that enables simulating the positions of objects in a smart factory DT, as well self-monitoring and self-diagnosing the status of processes and objects.

Anomaly detection in digital twins in manufacturing
• Vrabiˇcet al.
[54] in their work developed an approach where the learning capability is realised by introducing a learning agent.This agent can detect and diagnose faults to the DT, as well determine a response and self adapt the DT.

Anomaly detection in digital twins in manufacturing
• Most of the existing works develop digital models or digital shadows as they focus on unidirectional data flow from physical to digital.Therefore, the dynamic and fully integrated bidirectional mapping of data flow from digital to physical after executing DT simulation requires deeper research.

Agent-based approach in the development of digital twins
• ABM offers advanced capabilities including complexity management, decentralisation, intelligence, modularity, flexibility, robustness, adaptation and responsiveness.

In the area of agent-based approaches in DTs and CPSs,
• Laryukhin et al. [25] proposed a multi-agent approach for the conceptual development of an integrated CPS-DT for managing farms.
In this research, a knowledge base with domain ontology, DT agent and data mining methods for supporting decision making of farmers were considered.• Tran et al. [51] presented an approach for developing a smart cyber physical manufacturing system (CPMS).In their research, cognitive agent technology was integrated enabling the CMPS to have autonomous characteristics such as perception, communication and self-control.• In the context of complex manufacturing systems, Latsou et al. [26] developed a DT architecture integrated in a multi-agent CPMS where the DT represents the data flow for radio frequency identification (RFID) tagged products processed on a shop floor.

Agent-based approach in the development of digital twins
• Thenget al.
[63] proposed a conceptual DT modelling method based on a multi-agent architecture to examine the factors influence the product quality during the manufacturing phase.
• In the context of production systems,Dittrich & Fohlmeister [6] proposed a cooperative multi-agent system using reinforcement learning to handle the complexity of order scheduling and overcome the local optimisation problem.
• Such approaches where the model is composed of multiple digital representations for different cyber elements can facilitate the development of a flexible and adaptable digital architecture.
• The agent-based approach can, thus, empower the development of such architectures with advanced capabilities and be successfully applied to establish a multi-agent DT.

Agent-based approach in the development of digital twins
• The challenge, though, is to develop a DT that aggregates multiple cyber elements allowing data communication and integrity within a multi-agent CPS.In this regard, the existing literature shows that the majority of the current approaches discusses the development of multiagent CPS at a conceptual level.
• Compared to the other dynamic modelling techniques, i.e., system dynamics and discrete-event, ABM supports a higher degree of autonomy and offers opportunities for a more flexible, interactive and effective approach.
• The bottom-up approach and the multi-agent structure of ABM can, thus, provide innovative opportunities in the development of DT simulation models through computational experiments.

Literature review
• A well-known CPS structure that has been adopted by Jay Lee et al.
[28] is proposed to build the DT-CPS architecture of complex manufacturing systems

Digital Twin multi-agent cyber physical system
• The architecture considers a sequential workflow from data capturing and storage to development of simulation model as exact replica of the system in the physical space and analysis of the simulation results to gain insights for informed decision making.
• The DT is governed by the same control inputs as the system in the physical space, while automated knowledge feedback from the cyber to the physical spaces is provided for performance improvement.
• A dynamic system of multi-agents is deployed to model DTs for complex manufacturing systems.
• Additionally, the ABM approach, adopted by Farsi et al.
[7] and extended by introducing the 'monitoring agent', has been selected to create the global manufacturing system, multiple manufacturing phases, 'monitoring agent' and manufacturing components.

Digital Twin multi-agent cyber physical system
• Physical layer can be referred to any physical component including resource, system or process that is relevant to a given manufacturing system or installed at the shop floor and exists at the two spaces: the physical space and the twin space.
• Physical layer may include human, equipment and material resources, plants or facilities, storage containers and pallet racking storage systems, physical processes (e.g.picking, storing or assembling products, etc.), documents or paperwork (e.g.checklists and forms to perform inventory or quality assurance audit,etc.).

Digital Twin multi-agent cyber physical system
• Smart connection layer Smart connection layer is used to capture data measured directly by sensors, actuators or controllers, or obtained from enterprise manufacturing systems (e.g.product lifecycle management (PLM), enterprise resource planning (ERP), manufacturing execution systems (MES), etc.).
• In terms of the ABM approach, data acquired in the smart connection layer is referred to as manufacturing components, modelled at the micro-level agent.
• A database at this level is employed for storing the remotely collected sensor data (e.g.quantity of material resources, cycle times, etc.) for tracking and tracing physical components at a manufacturing system.Additional data available from enterprise systems refers to the status of human, equipment and material resources

Digital Twin multi-agent cyber physical system
• Conversion layer can be referred to as all the physical processes to convert data into information.• Once the data is acquired in the smart connection layer, data processing, including filtering process, artificial intelligence-based processing and data analytics, is required to transform the data into information.• This information, revealing key failures,brings self-awareness to physical components (e.g.product, machine,equipment, facility, etc.) and can later help users take actions to increase system's performance.• In this work, a novel 'monitoring agent' has been introduced at the exolevel agent to dynamically detect anomalies in the sensor data of the database, identify bottlenecks

Digital Twin multi-agent cyber physical system
• Cyber layer operates as a central hub of information that is acquired from the connection layer, stored and processed to develop the cyber twin model of system in the twin space.• Information from every physical component for sharing and exchanging data is being stored in an information base at the micro-level agent.This helps create the network of all connected physical components.• Information base may store the conditions under which an anomaly has been detected (e.g.date, time and location within the manufacturing system), equipment failure types, downtime and repair status, processing times of materials/ products, etc. • Utilising this information, cyber avatars for physical components can be created.Advanced data analytics can be then employed to extract knowledge from these avatars, providing insights beyond the status of individual components.

Digital Twin multi-agent cyber physical system
• Cognition layer is employed to transfer knowledge to the users to make appropriate decisions for improving system's performance and productivity.• Cognition of the monitored manufacturing system is achieved by thoroughly analysing and effectively visualising the results obtained from modelling and simulating the twin model as discussed in the cyber layer.• Results in terms of anomalies detection, bottlenecks identification, key performance indicators of manufacturing system, phases and components that can be also visualised in this layer may include system's throughput,these indicators are obtained at macro, exo, meso and micro level agents.

Digital Twin multi-agent cyber physical system
• Configuration layer acts as an intelligent control for self-configuration of resilience, self-adjustment of variation or of anomalies, by providing feedback to the smart connection, conversion and cognition layers.• The feedback, obtained at the macro-level agent of the manufacturing system, should be implemented at the other agent levels automatically to create a synchronised DT. • Feedback in manufacturing systems can be applied to achieve optimal throughput, resource planning, initial inventory capacity and inventory control, dispatch planning and space layout planning or job order control.• This work focuses on finding a decision strategy that proposes the optimal solution.

Digital Twin multi-agent cyber physical system
• Feed-back to the smart connection layer is applied to the micro-level agents which are updated based on the optimal solution; • To the conversion layer is employed to check if the bottleneck remains in the system.• To the cyber layer to simulate the cyber-twin model with the updated parameters at the micro level and obtain updated results visualised in the cognition layer.• This process stops only if the bottleneck is eliminated, otherwise, a new self-optimisation for handling anomalies and their emergent bottlenecks is performed.• Communication between physical and twin spaces for automated data capturing and transfer and knowledge feedback can be realised through wired or wireless network connections.
Digital Twin multi-agent cyber physical system • The 'monitoring agent', hosted by the macro-level agent for modelling the manufacturing system, is introduced within the exo-level agent using ABM approach.• The role of this agent is to provide a mechanism to constantly monitor the behaviour of manufacturing components of the database (i.e. the sensor data) at the micro-level agent (dependency relationship) and detect anomalies in this data by comparing it with corresponding values obtained from the steady state performance of the system.• Anomaly is, thus, any deviation between the behaviours of the sensor data and expected behaviour (i.e.historical data).• The experienced anomalies can consider variations in daily orders and delivery rates, cycle times, product type, batch size, etc. • Exo-level agent can also identify unplanned bottlenecks, as indicated by the dependency relationship that emerge from the detected anomalies, by comparing a set of performance metrics • Bottlenecks, are measured in terms of throughput at the macro-level agent, time in system and lead times at the exo and meso levels, and inventory levels or resource utilisation rates at the micro-level agent.

Digital Twin multi-agent cyber physical system
• Bottlenecks can be modelled as variables at the 'monitoring agent' to store information obtained from the bottlenecks identified during the system's operation including when (date and time), where (location at the manufacturing shop floor) and what type of bottlenecks has been identified.
• The types of bottlenecks can be associated to shortage of human resources (type B1), equipment resources (type B2) or material resources/storage space (type B3).
• Exo-level agents are created considering how the manufacturing phases operate over time.
• The 'monitoring agent' and manufacturing phases, created at this stage, can communicate and interact in parallel to each other during simulation.
• Similarly, meso-level agents are introduced as a single agent to simulate the interactive structure of repeated manufacturing modules.

Digital Twin multi-agent cyber physical system
• Key element in the presented architecture is a built-in integrated database that captures multidimensional (e.g.asset, time, activity) and heterogeneous data from multiple time periods from physical components.
• This is accomplished with the help of sensors, actuators or controllers located at the manufacturing shop floor.Database element included at micro-level agent reads data from the sensor data repository system.
• Each database is composed of one or more information bases (composition relationship).For the data to create meaningful information, such as date and time stamps, processing times of products or equipment failure types and repair status, retrieved data is further processed to create the information base.
• The information base is at the micro-level agent and its contents are used by the manufacturing components (dependency relationship) and, by extension, by the manufacturing phases and modules at exo and meso level agents to mimic the behaviour of physical components.

Digital Twin multi-agent cyber physical system
• In this work, the anomalies are identified in the remotely collected sensor data that measures the physical position or state of physical components in a manufacturing system.• Any identified anomaly that can lead to the emergence of unplanned bottlenecks requires action to meet the demand of the physical system.• The proposed DT-CPS is able to continuously update the cyber space, generate knowledge and actuate this as productive feedback to its physical space.• Therefore, if an anomaly is detected, the integrated DT-CPS is updated accordingly.• If a bottleneck is then identified by the 'monitoring agent', the simulation model (cyber layer) provides results for the impact on the performance of the physical system which are visualised and realised at macro-level agent as knowledge (cognition layer).• Deploying a multi-level ABM method, decision makers gain a better understanding of system structure, operation and abilities, as the multiple agents of a system are specified at various scales providing system granularity.

Digital Twin multi-agent cyber physical system
• A hybrid agent-oriented and process-oriented approach has been employed in the proposed DT-CPS architecture to model and then simulate the complex nature of manufacturing systems.
• The process followed to develop the hybrid multi-agent ABM-DES simulation method for detecting anomalies and recognising any associated bottlenecks in a dynamic and automated way is composed of four steps,

Digital Twin multi-agent cyber physical system
• This section emphasises on step 4 of the proposed multi-agent simulation method, i.e., on the development of the 'monitoring agent' using finite-state machine model.
• An agent-based simulation model is proposed to create the 'monitoring agent' at the exo-level that continuously observes the behaviour of the micro, meso, exo and macro level agents, while real-time sensor data is captured by the database at the micro-level agent.
• The system starts with an 'Initial State' entry point.Once initialise() transition is triggered, 'Idle' state is activated and average values for throughput (TH) and lead times (LTs) measured in various parts in the manufacturing system are obtained.

Digital Twin multi-agent cyber physical system
• Models the dynamic nature of the proposed agent-based model for automated detection of anomalies in the input data and for identification of any unplanned associated bottlenecks in complex manufacturing systems.

Digital Twin multi-agent cyber physical system
• WIP is the summation of number of delivered items for i = 1, ..., n, the initial stock size of stored items, the number of items stored after delivered items are received, the number of received orders and the number of dispatches for j = 1, ..., m.
• Employing the modified format of Little's Law proposed by Hopp and Spearman , the system's throughput (macro-level agent) can be formulated: TH = WIP/LT, where WIP is the work in progress i.e., the average number of items,and LT is the lead time i.e., the average time an item spends as WIP • When inputs, i.e., deliveries and/or orders are captured by the microlevel agent, busy() transition fires and 'busy' state where the system performance is measured is activated.The 'busymeasure process performance' state can dynamically calculate several metrics including stored items, items picked from storage, TH,WIP…

Digital Twin multi-agent cyber physical system
• Once real-time data is detected at the micro-level agent, detect Real Time Record() transition fires and 'detect Real Time record' state is activated and parameters associated to the real-time data, including tag ID, date and time stamps, location and user ID are detected.
• The model via assess Performance() transition moves back to 'busy' state.
• At this point, the detect() transition is enabled and the performance of the manufacturing system is assessed by comparing the real-time sensor data and performance metrics (i.e.TH and operating LT at macro and exo level agents, respectively) against average nominal values that either are provided as inputs to the model or derived from historical data (e.g.ERP system).

3.
Digital Twin multi-agent cyber physical system

Case Study: Digital twin cyber physical system development of cryogenic warehouse
4.1 Digital twin based multi-agent cyber physical system architecture • In this study, a complex manufacturing system of a cryogenic warehouse company in the Cell and Gene Therapy (CGT) sector has been selected as the case study to test the validity of the proposed architecture.• The impact of the DT-based multi-agent CPS model on the automated anomaly detection, and bottlenecks identification and removal in complex manufacturing systems is evaluated.• The complexity of CGT manufacturing systems, originated from multiple response time requirements and numerous policies and regulations, can result in parallel dynamic interactions within the system.• Common example of such parallel interactions is when operators are required to perform manufacturing activities at different locations within the facility at the same time.
• In this study, radio-frequency identification (RFID) technology has been installed at the cryostorage company for recording, monitoring and auditing of cryomaterials.Devices including sensors, RFID readers and tablets have been used to capture the real-time data from the shop floor.

Digital twin based multi-agent cyber physical system architecture
• The studied warehouse is responsible for receiving cryogenic material from manufacturers, storing and monitoring the material, and dispatching it when requested from manufacturers and healthcare institutions.

Case Study
• The manufacturing phases act in parallel with the 'monitoring agent' at the exo-level.
• The 'monitoring agent' interact with the phases to identify bottlenecks, by comparing performance measures (e.g.lead times and time in system) obtained from each phase to the corresponding average numbers observed during the normal daily operations of the cryogenic warehouse.

Case Study
• The authors tried to investigate DT-CPS approach in manufacturing or production systems and supply chains.More case studies could be included.
• The bidirectional communication between the physical and twin spaces is also considered.
• The model follow a stochastic bottom-up approach for DT-CPS, to detect anomalies and identify bottlenecks in complex manufacturing systems using the ABM.

•
According to these processes,three manufacturing phases are considered: Phase I -Receipt & Inventory; Phase II Storage & Monitoring; and Phase III Distribution.

•
If the performance metrics are greater than the nominal values, an anomaly is detected (i.e.set State (abnormal) transition is enabled) and the system state is set as disrupted, otherwise as normal (i.e.set State(normal) transition is enabled).
• Based on the state, 'Anomaly Detected' state or 'No Anomaly Detected' state is activated, respectively.The latter state leads to the 'idle' state, whereas if an anomaly is detected, the identify bottleneck() transition is triggered and the simulation model explores if the anomaly may cause potential unplanned bottlenecks related to human resources, equipment resources or inventory.