Skip to content
BY 4.0 license Open Access Published by De Gruyter (O) May 8, 2023

Cloud-based evaluation platform for software-defined manufacturing

Cloud-basierte Evaluierungsplattform für Software-defined Manufacturing

Stuttgarter Maschinenfabrik as a realization example of an Industry 4.0 production
Stuttgarter Maschinenfabrik als Umsetzungsbeispiel einer Industrie 4.0 Produktionsstätte
  • Michael Neubauer

    Michael Neubauer studied Mechanical Engineering at the University of Stuttgart. Subsequently he worked as research assistant in the team: Machine Technology at the ISW. From 2018 to 2021 he was group leader in the field of Drive Systems and Motion Control. Since 2021, he has been Chief Technology Officer (CTO).

    ORCID logo EMAIL logo
    , Colin Reiff

    Colin Reiff studied Mechanical Engineering at the University of Stuttgart. Since 2017 he has been working as research assistant and since 2022 he has been group leader of Mechatronic Systems and Processes at the ISW.

    , Moritz Walker

    Moritz Walker studied Mechatronics at the University of Stuttgart. Since 2020 he has been working as research assistant at the ISW.

    , Stefan Oechsle

    Stefan Oechsle studied Mechatronics at the University of Stuttgart. Since 2020 he has been working as research assistant at the ISW.

    , Armin Lechler

    Armin Lechler studied Technical Cybernetics at the University of Stuttgart. After that he worked as research assistant at the ISW. From 2009 to 2011 he was Head of the department Control Engineering. In 2011 he received the Dr.-Ing. degree. Since 2013 he has been Deputy Director.

    and Alexander Verl

    Alexander Verl studied Electrical Engineering at the Friedrich-Alexander-University in Erlangen-Nürnberg. In 1997 he received the Dr.-Ing. degree at the DLR Institute of Robotics and Mechatronics. Since 2005, he has been professor and head of the ISW.

Abstract

Software-Defined Manufacturing (SDM) is a new paradigm for how to manufacture in the Industry 4.0 factory of the future. The approach requires a fundamental change in thinking and reinvention of central business models in production technology. This paper discusses the main challenges on the way to a software-defined factory of the future and illustrates possible solutions. An exemplary future factory is presented, in which the validation of the solutions is carried out. For this purpose, cloud-based Computerized Numerical Control (CNC) is considered as use case. Experimental results demonstrate the proof of concept and show the potential for further applications.

Zusammenfassung

Software-defined Manufacturing (SDM) ist ein neues Paradigma für die Art und Weise, wie in einer Industrie 4.0 Fabrik produziert werden kann. Der Ansatz erfordert ein grundlegendes Umdenken in der Produktionstechnik. In diesem Beitrag werden die wesentlichen Herausforderungen auf dem Weg zu einer softwaredefinierten Fabrik der Zukunft diskutiert und mögliche Lösungen aufgezeigt. Es wird eine beispielhafte Zukunftsfabrik vorgestellt, in der die vorgeschlagenen Lösungen validiert werden. Dazu wird als Anwendungsfall eine CNC-Maschine aus der Cloud in Echtzeit angesteuert. Experimentelle Ergebnisse belegen die Machbarkeit und zeigen das Potenzial für weitere Anwendungsfälle auf.

1 Introduction

Industry 4.0 was announced as a strategic goal for the future of industrial production at the Hannover Messe 2011. Innovative methods from information and communication technology shall be introduced in Operational Technology (OT). The goal is a self-organized factory in which production systems, logistics, Information Technology (IT), products, and humans are connected and cooperate with each other. However, there is still a lack of holistic realization of this vision. The reason is that the realization represents a paradigm shift. This requires a fundamental change in thinking and reinvention of central business models. In addition, the solution approach is not clearly described. Consequently, there are different methods to bring the vision to realization.

One of them is SDM [1]. The basic idea is to separate software and hardware components. This allows largely independent development and dynamic combination of software and hardware components. Functionalities are not necessarily pre-designed, but can be added later by software changes. The concept is comparable to a smartphone (hardware) to which additional functionalities (calculator, weather forecast, games, messengers, etc.) can be added via apps. Software and hardware are no longer stiffly coupled with each other.

A prerequisite for this is a suitable software and hardware architecture. The individual components must be interoperably connected with each other. In production technology, a wide range of requirements must be satisfied, such as real-time capability and determinism in communication, standardized interfaces, open and reconfigurable platforms, data management, etc. This article illustrates these main challenges and presents possible solutions.

The paper is structured as follows: First, the state of the art is presented and fundamental requirements (labeled below with Rx) are stated. After that, the generalized structure and operating mode of a software-defined factory is described. Subsequently, the design of the required functional modules is presented using a layer model. In chapter 4, these concepts are transferred into a concrete realization of the software-defined factory. In Chapter 5, the functionality is demonstrated by means of the use case: cloud control. Throughout the main part, the selected standards/proposed solutions (Sx.y) and concrete contributions (Cx.z) of the paper are labeled. Finally, the results are discussed and concluded.

2 State of the art

To realize the concepts of Industry 4.0 and built on it SDM, fundamental changes in today’s production environments are required. Currently, production environments are often characterized by strictly separated, incompatible, and manually operated systems without clear division of tasks [2]. This concerns not only production resources, but also communication networks, control systems, and software services. This is caused by the use of different fieldbuses, network technologies (at IT and OT level), communication protocols, and missing interfaces or their specification. As a result, the lack of connectivity means that no machine-to-machine data exchange can be realized and higher-level software services such as data analytics cannot be directly applied to production.

Accordingly, future production environments require a system architecture based on the principle of separation of concerns (R1: architecture with separation of concerns). The extension, adjustment, maintenance and further development of a steadily growing number of software functionalities with increasing complexity has to be dealt with (R2: handling of complex business logic). A unified communication network is required that allows all participants (devices, services, etc.) to communicate with each other in both real-time (RT) & non-real-time (NRT) (R3: RT & NRT capable cross-layer communication). In addition, defined and, if available, standardized interfaces and data models have to be used (R4: defined interfaces). Another aspect that limits the degree of automation and flexibility of today’s production are the comparatively long development cycles of software applications and the increasing complexity of the cyber-physical systems. Changes in the running production, e.g. due to new product variants, are often associated with enormous expense and high risks (R5: continuous development life cycles). In addition, self-organization of the factory is a central solution component to further increase the degree of automation (R6: self-organized operation).

Since it is impossible to present the entire state of the art in terms of Industry 4.0, architectures, interfaces, communication technology, etc., we will focus on related work to SDM and evaluation platforms for SDM. A similar concept to SDM is Software-defined control (SDC) [3, 4], which combines information from the manufacturing and corporate levels. A centric controller takes configuration choices by application-level reconfiguration. Software-defined cloud manufacturing dynamically assembles applications at runtime to meet higher-level requirements [5]. Higher level systems define the application layer of manufacturing resources [6]. SDCWorks provides formal methods for SDC [7]. To support SDM, the manufacturing resources application layer should be fully customizable [8]. This goes beyond the possibilities of reprogramming within planned functions, such as replacing NC code. Based on a minimal platform adaptation layer (PAL), functionalities and interfaces can be defined within the physical constraints of cyber-physical systems. Orchestration, provisioning, and configuration of services for basic functionalities can realize such an approach [8]. A common control plane abstracts these low-level functionalities and provides generic interfaces to higher layers [9].

As noted, one of the most important aspects on the way to Industry 4.0 is the ability of machines, devices, and humans to communicate with each other. Accordingly, it is necessary to realize and validate even partial solutions in a complete ecosystem of an Industry 4.0 factory. A practicable way to achieve realistic proof of concepts are model factories, which replicate the production technology of a company. Model factories exist in different forms, levels of detail, and focuses. For education, there are learning factories, where the training of apprentices takes place [10]. In research, model factories have been established for experimental investigation of different aspects [11, 12]. A comprehensive overview of existing model factories is provided by Abele et al. [13]. Some of them even address Industry 4.0, for which the term Smart Factory has become established. Prominent representatives are, for e.g. Smart Factory Kl [14] and TU Wien Pilot Factory Industry 4.0 [15]. However, none of them has the suitable infrastructure to demonstrate and investigate SDM. Hence, there is a need for a new ecosystem in which this is possible.

3 Conceptual design

Starting point in a self-organized factory is an order using a suitable product description. From there on, manufacturing shall be automatic. Thus, we propose a hierarchical framework (C6.1) according to Figure 1. From the product description as User Interface (UI) (S6.1) required steps for manufacturing are derived, such as machining, assembly and logistics. Based on this manufacturing description and knowledge about the available production hardware, necessary automation services are defined by combining generic functionalities (C2.2). To handle the complexity, these functionalities are encapsulated into services that can be developed independently (S2.1). At this point, it is known which high-level functionalities a composite automation service provides, e.g. milling a part based on G-code. This defines the northern interface and allows higher-level services, such as a Manufacturing Execution System (MES), to interact with the newly defined automation service. Standardized order interfaces (S4.1), such as OPC UA-based order control compatible with ISA 95 [16] or OPC UA-based order management for glass processing machines, can be used as north interface. Subsequently, the composed automation services are tested using virtual commissioning, which enables verification of Quality of Service (QoS) and RT requirements (C6.3). After verification, the composed service descriptions are annotated with QoS and RT requirements (deployment description), such as maximum message latency. These functionalities are hereinafter bundled to be called SDM-controller.

Figure 1: 
SDM-controller: bundled functionalities for generating deployment descriptions from product descriptions and orders.
Figure 1:

SDM-controller: bundled functionalities for generating deployment descriptions from product descriptions and orders.

3.1 Multitier architecture

It is not sufficient to define only the interaction of involved software components. An architecture is required that also includes network, computing, and production hardware. Therefore, and to achieve the principle of separation of concerns, a multitier architecture is proposed (S1.1). Figure 2 shows the application of the concept to production engineering (C1.1). The SDM controller is assigned to the application layer and passes down deployment descriptions to underneath layers, which are strictly separated (C1.2). Application descriptions must be implemented on-site in the physical infrastructure, taking into account QoS and RT requirements for networking and data processing. The hardware layer consists of physical resources for computing, communication, and production. The infrastructure layer provides basic functionality to the hardware layer, e.g. by defining low-level interfaces for production resources or configuration interfaces for networking devices. The control layer uses those low-level interfaces to bring composed automation services into production while respecting the QoS annotations. It combines network management and software orchestration. The multitier architecture allows simple extensibility of further functionalities and customizability due to its design (C1.3).

Figure 2: 
Multitier architecture for SDM.
Figure 2:

Multitier architecture for SDM.

3.2 Hardware layer

The hardware layer consists of production-related resources, network devices, and computing hardware, see Figure 3. Production-related resources are physical components on shop floor level, which act as logical unit. Examples include Automated Guided Vehicles (AGVs), robots and machines. The communication infrastructure connects all hardware components to enable end-to-end data exchange, including both RT & NRT data. Thus, the suggested network standard is Time-Sensitive Networking (TSN), which enables convergent communication (S3.1). This requires TSN-capable devices on the shop floor through which individual components are connected.

Figure 3: 
Overview of computing and networking hardware.
Figure 3:

Overview of computing and networking hardware.

The computing hardware can be divided into four levels. The field devices are located on the lowest level, which allows, for e.g. the direct integration of intelligent sensors into an Ethernet-based network. On the next higher level are so-called edge nodes, which essentially decouple computing power from production resources. They mainly act as execution platforms for services, which can be executed both, in RT & NRT. The services are in turn deployed to the edge nodes by an orchestrator located on the (on-premise) cloud. This allows, for e.g. several machine tools to be controlled by one edge node, which in turn runs several CNC-kernel containers. This not only reduces the number of decentralized control systems, but also enables the creation of redundant and resilient control systems. The third level is the on-premise cloud, which is unlike the fourth-level cloud, operated by the company itself and not by a third-party provider. The on-premise cloud and the cloud can replace or complement each other depending on the use case and the company-specific regulations. Both have in common, that they only run services, which do not have any RT requirements.

3.3 Infrastructure layer

The infrastructure layer abstracts the low-level interfaces of the hardware layer in order to be configurable by the control layer. In the following, the two main elements, network and computing infrastructure are explained.

3.3.1 Network infrastructure

As already known from IT, TSN-based networks can have a flexible topology. To achieve RT capability, TSN adds deterministic functions to the existing Ethernet standards. In this way, both RT & NRT communication can coexist in the same network while guaranteeing determinism. For this purpose, the used switches must support the necessary standards for convergent communication. As mentioned before, suitable hardware is also required on the endpoint side: the network interfaces used must support the necessary mechanisms for convergent and RT data transmission (e.g. Intel i210 and i225). An example of this is the sending of data frames at a fixed time, which is oriented according to a defined, network-wide schedule (IEEE 802.1Qbv). In this way, the transmission cycle can be divided into different time slots for different traffic types. The implementation of these mechanisms, e.g. for time stamping of incoming and outgoing data frames, can be done by software or by hardware, the latter being preferred. As in conventional fieldbus-based networks, time synchronization is also required in TSN-based networks. The relevant substandard for this is IEEE 802.1AS which is based on the Precision Time Protocol (PTP) standard IEEE 1588. This ensures that all network nodes have the same understanding of time. For this purpose, one node in the network is defined as the grandmaster by the Best Master Clock Algorithm (BMCA) on the basis of several criteria, and the remaining network nodes are synchronized to its system time. The difference between the grandmaster clock and the respective local system clocks is used to assess the quality of synchronization. If the root mean square of this value is permanently below 100 ns, the network nodes can be assumed to be synchronous.

The implementation of such a convergent network requires the configuration of all network participants. An example of a necessary configuration is the calculation of the relative time stamps in relation to the interval start, when the data must be sent, taking into account the above-mentioned Qbv-schedule. Three basic approaches exist for configuring a TSN-based converged network: The centralized, the hybrid, and the decentralized configuration approach [17], whereby a tendency towards the centralized approach is emerging in the industry and is thus also the focus of this work (S3.2). In this configuration approach, the existence of a central network management instance is foreseen. In the context of TSN referred to as CNC (Central Network Configuration) we use “TSN-CNC”. In addition to knowledge of the physical topology of the network, the TSN-CNC also has an overview of the network resources available and already in use. This enables the TSN-CNC to calculate requests for additional communication relationships between network nodes and, if necessary, to reject them in the case of insufficient resources.

The interface between the communication requests and the configuration by the TSN-CNC is the so-called CUC (Central User Configuration, hereinafter “TSN-CUC”). This component can be represented several times in the network. Its task is to collect the communication requirements of the endpoints, i.e. the talkers and listeners, and to forward them to the TSN-CNC for calculation. The resulting configurations are then transmitted by the TSN-CUC to the corresponding endpoints. The systematic interaction is shown in Figure 4. Based on these configurations, the network interfaces are configured on the endpoints and the communication channels are established.

Figure 4: 
Components and interaction of the central configuration approach of TSN in accordance with [17].
Figure 4:

Components and interaction of the central configuration approach of TSN in accordance with [17].

3.3.2 Computing infrastructure

The use of virtualization enables the deployment of composed services, which are realized by a Service-Oriented Architecture (SOA). As units of deployment both, containers and virtual machines are considered, as shown in Figure 5 (C2.3). The reason for that is, that some legacy applications, e.g. CAD tools, are only available for specific operating systems. Because each service can be implemented in any programming language and deployed on any computer within a computer network, virtualization methods provide a sensible technological basis for SOAs. In addition to the bundling of runtime environment and services, one reason for this is that virtualization can achieve modularity at the resource level. Each service can be guaranteed a corresponding share of resources (e.g. memory). Apart from the disadvantage of possible overhead, the advantages of virtualization can be summarized as follows [18]: By operating several virtual servers or services on a few physical hardware resources, a reduction in operating costs can be achieved by running virtual servers or services with complementary resource requirement profiles on one physical computer node. Because entire virtual servers or individual services can be migrated and replicated between physical servers, uninterrupted operation is possible during data backups, hardware upgrades and failures.

Figure 5: 
Orchestration of services using continuos-X mechanisms.
Figure 5:

Orchestration of services using continuos-X mechanisms.

Virtualization at the operating system level (container virtualization) is used to encapsulate and deploy services (S2.2). This can be explained by comparing it with hypervisor-based virtualization (type II). The basis of type II hypervisor-based virtualization is a virtual machine monitor (VMM). Each Virtual Machine (VM) running on the host system has its own Operating System (OS). Container virtualization, on the other hand, deploys services with runtime environments in isolated areas. No additional guest operating systems are operated, but the required operating system functions are taken over from the host OS. The isolation between the individual containers takes place here by means of namespaces and control groups. With the help of namespaces, the kernel can provide different processes with different system views. Resource management is done through control groups, which can be used to restrict resource usage for containers. For e.g. the use of individual CPU cores for containers can be regulated. Namespaces thus regulate what a container sees as its host environment, while control groups regulate what resources a container can use.

A deployment methodology is also required for automatically distributing control containers to suitable compute nodes in the network as needed. The container scheduler, a component of container orchestration tools, is used for this purpose in office IT without RT requirements. Based on a set of containers to be executed, it decides on which compute node to execute a container based on information about required resources (CPU, RAM, fixed storage, access to peripherals, etc.). Well-known NRT capable implementations are Kubernetes and Rancher. We opt for a layered virtualization architecture. First, resources of on-premise cloud nodes are split up in multiple VMs. These VMs are then clustered as worker nodes for SDM-Services. This enables the separation of non-related services, e.g. those of staging and production environments. Edge nodes for RT workloads are directly integrated in the clusters, without additional VMs to enable lower latency and jitter for RT services. Every service is managed in a git repository and container-images are automatically created using corresponding pipelines. Furthermore, we opt for GitOps for operations (S5.1). This means, deployments, updates, and changes to the deployed services are managed declarative using a git repository. Thus, the state of the configuration of the cluster is known at any time and rollbacks are realizable. Furthermore, a Graphical User Interface (GUI) is provided, which enables laymen to deploy needed services in a simplified manner (C5.1). The user only has to provide service specific information, such as environment variables and configuration files via the interface. Corresponding Kubernetes resources are created automatically and the state is fed back into the git repository used as single point of truth regarding the target state of the cluster. While this enables the management of NRT SDM-workloads, special means are needed for RT systems, which are described in Section 3.4.2.

3.4 Control layer

The control layer is the intermediary between the application and infrastructure layer. One of its main purposes are network management and software orchestration. Afterwards, several concepts for the combination of RT-capable cloud computing and TSN are described (C3.1). The control layer is therefore connected to the low-level interfaces of the infrastructure layer.

3.4.1 Network management

Network management describes the realization and management of the network communication using the available mechanisms and the underlying hardware. In the context of engineering, the configuration of the network can be divided into two different approaches: Either it is created by a user or automatically, based on a self-description with defined requirements of the involved endpoints, or talkers and listeners. The latter variant corresponds to the idea of Plug & Produce and for a complete SDM realization it is planned to implement this in the future by means of Asset Administration Shells (AASs) (S6.2). Thereby, network components independently transmit their communication requirements to a network management instance (e.g. the TSN-CNC) and request a configuration. A prerequisite for this is a uniform definition of these requirements so as not to impair the interoperability achieved with TSN. However, the focus of this work is initially on the first variant. That means, the configuration of the network, consisting of communication relations and infrastructure components according to the central configuration approach (see Section 3.3.1), is driven by a user (C6.2).

The engineering of the network is thus done in a dedicated tool or GUI. The definition of the communication consists of two views: The logical and the physical. In the logical view, the communication relationships between the individual services are defined and thus describe which data are exchanged and with which QoSs (e.g. interval, redundancy, latest possible transmission times, and other constraints). The physical view, on the other hand, describes the actual topology of the network, consisting of the hardware components, such as switches, and the endpoints. The assignment of individual services to endpoints on which they are to be executed can then be performed manually or automatically depending on the resources required by the services and the resources available on the endpoints. The flexibility created by this dynamic mapping of services to physical components is of fundamental importance in SDM: Depending on the requirements of production and the current workload, or even in the event of failures of individual components, services can be dynamically deployed on other devices, thereby guaranteeing production.

Based on the mapping created, the configurations for TSN-based communication are then determined. For this purpose, the requests for a RT communication link are collected by the TSN-CUC or transmitted to it by the user and forwarded from there to the TSN-CNC. Due to the comprehensive overview of the resources in the network as a central network management instance, it is possible for the TSN-CNC to calculate the configurations of these communication requests or to reject them in case of insufficient resources. Once the configurations have been successfully calculated, they are deployed to the respective components. The TSN-CNC takes over the configuration of the TSN switches. The endpoints, on the other hand, receive their configurations from the TSN-CUC, which receives the calculated configurations as a response from the TSN-CNC. Finally, the endpoints or the respective applications are able to establish the corresponding communication channels (e.g. as OPC UA PubSub connection [19]) based on the received configurations.

3.4.2 Service orchestration

Services are encapsulated in containers or VMs. Linux patched with PREEMPT_RT is used as the OS, since RT containers and VMs are only available for this OS. We opt for reservation-based hierarchical RT scheduling based on the SCHED_DEADLINE policy to achieve a common resource interface for containers and VMs (C4.1). Corresponding and compatible scheduling mechanisms are available for containers [20] and virtual machines [21]. A Kubernetes-based RT orchestrator such as REACT [22] is used to assign RT services to compute nodes.

In addition to initial deployment, RT services in SDM benefit from dynamic reconfiguration at runtime. This means that new versions of control containers can be automatically deployed according to the principle of continuous deployment. This ensures, for example, that updates for troubleshooting can be installed quickly. A core problem with updates of RT software at runtime are internal states. For NRT services, updates of services are simplified by keeping them stateless. When replacing a service, only the underlying database needs to be migrated. Messages between services can be buffered briefly in brokers. This is not possible in control technology due to the latency to external data stores.

To solve this issue, we employ a custom Kubernetes controller, which enables uninterrupted updates of control applications and state transfer (C3.2a). Existing RT state transfer solutions for Kubernetes assume a strictly cyclical, often idempotent execution model, which is not valid for e.g. CNC kernel. This is especially true, if closed source libraries are used inside services, from which internal state variables cannot be extracted. Here the handover of control between old and new service has to be conducted in a synchronized manner. E.g. if the CNC kernel to be replaced is currently executing a NC program, the new container has to continue execution at the correct NC block. Thus, our approach not only provides state transfer but also handover at specific points in time, e.g. at a specific NC block. As these points in time are application dependent, services negotiate handover time points.

The approach is based on a Custom Resource Definition (CRD), called RealTimeApplication and a corresponding controller. If a RealTimeApplication is initially deployed, a Kubernetes deployment encapsulating a single pod, containing the RT service container, is created. When the RealTimeApplication is updated, i.e. the version of the RT service’s container image is changed, the RealTimeApplication controller triggers the update mechanism, which consists of the following steps (see Figure 6): A second deployment with the new service version is started in idle mode. When the corresponding container is ready the controller triggers the handover. Both services enter update mode, conduct the state transfer and negotiate the handover point. E.g. CNC services’ update mode is a special single step mode, where the execution of G-code is conducted stepwise for every NC block. The negotiated handover point in this case is a specific NC block. After the handover point is reached the new service enters operational mode and the old service enters done mode. In the case of a CNC service the service leaves the single step mode and continues the automatic execution of the NC program. After the controller detects that the first service has reached done mode, it deletes the corresponding deployment which leads to termination of the service’s container.

Figure 6: 
Components involved in the update mechanism (above) and flow chart of the update mechanism (below).
Figure 6:

Components involved in the update mechanism (above) and flow chart of the update mechanism (below).

4 Implementation

In this chapter, the systems described so far in general terms will be specified. The factory of the future with the capabilities described in the previous chapter was realized at the ISW. The so-called Stuttgarter Maschinenfabrik, see Figure 7, consists of the following machines so far:

  1. 3 axis CNC milling machine

  2. 5 axis CNC milling machine

  3. Laser processing machine

  4. CNC milling & printing industrial robot

  5. AGV (MiR250) with a handling robot (UR10).

Figure 7: 
Software-defined factory: “Stuttgarter Maschinenfabrik”.
Figure 7:

Software-defined factory: “Stuttgarter Maschinenfabrik”.

Endpoints with RT requirements have an Intel i210 or i225 network card. Kontron’s TSN capable KSwitch D10 MMT is used as industrial Ethernet switch. The edge node is also from Kontron (KBox C-104-TGL) and has integrated TSN ports. A Kenko server with dual Intel Xeon Gold 6238R 28/56x is used as on-premise cloud.

The services of the application, control, and infrastructure layers are implemented as described in chapter 3.

With reference to an existing factory, the following steps must be performed to realize the described architecture:

  1. Communication infrastructure: TSN-capable switches have to be installed on shop floor. Subsequently, all existing devices such as computers, actuators, and sensors have to be connected with the TSN network. Typically, the components are not TSN-ready itself and adapters and protocol converters have to be used. This can be done, by e.g. a simple computer with a RT operating system (i.e. a PC-based control system), two network interfaces, and the logic for translating between both interfaces.

  2. Computing infrastructure: In order to realize the service-oriented architecture, central computing hardware hosting the needed services is required. Here, a distinction between RT (e.g. Linux patched with PREEMPT_RT) & NRT capable hardware can be made. However, both have in common, that they are connected using the TSN network and Kubernetes is installed.

  3. Control layer: Once, the required hardware is installed, the TSN network has to be configured as described in Section 3.4.1. Therefore, the RT requirements of the single endpoints and applications have to be known. Next to the network management, the service orchestration needs to be set up (see Section 3.4.2).

5 Validation

After the evaluation platform has been realized, the SDM framework will be tested. A large number of scenarios are conceivable with the platform. In order to test as many solution modules of the framework as possible, the use case: cloud control will be considered. Another reason is that this use case has the toughest RT requirements in a factory. The scenario is described below, followed by a proof of concept with experimental results. Here, the main focus is on the communication and computing solutions.

5.1 Use case: cloud control

In a classical factory, each machine has its own and coupled computing hardware and control system. This is very expensive and not necessary in the factory of the future. One advantage of the proposed architecture is that services can be deployed and executed on arbitrary computing hardware. In the case considered below, two containerized CNC kernels are executed on one edge node, while the set points are communicated to two machines, i.e. a three axis and a five axis milling machine, in RT, see Figure 8 (C2.1, C2.4a). Certain requirements regarding e.g. latency and jitter have to be fulfilled. In our case, the required cycle time of set points is 2 ms. The hardware interface allows one cycle miss at a time, which is in line with typical servo amplifiers. To enable decoupling dependent RT & NRT services, such as the machines’ hardware interfaces, MES, and CNC kernels, we employ rolling releases of RT applications as described in Section 3.4.2.

Figure 8: 
SDM-framework applied to the use case: cloud control.
Figure 8:

SDM-framework applied to the use case: cloud control.

A network-wide Qbv schedule was selected for the network configuration in the use case. As already mentioned, this divides a cycle into several time slots for different types of traffic. A basic cycle time of 1 ms was selected, which means that all multiples of this time are also compliant with the schedule. The use case includes three different traffic classes: Isochronous, Prioritized and Best Effort. Isochronous traffic is reserved for the exchange of set point and actual values and has a duration of 300 μs. Prioritized traffic includes the transmission of an IP camera connected to the network to illustrate convergence and also lasts for 300 μs. The remaining 400 μs are for other data according to the best effort policy. This schedule was implemented both on the TSN switch and on the edge node. On the edge node, running under Linux, the Time Aware Priority Shaper (TAPRIO) Queueing Discipline (qdisc) is used for this. This can be used to define the time slots on the network card and to assign the various traffic classes to specific hardware queues. In addition, an Earliest TxTime First (ETF) qdisc was created. This is used to send individual packets at predefined times.

5.2 Experimental results

First, the classical communication metrics are determined: latency and jitter (here: jitter ≔ average deviation of latencies). For this purpose, a measurement between the Network Interface Cards (NICs) of a sender and a receiver with an interconnected switch is considered over a period of approx. 1.5 h. This corresponds to roughly 2.7 million packets with a cycle time of 2 ms. The results are an average latency of 17.366 μs and a jitter of ±839 ns (C2.4b).

Figure 9: 
Virtual network setup and qualitative timing diagram for runtime updates of containerized CNC kernels.
Figure 9:

Virtual network setup and qualitative timing diagram for runtime updates of containerized CNC kernels.

Next, the update mechanism is validated by performing 250 updates of the CNC kernels, while they were executing G-code and by measuring the delay between the last UADP frame (i.e. the OPC UA PubSub frame format) from the old CNC container and the first UADP frame from the newly deployed CNC container (C2.5). The setup of containers, virtual network interfaces and a qualitative timing diagram are shown in Figure 9. The change of kernels requires a synchronized state of both (v1.0 and v2.0 of CNC 1 in Figure 9). For this, both kernels need the current actual values. This is implemented by using Virtual Local Area Network (VLAN) and MacVLAN interfaces (C3.2b). For a logical separation of the network traffic, each machine is assigned to its own VLAN. These VLANs are subdivided again by means of MacVLAN, whereby one MacVLAN is assigned to each CNC kernel (see upper part of Figure 9). The machines in turn send their actual values to the broadcast address in their respective VLAN. In this way, data are received on the edge node by the VLAN interface assigned to the machine and passed on to the two connected MacVLANs via broadcast. In this way, both CNC kernels (v1.0 and v2.0) can receive the current actual values.

The results are shown for the container’s MacVLAN interfaces and the hosts VLAN interface in Figure 10. It shows the distribution of latencies between the last frame of kernel v1.0 and the first one of kernel v2.0, after the handover of control has taken place. It can be seen, that at most one cycle, i.e. 2 ms in our use case, is missed while updating a CNC container at runtime. Furthermore, the jitter, i.e. the deviation from the NC’s cycle time, between the final frame of the old container and the first frame of the new container is relatively high with about 140 μs at the containers MacVLAN interface. By using the TSN features described above, the VLAN interface reduces this jitter significantly, as shown in detail in Figure 11. The figure shows an excerpt of Figure 10 at around 2 ms.

Figure 10: 
Frequency of occurrence of the delay during updates of CNC containers.
Figure 10:

Frequency of occurrence of the delay during updates of CNC containers.

Figure 11: 
Excerpt of the distribution of the delay during updates of CNC containers.
Figure 11:

Excerpt of the distribution of the delay during updates of CNC containers.

One drawback of this approach is, that frames, which come in late from the service’s user space (memory area where application software is executed) are sent out with a delay of the TSN network’s cycle time. This explains the appearance of delays of 4 ms for the VLAN interface in Figure 10, which are missing on the MacVLAN interfaces. Some update delays were below the cycle time, which our machine’s hardware interface can handle. However, if sending frames out early is not allowed, this can be simply solved by adding a delay at user space level.

6 Discussion and future work

The article discusses main challenges on the way to the software-defined factory, from which requirements were derived. By selecting, composing, and adapting existing standards/solutions in combination with own contributions, they are well met with the developed Stuttgarter Maschinenfabrik. Due to the proposed architecture, the platform allows to investigate a variety of use cases, such as virtual Programmable Logic Controller (vPLC), Condition Monitoring, Cloud-based Safety etc. The ability to handle RT & NRT services as well as the unlimited extendability of computing power allows the realization of a high range of applications. The principle of separation of concerns ensures the independence and defined interfaces the interoperability of existing services. Based on the open and flexible character, adjustments and extensions can be easily integrated and evaluated. In summary, Table 1 gives an overall view of identified requirements, proposed standards/solutions, and contributions of the article.

Table 1:

Overview of the identified requirements, selected standards/proposed solutions, and contributions of the paper.

Requirement (Rx) Selected standard/proposed solution (Sx.y) Contribution of the paper (Cx.z)
R1: Architecture with separation of concerns S1.1: Multitier architecture C1.1: Transfer of the multitier architecture approach from software engineering to production engineering

C1.2: Concept for controllability of underlying complexity through abstraction to separate networking, computing, and applications

C1.3: Concept for simple extensibility and customizability of a production factory
R2: Handling of complex business logic S2.1: Service-oriented architecture

S2.2: Use of virtualization technology for encapsulation
C2.1: Application of the service-oriented architecture to production engineering

C2.2: Concept for automatic composition of applications

C2.3: Composition for simultaneous management of VM and container

C2.4: Proof of concept for service-oriented architecture (multiple CNC kernels on one edge device, matching latency of 17.366 μs and jitter of ±839 ns)

C2.5: Proof of concept and demonstration of advantages of a service-oriented architecture for RT applications (update of RT services during runtime)
R3: RT & NRT capable cross-layer communication S3.1: TSN-based converged network

S3.2: Centralized configuration approach
C3.1: Concepts for RT-capable cloud computing in combination with TSN

C3.2: Kubernetes extension with RT update controller and validation. Use of MacVLan in combination with TSN for container connectivity during updates
R4: Defined interfaces S4.1: Use of standards (e.g. OPC UA (CS), AAS, REST etc.) C4.1: Composition of deployment and orchestration mechanisms for containers and VMs based on common resource interfaces
R5: Continuous development life cycles S5.1: GitOps as version control and continuous integration & deployment system C5.1: Combination of a GUI and GitOps for orchestration
R6: Self-organized operation S6.1: Product descriptions and orders as user interface

S6.2: Self-description of assets via AAS
C6.1: Concept for generating deployment descriptions from product descriptions and orders

C6.2: Proof of concept for a configuration of a RT TSN network

C6.3: Virtual commissioning for determination of non-functional requirements

The next planned steps are:

  1. Scaling: Operating multiple instances of services to investigate boundaries in terms of communication and computing infrastructure, service handling, etc.

  2. Cloud-Computing: Realization of the OpenStack architecture to manage the computing resources.

  3. Communication: Detailed examination of essential network parameters such as latencies, jitter, utilization, etc. for different number and types (network card models, switches, etc.) of network participants, network structures as well as configurations.

  4. Network configuration: Development of an approach to realize automatic configuration of TSN-networks using AAS-based self-descriptions of the endpoints. In addition, investigation of dynamic reconfiguration.

  5. Data management: Consistent use of AAS according to the IDTA-standard as data layer.

  6. Cloud control: Migration of the machines’ position and velocity control loops to the edge node. Measured communication metrics show the feasibility.

  7. Further use cases: Investigation and exploitation of the abilities of a SDM-factory. For this purpose, an iterative production planning approach according to [23] shall be considered. The intention is to respond to deviations and unplanned disturbances during production. In addition, generated G-code shall be tested virtually before deploying it to the real factory.

7 Conclusions

SDM is a new paradigm for how to manufacture in the future. This paper presents a general overview of architectural structures and methods in order to realize a SDM capable Industry 4.0 factory of the future. The implementation was achieved as a cloud-based evaluation platform. The concepts and approaches were validated using cloud control as use case. Experimental results show, that two industrial machines could be controlled via RT TSN using two containerized CNC kernels. In addition, it was shown that a CNC container can be updated during runtime.


Corresponding author: Michael Neubauer, Institut für Steuerungstechnik der Werkzeugmaschinen und Fertigungseinrichtungen (ISW), Universität Stuttgart, Seidenstraße 36, 70174 Stuttgart, Germany, E-mail:

Funding source: Federal Ministry for Economic Affairs and Climate Action

Award Identifier / Grant number: BMWK

Award Identifier / Grant number: SDM4FZI

About the authors

Michael Neubauer

Michael Neubauer studied Mechanical Engineering at the University of Stuttgart. Subsequently he worked as research assistant in the team: Machine Technology at the ISW. From 2018 to 2021 he was group leader in the field of Drive Systems and Motion Control. Since 2021, he has been Chief Technology Officer (CTO).

Colin Reiff

Colin Reiff studied Mechanical Engineering at the University of Stuttgart. Since 2017 he has been working as research assistant and since 2022 he has been group leader of Mechatronic Systems and Processes at the ISW.

Moritz Walker

Moritz Walker studied Mechatronics at the University of Stuttgart. Since 2020 he has been working as research assistant at the ISW.

Stefan Oechsle

Stefan Oechsle studied Mechatronics at the University of Stuttgart. Since 2020 he has been working as research assistant at the ISW.

Armin Lechler

Armin Lechler studied Technical Cybernetics at the University of Stuttgart. After that he worked as research assistant at the ISW. From 2009 to 2011 he was Head of the department Control Engineering. In 2011 he received the Dr.-Ing. degree. Since 2013 he has been Deputy Director.

Alexander Verl

Alexander Verl studied Electrical Engineering at the Friedrich-Alexander-University in Erlangen-Nürnberg. In 1997 he received the Dr.-Ing. degree at the DLR Institute of Robotics and Mechatronics. Since 2005, he has been professor and head of the ISW.

  1. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: The authors would like to thank the Federal Ministry for Economic Affairs and Climate Action (BMWK) for funding the joint project: SDM4FZI as part of the, “Future Investments in the Automotive Industry” program. The presented results were created in this project.

  3. Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

References

[1] M. Neubauer, F. Frick, C. Ellwein, A. Lechler, and A. Verl, “Software-defined Manufacturing - ein Paradigmenwechsel für die industrielle Produktion,” WT Werkstattstech., vol. 112, no. 6, pp. 383–389, 2022. https://doi.org/10.37544/143649802022633.Search in Google Scholar

[2] L. D. Xu, E. L. Xu, and L. Li, “Industry 4.0: state of the art and future trends,” Int. J. Prod. Res., vol. 56, no. 8, pp. 2941–2962, 2018. https://doi.org/10.1080/00207543.2018.1444806.Search in Google Scholar

[3] F. Lopez, Y. Shao, Z. M. Mao, J. Moyne, K. Barton, and D. Tilbury, “A software-defined framework for the integrated management of smart manufacturing systems,” Manuf. Lett., vol. 15, pp. 18–21, 2018. https://doi.org/10.1016/j.mfglet.2017.12.015.Search in Google Scholar

[4] N. G. Nayak, F. Durr, and K. Rothermel, “Software-defined environment for reconfigurable manufacturing systems,” in 2015 5th International Conference on the Internet of Things (IOT), IEEE, 2015, pp. 122–129.10.1109/IOT.2015.7356556Search in Google Scholar

[5] L. Thames and D. Schaefer, “Software-defined cloud manufacturing for industry 4.0,” Procedia CIRP, vol. 52, pp. 12–17, 2016. https://doi.org/10.1016/j.procir.2016.07.041.Search in Google Scholar

[6] C. Yang, F. Liao, S. Lan, L. Wang, W. Shen, and G. Q. Huang, “Flexible resource scheduling for software-defined cloud manufacturing with edge computing,” Engineering, in press, pp. 1–11, 2021. https://doi.org/10.1016/j.eng.2021.08.022.Search in Google Scholar

[7] M. Potok, C. Chen, S. Mitra, and S. Mohan, “SDCWorks: a formal framework for software defined control of smart manufacturing systems,” in 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS), IEEE, 2018, pp. 88–97.10.1109/ICCPS.2018.00017Search in Google Scholar

[8] A. Lechler and A. Verl, “Software defined manufacturing extends cloud-based control,” in Proceedings of the ASME 12th International Manufacturing Science and Engineering Conference – 2017, New York, N.Y., 2017.10.1115/MSEC2017-2656Search in Google Scholar

[9] C. Yang, S. Lan, S. Weiming, L. Wang, and G. Huang, “Software-defined cloud manufacturing with edge computing for industry 4.0,” in IWCMC 2020, A. Rachedi and M. Guizani, Eds., Piscataway, NJ, IEEE, 2020, pp. 1618–1623.10.1109/IWCMC48107.2020.9148467Search in Google Scholar

[10] M. Sudhoff, C. Prinz, and B. Kuhlenkötter, “A systematic analysis of learning factories in Germany – concepts, production processes, didactics,” Procedia Manuf., vol. 45, pp. 114–120, 2020. https://doi.org/10.1016/j.promfg.2020.04.081.Search in Google Scholar

[11] D. Dhungana, A. Haselbock, S. Meixner, et al.., “Multi-factory production planning using edge computing and IIoT platforms,” J. Syst. Software, vol. 182, p. 111083, 2021. https://doi.org/10.1016/j.jss.2021.111083.Search in Google Scholar

[12] B. H. Huynh, H. Akhtar, and W. Li, “Discrete event simulation for manufacturing performance management and optimization: a case study for model factory,” in 2020 9th International Conference on Industrial Technology and Management (ICITM), IEEE, 2020, pp. 16–20.10.1109/ICITM48982.2020.9080394Search in Google Scholar

[13] E. Abele, G. Chryssolouris, W. Sihn, et al.., “Learning factories for future oriented research and education in manufacturing,” CIRP Ann., vol. 66, no. 2, pp. 803–826, 2017. https://doi.org/10.1016/j.cirp.2017.05.005.Search in Google Scholar

[14] D. Zuehlke, “SmartFactory—towards a factory-ofthings,” Annu. Rev. Control, vol. 34, no. 1, pp. 129–138, 2010. https://doi.org/10.1016/j.arcontrol.2010.02.008.Search in Google Scholar

[15] M. Hennig, G. Reisinger, T. Trautner, P. Hold, D. Gerhard, and A. Mazak, “TU wien Pilot factory industry 4.0,” Procedia Manuf., vol. 31, pp. 200–205, 2019. https://doi.org/10.1016/j.promfg.2019.03.032.Search in Google Scholar

[16] OPC Foundation, OPC 10031-4: OPC UA for ISA-95: Part 4: Job Control, Scottsdale, OPC Foundation, 2021.Search in Google Scholar

[17] LAN/MAN Standards Committee of the IEEE Computer Society, IEEE Std 802.1Qcc-2018. S.l., New York, IEEE, 2018.Search in Google Scholar

[18] G. Bengel, C. Baun, M. Kunze, and K. Stucky, Masterkurs Parallele und Verteilte Systeme, Wiesbaden, Springer Fachmedien Wiesbaden, 2015.10.1007/978-3-8348-2151-5Search in Google Scholar

[19] A. Eckhardt, S. Muller, and L. Leurs, “An evaluation of the applicability of OPC UA publish subscribe on factory automation use cases,” in 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, 2018, pp. 1071–1074.10.1109/ETFA.2018.8502445Search in Google Scholar

[20] L. Abeni, A. Balsini, and T. Cucinotta, “Container-based real-time scheduling in the Linux kernel,” ACM SIGBED Rev., vol. 16, no. 3, pp. 33–38, 2019. https://doi.org/10.1145/3373400.3373405.Search in Google Scholar

[21] L. Abeni, A. Biondi, and E. Bini, “Hierarchical scheduling of real-time tasks over Linux-based virtual machines,” J. Syst. Software, vol. 149, pp. 234–249, 2019. https://doi.org/10.1016/j.jss.2018.12.008.Search in Google Scholar

[22] V. Struhar, S. Craciunas, M. Ashjaei, M. Behnam, and A. Papadopoulos, “REACT: enabling real-time container orchestration,” in Proceedings, 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Piscataway, New Jersey, IEEE, 2021, pp. 1–8.10.1109/ETFA45728.2021.9613685Search in Google Scholar

[23] D. Dietrich, M. Neubauer, A. Lechler, and A. Verl, “Fehlerfreie Produktion durch flexibilisierte Prozesse,” WT Werkstattstech., vol. 112, no. 5, pp. 276–281, 2022. https://doi.org/10.37544/143649802022056.Search in Google Scholar

Received: 2022-10-31
Accepted: 2023-02-24
Published Online: 2023-05-08
Published in Print: 2023-05-25

© 2023 the author(s), published by De Gruyter, Berlin/Boston

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 29.3.2024 from https://www.degruyter.com/document/doi/10.1515/auto-2022-0137/html
Scroll to top button