Enabling End-Users in Designing and Executing of Complex, Collaborative Robotic Processes

: Over the last years, capabilities of robotic systems have quantitatively and qualitatively improved. But going beyond isolated robotic systems, the integration and interoperability of robotic capabilities in complex work processes remains a major challenge. This lack of tools to integrate robots needs to be addressed on technical, semantic and organizational level. In the ROBxTASK research project, we developed an approach to support cooperation between different types of users in order to enable domain experts, with no robotic know-how, to work with robot-assisted workﬂows. By engineering robotic skills at a useful and usable level of abstraction for experts in different domains, we aim to increase re-usability of these skills on two different levels, (robotic) device level, and on level of application speciﬁc workﬂows. The researched prototype consists of a web platform, which allows (a) engineers to register (robotic) devices and the implemented skills of the devices, (b) domain experts to use a graphical task design environment to create workﬂows across multiple robotic devices and lastly (c) robot co-workers to download and execute the workﬂow code in a local environment with digital twins or real robots. Additionally skills and workﬂows can be shared across organisations. Initial user studies have shown that the visual programming environment is accessible and the deﬁned skill-set is easy to understand even for domain experts that are inexperienced in the ﬁeld of robotics.


Introduction
Robot technologies are already used in very varying areas of our society. According to the International Federation of Robotics (IFR) [1] the World Robotics 2021 Industrial Robots report shows a record of 3 million industrial robots operating in factories around the world-an increase of 10% compared to 2020. The capabilities of robots have increased, with a high velocity where, for example between 1995 and 2016, the number of robot-related patent families worldwide have increased exponentially [2].
Within the service sector the market for Autonomous Mobile Robots (AMRs) is growing fast [3]. According to a recent market research study [4] containing the analysis of more than 500 players and 15+ geographical countries/regions, Automated Guided Vehicle (AGV) and AMR Market is expected to reach $13.2 Billion by 2026 with a growth rate of around~35%. "Mobile robots and AGVs are used in industrial automation applications and as service robots in new environments such as hospitals." [5] (p. 2). Other application areas for mobile robots are production logistics, agriculture, and service [6].
New domains for the application of social robots [7] have emerged. Among others, in health and daily care services the potentials of social robots have been demonstrated [8].
Implementing robotic solutions in a domain requires the expertise and collaboration of human resources from many disciplines. This is in particular true in complex environments like manufacturing [9]. Here, the missing workforce requires an increased level of flexible automation [10].
Approaches that promise less manual effort, such as data-driven approaches [11,12], are likely to fail due to missing data and changing processes. A model-driven approach is desired, requiring expert knowledge of both the robotic systems and the application domain in which the robotic processes are executed. In the industrial environment, holistic robotics solutions are often easier to implement due to the proximity of technical tasks that usually dominate there. In domains like healthcare, involving domain experts can be more difficult, because in many cases they do not have any robotic know-how. In any case, knowledge of a complex and unique situation is needed, where requirements and processes of the application domain, in general, and in the particular of the to-be automated use-case, is combined with the technical knowledge how to program collaborative robots interacting with humans and the other system along the processes. Interoperability needs to be established on technical, semantic and organizational process level [13,14].
Training people from the application domain in computational thinking for being able to program robotic processes requires effort [15]. The same is true in training robot programmers in understanding the particularities of the end-user domain. What is needed, is an approach that allows people either with robotic know-how to easily understand the requirements and the processes of the application domain, or an approach where (application) domain experts can easily model robotic processes.
In the project ROBxTASK, we have focused on the second approach [16][17][18][19]. In brief, in this project we support multiple user groups working together on the task of creating robotic programs. The three roles supported are: (a) the Robotic Engineer to provide concrete implementations of robotic skills for a specific hardware; (b) the Task Designer, a person with partial domain and robotic know-how, is working on robotic workflows that build on the skills provided by (a); (c) the Robot Co-Worker is an end-user domain expert with no robotic knowledge, but who is working with robots and has the responsibility to execute the workflows designed by (b).
Application-oriented research of accessible interfaces that allow end-users to build robot-based applications has already been researched [20,21] including applications for robot-based therapy [22][23][24], education [25], social robotics [26,27], industry [28] and service robots [29]. A broad overview on the topic of end user robotic programming is presented by Ajaykumar, Steele, and Huang [30]. In general we found the following platforms and research of particular interest: • Trigger-Action Programming: The trigger-action paradigm [31], is a possible approach to help application domain experts, without specific programming know-how, to personalize the behaviour of humanoid robots. The tool IFTTT (https://ifttt.com, accessed on 8 May 2023) is a well known example, implementing this paradigm. • Code3: A System for End-to-End Programming of Mobile Manipulator Robots for Novices and Experts [32]. Code3 builds on Google's ™Blockly (https://developers. google.com/blockly, accessed on 8 May 2023) [33] environment. However there the focus is on single robots. The focus with respect to targeted user group is on programmers (with and without robotic coding skills). Code3 enables use-case-specific code using three parts "(1) CustomLandmarks for developing perceptual capabilities, (2) CustomActions for programming manipulation actions, and (3) CodeIt for combining the two in a high-level program that captures task requirements" [32] (p. 2).
Use-cases and robot specific Landmarks and Actions are defined in Code3 and can then be used in the CodeIt environment. This environment shows a lower level of abstraction, which gets visible by code that includes detailed modelling of gripper movements [32]. • CoSTAR [34] a system that allows to use natural abstractions and perception in a way users can both understand and utilize to author plans using behavior trees and high-level information from perception of known objects. A certain skill for behaviour trees is needed. Also the behaviour elements needs to be programmed in a specific way for the targeted systems. • RAZER [35] is a software framework for task-level programming. The system targets both user groups, robotic experts and shop-floor workers. The first user group is supported by a web-based graphical interface providing robotic skills and their interface. The workers are supported by the graphical interface in creating tasks from parameterized skills. Although  While these applications are highly valuable for making robot programming more accessible to end-users, they are partially constrained to certain use cases, application domains, commercial technology providers or specific hardware. In contrast to domainspecific approaches, interfaces for end-user-friendly robot programming that are applicable across different domains and hardware would bring remarkable benefits, since robotic tasks in different application domains are usually similar. For example, picking up an item and placing the item in another location is a task that is common in both industrial and healthcare domains [18]. Thus, applications that enable both the easy development of robot-based tasks and the exchange of tasks between different application domains bear great potential, as they can support transfer of knowledge between different domains, decrease redundancies, and ultimately reduce time and costs.
The contributions of our research can be summarized as follows: • User-centered evaluation of the platform for task designers based on three user studies to test the usability of the developed platform for the selected target group.
The paper is organized as follows. Section 2 gives an overview of the ROBxTASK platform. In Section 3 the use cases that were used in the project to evaluate the platform are explained. The user studies evaluated in the research project are then described in detail in Section 4. Section 5 discusses the methods and results. Finally, our paper is concluded in Section 6.

ROBxTASK Overview
In order to introduce the reader to the general context of this work, we describe user roles, followed by the interaction between roles.
As briefly mentioned in Section 1, the collaboration platform provides an infrastructure that connects three types of users. This allows to build an organisational workflow to link detailed and hardware-specific implementations with domain experts and robotic end-users.
The three user roles are described in more detail in Figure 1 and in the following. • Robotic Engineer: Users in this role, do not have and do not need any end-user specific know-how. They focus on implementing skills for a specific hardware. • Robot Co-Worker: Users in this role are focusing on the execution of processes. The robot is a tool only. Users in this role have no specific robotic know-how. For this type of user, the focus is on efficiency and effectiveness of execution. • Task Designer: Users in this role bridge the gap between end-user domain-specific needs for executions of processes on the one side and robot-specific implementations of tasks that are needed within the processes. Task Designers need know-how in robotic and end-user domain. However, ROBxTASK support eliminates the need for detailed knowledge for users in this role. The tools and approach supports abstraction of details from both domains. Organizational and tooling support includes reuse and sharing of both process models and task module implementations. The task designer is a system integrator on technical level and on organisational level. The platform supports users in this role to collaborate with both other roles. Robotic engineers are contacted to create task-implementations [38] for robotic systems and on the other side robot co-workers need workflows to execute their processes ( Figure 2).
The ROBxTASK platform uses Google's ™Blockly (https://developers.google.com/ blockly, accessed on 8 May 2023)-Editor on the frontend, which allows users to build appropriate actors/software agents [39][40][41] by using the skills of the corresponding robotic devices. The full workflow to create and run an application involves all previously described user roles and can be simplified into five steps:

1.
Robotic Engineer: Implement skills and register a robot on the platform. An engineer needs to implement a server application (e.g., ROS action Server or OPC UA Server), create a JavaScript Object Notation (JSON) based registration file containing all available skills of the target robot, and register the device on the platform.

2.
Task Designer: Select a robot target platform for task programming.

3.
Task Designer: Create a task design in the Blockly task design editor by using the available robotic skills and combining task modules in any way. May also design a distributed process using multiple robots/robotic devices (see Figure 3).

4.
Task Designer: Save the task design project.

5.
Robot Co-Worker: Start the project download and run the generated program agents/ actors locally. Note that all robots should be in the same intranet for shared execution.

Skills in ROBxTASK
Skill-based programming simplifies the creation of robot programs, and helps to enable non-experts to create robot applications [42]. Thus, a lot of research with respect to skill-based programming can be found in literature especially for the the industrial area [43][44][45][46]. Following Pedersen et al. [47] approaches of task-level programming using skills can be divided into three layers: • Primitives: Perform a single operation related to the robot system. • Skills: Are combination of primitives. • Tasks: Are directly related to solving goals and can be either explicitly programmed or planned ad hoc, if skills are e.g., conform to the STRIPS [48] notation.
Some approaches also take care of supporting to find the right skill. Bøgh et al. [49] developed first a method to find re-usable skills through analysis of industrial tasks, industrial implementations and laboratory experiments; secondly they analyse Standard Operation Procedures (SOP).
The result of the analysis finds that only a few standard skills are needed e.g., to support flexible mobile manipulators for industrial logistics and assistive scenarios: Move To, Locate, Pick up, Place, Unload, Shovel, Check, Align, Open, Close, Press, Release and Turn.
According to [42] (p. 1) "the Skill Based Programming eases the robot program generation, its similarity to human behavior allows non expert operators maintaining, adapting or creating robotic applications", so this approach was also chosen for implementing a visual programming environment in ROBxTASK.
The initial set of skills was drafted in a series of workshops with stakeholders from healthcare and industry domains. One of the major criterion for developing the initial skill was that a specific skill is needed for the description and later implementation of a use case (see Section 3) from the perspective of the task designer: In the workshops the MIRO boards (https://miro.com, accessed on 8 May 2023) (see Figure 4) were used to identify the skills, skill slot parameters and skill slot types presented in this chapter. The MIRO software has proven to be extremely useful in picking up the different groups of participants in the research project with their different levels of knowledge at an appropriate level. Multiple evaluation studies have been conducted to find the right level of abstraction regarding specifically the skill formal definition, slot parameters, return values and applied skill semantics. The slot types that can be used in module descriptions have been initially defined in the workshops of the project. However, during progress of work we have been reducing the slot types to: • "String" • "Numeric" • "Enum" (e.g., list of predefined position from which the user can choose) • "Bool" To enable heterogeneous robots to agree on a basis of skills that can be programmed in the editor, the project developed a skillset that can be targeted by concrete robots while registration of a new robotic system on the platform. The target version of the skill set implementation of the robot has to be provided upon registration. Table 1 shows our defined skills (ROBxTASK skillset version 1.0) and all blocks that are available in the task design editor for programming agents/actors.
Concrete robots can communicate their provided subset of the skillset using a JSONbased registration file (details about registration process following in Section 2.2).
Similar to the servicelevel used in [50] we can add other runtime environment (RTE) features later to provide additional blocks for task programming. This architecture enables us to support building e.g., more intelligent software agents later, by providing AI features, database access, . . . if needed.

ROBxTASK Platform
The ROBxTASK cloud platform is based on the microservice architecture of the "collaboration network for industry, manufacturing, business and logistics in Europe platform" (NIMBLE) [51]. NIMBLE (https://www.salzburgresearch.at/projekt/nimble, accessed on 8 May 2023) is a cloud-based, Industrie 4.0, Internet-of-things-enabled B2B platform on which European manufacturing firms can register, publish machine-readable catalogues for products and services, search for suitable supply chain partners, negotiate contracts and supply logistics, and develop private and secure B2B and M2M information exchange channels to optimise business workflows. Using the existing technology stack from NIMBLE, the ROBxTASK platform can support features like, e.g., user management, registration of items on the platform, searching for registered items or chatting with other users. Some sample dialogs of the platform can be found in Appendix A. Figure 5 shows the Task Design editor, where all registered device skills are available as blocks to be used in Google Blockly. In the Task Design editor, the total workflow can be saved and restored in XML. Finally, the code for a concrete RTE (e.g., "ROS") can be downloaded as a zip file which contains for any device used in the workflow the concrete python agent code. Thus, these agent files can be used by the robot co-worker then to execute the workflow.
When the Robot Co-Worker presses the button "</> OPCUA", the flexible generator (https://github.com/ROBxTASK/codegen-service, accessed on 8 May 2023) generates Python code that supports our ROBxTASK RTE (RxRTE) developed within the project [19]. RxRTE also supports the use of different test environments during the execution of the workflow, e.g., to test workflows without hardware by using mock objects (skill calls are then only emulated by waiting, e.g., 2 s) or to simulate the workflow using a digital twin, and enables the monitoring of the distributed workflow using some additional tools (see Section 3.2.1). The ability to support digital twins proved to be helpful for task designers too. In a later stage of the project we developed a docker based testing environment where task designers can now download their programs from the platform and test them using digital twins which support our use cases. This new environment simplifies the understanding of the programs for task designers, and is described in more detail in Section 4.3.

Use Cases
A central goal of the ROBxTASK project is to demonstrate and validate the functionality of the platform based on practice-oriented use cases. To promote the cross-domain usability of the platform, use cases in healthcare and industry have been planned.
To identify meaningful use cases for both application domains, design fiction workshops with stakeholders from industry and healthcare domains were conducted as part of the human-centered development process in ROBxTASK. In the workshops, stakeholders were asked to brainstorm useful robot-based applications for the respective domains. Next, these ideas were further developed into concrete use cases, also taking their practical feasibility into account. The different use cases were further refined by members of the project consortium as described in Sections 3.1 and 3.2. To test the cross-device use of our skills, we planned to use different robotic devices in the use cases for the same robotic tasks, e.g., for grasping/lying operations (skills: GrabObject, PutObject) and transport tasks (skill: MoveToLocation).

Use Case Healthcare (UCH)
In contrast to the manufacturing industry the Nursing and Care industry is dominated by direct human interaction in a physical and social manner. That fact rises specific considerations about the implementation of robotic systems within this environment. Societal and ethical issues like autonomy, dignity, acceptance and responsibility are the primary subjects of the debate nevertheless, in view of the increasingly aging society and simultaneous shortage of skilled workers within the nursing and care sector, assisting robotic systems can be a building block to relive that tensioned situation [52].
The knowledge of predictable physical activities is crucial to identify further application fields. Estimates assume the proportion of such activities within nursing and care industry around 30-36% opposed to the manufacturing industry, where the amount of predictable activities is up to 59% [56].
In order to identify an applicable use case for nursing and healthcare scenarios in ROBxTASK, an analysis on the basis of "nursing phenomena" [57] in combination with a fictitious "nursing process" [58] was conducted. Therefore, two possible scenarios were selected, namely "malnutrition" and "high risk of falls". Both variants were presented and discussed in the consortium, finally a consensus was reached on the malnutrition scenario. This is mainly explained by the consideration of possible technical implementation and practical added value for nursing professions. The process design was guided by the view of robotic systems as assistive devices in the day-to-day nursing process.
The flowchart in Figure 6 illustrates the whole malnutrition process. This process is characterized by the fact that intersections between health professions (nurses), task designers and the robotic system become visible. The process starts on the task designer side with the decision for whom the use case should be active and which assessment or screening tool should be applied. The robotic system independently performs the selected assessment or screening tool at predefined intervals and compares the data to predefined thresholds. If no deviation is detected, the robotic system performs a re-assessment at a specified interval. In the event of a detected deviation, an action message is sent by the robotic system to the health professionals. In addition, predefined measures take place with regard to close surveillance.
During the project implementation the use case was analyzed and discussed in detail. The joint decision was made from a feasibility perspective and represents a sub-process in the main malnutrition process, namely receive the request to bring beverages and monitor the fluid intake. The next step was to examine the use cases of industry and healthcare for synergies. The goal was to determine which robotic motion sequence can be similarly or even identically mapped. As a result, elements of the initial process could be used for further processing in the project.
In the healthcare domain we want to show that we can use the presented visual programming approach to enable an immobile patient to get drinks and food delivered to their patient bed. Since we want to show robot-cross execution we separated this delivery task into three different robots, each equipped with different individual unique skills. No single robot is able to perform the task alone, the robot always needs the skills of the others as well. Their shared task execution should work as described by the task designer in Blockly task design editor and collectively deliver an order to the patient bed.
For the healthcare domain we implemented a ROS-based local environment and registered three robots (see Figure 7) to the platform via JSON upload:  In the registration of the three robots the task designer can get additional information on how the robots are registered and which skills they do implement (decided and implemented by the robotic engineer). For the healthcare demonstration lab setup, we registered and implemented skills based on the minimum requirements needed to perform the scenario presented. After successful registration of these three robots, the task designer may for example design the workflow of the robots in the Blockly task design editor as follows (see also Figure 8):

1.
Patients can interact with the social robot 'Q.Bo One' and place an order via audio input (e.g., say 'Hugo, I need a cup of water!'). In our lab setup, there are nine different storage positions to choose food and drinks from, so the designer needs to hard-assign storage positions to audio commands.

2.
Now the mobile robot 'Chasi' will autonomously drive to the drink station. 3.
Next, the arm robot 'Franka Panda' will grasp the desired item and places the item on the mobile robot. The designer here needs to make sure that the mobile robot is always correctly parked in the parking slot first. 4.
Then the mobile robot 'Chasi' will autonomously drive to the patient bed. Finally, after the item has been picked up, the robot should drive back to the original parking position. In the end, how they want to implement the specific scenario is left up to the task designers. The goal however is to deliver an ordered item to an immobile patient lying in a bed and communicating with Q.Bo robot. Figure 9 shows an exemplary task design for the healthcare use case.  The ROBxTASK lab translation modules take commands from the ROS action clients (Python client) and communicate them to the robot's action server implementation (Python server). Each robot hosts his own action server and waits for ROBxTASK commands to execute given tasks. These action servers perform the action on the robot accordingly. Commands are communicated via intranet in Python-based ROS action topics with strictly defined message formats. Command sending success is evaluated using the Wireshark tool (https://www.wireshark.org, accessed on 8 May 2023). The ROS action clients are automatically generated by the codegenerator based on the compiled Task Design Editor project.
While the robots execute the command the planned paths of every movement are visualized for testing purposes in RViz before the actual movement is executed. This also allows stopping the robot before executing the movement by means of emergency stop functions and prevents self-damage of the robot due to bad ROBxTASK command designs. Since commands are derived from the graphical user interface Blockly application they will be accessible to non-technical users, which requires the robots to have this additional safety layer. Figure 11 shows what the RViz tools graphical user interface looks like when used with Franka Panda arm robot.
The autonomous driving of ARTI Chasi robot is achieved through Monte Carlo algorithm Simultaneous Localization and Mapping (SLAM), ROS navigation stack and Google Cartographer (https://google-cartographer-ros.readthedocs.io/en/latest, accessed on 8 May 2023). The robot uses light detection and ranging (LIDAR) and 3D cameras for indoor navigation. We applied existing outdoor navigation algorithms to work for indoor navigation in corridors and rooms. Figure 12 shows a RViz visualisation of a sample robot navigation within the map of the Salzburg Research lab. On the left image one can see the robot in the lab trying to find his way down the narrow corridor to reach the room next to the lab (on the right side of the left image). On the right image we can see a clean bird perspective of the lab where the room on the right and the corridor have not yet been discovered by the cartographer. The food and drink pickup station is located in the lab (left room), the patient bed with Q.Bo social robot is situated in the neighboring right room.  However, this technical RTE toolchain is only of interest for robotic engineers. Task designers can simply use the final skills to implement appropriate tasks.

Use Case Industry (UCI)
The conceptualized industrial use case application targets the domain of manufacturing, specifically the pre-sorting, commissioning and delivery of finished parts or raw materials. Similarly, as for the healthcare domain, the primary focus is on applying the ROBxTASK programming approach to implement the required system behavior. Highly optimized primitives for, e.g., precise handling of parts, or visual perception are beyond the scope of this use case implementation. The following actors and sensors are integrated and coordinated within an OPC UA-based environment: Note that the mentioned peripherals (sensors and camera system) are physically linked and managed by the robot systems UR10 and MiR100. This is because of practical reasons, e.g., a robot controller usually comes with digital/analog I/O ports, and simplicity reason, i.e., to reduce system complexity.
The system layout considered for the industrial use case is depicted in Figure 13. The stationary UR10 robot system ('UCIUr10'), equipped with an electric gripper and a camera system, is configured to (a) grab objects of type 'Part_Cube' and 'Part_Cylinder' and (b) to put the same type of objects to predefined target locations. Those target locations include a 'Box', two slides (gravity chutes 'Slide.Part_Cube' and 'Slide.Part_Cylinder') are used as storage for sorted parts, and a tray to hold pre-commissioned parts of both types. The tray is mounted on an autonomously guided vehicle (AGV) of type MiR100 ('UCIMir'), which enables transport of parts to the required storage or production cell. Both slides include sensors at the entrance and bottom of the slides to enable interpretation of the storage level. As an application use case, a three-step process shall be considered: (a) Bin-picking of parts (both types) out of the 'Box' and placing them onto the slide storage in a sorted fashion. (b) Picking sorted parts (both types) from the slide storage and commissioning them onto the tray at the designated tray-sections. (c) Transportation of the commissioned parts to the target destination.
The skills for UCI conform with the ROBxTASK skill definitions according to Table 1. To reduce implementation complexity for the task designer, the 'GrabObject' skill also includes the object localization process required to retrieve unordered parts from the 'Box'. Moreover, vision systems are tightly linked to the robot system as extrinsic calibration is often inevitable to achieve reasonable accuracy during part manipulation. The implementation of 'GrabObject' allows for an empty string passed to the 'ÒbjectType' slot if the type of object can be deduced from the specified 'ObjectPosition'. To give an example: 'Slide.Part_Cube' specified as position directly suggests that type 'Part_Cube' is of interest. The slots 'ObjectPosition' and 'TargetPosition' also support alias-names for the slide storages. This need was identified during skill implementation, when initially the IDs 'Slide_A' and 'Slide_B' were introduced. However, while such identifiers can be intuitive enough for a robotic engineer, they are less self-explanatory for a task designer.
Using the setting described in Figure 13 we now wanted to research how to solve an industrial part delivery scenario. Figure 14 summarizes all available robots, skills, objects and locations. We recommend that such an overview graph also is provided to task designers so that they can understand all available robotic skills and and their supporting arguments.    For our UCI we first generated OPC UA servers that support the planned robotic skills by using XRob [59] and a SCXML based workflow designer tool.
Using some helper tools we than scanned the OPC UA servers to generate appropriate agent type stubs that can be used by agents to call the robotic skills using OPC UA, and the registration files for the ROBxTASK platform to register theses skills. The code generator then was adapted to be able to call the appropriate software layer to support our RxRTE. Now, when the Robot Co-Worker presses the button "</> OPCUA", the flexible code generator generates Python code that supports RxRTE to download for all Blockly agents and theses agents can then be started as individual processes.
In the project we can also generate mock objects [60], where skill calls are only simulated by pausing the program for 2 s. This feature is helpful for testing without robotic hardware and was used for our initial tests. Finally, a monitoring application was developed in the project, that enables users to follow the message exchange between devices and see the execution of the skills. A sample ROBxTASK monitor [19] webview of the running UCI2 process of Figure 16 is depicted in Figure 17. Later both industrial use cases were also tested successfully by usage of the simulation system twin (https://www.digifai.com/twin, accessed on 8 May 2023) which was provided by our project partner Eberle Automatische Systeme GmbH & Co KG. twin is a physic based tool for the virtual commissioning of special purpose machines. In the project, e.g., a simulation model was built that works together with our robotic OPC UA servers for UCI (see Figure 18).
A video of the running simulation in twin that also shows the usage of the monitoring application is given in [61].
In the further progress of the project the scenario was then developed and tested using real hardware (see Figure 19). However, except for configuration changes, this did not mean any changes for the agents or the already developed code, since the OPC UA interface remained the same.
The simulation model developed in twin can also support task designers to test their robotic programs without a need for concrete hardware (see Section 4.3). Thus, within our project task designers can develop and test robotic workflows which are then transferable to run on real hardware.

Discussions Concerning the UCI Scenarios
Although at first glance these scenarios seem simple. However, a closer look at the details raises some questions: Or what if the sensor-name differs on another slider scenario? An initial finding is that using real sensor-names in programs makes them less portable to other settings. Also if someone wants to enlarge the space for parts at the table, e.g., by adding additional slides 'Slide_C' and 'Slide_D' later, the programs then must be massively adapted by the task engineer. This might even be too complex.
Thus, we propose to hide these details for the task designer. Actually, the plan in the project was to include only the obvious skills of robotic devices via registration to the platform. But if we look deeper in this grab and put scenarios, we immediately see, that some kind of storage concept would be very useful for manipulators, boxes, slides and trays. If we grab objects we may have a camera at this position which can tell us if there's a part available to grab. Or some technical sensors could have this knowledge. Or some algorithm is internally counting parts on locations. One question in our use case was also whether to model the camera on the UR10 as an individual device. But we decided against this because the camera was mounted directly on the UR10. However, this now led to a sort of 'combined device' of a UR10 and a camera, which left room for discussion about what skills to provide then. For example, if someone implements only a skill GrabObject and internally hides the cameras skill DetectObject, then the error case when the camera did not detect a part must be included in the GrabObject skill.
However, if reuse and program simplicity are to be the primary concerns, we suggest that the task engineer not be burdened with these technical details. Instead we propose to add a "Virtual Storage" concept providing some appropriate skills: At a minimum one storage skill shall give task designers the ability to count how many objects of a given objecttype are at a given location, e.g., CountObjects(sObjType, sObjPosition). And a second skill, e.g., CountObjectSpaces(sObjType, sObjPosition) is needed to give the numbers of spaces available for a special object type at a given Position. Some additional comfort functions may, e.g., be ExistsObject(sObjType, sObjPosition) and ExistsOb-jectSpace(sObjType, sObjPosition).
The usage of these skills now can increase the interoperability between different technical robot scenarios in the future by abstracting the concrete technical implementation. This also leaves enough room for the robotic engineer for later process improvements by, e.g., adding cameras, additional sensors or slides. So in UCI1 instead of 1 while not UciUr10 . GetData ( S l i d e _ A _ S t a t u s _ P a r t _ C u b e _ S t o r a g e I s F u l l ) :  Table ' , ' Part_Cube ') : can be used to test if there's enough space left to put cubes on the (slide on the) table. And in both use cases CountObjects can be used before a grab operation to not grab parts if there are none available. Whereas CountObjectSpaces can prevent to try to put objects at positions where there is no space available, if this is needed in user programming.
Thus, in order to develop more robust, portable and adaptable tasks, we recommend determining the appropriate skills based on more than just the obvious robotic skills -we see a need for some kind of environmental skills too.

User Studies
In the context of developing interfaces for robot-programming by end-users, explicit emphasis of principles related to human centered design is called for [20]. Thus, the development of the ROBxTASK platform is embedded into an continuous human-centered design process involving various stakeholders from the industry and healthcare domains. In the following, we provide an overview of the methodology and results of user studies that have already been conducted in the context of the human-centered design process. Since the ROBxTASK platform is still under development, we also preview future studies aimed at evaluating refined versions of the platform.

Methodology
In the studies, we mainly focused on small scale usability tests as they are an efficient tool to identify usability issues at an early development stage. According to Faulkner [63], in a usability test involving only 5 participants 86% of usability problems can be identified on average. Each of our studies included task-based evaluations of the respective prototype employing the thinking aloud method [64] (i.e., participants were asked to solve various tasks with the platform while articulating their thoughts in the process) and additional interview questions. So far, 3 user studies (RxT01, RxT02 and RxT03) have been carried out in the project. The studies were conducted in an iterative process that evaluated various prototypes of the platform and incorporated feedback from the studies into technical development. In each study, a refined version of the platform prototype was evaluated (compared to previous studies).
All studies involved stakeholders from industry and healthcare domains. Emphasis was put on early identification of obstacles related to the interaction with the platform and insights into possibilities for improvements. Due to restrictions related to the ongoing COVID-19 pandemic, studies were conducted in an online or hybrid setting. A video conferencing software was used for remote communication (speech and video) and recording of the study sessions (compliant with data protection regulations). Prior to the respective study, participants were informed about the goals and content of the study and gave their informed consent to participate.
While the first study did not include any training, training with the platform was applied for the second and third study. This means that before solving the study tasks, participants received a short briefing on how to use the platform. In addition, all persons who participated in the second study also took part in the third study to simulate a realistic environment in which persons become familiar with using the platform over time. Table 2 provides an overview of the sample, procedure and study tasks in the 3 different ROBxTASK studies conducted so far. Register an account or log in to the participant account.
Add a new asset to a specific process.
Define a process (using the task design editor) to exchange a message between two devices.

Search for devices.
Search for a new task module.
Define a process to grab and place an object.
Register a device.
Edit and save a specific task module.
Define a process in which a robot reacts to human speech input.
Use the task design editor to define the process for the basic industrial use case. Recreate the described processes with the help of "skills" (blocks) provided (Industry domain).
Define a process to move a robot to a different position.
Use the task design editor to define the process for the basic healthcare use case. Recreate the described processes with the help of "skills" (blocks) provided (Healthcare domain).
Define a process for measuring dehydration in a patient.
Define a process to pick up and place multiple objects by a robot.
Freely define a process.
For analysis, recordings from the study sessions were transcribed. Based on this data, statements were grouped into overarching themes and implications for the further development of the platform were derived. The results presented in the following focus on positive and negative feedback from users and respective implications.

User Study RxT01
In general, participants expressed positive feedback about the cloud platform mock-up in the first study. The design of the user interface was highly appreciated and all participants were able to complete the tasks using the VPL prototype, describing them as fairly simple. However, several participants suggested that initial training and additional documentation would be needed to make the platform even more accessible. All participants indicated that they could imagine using such a platform. More detailed results from the first (positive and negative feedback from participants) study are presented in Table 3. A list of implications was derived from the analysis of the participants comments and suggestions. Each implication was assigned a priority "high", "middle" or "low". These implications will be considered as the platform continues to evolve. For example, implications rated as "high" focused on maintaining a consistent tab layout and terminology, defining some functionalities more clearly, displaying more information when hovering, necessity for documentation, grouping blocks into categories. The implications also contributed to the improvement of the studies, e.g., the necessity for documentation and training was rated as "high" and therefore a training concept was developed for the following studies.

User Study RxT02
Generally, the second user study showed positive feedback from the participants from both the healthcare and the industry domains. The participants appreciated the design of the platform prototype and the simplification of programming. Participants highlighted the benefit of initial training to get an introduction to the platform functionalities and the programming. They also showed interest in using the platform in the future. In addition, participants pointed out some negative aspects of the platform and future improvements they would like to see. Participants from both areas commented as follows on the tasks they had to complete during the assessment: • Interaction with user registration on the platform was considered very easy. • The tasks that focus on an action that requires only a few blocks, such as exchanging messages, grabbing an object, or moving to a position, were considered easy to perform and required minimum assistance from the study facilitator. • Tasks requiring the design of a complete process were considered more challenging and required more support in interpreting the task and translating the task into blocks.
The feedback is presented in more detail in Table 4. • For non-programmers some information or explanations are missing (e.g., what is a string object). • Better information or explanation for blocks is missing (e.g., some blocks seem to have the same function and for some blocks participants don't understand how to use them). • To make the platform more practical, some functions should be improved (select multiple blocks together, scroll function in the TaskDesign editor). • More information for blocks, command and devices would be useful. • Lack of feedback from the platform on whether the user is using the blocks as intended.

Industry Domain
• Platform design is very pleasing. • Programming is simplified and easy to handle for a first time user thanks to the existing blocks and the hovering help. • Platform could be used for training purposes and show many robotics applications.
• Some blocks need more information to use them (e.g., data unit and format). • Blocks need some optimizations to make the programming more natural (e.g., use blocks inside blocks, blocks prefilled with example). • Saving and loading of a program is not user-friendly. • Make the saving/loading of a program more standard. • Make a short tutorial included in the platform for beginners. • More information about blocks.
In the second study, the implications focused on specific aspects of certain functionalities. For example, the "Save" button should directly save the user's program as an XML file and not open the XML code in another browser tab, the "Load" button should open the Windows Explorer instead of providing an interface for copying and pasting the XML file, the type of the blocks' parameters should be clearer, access to the free text field should be easier, or more information should be added when hovering.

User Study RxT03
In the third study, the less complex tasks (e.g., logging in) were considered easy-except for the registration of a device. In contrast, the more complex tasks involving the task design editor (i.e., designing complete use cases for health and industry domains) required the assistance of the study facilitator and were more challenging for the participants. Detailed feedback from participants from the third study is presented in Table 5. Since the participants in the third study already had more experience with the platform (see Section 4.1), the implications derived from the third study focused primarily on the newly introduced functionalities. For example, device skills could be more accurately described in search results, device registration could be made simpler (no use of a JSON file), runtime errors should be fixed, and the user account password restoration could be improved. Finally, the fact that most participants required help from the study facilitators when designing more complex use cases again shows that-while the platform is considered easy to use by most users-approaches for end-user training are still important to consider.
As part of the third user study (in addition to the evaluation with potential end-users), a checklist of 27 functional criteria (i.e., functionalities that should eventually be implemented in the final version of the platform) was created and evaluated by two study facilitators. Based on the checklist, the current prototype of the platform was explored and the individual criteria were reviewed. The results are depicted in Table 6 and provide an comprehensive overview of the functionalities implemented in the current ROBxTASK prototype. Table 6. ROBxTASK platform functional criteria ( fulfilled, partly fulfilled, not fulfilled, not yet tested).

Group Criteria
Platform Users can log in to the platform using their name and password.
A new user account can be created on the cloud platform.
(New) devices can be added to a process in the cloud platform. Actual status: JSON Files only.
Processes defined using blocks or task modules can be saved in the cloud platform. Actual status: JSON Files only, download button missing.
Operations in the task design editor can be undone (optional). Operations in the task design editor can be repeated (optional). Task modules, processes and devices (assets) can be searched for in the cloud platform. Actual status: only available for some devices.
A flag for public access (i.e., public or private) can be set when creating/saving task modules or processes to allow access only for certain user roles on the platform If the access to a task module or process is set to private (i.e., non-public) only certain user roles can view and access the respective item.

Programming
All blocks or task modules required for implementing the basic use cases are available in the cloud platform.
All devices required to implement the basic use cases are available in the cloud platform.
Different blocks or task modules can be combined with each other in the cloud platform ("workspace" of the task design editor) using drag and drop.
Blocks or task modules are organized into categories in the task design editor. Blocks or task modules can be removed from the task design editor workspace. Blocks, task modules and combinations of them can be copied from the task design editor workspace. Usability improvements required.
Blocks, task modules and combinations of them can be cut from the task design editor workspace. Usability improvements required.
Blocks, task modules and combinations of them can be pasted from the clipboard into the task design editor workspace.
Blocks or task modules can only be assigned to a device if that device can execute them.
Arguments can be passed to blocks and task modules in the cloud platform. Predefined variables required for the implementation of the use cases (e.g., position specifications) are defined in the cloud platform.
Communication between different devices is visualized in the task design editor.
Additional information on the functionality of a block or task module is displayed when hovering over the item in the task design editor.

Runtime
Program code is generated by combining blocks or task modules in the task design editor.
Program code generated by combining blocks or task modules can be downloaded or exported.
Downloaded or exported program code can be transferred to the simulation environment.
Code transferred to the simulation environment correctly represents the use case for the industry domain and can be executed without errors.
Code transferred to the simulation environment correctly represents the use case for the healthcare domain and can be executed without errors.

Outlook
An important aspect raised by the test users was that they also wanted to see the robotic actions when executing their programs. Of course, this is not easy to solve with real hardware. But in a later stage of the project we developed a docker based testing environment where our usecases UCH and UCI can be tested virtual by help of the twin models that were developed in earlier stages of the project. Users can (1) download their robotic programs from the ROBxTASK platform to some shared folder of a docker-container, (2) start the program within a docker-container, and (3) see the simulation in the 3D-view of twin while (4) observing the monitoring output of the distributed workflow (see Figure 20). An example video that shows how this testing environment can be used by end-users is given in [65].
The upcoming user studies will use the new testing environment, and focus on evaluating revised and improved, as well as newly implemented, functions of the platform and the VPL. The studies will be based on the healthcare and industrial use cases defined in the ROBxTASK project and will collect objective metrics (e.g., task time) and subjective data (e.g., ratings on user experience, qualitative feedback) to continuously monitor and improve the quality of the application. Furthermore, future studies will evaluate the platform in real-world settings and explore factors that may facilitate or hinder the practical application of the task design process from an end-user perspective.

Discussions
By leveraging NIMBLE's [51] existing technology stack, the ROBxTASK platform could be faster developed than starting from scratch. With the help of the flexible architecture, further improvements can be added more easily in the future (e.g., self-registering of devices).
In the project a flexible code generator was developed that allows to generate, e.g., Python code for different RTEs. For our healthcare use case the code generator, e.g., generates code to call ROS actions. Whereas for the industrial use case the code generator produces source code to be run by using a RxRTE that was developed to be able to also support exotic runtime environments and robotic devices. The ROBxTASK platform can therefore be used with various robotic RTEs and thus, the concrete RTE used can be hidden for the task designers.
In our industrial use case we contact the robotic devices via OPC UA. For the OPC UA environment, we also developed a parser and code generator so that as new skills become available on a suitable robotic OPC UA server, we can generate agent type stubs and JSON files to register theses skills on the platform and then let task designers use them, speeding up development.
The logging and monitoring capabilities of the RxRTE environment proved to be helpful in observing distributed tasks. Displaying message flow paths enables robotic co-workers to comfortable follow the current state of skill execution. Even task designers can pre-test robotic workflows to find problems in a workflow, when using a mocking environment and RxRTE. The platform currently supports only basic interoperability (messaging) to be used between agents for the ROBxTASK platform. Further improvements that support service discovery and mediation have been implemented in the RxRTE [19] but are currently not available using Google Blockly blocks to be used in the platform. In general additional RxRTE features added later, can also be registered on the platform to provide additional blocks for task programming. Since AI capabilities, database access and other additional functions can be provided in this way, the development of more intelligent and autonomous software agents shall be supported in the future.
By integration twin simulation, problems in the environment can now be more easily detected by robot co-workers before they work on the real hardware. Using twin for testing is also helpful for task designers to test the robotic programs and improve their understanding on how to properly use the robotic skills. But of course, this requires manual development of a suitable simulation environment.
If advanced error handling is to be integrated in the future, more research is needed. In this context, we see maintaining usability for task designers while developing robust robot workflows as a challenge yet to be solved.

Conclusions
In the research project ROBxTASK, the partners developed a platform to support end-users in designing and executing robotic processes. Using the platform, multiple user groups work together. Robotic engineers can register (robotic) devices including their skills by using appropriate JSON files. Task designers can use these skills in the "Google Blockly"-based, graphical task design editor to create workflows across multiple robotic agents. And robot co-workers can download and execute the workflow code in a local environment with real robots or digital twins. The platform's operational capability was evaluated by end users based on healthcare and industrial use cases.
In workshops MIRO boards proved to be useful to identify skills at the right level of abstraction to be used by task designers. These skills where then applied to use cases in the industrial and healthcare domain. The project used the same skills in different domains and implemented them through different robotic devices to demonstrate the usability of the skills across domains and devices.
In ROBxTASK we have created the three roles described above that collaborate to enable end users without robotic expertise to focus on their tasks, while being able to get support from robotic engineers. Application specific workflows are used to integrate heterogeneous systems (including robots). Individual tasks in these workflows are realized by skills, implemented in the technical systems. In order to develop more robust, portable and adaptable tasks, we recommend determining the appropriate skills based on more than just the obvious robotic or device skills. For example, using "Environmental skills" according to our "Virtual Storage" concept proposed in Section 3.2.2, can help to develop more stable grasping/lying programs.
In the project three user studies were carried out. The user studies showed that although the platform is relatively easy to use, a training concept for end users is necessary. Tasks that focus on an action that requires only a few blocks, such as exchanging messages, grabbing an object, or moving to a position, were considered easy to perform but design of a full process is considered more challenging.
However, participants generally indicated that they could imagine using such a platform, so the platform was considered interesting.
Overall, the ROBxTASK platform is able to help organisations to integrate robotic systems along a workflow. Through features like sharing skills and workflows the platform enables knowledge exchange even across multiple organizations. The three user roles that have been created for ROBxTASK allow that individuals focus with their expertise on certain tasks, where, most importantly, end users can work in their domain and make use of robots without much training. Also the robotic programmer can focus on implementing skills, without the need to understand in detail the application domain(s). Only the role of the process designer needs knowledge of both worlds. However, that role receives the most support from the implemented tools. The graphical process editor allows to stay on a higher level of abstraction with respect to robotic know-how.
The ability to provide simulations for end-user testing proved to be a great added value for end users. However, developing individual simulation environments for each robot environment is time consuming. To this end, some ideas have already been discussed by the project participants to possibly create suitable environments more automatically in a future follow-up project. Further research has to be taken especially when e.g., advanced error handling shall be integrated. Here, ease of use for task designers while building robust robotic workflows can be a challenge.
In the next steps of the project, further studies will be conducted to evaluate the platform with end users in real-world environments.  Institutional Review Board Statement: Ethical review and approval were waived for this study due to experience from a similar projects (e.g., RoboGen, PNr. 866694, AT programme FEMtech), for which ethical commissions clarified non-necessity of approval at this development stages. In the project ROBxTASK ethical monitoring of the requirements regarding ethical, privacy and data protection issues throughout the project lifetime will be achieved by means of designated audit forms and questionnaires (e.g., MEESTAR analysis). This task will also examine the extent to which the project solutions affect ethical, privacy and data protection issues.
Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Conflicts of Interest:
The authors declare no conflict of interest.

Abbreviations
The following abbreviations are used in this manuscript:

Appendix A. Sample Dialogs for the ROBxTASK Platform
The ROBxTASK platform is based on the existing technology stack of (NIMBLE) [51]. Therefore, a lot of documentation can already be found online on the NIMBLE documentation website (https://www.nimble-project.org/docs/, accessed on 8 May 2023).
In addition, only some of the most important ROBxTASK specific dialogs that task designers come into contact with in connection with the development of robotic programs are shown below.
After successfully login to the ROBxTASK platform, the initial dashboard shown in Figure A1 is presented. Users then can register devices or start writing programs using the TaskDesign editor. Robotic Engineers can register new devices using JSON based registration file containing all available skills of the target robot by using the form depicted in Figure A2. This architecture also enables the registration of devices that are currently not yet available in this form, which in turn enables initial usability tests before the corresponding skills have to be developed in simulation or reality. All registered devices can be searched by task designers, to use them for their robotic workflows (see Figure A3). The result of the query explains the available devices, their skills and parameters.