A Taxonomy for Heavy-Duty Telemanipulation Tasks Using Elemental Actions

In the maintenance of large scientific facilities, telemanipulation procedures can involve various subprocedures which in turn are made up of a sequence of subtasks. This work presents a taxonomy which describes a set of elemental actions for heavy-duty telemanipulation, along with an example of these actions in a standard maintenance subprocedure. As maintenance tasks are often very different at high-level, this generalized way of deconstructing tasks allows a highly adaptable approach to describe the sequence of any procedure, which can then be used for such applications as task monitoring, automation or detection of incomplete tasks. We describe in detail the properties of each elemental action and apply the taxonomy to an example subprocedure to show how the process can be generalizable. An automatic state-machine creation stage is shown, which would be used at the task scheduling stage to simplify calculations carried out during the moment-by-moment execution of the task.


Introduction
Telemanipulation involves the direct control of a robot which is situated in a remote environment for manipulation tasks.Often it is a robotic arm controlled by a user with a force-feedback master device.Many previous studies have investigated the properties of their bilateral control systems [1][2][3][4][5] but little research has been conducted into the tasks which are performed using this type of system, and what these tasks consist of at a basic level.
In the maintenance of large scientific facilities, telemanipulation procedures can involve various subprocedures which are made up of a sequence of subtasks.For example, the task of "disassembly of two joined parts" may involve subtasks of: securing the part, cutting a weld line, unscrewing bolts and then disassembly of the parts.In turn these subtasks can be broken down further into a sequence of elemental actions, e.g., "cutting a weld line" would involve: aligning the welding iron, following the line carefully and retreating.
We propose a taxonomy of basic actions from which all higher-level telemanipulation tasks in heavy-duty maintenance are built.While the taxonomy has been developed with direct telemanipulation in mind, where the manipulator tracks the operator's movements to determine the current stage in the procedure, it could also be applicable to autonomous or semi-autonomous teleoperation, with the autonomous system taking control of the robot during non-critical parts of the task (such as retreating) to reduce the cognitive load on the operator.
Currently in environments where teleoperation is used, the progression from one subtask to another is overseen by an additional operator, who has the sequence of tasks written in front of them.Telemanipulation operators, we have found from discussion with staff from both particle accelerator and nuclear fission facilities, commonly work for shifts of 3-4 hours on a procedure before swapping with another operator.
As well as reducing cognitive load, this type of sequential, low-level monitoring of a task could be used for Fault Detection Isolation -currently a hot topic in robotics [6][7][8][9] -which is concerned with detecting faults in a robotic system to improve their robustness and reduce risk.This approach could be used to detect deviations from a planned procedure which indicate that a fault has occurred in the system.
The motivation behind this research is twofold.Firstly, planning complicated procedures in this manner can aid in reducing the workload of the additional operator during these long shifts by describing the tasks at a more basic level in a state-machine format, allowing them to easily progress from one subtask to another.Secondly, the work in the paper could be extended into the classification of the transitions between elemental actions and thus allow the computer to advance the task status automatically based on the telemanipulator movements.We propose that our approach of defining the elemental actions which make up any task could be a good way of building more generalizable classifiers for a given system.Exact implementation of such an autonomous classification would vary with different manipulators and thus is beyond the scope of this paper, which aims to remain general.

Taxonomies
Taxonomies are a way of organizing information into descriptive subgroups.The classification of movements has been used, for example, to group the different types of movements over agents moving as groups (e.g., flocks of sheep, football teams, etc.) [10].In robotics specifically, taxonomies are often used to define the possible grasps of dexterous robotic hands [11][12][13][14].
Though highly applicable to detailed manipulation with many Degrees of Freedom (DoF) manipulators, these focus more on in-hand movements and do not take into account larger movements of a robotic arm, as our taxonomy does.
The taxonomy in this paper defines the possible elementary actions made by a human operator when operating a robot for heavy-duty telemanipulation.A motion-centric approach has been chosen as it allows for greater flexibility than an object-centric approach, which would require a priori knowledge of the object being manipulated.

Hierarchical Task Description
Maintenance procedures are often described hierarchically, starting from long-term plans which may span several years all the way down to individual maintenance procedures lasting hours, or even minutes.One existing description of such a breakdown is the NASA/NBS standard reference model for telerobot control system architecture (NASREM) which [15] defines in a six-level hierarchy of telerobot control ranging from the lowest level (Level 1) in which coordinate frames are transformed, up to the highest level (Level 6) in which entire mission plans are described.The taxonomy proposed here would be placed at around Levels 3 and 4 of the NASREM hierarchy, with our elemental actions somewhat comparable to their "E-Moves", which describe elementary movements in a sequence to make up a single task command (such as "disassemble part").However, our "elemental actions" differ from the "object-centric" "E-Moves", as they are "arm-centric" and thus do not depend on the object being manipulated.This is an advantage as it does not require that the system should know details of the geometry or physical properties of the environment and objects, only those of the manipulator itself which are likely to be known long in advance.

Proposed Taxonomy
Our proposed taxonomy, shown in Fig. 1, is derived from the hand-centric, motion-centric taxonomy presented by Bullock and Dollar [13] for the different grasps of a human hand.This is a good starting point as it is not object-centric, as are other manipulation taxonomies [16,17], and so is applicable no matter what object is being manipulated.Our taxonomy is arm-centric and motion-centric, specifically the motions of heavy-duty telemanipulation (heavy-duty being defined here as scaled-force manipulation of objects over 20kg).

General terms used in this paper
Task Any high-level work to be done by telemanipulation, e.g., disassembly of two pipes.
Subtask The low-level work which is involved in this task, e.g., securing the pipe, cutting a weld line, unscrewing bolts and then disassembly of the pipes.
Procedure/subprocedure A sequence of tasks make up a procedure.
A sequence of subtasks make up a subprocedure.
Elemental action One of the basic movement types defined in the taxonomy.Several of these may be involved in one subtask.

Primary axis/axes
The main axis/axes along which the elemental action is performed.For example, to weld along a straight line the primary axis will be the collinear axis.

Secondary axes
The axes which support the elemental action.For example, to weld along a straight line the secondary axes will be two perpendicular axes to the line and three rotational axes, all of which would be applied a Hold Steady elemental action.

Terms used within the taxonomy
Contact Force Contact with an external object which is either fixed, such as a wall, or being manipulated, such as a heavy iron bar.The holding of tools, such as power drills, does not come under this category when the high power and scaled force feedback of heavy-duty manipulators makes them almost imperceptible to the user once held.
Movement Intentional, significant movement of the whole arm is considered here.Although some movement may exist during actions, such as Applied Pressure, due to the shaking of the operator's arm from the applied force and weight of the master device, these are considered as no movement.
Rough(Fast) Movements are imprecise, such as pushing a box across a table.Accuracy is not important.Fine(Slow) Movements are required to be precise, such as inserting a part to be assembled or following a weld line.
Pushing/Pulling A force is applied along the primary axis and the object being manipulated is moving as a result of this force.
Applied Pressure A continuous force is applied along a primary axis to an object which does not move, such as a wall.
Path Following A motion following a path along a primary axis which does not require any contact force.For example, when spray painting.
Path Tracing A motion following a path along a primary axis which does require a contact force.For example, when scribing a line into metal.
Approaching Motion towards a point.This differs from Path Following in that the line of movement is not as critical as the end point.Fine approaching motions may be used to align the end effector with a target, say when assembling a part.
Retreating Motion away from a particular subtask.This differs from Approaching in that it is not likely to have an intended end point, and thus will be less controlled.

Hold Steady
The arm is held in place in space.The only force which the user applies is that required to keep the master arm in position.

Relevant Transitions
Some transitions between these elemental types will never occur in a real task.For example, Retreating will never follow Rough Approaching in a sequence, as to do so would be considered part of the same Rough Approaching movement.Table 1 shows a matrix of all 43 possible transitions.In general, Approaching and/or Retreating movements happen between any of the different types of movements, as the operator readjusts their position before beginning their next movement.The only exception to this is between Applied Pressure and Pushing/Pulling movements, which would occur when high pressure is required to overcome the static inertia of a heavy object.The Held Steady movement type could happen between any stage of a task.

Examples of Primary Axis in Maintenance Subtasks
Table 2 shows examples of the common types of maintenance subtasks and their respective elemental actions.These actions are applicable along the main line of motion with secondary actions along the secondary axes to support the action being performed.
Observational tests were performed using a hydraulic telemanipulator, shown in Figure 2, for all of the elemental actions to determine the correct primary and secondary axes for each action.

Intervention Planning
The initial goal of this approach is to implement it at the planning stage of a remote handling intervention, when human operators are deciding what sort of a procedure they are going to carry out with a telemanipulator.A graphical tool, designed to be easily integrable into an existing intervention planner, was developed to automatically extract the relevant elemental actions for a given subtasks and apply them in a state-machine format, with the corresponding Approaching, Hold Steady and Retreating movements.
Figure 4 shows a screenshot of this add-on to the planner.It dynamically creates a menu of types based on a standard format .csvfile.New types of subtask can be added to the system by a simple addition to this file, which specifies the elemental actions which make up the subtask, shown in Table 3.This set of these subtasks types are only required to be assigned once and can then be reused for any procedure.
By introducing the elemental actions at the level of human planning it both encourages planners to think about what sort of actions will be performed in the teleoperation procedure and allows the system to be broken down in a way that is generically applicable to robotic movements.

State-machine Generation
When task names and types have been entered a state-machine is automatically created following the possible transitions as described previously in Table 1.A series of sequential, hierarchical state-machines generated using the python SMACH (state-machine-based execution and coordination system) executive controller libraries [18] for task-level planning and integrated into ROS (Robot Operating System [19]) on a computer running Ubuntu Linux.
Figure 5 shows the top level of this generated state-machine shown in the library's state-machine visualization tool (smach_viewer) which creates a dynamic view of the state-machine.The library has in-built capability to view the task currently being executed, based on simple transition functions which can be simply coded at each node.

Complexity
The hierarchical state-machine is designed to simplify the process of advancing through an entire procedure from the point of view of a human operator and/or any automatized system which could monitor or carry out some part of that procedure.
To calculate the complexity of the resulting procedure the Cyclomatic Complexity metric [20], commonly used to measure the complexity of a graph-based software system, was used -see Equation 1.
where v(G) is the cyclometric complexity of a system, e is the number of edges, n is the number of nodes and p is the number of exit nodes. .
In this state-machine, a node refers to a single elemental action and an edge is the transition between elemental actions.An exit node is the final node in a state-machine (i.e., the end of a procedure), which in the example case will always be 1 as the procedure does not allow for different possible end states.
A higher value of cyclometric complexity indicates a more complex procedure.Although the task to be achieved may look simple from a general level (e.g., Figure 5) we aim to show that at a moment-to-moment movement level even such an apparently simple task is in fact quite complex.Thus, to be able to follow the task in real time, it is worthwhile simplifying the subprocedure which describes the task such that the task state is detectable from moment-to-moment.
Additionally, the number of possible transitions at each node is an important factor in the task complexity, shown in Equation 2.
where α is the maximum number of possible transitions at any single point during the entire subprocedure, n is the total number of nodes in the graph and e i is the number of exit edges for an individual node.This can be compared to the maximum number of possible transitions from any one elemental action to another α max , which is taken from Table 1 as the number of possible transitions from a "fine approaching" elemental movement.

Example Procedure
To demonstrate this approach we have taken an example procedure of "removal of a beam dump target" to demonstrate the application of the elemental actions and automatically generated state-machine.
This example has been adapted for heavy-duty telemanipulation from a real-life procedure in the setting of maintenance of facilities on equipment emitting ionising radiation.

Removal of Beam Dump Target
A simplified subprocedure for the task of removal of a beam dump target is shown below.Subtasks are shown along with their {primary axis} and their associated primary (and sometimes secondary) elemental movements.Figure 5 shows the top level state-machine of this procedure.This is the level of detail at which planning for teleoperation procedures usually is provided.
Each individual subtask is broken down automatically into a series of elementary actions of "Approaching > TASK > Retreating ", and the applicable transitions between these stages are entered to the state-machine transition table, Figure 6.
At the most detail level, shown in Figure 7, each of these elemental actions is further decomposed into a state-machine by including all possible holding actions which could be performed during the action itself.For example, at any point the operator could perform a Hold Steady action, while thinking about the task and during some elemental actions an additional Rough or Fine Approaching action may be used to reorient the manipulator before continuing with the subtask.Using the cyclomatic complexity equation on the highest level (Level 0) and lowest level (Level 2) of the subprocedure, Equations 4 and 5, we can see that the simple subtasks names (e.g., "turn off water lever") hide an underlying 40 times more complexity in respect to the transitions between elemental movements.
This would greatly reduce the problems of detection of transitions, as it means that the set of possible elemental actions at any given point which must be evaluated to determine the following stage in the subprocedure is never going to be more than four.

Conclusion
In this paper we have proposed a taxonomy which describes all elemental actions which can be performed using a heavy-duty telemanipulator.
The taxonomy provides a way of breaking down any subtasks, such as assembling a part or cutting a weld line, into a distinct series of elemental actions.The terms used in this taxonomy were given and explanation made as to how these terms fit into real-world movements.All of the possible transitions between these elemental actions were given along with examples of some common maintenance subtasks to demonstrate how the proposed actions can be used to describe any subtask.
A state-machine implementation was described, and shown to reduce the possible difficulties of moment-by-moment detection of inter-node transitions, in order to simplify intrinsically complicated telemanipulation tasks.

Acknowledgements
The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7-PEOPLE-2010-ITN) as part of the PURESAFE Initial Training Network.

1Figure 1 .
Figure 1.Taxonomy of Elemental Actions in Heavy-Duty Telemanipulation.Actions are defined by contact with an external object and movement of the arm.Each elemental action is reinforced by a pictorial view of the slave end effector and type of movement which this represents.

Figure 2 .
Figure 2. Industrial, hydraulically powered Grips Telemanipulator and master from Kraft Telerobotics.Master and Slave are kinematically similar, with equivalent joints shown.

Figure 3
Figure 3  shows an example of primary and secondary axes for an Applied Pressure action.The Applied Pressure elemental action is applied along the primary axis and all other axes apply a Hold Steady elemental action.

Figure 3 .
Figure 3. Example representation of the primary and secondary axes for an Applied Pressure action.The path taken during the associated Approaching motion is also shown here.Rotational axes are not shown for clarity, but would each involve a Hold Steady elemental action.

Figure 4 .
Figure 4. GUI in which procedure tasks are entered.

1 .
Turn off water lever {perpendicular to lever} -Applied Pressure (Fine Path Following ) 2. Disconnect pipe nut {about axis of nut} -Fine Path Following 3. Removal of cable {axis of insertion} -Fine Pulling 4. Removal of torque limiter {axis of insertion} -Fine Pulling 5. Extracting a screw {axis of screw} -Applied Pressure 6. Removal of block {axis against gravity} -Rough Pushing/Pulling

Figure 5 . 1 Figure 6 .Figure 7 .
Figure 5. Automatically generated top level (Level 0) task view of state-machine with corresponding subtask type as entered into the .csvfile.

Table 1 .
Possible transitions between the different elemental actions.An "x" indicates that the transition could occur in a real-world situation.

Table 2 .
Example of common maintenance subtasks and their respective elemental actions.Angular axes are not shown here for clarity but will usually be comparable to the secondary axes.

Table 3 .
Example of task types data file.