1 Introduction

Robots are ubiquitous, with many areas of use. Each such area is called an application. An important application is robotic assembly manufacturing. Consumer products are often the result of assembly manufacturing. This means that complex sub-parts, such as cameras and circuit boards, are put together in a manner akin to a 3D puzzle. These products are mainly manufactured by hand, in countries with competitive labor cost.

Recently lightweight industrial robots have been introduced, such as YuMi by ABB Ltd [1] and Sawyer by Rethink Robotics [37]. Many of these robots are designed to take over repetitive and tedious tasks from humans, occupying similar floor area and having similar reach. YuMi in particular caters to small-parts assembly manufacturing [1].

However, robot-based assembly manufacturing is too expensive for many products. The throughput of a robot installation determines the value it creates. The cost is largely determined by the time it takes to deploy the robot, and this often costs significantly more than the robot itself. Deploying industrial robots is a time-consuming, iterative process, and requires trained developers. This incurs cost, and production downtime means loss of revenue. At the same time, production series have become smaller, hence fewer produced items will share the cost.

The initial inspiration for our work came from a real-world installation of a YuMi, where a highly skilled programmer and researcher from ABB spent six weeks to program the robot and to determine where to put equipment in the robot workspace. At deployment, unforeseen obstacles were present in the robot workspace, and large parts of this work needed to be redone, which took another week.

So, while it is important that a robot installation obtain good enough throughput, the incurred cost from deployment time has the greatest impact on the net value of the robot installation. One of the main reasons why deploying a robot takes so much time is that several interconnected problems need to be solved, and there is no prescribed order in which to solve them. For the programming, we know what tasks to perform, but we do not know where to perform them. This is decided by the robot programmer and is called the (A) layout design of the robot workspace. This means deciding where in the robot workspace to put trays for picking components and where to put fixtures for assembly. It also means deciding where to put cameras for detailed arm-to-component calibration, and deciding which camera to use for each individual task.

Robot programming itself can be subdivided into the (B) allocation of tasks to arms and (C) task sequencing. For multiple robot arms, the programmer must schedule all tasks, that is (D) decide their start times in a way that (E) optimizes the throughput. For multiple robot arms, the programmer must also consider the problem of coordinating the arms toward (F) collision avoidance during execution. The need to coordinate arm movements greatly adds to the complexity of the problem. The two-dimensional case is NP-hard [48] and is an active field of research [25]. The workspace layout can be designed given the robot program instead, or vice versa; there is no prescribed order in which to solve these two sub-problems. The key insight here is that all sub-problems (A) to (F) are tightly interconnected.

An important aspect of assembly manufacturing is its cyclic nature: a robot installation typically produces the same assembly repeatedly. Thus, it makes most sense to minimize the time span between the start time of any task in one assembly and the start time of the same task in the subsequent assembly. This is called the period of the recurring schedule. Scheduling using the period as the objective function is called (E) cyclic scheduling. Scheduling using the makespan as the objective function, i.e., optimizing the completion time of a single run, typically leads to a longer period. Cyclic scheduling has been handled extensively over the years, both for the general case [16, 18, 24, 39, 46] and for robotics applications [4, 7, 8, 15, 20, 50]. In our work, we minimize the period, allowing one robot arm to finish the last tasks of one assembly, while the other arm starts working on its first tasks of the subsequent assembly.

Each task, such as picking a component, is itself a small robot program. Each such program is currently generated by a human. However, for the level of abstraction of our and related work, each task can be abstracted as its start time, duration, and work location. Similarly, travel between task locations is a small robot program, which, in our and most of the related work, is abstracted as a constraint on the travel time between two consecutive tasks. Such free-space motion can be generated automatically; however, one should note that the computational complexity of single-arm and multi-arm motion planning [23, 26, 44] is exponential in the total number of joints of the arms. As the arms can be seen as a composite robot [12], motion-planning methods that search in this composite space allow us to achieve global properties like completeness and optimality, at the cost of high computational overhead. Efficient sampling-based and quadratic-optimization-based methods [30, 40] have been developed to mitigate the problem of intractability. However, solving the motion-planning problem for every candidate program is not tractable. This makes it necessary to decouple the motion-planning problems for each arm [12] and precompute the motion for every pair of candidate locations. By also partitioning the workspace into areas, their occupation can be scheduled as a part of the overall problem. We note that such precomputed motion might report arm collisions in situations where collisions can be avoided, thus sacrificing some feasible solutions for tractability. We illustrate the components of a robot program as a Venn diagram in Fig. 1.

The artificial intelligence (AI) community has a long history of addressing model-based robot programming, such as for automated ground vehicles using planning [11] and first-order logic [5] in order to achieve (C). AI for robotics is a long-standing topic [36], and robotics is an active part of the AI community [35]. Task planning, including temporal planning, is a field of active research [17], and the sub-field of integrated task and motion planning [14, 19, 22, 27, 32, 33, 45] addresses (C) while also generating the motion plans between tasks. Most work addresses single arms, while a recent CP approach includes a second arm [2], thus addressing (B), (D), and (F), whereas (C) was given. Another recent work [31] includes multiple arms and addresses (B), (C), (D), and (F), while optimizing for makespan instead of period as in (E). That work also presents a CP model, however it reports that the implementation did not work well due to the large number of collision constraints, in the form of blocking constraints [29].

In our work, we remove some of the complexity of the multi-arm motion-planning problem by partitioning the workspace into zones and scheduling their use by the arms. In this way we model zones as unary resources, shared by all robots and tasks. This models the same thing as blocking constraints [29, 31], which invoke a non-overlap constraint for each pair of tasks, conditioned on the robots performing those tasks and their locations. We also present a second method that assign parts of the workspace to arms. A similar intuition is used across other applications, including industrial manufacturing [40], space exploration [49], collaborative manipulation [10, 41], and assembly [43]. All of these papers address multiple problems, including task allocation (B), collision avoidance (F), and motion planning. To the best of our knowledge, no work approaches (B), (C), (D), (E), and (F) in unison, and no work addresses (A).

Fig. 1
figure 1

Installing an assembly robot includes programming the robot as well as designing its workspace layout. Since the production throughput depends on the interaction of the two, we include both in our model. However, we assume that arm motion plans and task implementation are given: we call these motion planning and execution

Full size image

The main contribution of our work is a unified CP-based optimization model spanning all aspects (A) to (F) of the robot deployment process, but not motion planning. The model is given in Section 3 and contains elements of scheduling, routing, and spatial separation. The efficiency of a robot program and layout design will in turn depend on other choices made by the programmer as well as on outside factors. Therefore the installation of an assembly robot will still be iterative work. So we will need to solve many variations of this problem during a robot installation. Hence we need to obtain solutions of a satisfactory quality robustly and quickly.

We here continue the work of two MSc thesis projects [9, 21] supervised by the first author, by addressing a more general problem. We extend our preliminary work [47]: we improve the space complexity of its arm-to-arm collision avoidance method from quadratic to linear and present an alternative to that method; we make fewer assumptions about underlying robot motion; we conduct a much more extensive evaluation; and we model a cyclic robot schedule for continuous production. We present the following contributions:

  • the first optimization-based model integrating (A) workspace layout design, (B) task allocation, (C) task sequencing, (D) task scheduling, (E) cyclic scheduling, and (F) collision avoidance for a dual-arm robot, to the best of our knowledge: the model has features from routing and scheduling, with variable task locations and travel times that depend on the locations and the performing robot arm;

  • two effective methods for avoiding arm-to-arm collisions as part of robot program generation: a generic method relies on partitioning the workspace into zones, which the scheduling part of the model treats as bookable resources; a specialized method is based on a static division of the workspace into two exclusive-use sides;

  • an evaluation showing that the model quickly delivers good solutions: the evaluation is performed on problem instances of realistic size, as well as of sizes that exceed the number of tasks of real-world problem instances, with different granularity of the partitioning of the workspace into bookable zones; we also evaluate the different collision avoidance methods, as well as the impact of redundant constraints on solution times.

The rest of the article is organized as follows. In Section 2, we present the dual-arm robot, the assembly cell, and the assembly robot programming problem. In Section 3, we introduce our CP model capturing this problem, with two methods for avoiding arm-to-arm collisions. In Section 4, we describe the experiments we have conducted. We end the article by concluding on the findings in Section 6. Some notation and data are defined in two appendices: the definitions of the used global constraints and search annotations (see Appendix A); and the data that define a problem instance, which depend on the characteristics of the robot arms, the workspace, and the application (see Appendix B).

2 Problem description

We present the robot assembly application we have captured in our model, and where sub-problems (A) to (F) come in. We outline assembly manufacturing in Section 2.1. We present the YuMi robot in Section 2.2, introduce the concept of fixture in Section 2.3, the specifics of a workspace in Section 2.4, assembly robot programming in Section 2.5, and collision avoidance in Section 2.6.

2.1 Assembly manufacturing

Assembly manufacturing is a type of production where several components are combined into an assembled product. This is very common when producing consumer electronics. Components could be of any kind such as electronic boards, motors, and sensors. Each component is presented on a component tray. Such a tray holds multiple copies of its component type, and only of that component type, as provided by the component manufacturer. The products are assembled by adding one component at a time in a prescribed order. Until all components have been added to the assembly, we call it a sub-assembly. When the assembly is finished, it is placed on an output tray.

Fig. 2
figure 2

ABB Ltd [1]. A video demonstrating the robot can be found at: https://youtu.be/wh3zv5kY0k4. Right: A YuMi robot hand with one gripper and two suction cups

Full size image

2.2 The YuMi robot

For this work, we assume that we have a YuMi robot [1]. The robot, shown on the left in Fig. 2, is built to resemble a human in size and appearance, so we refer to its parts as arm, hand, elbow, and torso. Each hand, shown on the right in Fig. 2, features one gripper tool and two suction cup tools. These tools are designed to carry out the tasks of assembling a product. Two important tasks are picking and placing objects, where an object is a component, an assembly, or a sub-assembly.

A gripper picks an object with its two parallel fingers. A suction cup picks an object by adhering to a flat surface of the object. The robot programmer chooses which tool to use for each object. When holding an object with a gripper, the position and orientation, or pose, of the object are sufficiently well known to place it onto the growing assembly. This is not true for a suction cup; however, the object pose can be determined by processing an image of the object held by the tool. Thus, when using a suction cup, such a calibrating camera task must be performed at the focal point of a camera before placing the object. This is the third type of task of assembly manufacturing.

Each tool can hold one object, independently of the other tools. This means that one robot hand can hold up to three objects simultaneously. This fact is often used by experienced programmers in order to improve throughput.

2.3 Fixtures

In assembly manufacturing, one repeatedly adds one component at a time to the current sub-assembly. Humans would typically use both hands. Similarly, we would like to utilize the two robot arms’ ability to work in parallel. While fixating the sub-assembly and adding components using one robot arm, the other arm is free to work on other tasks. Assigning tasks to arms is sub-problem (B) task allocation.

The fixation is done by using a device to hold the growing sub-assembly. Such a device is called a fixture and is firmly attached at its designated location. After the first component has been mounted on the fixture, we continue adding the other components in the prescribed order. Every placement of a component onto a sub-assembly is followed by applying force to ensure proper incorporation of the component: this is called a press task and is the fourth and last type of task in assembly manufacturing.

On a given fixture, there could be intermediate tasks that are not press tasks, such as peeling off protective tape. There might be components that do not need a press task. Additional tasks might be required between picking and placing for some components, such as blowing an air gun. However, our work is restricted to cases where the order is prescribed, each component type is picked exactly once, and components do not need any tasks other than the outlined standard ones: pick, (camera), place, and press.

By the way assembly products are designed, we can use one fixture to finish an entire assembly, and this is probably how most humans would do it. However, due to the hazard of arm-to-arm collisions, only one robot hand can work at a fixture at any time. Typically, it is possible to divide the assembly into two sub-assemblies, to be assembled at different fixtures. Subsequently, one sub-assembly is then picked up and merged onto the other sub-assembly. This adds three new tasks to perform: picking the first sub-assembly, placing it onto the other sub-assembly, and finally pressing the entire assembly in order to ensure a proper incorporation. When having two robot arms, it is typically more efficient to have two fixtures in order to allow parallel work. More than two fixtures for a dual-arm robot can only incur overhead, because there would be no extra parallelism and every new fixture increases the need to move sub-assemblies. Consequently, in this work, we use two fixtures, and each relevant task is preassigned to one of the two. Note that the location of the fixtures is subject to optimization: sub-problem (A) workspace layout design.

2.4 The assembly cell

In industrial robotics, the robot workspace is the space used by one or more robot arms and the equipment required to assemble a product; if a workspace is used for assembly manufacturing, then it is called an assembly cell.

An example of a real-world assembly cell is shown in Fig. 3. The fixtures and horizontal trays are on the bottom-most horizontal plane, called the table. Each component tray is paired with a camera facing it, for calibration and in order to keep track of component availability. The focal point of each camera is located between the camera and its tray. Additional cameras can be mounted if there is a need to cover additional locations. Typically, there is a single output at a given location. Without loss of generality, we assume it to be on the right-hand side of the robot; it is not shown in the figure.

Fig. 3
figure 3

An assembly cell with an ABB YuMi, two fixtures, five component trays, and five cameras, one per tray

Full size image

Related to sub-problem (A) layout design, we address the decision of where best to put component trays and fixtures, which generally can be located anywhere, as long as they can be reached, even mounted on walls. Typically, the fixtures are located in front of the robot, as in Fig. 3, or possibly on the small horizontal surface in the middle behind the shoulders of the robot arms. Towards optimizing the layout, we let the number of candidate tray locations be greater than the number of component trays to position, and add the horizontal surface on top of the robot as a third candidate fixture location.

2.5 Assembly robot programming

The software controlling a robot with its cameras is called a robot program. It consists of two sequences of tasks, one per arm, and of synchronization between these sequences. Each arm performs a task, moves to the next task, possibly waits for the other arm to move away, before repeating the sequence. Deciding on an order is sub-problem (C) task sequencing. The decision at which time point to perform each task is sub-problem (D) task scheduling. When the robot should output multiple copies of the same assembly product, this is sub-problem (E) cyclic scheduling.

Each task is itself a program, provided by the robot programmer. However, for our purposes, it suffices to know the duration of each task. Similarly, for arm movements, it suffices to know the travel time between any two locations.

2.6 Arm-to-Arm collision avoidance

We must avoid arm-to-arm collisions: this is sub-problem (F). A robot programmer often spends considerable effort to keep robot arms apart. We assume there is a small (implicit) volume around each candidate location, dedicated to the arm working at that location. When using zone resources as collision avoidance method (see below), each such volume corresponds to a zone. However, when using a static division of the workspace as collision avoidance method (see below), leaving the location upon completion of the task is an assumption for this approach to be viable and is part of the task implementation: this is included in the task duration. Collision avoidance is especially important for fixture tasks, since multiple tasks occur at the same location. We now present these two methods, with their different benefits and drawbacks.

2.6.1 Zone-resource collision avoidance

The zone-resource method for collision avoidance does not require any additional assumptions on tasks and travel. The robot programmer, or a collision detection algorithm, defines zones and determines, for any arm movement between two locations, which of these zones are entered and exited. The approach then considers these zones as resources that only one arm can utilize at any given time. In Fig. 4 we show a robot and a cuboid-shaped zone, as in the experiments. The details can be found in Section 3.9.1.

Fig. 4
figure 4

Zone-resource collision avoidance. Left: A YuMi dual-arm robot from ABB Ltd [1], and a zone, depicted as a gray cuboid. Right: The same YuMi robot and zone, seen from above

Full size image

2.6.2 Static collision avoidance

The static method for collision avoidance, described in Section 3.9.2, relies on the following assumptions on tasks and travel:

  • Parallel work: Any task at one fixture, performed by one arm, does not interfere with any task at the other fixture, performed by the other arm.

  • Elbows out: If the left robot hand is located to the left of the right robot hand, then no parts of the arms are colliding.

  • Straight-line travel: Travel between fixtures and other locations does not pass through volumes to the left (right) of that location when using the right (left) arm.

  • Shared fixture locations: Both arms reach all non-fixture locations without blocking any fixture location. An arm blocks a location if their projections onto the horizontal plane overlap.

This method requires the robot programmer to rank the non-fixture locations from left to right. By partitioning those locations into a left part and a right part, a static division of the workspace is achieved. In Fig. 5 we show a robot and a dividing plane, which is not given up front but part of the solution, where every non-fixture location to the left (right) side of this plane can only be visited by the left (right) arm.

Fig. 5
figure 5

Static collision avoidance. A YuMi robot, with arrows indicating which arm the non-fixture locations belong to

Full size image

3 Model

Our CP model captures the problem of continuous dual-arm robot assembly manufacturing. It features variable task locations, thus optimizing component-tray layout, fixture layout, and which camera to use for each camera task. It is a scheduling model integrated with a pickup and delivery vehicle routing model, augmented with constraints to avoid collisions of the arms. The routing part of the model is implemented using a Giant-Tour formulation [13]: while all tasks form a Hamiltonian cycle, this cycle consists of the two arms’ task sequences, connected at the start and end by dummy tasks. This is done by forcing the start dummy task of each arm to be the successor of the other arm’s end dummy task.

Since the problem is the continuous production of assemblies, we repeat the set of tasks perpetually using the same schedule. The objective to be minimized is the completion time of one assembly during continuous production, called the period of the schedule. Therefore the locations of the dummy start and end tasks are constrained in such a way that both arms travel back to their initial locations at the end of the schedule. Thus we obtain the correct time and location for the last task of each arm.

In the following subsections, we introduce the main simplifications we have made (Section 3.1); we introduce notation for constants and sets (Section 3.2); we introduce the CP model’s core decision variables (Section 3.3); we give the details of a Giant Tour formulation that captures task order (Section 3.4); we present constraints on time variables (Section 3.5), for cyclic scheduling (Section 3.6), on tool capacity and load (Section 3.7), on locations (Section 3.8), and for avoiding arm-to-arm collisions (Section 3.9). We then give a dual Giant Tour representation (Section 3.10) and present some redundant constraints that help reduce the search effort (Section 3.11). Finally, we describe our search strategy (Section 3.12). The global constraints and search annotations used in this section are defined in Appendix A.

3.1 Simplifications

In our model, we have made a few noteworthy simplifications. These are design choices aimed at making the problem easier to solve, but which might remove solutions of higher quality:

  • We pre-compute a unique path between every pair of locations, for each arm. Alternative pre-computed paths or dynamic path planning could increase solution quality. This is discussed as future work (Section 5.2).

  • The size and shape of zones are user-defined. In the experiments, the zones are shaped as tall cuboids, which implies that the two arms cannot coexist in the same zone even if they are laterally separated. This means that we are over-conservative with regards to collisions in the experiments.

3.2 Instance data

In the given problem instance, let T be the number of tasks, L the number of locations, W the number of discrete time points, and Z the number of workspace zones in the case of zone-resource collision avoidance. In addition to the actual tasks, the model uses four dummy tasks: \(\text {StartLeftArm} = T-4\) and \(\text {EndLeftArm} = T-2\) are the dummy start and end tasks of the left arm, respectively, while \(\text {StartRightArm} = T-3\) and \(\text {EndRightArm} = T-1\) are the dummy start and end tasks of the right arm, respectively. We model the suction cups of a particular arm as one tool, capable of holding two objects, because there is never any need to distinguish its two suction cups for our purposes. The load of the suction cup or gripper of a particular arm is the number of objects it currently holds. We introduce the following sets of values:

  • \(\mathbb {T} = \{0,\dots ,T-1\}\), the set of tasks

  • \(\mathbb {L} = \{0,\dots ,L-1\}\), the set of locations

  • \(\mathbb {W} = \{0,\dots ,W-1\}\), the set of time points and time spans

  • \(\mathbb {A} = \{\text {LeftArm}, \text {RightArm}\}\), the set of robot arms

  • \(\mathbb {H} = \{\text {Gripper}, \text {SuctionCup}\}\), the set of types of robot hand tools

  • \(\mathbb {B} = \{0,\dots ,Z-1\}\), the set of bookable workspace zones

It is useful to partition the task and location sets:

  • \(\mathbb {T}_{\text {start}} = \{\text {StartLeftArm}, \text {StartRightArm}\}\), the dummy start tasks

  • \(\mathbb {T}_{\text {end}} = \{\text {EndLeftArm}, \text {EndRightArm}\}\), the dummy end tasks

  • \(\mathbb {T}_{\text {Actual}} = \mathbb {T} \setminus \mathbb {T_{\text {start}}} \setminus \mathbb {T_{\text {end}}}\), the actual tasks

  • \(\mathbb {T}_{\text {TrayPick}} \subset \mathbb {T}\), the set of actual tasks that occur at some component tray

  • \(\mathbb {T}_{\text {Camera}} \subset \mathbb {T}\), the set of camera tasks

  • \(\mathbb {T}_{\text {Fixture1}} \subset \mathbb {T}\), the set of actual tasks that occur at fixture 1

  • \(\mathbb {T}_{\text {Fixture2}} \subset \mathbb {T}\), the set of actual tasks that occur at fixture 2

  • \(\mathbb {T}_{\text {Output}} \subset \mathbb {T}\), the singleton set of actual tasks that occur at the output tray

  • \(\mathbb {L}_{\text {TrayLocation}} \subset \mathbb {L}\), the set of candidate component-tray locations

  • \(\mathbb {L}_{\text {CameraLocation}} \subset \mathbb {L}\), the set of candidate focal-point locations for camera tasks

  • \(\mathbb {L}_{\text {FixtureLocation}} \subset \mathbb {L}\), the set of candidate fixture locations

  • \(\mathbb {L}_{\text {OutputLocation}} \subset \mathbb {L}\), the singleton set of the given output-tray location

Additional instance data, mentioned in the following subsections, is compiled in Appendix B. Some straightforward aspects of the model are omitted for brevity (e.g., sequencing constraints), but the complete set is provided in that appendix.

3.3 Core decision variables

Below are the core decision variables. We introduce additional variables for collision avoidance later. For each task \(t \in \mathbb {T}\), we have the decision variables:

  • \(arm[t] \in \mathbb {A}\), the arm performing t

  • \(succ[t] \in \mathbb {T}\), the successor task of t; see Fig. 6

  • \(loc[t] \in \mathbb {L}\), the location of t

  • \(load[{t},{\tau }] \in \{0,1,2\}\), the load of its tool \(\tau \in \mathbb {H}\) when arm[t] arrives at loc[t]

  • \(arrival[t] \in \mathbb {W}\), the time point at which arm[t] is ready to perform t

  • \(wait\_time[t] \in \mathbb {W}\), the waiting time before arm[t] starts performing t

  • \(start[t] \in \mathbb {W}\), the time point at which arm[t] starts performing t

  • \(duration[t] \in \mathbb {W}\), the duration when arm[t] performs t

  • \(end[t] \in \mathbb {W}\), the time point at which arm[t] finishes performing t

  • \(travel\_time[t] \in \mathbb {W}\), the travel time from loc[t] to loc[succ[t]]

  • \(next\_arrival[t] \in \mathbb {W}\), the time point at which arm[t] is ready to perform succ[t]

We introduce \(next\_arrival[t]\) solely as a shorthand for arrival[succ[t]]. We construct three consecutive time intervals, as shown in Fig. 7. The right-open interval [arrival[t], start[t]) captures the robot arm waiting to start performing the task. The interval [start[t], end[t]) captures the time spent on performing the task. The interval \([end[t], next\_arrival[t])\) captures the travel time from the current task to the successor task. A task t is said to be active during the interval \([arrival[t], next\_arrival[t])\). In this way, we model an unbroken chain of time intervals for each arm, thus keeping track of the arm over time. This is used in Section 3.9.1 when preventing arm-to-arm collisions.

The following variables are used to define the cost function, period, which is to be minimized:

  • \(makespan \in \mathbb {W}\), the maximum arrival[t] for \(t \in \mathbb {T_{\text {end}}}\)

  • \(period \in \mathbb {W}\), the time span between the start time of any task of one assembly and the start time of the same task of the subsequent assembly: this equals the minimum arrival[t] for \(t \in \mathbb {T_{\text {end}}}\)

  • \(cycle\_overlap \in \mathbb {W}\), the time span between one arm starting work on an assembly and the other arm starting work on the same assembly: this equals the maximum arrival[t] for \(t \in \mathbb {T_{\text {start}}}\), and also equals the difference between makespan and period.

Figure 8 conveys the intuition of how they relate to each other.

Fig. 6
figure 6

The value of succ[t] equals the successor of task t. Having the dummy tasks as break points allows succ to represent a single Hamiltonian cycle consisting of all tasks, and allows us to encode two task sequences

Full size image

3.4 Giant-tour representation

At the heart of the model is the concept of route. We refer to an arm route as a sequence of tasks performed by that arm, beginning with its dummy start task and finishing with its dummy end task.

Fig. 7
figure 7

The time variables of each task t. Note that there are neither gaps between the intervals for a task, nor gaps between tasks

Full size image
Fig. 8
figure 8

Illustration (best viewed in color) of a dual-arm schedule of four actual tasks \(t_1\) to \(t_4\), with the arms shown in the two top rows. Grey hashing denotes cycle overlap. Green vertical bars denote \(arrival[t] \text {\ for\ } t \in \mathbb {T_{\text {start}}}\); red vertical bars denote \(arrival[t] \text {\ for\ } t \in \mathbb {T_{\text {end}}}\); note that these four dummy tasks have zero duration. The four bottom rows show variables related to the cost function. Due to the possibility of imposing a waiting time upon arrival to a task, we can enforce several equalities involving makespan, \(cycle\_overlap\), and period

Full size image

We have chosen to represent a Giant-Tour formulation [13] as a successor list \(t, succ[t], succ[succ[t]], \dots , t\), for some first task t. The list starts with the dummy StartLeftArm, followed by the left arm’s route, followed by the dummy EndLeftArm, then the dummy StartRightArm, followed by the right arm’s route, followed by the dummy EndRightArm, and finally by StartLeftArm: see Fig. 6. We constrain each arm’s dummy tasks to take the location of its first task. In this way, each robot arm returns to its original location. Thus the cyclic schedule continues. Of course, the same arm must be used within each arm route. We have the following constraints, which are relevant for sub-problems (A) layout design, (B) task allocation, and (C) task sequencing:

$$\begin{aligned} \left. \begin{aligned} {\textbf {circuit}} ([succ[t] \mid t \in \mathbb {T}]), \\ succ[{\text {EndLeftArm}}] = \text {StartRightArm}, \\ succ[{\text {EndRightArm}}] = \text {StartLeftArm}, \\ arm[t] = arm[{succ[t]}] \quad \forall t \in \mathbb {T} \setminus \mathbb {T_{\text {end}}}, \\ arm[{\text {StartLeftArm}}] = arm[{\text {EndLeftArm}}] = \text {LeftArm},\\ arm[{\text {StartRightArm}}] = arm[{\text {EndRightArm}}] = \text {RightArm},\\ loc[{succ[{\text {StartLeftArm}}}]] = loc[{\text {StartLeftArm}}] = loc[{\text {EndLeftArm}}], \\ loc[{succ[{\text {StartRightArm}}}]] = loc[{\text {StartRightArm}}] = loc[{\text {EndRightArm}}] \end{aligned}\right. \end{aligned}$$
(1)

3.5 Time constraints

We now come to the constraints that are relevant for sub-problem (D) task scheduling. As the location of each task is a variable, the travel times are defined between locations, not tasks. All travel times are dependent on the arm and direction of travel (such as upward vs downward). The matrices \(W_\text {D}\) of task durations and \(W_\text {L}\) of travel times are provided by simulation, other algorithms, manual estimation, or real-world measurements. We define \(travel\_time[t]\) and duration[t] of each task t as follows:

$$\begin{aligned} \begin{array}{c} \left. \begin{aligned} travel\_time[t] = W_\text {L}(arm[t], loc[t], loc[{succ[t]}]) \quad \forall t \in \mathbb {T}_{\text {Actual}}, \\ duration[t] = W_\text {D}(arm[t], t) \quad \forall t \in \mathbb {T}_{\text {Actual}} \\ \end{aligned}\right. \\ \ \\ \text {where} \\ \ \\ \left. \begin{aligned} W_{\text {L}}(a,\ell ,\ell ') \in \mathbb {Z} = \text {travel time of arm }a\text { from location }\ell \text { to location }\ell ', \\ \text {or }-1\text { if infeasible}, \\ W_\text {D}(a,t) \in \mathbb {N} = \text {duration of task }t\text { if assigned to arm }a \end{aligned}\right. \\ \end{array} \end{aligned}$$
(2)

The following constraint relates the time variables of each actual task (see Fig. 7) and connects them with the successor variables:

$$\begin{aligned} \left. \begin{aligned} arrival[t] + wait\_time[t] = start[t] \quad \forall t \in \mathbb {T}_{\text {Actual}}, \\ start[t] + duration[t] = end[t] \quad \forall t \in \mathbb {T}_{\text {Actual}}, \\ end[t] + travel\_time[t] = next\_arrival[t] \quad \forall t \in \mathbb {T}_{\text {Actual}}, \\ next\_arrival[t] = arrival[{succ[t]}] \quad \forall t \in \mathbb {T}_{\text {Actual}} \cup \mathbb {T}_{\text {start}} \\ \end{aligned}\right. \end{aligned}$$
(3)

The following constraint defines the time variables of each dummy task, and requires at least one arm to start at time zero:

$$\begin{aligned} \left. \begin{aligned} arrival[t] = start[t] = end[t] = next\_arrival[t] \quad \forall t \in \mathbb {T}_{\text {start}} \cup \mathbb {T}_{\text {end}}, \\ wait\_time[t] = duration[t] = travel\_time[t] = 0 \quad \forall t \in \mathbb {T}_{\text {start}} \cup \mathbb {T}_{\text {end}}, \\ 0 = \min ([arrival[t] \mid t \in \mathbb {T}_{\text {start}} ]) \end{aligned}\right. \end{aligned}$$
(4)

Precedences at fixtures are encoded as a set of task sequences \(\mathbb {O}_{\text {F}}\) (see Appendix B). Similarly, precedences among tasks using the gripper (the suction cup) are encoded as a set of task sequences \(\mathbb {O}_{\text {G}}\) (\(\mathbb {O}_{\text {S}}\)). This is typically a pick task and a place task, and possibly an intermediate task such as a camera task. They are enforced by the following constraints:

$$\begin{aligned} \left. \begin{aligned} end[t] \le start[{t'}] \\ \forall \textbf{t} \in \mathbb {O}_{\text {F}} \quad \forall t,t' \in \textbf{t} \text { where }t\text { precedes }t' \end{aligned}\right. \end{aligned}$$
(5)
$$\begin{aligned} \left. \begin{aligned} (next\_arrival[t] \le arrival[{t'}]) \wedge (arm[t] = arm[{t'}]) \\ \forall \textbf{t} \in \mathbb {O}_{\text {G}} \cup \mathbb {O}_{\text {S}} \quad \forall t,t' \in \textbf{t} \text { where }t\text { precedes }t' \end{aligned}\right. \end{aligned}$$
(6)

Note that the same arm for t and \(t'\) is required in Constraint (6), which is why Constraint (5) is on the performing phase of the tasks whereas Constraint (6) is on the whole time span of activity of the tasks; see Fig. 7. As mentioned in Section 2.6, arms leave locations as part of their task duties. This means that at the end time of any fixture task t, end[t], the performing robot arm has left the fixture zone. This is of course a bit on the conservative side for cases when the same arm performs the subsequent task on that fixture. Constraint (5) ensures that arms do not collide on a fixture; see the continuation in Section 3.9.2.

3.6 Cyclic scheduling

Since our target is continuous production, we want to minimize not the finish time for a single assembly (makespan), but the cycle time (period): this is sub-problem (E) — please refer to Section 2.5. These constraints define the period, which is the cost function, but are also required for imposing zone-resource collision avoidance across cycles. See Section 3.9.1 for details. The cyclic scheduling constraint is as follows (see Fig. 8):

$$\begin{aligned} \left. \begin{aligned} makespan = \max ( [arrival[t] \mid t \in \mathbb {T}_{\text {end}}] ), \\ cycle\_overlap = \max ([arrival[t] \mid t \in \mathbb {T}_{\text {start}} ]), \\ period = \min ([arrival[t] \mid t \in \mathbb {T}_{\text {end}} ]), \\ period = makespan - cycle\_overlap \end{aligned}\right. \end{aligned}$$
(7)

To guarantee a cyclic schedule with the same period for both arms, we must ensure that \(cycle\_overlap\) equals not only the time span between the two arms starting work on an assembly, but also the time span between the two arms ending work on an assembly, by adding the constraint:

$$\begin{aligned} \left. \begin{aligned} arrival[{\text {StartLeftArm}}] - arrival[{\text {StartRightArm}}] \\ = arrival[{\text {EndLeftArm}}] - arrival[{\text {EndRightArm}}] \end{aligned}\right. \end{aligned}$$
(8)

Let \(first(\textbf{t})\) (\(last(\textbf{t})\)) denote the first (last) item of a sequence \(\textbf{t}\). Since the two arms can work on the current and subsequent assembly simultaneously we must ensure that each fixture is emptied before it is being used for the subsequent assembly. This is achieved by adding the following constraint, again using instance data \(\mathbb {O}_\text {F}\) (see Appendix B):

$$\begin{aligned} \left. \begin{aligned} end[{last(\textbf{t})}] \le start[{first(\textbf{t})}] + period \quad \forall \textbf{t} \in \mathbb {O}_\text {F} \end{aligned}\right. \end{aligned}$$
(9)

3.7 Tool capacity constraints

Relevant for sub-problem (C) task sequencing, a feasible task sequence must not exceed tool capacities. Also, the gripper must be empty for press tasks. Please refer to Section 2.2 for a description of the YuMi robot and its tools. To this end, we have variables \(load[{t},{\tau }]\) for each task t and tool \(\tau \in \mathbb {H}\). They are linked in the constraint below. Also, the press tasks use the gripper for the pressing; they require the gripper to be empty on arrival:

$$\begin{aligned} \begin{array}{c} \left. \begin{aligned} load[{t},{\tau }] = 0 \quad \begin{array}{l}\forall t \in \mathbb {T}_{\text {start}} \cup \mathbb {T}_{\text {end}},~ \forall \tau \in \mathbb {H} \\ \end{array}, \\ load[{succ[t]},{\tau }] = load[{t},{\tau }] + \Delta (t,\tau ) \quad \begin{array}{l}\forall t \in \mathbb {T},~ \forall \tau \in \mathbb {H} \\ \end{array}, \\ load[{t},{\tau }] \le Cap (t,\tau ) \quad \begin{array}{l}\forall t \in \mathbb {T},~ \forall \tau \in \mathbb {H} \\ \end{array} \\ \end{aligned}\right. \\ \ \\ \text {where} \\ \ \\ \Delta (t,\tau ) = \left\{ \begin{array}{ll} +1, &{} \text {if task }t\text { uses tool }\tau \text { to pick something,} \\ -1, &{} \text {if task }t\text { uses tool }\tau \text { to place something,} \\ 0, &{} \text {otherwise} \\ \end{array}\right. \\ \ \\ Cap (t,\tau ) = \left\{ \begin{array}{ll} 0, &{} \text {if task }t\text { requires tool }\tau \text { to be empty,} \\ 1, &{} \text {if tool }\tau \text { is the gripper,} \\ 2, &{} \text {if tool }\tau \text { is the suction cup} \\ \end{array}\right. \\ \end{array} \end{aligned}$$
(10)

3.8 Location constraints

Variable locations are a means to optimize the layout for best performance: this is sub-problem (A) layout design. Whereas a typical routing problem has fixed locations, we can perform tasks on the same arm close to each other in order to minimize travel time, and separate apart tasks performed on different arms in order to avoid collisions.

We have several types of tasks, each with a dedicated set of candidate locations. Refer to Sections 2.3 and 2.4 for a description of fixtures and the assembly cell, respectively. No location can accommodate more than one task at a time. This assumption is not strictly necessary for all assembly problems, but is most often the case. We have the following constraints, which constrain the domains of the location variables:

$$\begin{aligned} \left. \begin{aligned} loc[t] \in \mathbb {L}_{\text {TrayLocation}} \quad \forall t \in \mathbb {T}_{\text {TrayPick}}, \\ loc[t] \in \mathbb {L}_{\text {CameraLocation}} \quad \forall t \in \mathbb {T}_{\text {Camera}}, \\ loc[t] \in \mathbb {L}_{\text {FixtureLocation}} \quad \forall t \in \mathbb {T}_{\text {Fixture1}} \cup \mathbb {T}_{\text {Fixture2}}, \\ loc[t] \in \mathbb {L}_{\text {OutputLocation}} \quad \forall t \in \mathbb {T}_{\text {Output}} \\ \end{aligned}\right. \end{aligned}$$
(11)

We have other location constraints on two types of tasks. First, recall that we assume that each component is picked by exactly one task, and that there is only room for one component tray at each candidate tray location:

$$\begin{aligned} \left. \begin{aligned} {\textbf {all\_different}} ([loc[t] \mid t \in \mathbb {T}_{\text {TrayPick}} ]) \end{aligned}\right. \end{aligned}$$
(12)

Second, each fixture task must occur on its dedicated fixture, while the two fixtures themselves must be at different locations:

$$\begin{aligned} \left. \begin{aligned} loc[t] \ne loc[{t'}] \quad \forall t \in \mathbb {T}_{\text {Fixture1}},~ \forall t' \in \mathbb {T}_{\text {Fixture2}}, \\ {\textbf {all\_equal}}([loc[t] \mid t \in \mathbb {T}_{\text {Fixture1}}]), \\ {\textbf {all\_equal}}([loc[t] \mid t \in \mathbb {T}_{\text {Fixture2}}]) \\ \end{aligned}\right. \end{aligned}$$
(13)

This completes our description of the core model, i.e., Constraints (1–13), which covers all the essential constraints, except collision avoidance, which is the subject of the next subsection.

3.9 Arm-to-Arm collision avoidance constraints

We present two methods for sub-problem (F) collision avoidance. This sub-problem is described in Section 2.6. The first method is new compared to our previous paper [47]: it is a zone-resource method, where each arm books a zone each time it occupies it. The second method enforces a static division of the workspace, and improves on its version in [47] by needing a linear instead of quadratic number of constraints.

3.9.1 Zone-resource collision avoidance

The idea is to avoid collisions by constraining the arms not to be at the same place at the same time. To model the spatial whereabouts of the arms, we introduce a set \(\mathbb {B}\) of disjoint zones, where a zone is an instance-specific volume of space in the robot workspace. We treat every zone as a non-consumable unary resource. This abstraction prevents collisions in any given zone; thus we require the zones to partition the reachable workspace. This abstraction also requires us to keep track of the robot arms at all times. We can do that because the tasks of each arm form an unbroken chain of the three kinds of time intervals [a, b) in Fig. 7.

In order for a task t to use a zone z during one of those time intervals, it must book z, which prevents z from being used by any other task during that time interval. For that purpose, we introduce the following variables:

  • \(waitZB[t,z] \in \{0,1\}\), whether task t books zone z when arm[t] is waiting at loc[t]

  • \(performZB[t,z] \in \{0,1\}\), whether task t books zone z when arm[t] is performing at loc[t]

  • \(travelZB \in \{0,1\}\), whether task t books zone z when arm[t] is traveling from loc[t] to loc[succ[t]]

These variables are constrained as follows, using the zone data defined in Appendix B:

$$\begin{aligned} \left. \begin{aligned} waitZB[t,z] = ZB_{\text {Wait}}(arm[t], loc[t], z), \\ performZB[t,z] = ZB_{\text {Perform}}(arm[t], loc[t], z), \\ travelZB[t,z] = ZB_{\text {Travel}}(arm[t], loc[t], loc[{succ[t]}], z) \forall t \in \mathbb {T}_{\text {Actual}},~ \forall z \in \mathbb {B} \end{aligned}\right. \end{aligned}$$
(14)

In order to prevent collisions, imagine for zone z the set of time intervals during which some task books z. If these sets are non-overlapping for all zones, then no collision can occur. With cyclic work schedules, we also need to take into account collisions across cycles. Thus, the sets of relevant time intervals are formed by taking all time intervals [a, b) as well as \([a+period,b+period)\) for all active tasks. The non-overlapping interval view is encoded by constraints (15) and (16) below. Alternatively, these intervals can be viewed as rectangles of length \(b-a\) and height 1, placed at coordinates (a, z), and constrained not to overlap. Then one global constraint suffices; see constraint (17) below. Let \(A[\varOmega ]\) be a shorthand for the slice \([A[x] \mid x \in \varOmega ]\) of array A for set \(\varOmega \), and let \(++\) denote array concatenation. We define:

  • \(\textbf{s} = arrival[{\mathbb {T}_{\text {Actual}}}] ++ start[{\mathbb {T}_{\text {Actual}}}] ++ end[{\mathbb {T}_{\text {Actual}}}]\)

  • \(\mathbf {s'} = [x + period \mid x \in \textbf{s}]\)

  • \(\textbf{d} = wait\_time[{\mathbb {T}_{\text {Actual}}}] ++ duration[{\mathbb {T}_{\text {Actual}}}] ++ travel\_time[{\mathbb {T}_{\text {Actual}}}]\)

  • \({\textbf{r}}[z] = {waitZB}[{\mathbb {T}_{\text {Actual}}},{z}] ++ performZB[{\mathbb {T}_{\text {Actual}}},{z}] ++ travelZB[{\mathbb {T}_{\text {Actual}}},{z}]\)

Defining \(\textbf{0}\) as the array of appropriate length containing all zeros, we give three alternative encodings of the zone booking constraint (see Appendix A.1):

$$\begin{aligned} \left. \begin{aligned} {\textbf {cumulative}} ( \textbf{s} ++ \mathbf {s'}, \textbf{d} ++ \textbf{d}, {\textbf{r}[z]} ++ {\textbf{r}[z]}, 1) \quad \forall z \in \mathbb {B} \end{aligned}\right. \end{aligned}$$
(15)
$$\begin{aligned} \left. \begin{aligned} {\textbf {diffn}} (\textbf{s} ++ \mathbf {s'}, \textbf{0}, \textbf{d} ++ \textbf{d}, {\textbf{r}[z]} ++ {\textbf{r}[z]} ) \quad \forall z \in \mathbb {B} \end{aligned}\right. \end{aligned}$$
(16)
$$\begin{aligned} \left. \begin{aligned} {\textbf {diffn}} ( [s \mid s \in \textbf{s} ++ \mathbf {s'} ,~ z \in \mathbb {B}],\\ [z \mid s \in \textbf{s} ++ \mathbf {s'} ,~ z \in \mathbb {B}],\\ [d \mid d \in \textbf{d} ++ \textbf{d } ,~ z \in \mathbb {B}],\\ [r \mid r \in {\textbf{r}[z]} ++ {\textbf{r}[z]},~ z \in \mathbb {B}]) \end{aligned}\right. \end{aligned}$$
(17)

These alternatives will be raced against each other in the evaluation.

3.9.2 Static collision avoidance

The idea is to allow access to both fixtures by both arms, but to allow access to non-fixture locations by one arm only, to be decided for each such location, as described below. Collisions at fixture locations are already avoided by Constraint (5). This approach may imply that solutions are sub-optimal compared to the previous collision avoidance method, but will be shown in Section 4.2 to be much faster.

Each non-fixture location is pre-assigned a relative position on the left to right dimension of the workspace, as encoded in the function \(P_\text {L}\), defined in Appendix B. We introduce a decision variable \(p_\text {mid}\) to define the rightmost non-fixture location of the left arm. All tasks performed by the left arm are required to be performed at or to the left of \(p_\text {mid}\). Similarly, all tasks performed by the right arm need to be performed strictly to the right of \(p_\text {mid}\). Also, while arms perform at, or travel to, non-fixture locations, they may pass fixture locations. To avoid such collision hazards, we must constrain how far right the left arm can move, and vice versa, using the quantities \(P_{\text {L}_{\min }}\) and \(P_{\text {L}_{\max }}\):

$$\begin{aligned} \left. \begin{aligned} p_\text {mid} \in \mathbb {N}, \\ (arm[t] = \text {LeftArm}) \iff (P_\text {L}(arm[t], loc[t]) \le p_\text {mid}), \\ (arm[t] = \text {LeftArm}) \implies (P_\text {L}(\text {LeftArm}, loc[t]) \le P_{\text {L}_{\max }}(\text {LeftArm})) , \\ (arm[t] = \text {RightArm}) \implies (P_\text {L}(\text {RightArm}, loc[t]) \ge P_{\text {L}_{\min }}(\text {RightArm})) \\ \forall t \in \mathbb {T} \end{aligned}\right. \end{aligned}$$
(18)

Thanks to an anonymous reviewer of our earlier paper [47], we need only a linear number of constraints on \(p_\text {mid}\), instead of the quadratic number of constraints in that paper.

3.10 Dual giant-tour representation

For improved search and redundant constraints (see Section 3.11), we add a dual representation of the Giant Tour. We define the value set and variables:

  • \(\mathbb {P} = \{0,\dots ,\text {T}-1\}\), the set of positions in the Giant Tour

  • \(task[{p}] \in \mathbb {T}\), the task at position \(p \in \mathbb {P}\) in the Giant Tour

Even though \(\mathbb {P}\) contains the same integers as \(\mathbb {T}\), they enumerate different things. We arbitrarily choose the dummy start task of the left arm to be the task at the first position of the Giant Tour; thus the dummy end task of the right arm is the task at its last position, by line 3 of Constraint (1). To link succ and task, we have:

$$\begin{aligned} \left. \begin{aligned} task[{p+1}] = succ[{task[{p}]}] \quad \forall p \in \mathbb {P} \setminus \{ T-1 \}, \\ task[{0}] = \text {StartLeftArm}, \\ task[{T-1}] = \text {EndRightArm} \\ \end{aligned}\right. \end{aligned}$$
(19)

3.11 Redundant constraints

We can significantly improve the performance by introducing a number of redundant constraints, that is constraints that are implied by constraints in the core model and that do not remove solutions. We now describe these redundant constraints, which use task precedence data, defined in Appendix B.

3.11.1 Order-enforcing vonstraints

The following constraint is redundant with Constraint (5). It enforces a pairwise order on the Grand Tour for fixture tasks that are performed by the same arm. If the two tasks are performed by different arms, then their order on the Grand Tour cannot be enforced, and the constraint is effectively disabled, relying on Constraint (5) for the fixture schedule. The constrained term \(\epsilon (i,j)\) implements this mechanism:

$$\begin{aligned} \left. \begin{aligned} {\textbf {value\_precede}} ({\textbf{t}}[{i}],{\textbf{t}}[{j}], [\epsilon (i,j)] ++ task)), \\ \text {where\ }\epsilon (i,j) = (arm[{\textbf{t}}[{i}]] \ne arm[{{\textbf{t}}[{j}]}]) \cdot {\textbf{t}}[{i}] \\ \forall \textbf{t} \in \mathbb {O}_\text {F},~ \forall i,j \in \{ 1, \dots , |\textbf{t}| \} \text { where } i < j \end{aligned}\right. \end{aligned}$$
(20)

The following constraint is redundant with Constraint (6). It enforces a Grand-Tour order on pick-(camera-)place sequences:

$$\begin{aligned} \left. \begin{aligned} {\textbf {value\_precede\_chain}} (\textbf{t}, task) \quad \forall \textbf{t} \in \mathbb {O}_\text {G} \cup \mathbb {O}_\text {S} \\ \end{aligned}\right. \end{aligned}$$
(21)

3.11.2 Regular expressions

We constrain the array task to be recognized by some regular expressions, utilizing the regular constraint. The resulting constraints are redundant with Constraints (5, 6, 10).

Like in Constraint (20), we use \(\mathbb {O}_\text {F}\) to post constraints that are redundant with Constraint (5): they enforce an order on the tasks at the fixtures. Similarly, like in Constraint (21), we use \(\mathbb {O}_{\text {G}}\) and \(\mathbb {O}_{\text {S}}\) to post constraints that are redundant with Constraint (6): they enforce an order on pick-(camera-)place sequences.

We also post constraints that are redundant with Constraints (10): they enforce that tool capacities are never exceeded. Each robot hand has two suction cups and one gripper, which results in different regular expressions for the two tool types. Also, recall that press tasks require the gripper to be empty on arrival.

To sum up, for various regular expressions \(\sigma \), we have redundant constraints of the form:

$$\begin{aligned} {\textbf {regular}}(task,\sigma ) \end{aligned}$$
(22)

For the sake of brevity, we do not present the regular expressions themselves.

3.11.3 Cumulative constraints

The following constraints are redundant with Constraint (3) and provide a global view of the time variables, aimed at improving propagation:

$$\begin{aligned} \left. \begin{aligned} {\textbf {cumulative}} ( \\ [arrival[t] \mid t \in \mathbb {T_{\text {Actual}}} ] , \\ [wait\_time[t] + duration[t] + travel\_time[t] \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ [1 \mid t \in \mathbb {T_{\text {Actual}}} ] , \\ 2), \\ {\textbf {cumulative}} ( \\ [arrival[t] \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ [wait\_time[t] + duration[t] + travel\_time[t] \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ [arm[t] = \text {LeftArm} \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ 1) \\ {\textbf {cumulative}} ( \\ [arrival[t] \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ [wait\_time[t] + duration[t] + travel\_time[t] \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ [arm[t] = \text {RightArm} \mid t \in \mathbb {T}_{\text {Actual}} ] , \\ 1) \\ \end{aligned}\right. \end{aligned}$$
(23)

These constraints express that, at any time point, (i) at most two tasks can be active, (ii) at most one task can be active for the left arm, and (iii) at most one task can be active for the right arm.

3.12 Search strategy

We perform the following branching decisions in order. The annotations are defined in Appendix A.2:

  1. 1.

    dom_w_deg(arm), assigning each variable a random value from its domain.

  2. 2.

    dom_w_deg(loc), assigning each variable its smallest domain value first.

  3. 3.

    input_order(task), assigning each variable its smallest domain value first.

  4. 4.

    smallest(arrival), bisecting the variable domains and excluding the upper half first.

The search is restarted from scratch according to the schedule given by \({\textbf {restart\_luby}}(100)\).

4 Evaluation

We have implemented our model of Section 3 in the high-level modeling language MiniZinc [34]. We now evaluate it by using a set of problem instances ranging from realistic to large.Footnote 1 As the model has several alternative constraints ensuring collision-free schedules, we compare them. We also evaluate the benefit that the redundant constraints bring.

All instances are solved using three threads, using the Gecode 6.1.1 Flat-Zinc solver with the default MiniZinc to FlatZinc compilation of Mini-Zinc 2.4.2, on an Intel Core i7-4600 2.1 GHz CPU. We use a timeout of 2 minutes and each instance is run 5 times, each with a unique random seed. Unless noted otherwise, we report the median performance of those 5 runs.

We first describe the problem instances (Section 4.1) for which we then conduct experiments over the model variants (Section 4.2).

4.1 Problem instances

We describe the generation of the problem data related to the assembly instance (Section 4.1.1) and workspace instance (Section 4.1.2).

4.1.1 Assembly instances

The typical assembly problem involves 3 to 5 components. To evaluate the model, we consider 4 to 10 components. For each of these 7 numbers of components, we generate 4 assembly instances: one where all components are handled with the gripper, one where all components are handled with suction cups, and two where half of the components are handled with suction cups and the other half with the gripper. Half the components are to be assembled at each fixture. The finished sub-assembly of one fixture is picked and placed on top of the sub-assembly of the other fixture; no further sub-assemblies are picked and placed. We include a press task after each place task. Also, we include a camera task after each suction-cup pick and before the corresponding placing of the object. Moving a sub-assembly from one fixture to the next is always done using the gripper. The duration for each task and arm is determined by using a template value for the task type, estimated by experienced robot programmers and randomly perturbed by up to \(\pm 25\%\). This results in \(7 \cdot 4 = 28\) assembly instances.

4.1.2 Workspace instances

For fixtures, we have three candidate locations (see Fig. 9): two directly in front of the robot torso on the table, and a third at the center of the flat surface on top of the robot.

Fig. 9
figure 9

A YuMi robot with the outer two rings indicating the reach of each arm. The flat area on top of the robot serves as a candidate location for fixtures

Full size image

The trays are, in the real world, either at a small fixed number of locations in a standardized assembly cell, or freely positioned by the robot programmer within the reach of the robot arms. A grid is a proxy for both scenarios, as sparse grid points emulate the first scenario and dense grid points emulate the latter.

We distribute candidate tray locations within a grid in a horizontal x, y-plane. Trays are typically quite large and there is only room for a few of them. We consider workspaces with a realistic number of candidate tray locations, and also some workspaces with a larger number of candidate tray locations, in order to demonstrate that the model can handle this with reasonable performance. To do this, we generate workspace layouts with candidate tray locations on an \(n \times m\) grid, with the YuMi robot along one edge of the grid. For our experiments, we consider the use of a \(3 \times 3\)-grid, a \(3 \times 4\)-grid, a \(5 \times 5\)-grid, and a \(7 \times 7\)-grid. This results in 4 workspace instances, where each grid cell is a candidate tray location, thereby forming \(\mathbb {L}_{\text {TrayLocation}}\).

By default, a tray is situated at table level. However, if a grid cell is out of reach at table level for both arms, then an elevated tray is considered at that grid cell, emulating a slanted tray as seen in Fig. 3. We then apply a data filtering step where a candidate tray location is removed from \(\mathbb {L}_{\text {TrayLocation}}\) if it overlaps with a candidate fixture location or is out of reach of both robot arms even if elevated. This results in workspaces with 8, 12, 21, and 41 candidate tray locations respectively. This means that our two assembly instances with 9 or 10 components are trivially unsatisfiable for the smallest workspace: the resulting problem instances are removed when running the experiments.

Candidate camera locations are generated similarly to the candidate tray locations: we introduce one per grid cell, followed by a data filtering step. The only difference is that camera focal points are slightly above the candidate tray locations.

A collision-avoiding path planner is used to generate paths. We implemented a non-optimal path planner that ensures that the elbows point outward. It computes a path and traversed zones in around 0.02 seconds. This results in a computation time per arm of 4 s, 9 s, 18 s, and 1 min respectively for the four workspaces.

Travel times are in turn approximated using the distance for the robot hand to travel and arm-speed estimates, obtained by simulation.

For zone-resource collision avoidance (see Section 2.6.1), we introduce one zone per grid cell. Each zone is a polyhedron that extends infinitely in the vertical direction, and whose projection onto the horizontal plane equals the grid cell. We compute the zones that are booked when an arm is waiting at a location, so as to define the matrix \(ZB_{\text {Wait}}\) (see Appendix B). Similarly, we compute the zones that are booked when an arm is performing a task at a location or traveling between locations, so as to define the matrices \(ZB_{\text {Perform}}\) and \(ZB_{\text {Travel}}\) (see Appendix B).

For static collision avoidance (see Section 2.6.2), we compare every non-fixture location with every other such location in order to evaluate whether the first location is to the left or right of the second location, or neither, We also check for the left arm how far right it can travel without blocking any fixture location, and vice-versa for the right arm: this defines the values \(P_{\text {L}_{\min }}\) and \(P_{\text {L}_{\max }}\) (see Appendix B).

Recall that the workspace with a \(3 \times 3\)-grid results in 8 candidate tray locations after the data filtering above: since 2 of them cannot be reached without blocking some fixture, only 6 candidate tray locations are available when using static collision avoidance. This means that our two assembly instances with 7 or 8 components are then also unsatisfiable for this workspace. Instead of removing the resulting problem instances, we consider them as failures when analyzing the results, because we are interested in how much the solution quality is impaired by using static instead of zone-resource collision avoidance.

This methodology results in problem instances with \(\mathbb {T}\) comprising 17 to 45 tasks, \(\mathbb {L}\) comprising 20 to 86 locations, and \(\mathbb {B}\) comprising 10 to 45 zones. Using 10 discrete time steps per second and a trivial upper bound, \(\mathbb {W}\) comprises 2,500 to 7,500 discrete time steps. The upper bound of W is calculated by assuming that each arm performs all tasks it can perform, and uses the longest path possible from every task to the next. The longest possible path from a task is the longest path from any reachable location of said task to any other task location. Finally, we use the maximum time for any arm as the value W.Footnote 2

4.2 Experiments

Using the 28 assembly instances and 4 workspace instances described above, we get a test suite of \(28 \cdot 4 = 112\) problem instances, of which \(2 \cdot 4 = 8\) are trivially unsatisfiable due to too few candidate tray locations, leaving us with 104 problem instances.

In Table 1, we list the model variants that we evaluate: they differ in the collision-avoidance method to use, and which redundant constraints to include.

We summarize model quality by using cactus plots [3], which are best viewed in color. The y axis shows the normalized period. We normalize with respect to the best period found for that problem instance by any model variant, making 1.0 the smallest possible period. In this way, we put all problem instances on the same scale. The x axis indicates the number of instances having a period less than or equal to the threshold of the y axis. For example, a curve crossing \(x = 10\), \(y = 1.3\) means that the corresponding model yields solutions with at most 1.3 times the period of the best approach for 10 of the problem instances. Note that for plots showing median solution cost, we still normalize with respect to the best period found.

Table 1 Model variants and the constraints they include
Full size table

Thus: the lower a curve, the better. A line y = 1.0 for all x is an oracle.

Evaluating collision-avoidance variants

The static collision-avoidance method removes another 2 candidate tray locations for the smallest grid since they cannot be reached without blocking some fixtures. Thus another \(2 \cdot 4 = 8\) otherwise satisfiable assembly-workspace combinations become unsatisfiable, leaving us with \(104 - 8 = 96\) satisfiable problem instances. The order-enforcing Constraints (20, 21) are included, because otherwise too many problem instances would time out. As can be seen in Fig. 10, a static division of the workspace yields, as expected, the best performance of the problem instances it finds solutions for. The three alternatives of the zone-resource method have very similar performance, but using cumulative (see Constraints (15)) solves all but two problem instances, while the other two fail on three problem instances each, which corresponds to less than \(2\%\) and \(3\%\) of the problem instances respectively

Fig. 10
figure 10

Evaluating collision-avoidance model variants

Full size image
Fig. 11
figure 11

Evaluating redundant constraints using static collision avoidance

Full size image

Evaluating redundant constraints with static collision avoidance

It is evident from Fig. 11 that the redundant constraints enforcing an order on the sequence of tasks give the best results, in terms of both quality and the number of instances solved. However, adding redundant cumulative constraints on the time variables (see Constraints (23)) worsens the performance compared to the core model and we therefore recommend not to add them.

Evaluating redundant constraints with zone-resource collision avoidance

Figure 12 paints a picture similar to Fig. 11, with the same ranking of the model variants. However, the number of solved instances differs more between the better and the worse model variants.

From the two latter experiments, it is clear that extending the core model with the redundant Constraints (20, 21) gives a significant performance boost.

Fig. 12
figure 12

Evaluating redundant constraints using zone-resource collision avoidance

Full size image

Robustness

To evaluate robustness, we present in Table 2 the rate of success of each model variant, i.e., the percentage of runs that led to a solution. We find that ImpVC-ZoneCumul has the highest rate of success. It is worth noting that the only instances that ImpVC-Static and ImpReg-Static are unable to solve are the ones that are unsatisfiable due to the more conservative, static collision avoidance.

Longer timeouts

We ran the same experiments with a 4-minute timeout. However, an improvement to solution quality was often not achieved and was rarely more than \(0.1\%\). We also conducted experiments with a 30-minute timeout by using the best model variant for each collision-avoidance method, namely ImpVC-ZoneCumul and ImpVC-Static. The gains were greater, but not proportional to the additional effort. The convergence of the normalized median cost for the respective model variant is illustrated in Figs. 13 and 14. The plots show the results aggregated by number of components. For example, in Fig. 13, the curve “7 components” crosses the coordinates (1, 1.3), which means that after one minute, the mean normalized cost of the instances with 7 components is 1.3. For all instances, we assume a normalized cost of 3 initially, before any solution has been found. For both models, we observe good convergence toward the best solution. The convergence is faster for ImpVC-Static.

Table 2 The percentage of all runs that returned solutions within 2 minutes of search. ImpVC-ZoneCumul has the highest rate of success (in boldface)
Full size table
Fig. 13
figure 13

Tracking normalized solution quality for ImpVC-ZoneCumul median solutions

Full size image

Scalability

The tray granularity has physical meaning. A smaller tray size than in our experiments is not meaningful in practice, as the trays for the \(7 \times 7\) grid workspace are \(9 \times 19\) cm, which is already much smaller than current industry standards. The evaluated problem instances are all industry-size, or beyond. However, in order to truly test our model we scaled up the number of tasks 4-fold, until we pick one component per tray location of the \(7 \times 7\)-grid workspace, that is 41 components. The largest instance has 169 tasks. We ran each instance until the first solution is found. We used a 30-minute timeout. The best model variant for each collision-avoidance method was used, namely ImpVC-ZoneCumul and ImpVC-Static. As seen in Fig. 15, for every problem instance, at least three out of the five runs found a solution.

Fig. 14
figure 14

Tracking normalized solution quality for ImpVC-Static median solutions

Full size image
Fig. 15
figure 15

Evaluating very large assembly instances on the most granular workspace. The two collision-avoidance methods are contrasted

Full size image

Spatial distribution of locations in solutions

For the experiments with longer timeouts, as well as the experiments to test scalability, we analyzed the statistics of the layout of the solutions. The locations within the solutions reported at timeout are not symmetrically distributed as an artefact of our location numbering and our value selection heuristic for loc[t] variables, given in Section 3.12.

Effectiveness of joint solving

In order to evaluate what is best, to solve some subproblems separately or to solve everything jointly, we design two further experiments. The first experiment emulates the situation where a good workspace design (A) is given, with (B,C,D,E,F) to be solved by our approach. Conversely, in the second experiment we emulate the situation where we are given (B,C), with (A,D,E,F) to be solved. For the first experiment, we use the locations of the first solution found for the full problem, and re-run the experiments in Section 4.1.1 with a 30-minute timeout. Similarly, for the second experiment, we use the task allocation and task sequencing of the first solution found for the full problem, and re-run the experiments in Section 4.1.1 with a 30-minute timeout. The results for the finest-granularity workspace (\(7 \times 7\)) are presented in Fig. 16: with our method of providing (A) and (B,C), joint solving is clearly more effective than solving subproblems (A) or (B,C) separately.

Fig. 16
figure 16

Solving the sub-problems in unison vs solving with some of the sub-problems given, on the finest-granularity workspace (\(7 \times 7\))

Full size image

5 Model generalizations and future work

Our application is a very prevalent type of robot installation. However, it is merely a specialization of a broader model class, addressing a broad spectrum of robot applications. By utilizing the versatility of CP it is straightforward to apply variations to our model, resulting in models for a wide range of other applications. We present some of the possible variations in Section 5.1. Then, we outline avenues for future work in Section 5.2.

5.1 Model variations

Without going into details, we present a non-exhaustive list of interesting applications that are easily modeled by using our model as template and applying an atomic variation. Compound variations can then be achieved by simply performing several of these atomic variations:

  1. 1.

    As robot perception becomes more mature we might be able to place objects without precise calibration in the suction-cup tool. In such a scenario we do not need camera tasks and remove them from the model.

  2. 2.

    Some assembly applications have multiple intermediate tasks between the pick and place tasks, such as the airgun described in [47]. These can be with or without internal ordering. We can easily add such tasks to the model, and impose arbitrary combinations of precedence between tasks by employing the same constraints used for our model, but for other variable tuples.

  3. 3.

    Some assembly applications have some flexibility on the ordering of placing components on the fixtures. This can easily be addressed as we can impose arbitrary combinations of precedence constraints between placement tasks by simply using our model.

  4. 4.

    If there is only one robot arm, then this can be modeled by fixing all agent variables to 1, and removing all arm-to-arm collision-avoidance constraints.

  5. 5.

    If there are additional robot arms, then we can add values to the domains of the existing agent variables, two dummy tasks, and another distance matrix. For zone-resource collision avoidance we add the corresponding zone data. The static collision avoidance model would need modification.

  6. 6.

    If there are more robot arms, then we might want to evaluate using more fixtures. This can be achieved by adding one task for picking up from the new fixtures and one task on where the sub-assembly should be placed. The model is easily adapted by defining a new set of tasks \(\mathbb {T}_{\text {Fixture3}} \subset \mathbb {T}\) in Section 3.2 and straightforward update of constraints (11) and (13). Similarly, it is straightforward to only use one fixture.

  7. 7.

    As robots can have a maximum load or exert large forces on joints when carrying heavy objects, we can model the maximum weight of carried components by using the same type of capacity constraints as in Section 3.7, but using weight instead of 1 for pick, \(-1\) for place and \( Cap (t,\tau )\) depend on robot arm instead of tool.

  8. 8.

    In some applications the same type of component is used more than once for a particular assembly. Such components could then be picked from the same tray. This can be modeled by posting equality constraints on the location variables of such pick tasks, and excluding one of the variables from the all_different constraint (12).

  9. 9.

    There are applications where the component tray or output tray are available in fixed or recurring time intervals. Such trays can be brought by an autonomous ground vehicle (AGV) or a conveyor belt, delivering components in a just-in-time fashion. This model can easily be implemented by adding a constraint on the start and finish time of the task in question.

We can create compound model variations by combining such atomic variations. An interesting robotics application is the emptying of boxes in a fulfillment center, and putting the content into new boxes: this is captured by our model combined with variations 3, 6, and possibly 5.

5.2 Future work

Our collision-avoidance methods relies on unique, precomputed paths between locations. In our approach the entire swept volume is booked throughout the travel time of such paths. This makes the model tractable, but quite conservative. There are many potential remedies to this. One avenue for future work is to book zones only for the time span during which they are actually occupied, and how to solve this in an efficient way. Another avenue could be to precompute multiple paths between locations, and assign path as part of the optimization. How to generate such paths to make them useful is an interesting direction.

In general, one would use one of many available algorithms to get collision-free paths, as well as travel times and traversed zones for such paths. There are time-optimal path planners, such as those in the MoveIt package [6]. However, such general planners might take 10 to 100 times the computational time of the planner we used, while most paths will never be part of a robot program. A way to minimize computational overhead could be to delay computations of paths and zone booking for paths until paths are part of a candidate program, akin to an admissible heuristic of A* search [38].

Future work also includes integrating path planning into the model. Also, the conservative nature of our static collision avoidance makes zone-resource collision avoidance much more general and attractive. Therefore, we will examine ways to make it more efficient.

Finally, we could recover from some types of failures without any change to the approach. Such failures would be handled in a feedback loop by changing the input data and re-solving. Examples are a faulty camera or an arm breaking down, which would be removed from the data. A final avenue would be to introduce optional tasks for handling contingencies.

6 Conclusions

This work presents the first CP-based model integrating (A) workspace layout design, (B) task allocation, (C) task sequencing, (D) task scheduling, optimization for (E) cyclic scheduling, and (F) collision avoidance for a dual-arm robot, to the best of our knowledge. We presented this as a core model containing all necessary constraints, plus two alternative methods for collision avoidance and three redundant constraints that speed up the search. The resource-based collision-avoidance method requires arms to book user-defined zones as non-consumable resources, and can be generalized to scheduling more than two robot arms. The static collision-avoidance method implements a division of the workspace into two sides, but is limited to two arms, is more conservative thus rendering some problem instances unsatisfiable, and requires some additional assumptions. However, when solutions are found, it outperforms the resource-based method, in terms of both solution quality and number of instances solved.

We compared the model with and without redundant constraints, and found that the redundant constraints on the routing part of the model significantly increase the solution quality compared to the core model. Notably, the best choice of redundant constraints increases the number of instances solved from \(37.9\%\) to \(98.5\%\) for zone-resource collision avoidance. Furthermore, the redundant constraints that use cumulative constraints for the scheduling part of the model actually decrease the performance on both measures compared to the core model.

We performed an extensive evaluation of problem instances by combining assembly instances and workspace instances. The assembly instances are of realistic size, or larger, having 17 to 45 tasks, which exceeds the typical number of tasks for an assembly robot. The workspace instances vary from a few candidate tray locations to a large number of candidate tray locations. We allow fixture locations to be variable, leaving almost all locations available for optimization. The empirical results show that the model delivers solutions within a two-minute timeout for almost every problem instance, for both collision-avoidance methods. We also performed longer experiments using the same experiment setting: they gave the same efficacy ordering of the redundant constraints. Using the best choice of redundant constraints, we ran experiments with a very large number of tasks, for the most fine-grained workspace: they indicate that the approach displays an increase of computation time that is exponential in the number of tasks. Even for instances well beyond the typical number of tasks in an industrial application, our approach always finds solutions within 30 minutes.

We also demonstrated that joint solving of all subproblems is more effective than solving subproblems separately.

We emphasize the ability to generalize our CP model beyond our application: other interesting robot applications can be modeled by applying small variations to our model.