Robotic Pick-and-Place Time Optimization: Application to Footwear Production

This article considers the problem of optimizing the task sequences carried out by a dual-arm manipulator robot in a footwear production setting. The robot has to identify the pieces of a shoe put in a tray and pick-and-place them in a shoe mould for further processing. The shoe pieces arrive on a tray in random positions (patterns) and can be picked up in different order. In such a setting, a decision tree model is developed to recognize the pattern and predict the optimal sequence for picking the pieces up, thus, the picking and decision-making time is minimized. Two shoe models are considered for training and validating the solution proposed and the developed algorithm is applied in the real setting. There are not many studies which use the decision trees in sequencing and scheduling problems in robotics. The findings of this article show that the decision tree method has advantages in task planning in a complex environment consisting of multiple trajectories and possible collisions between robot arms.


I. INTRODUCTION
There is an increasing trend in the current industry automation according to the pursuit of the performance in different features. The automation in the manufacturing lines allows the improvement of production time, precision in complex tasks, repeatability, etc.
Current production processes in the shoe industry are characterized by high material consumption and low automation of production lines, but some technologies such as Knitting, 3DPrinting, Direct Injection or 3DBonding are increasing the level of automation in the shoe industry. 3DBonding joins pieces of different materials by injecting polyurethane through a network of channels previously created inside a mould.
3DBonding process removes many of the traditional operations, such as sewing, which reduces production costs and delivery time. It also improves product performance and allows new styles and designs. The main task of shoe production is reduced to a pick-and-place process where the upper The associate editor coordinating the review of this manuscript and approving it for publication was Hui Xie . pieces are placed inside a mould that will generate the shoe in a few seconds of polyurethane injection. This is an example of the rise up in the use of dual arm manipulators in the current industry according to the pickand-place tasks. This is due to the new trend in transporting flexible pieces and obtaining an improvement in the positioning of them, therefore, the use of a dual arm manipulator is essential. In [1] the approach is to use an algorithm that develops a path for each arm which executes a task, rather than using one arm to hold one of the objects while the other arm performs the task on the object. Dual arm manipulator must be coordinated to achieve the optimum time in the pickand-place task, because with a good coordination two tasks can be performed at the same time, increasing complexity in benefit of time reduction.
For optimizing the time, an important feature is the sequence of pieces in the pick-and-place. In [2] authors develop an algorithm called Best Uniform Algorithm (BUA) to solve the problem while other solutions are based on genetic algorithm (GA) proposed by [3].
It will be necessary to optimize the trajectory in order to achieve a successful task. The first step to follow is the optimization of the reference trajectory for each arm. In [4] authors present a way to optimize the reference path for a dual arm manipulator in industrial tasks. Two arms are working in the same workspace, hence, each arm should take into account the other arm and could be considered like an obstacle which continuously changes its position. In [5], a research about a solution to perform the optimization of the motion time of robotic manipulators with the presence of hurdles is presented.

A. BIMANUAL ROBOTICS
The increase in the complexity of industrial tasks and the pursuit of minimizing time has given rise to adapt robot designs as similar as possible to the way in which human beings develop their activities. In industry, anthropomorphic robots of human size are expected to replace operators without more changes in the industrial task. Accordingly, the use of anthropomorphic or dual arm manipulators is an increasing field in the research of new technologies. The research and history of single and dual manipulators are studied with detail in some articles: in [6], the time-optimal motions of robotic manipulators in presence of obstacles is developed; in [7] the undersea robot is presented and in [8] the Robonaut space humanoid is shown.
The use of robot manipulator, is a field widely studied during the last years in order to optimize and automatize industrial tasks, but when a dual arm manipulator task is designed, a high complexity compared with single arm is introduced. In [9] a survey about the state of the art about the dual arm manipulators was done.
The hiccup in the use of dual arm robots or multisystem robots lies in some features that in a single arm is not so important: · Coordination-It is an essential feature for obtaining successful tasks. One of the factors that motivates the use of this multisystem setups is the big flexible pieces transport. In this case, the coordination between both arms for moving the piece is essential and includes constraints such as, distance, orientation, etc., in order to ensure a successful pick-and-place. In [10], a solution for a dual arm robot coordination is proposed, where authors present an algorithm that is able to automatically coordinate independently planned motions of a dual-arm manipulator during the task.
· Trajectory-When a single robot is working by itself, the fixed environment is taken into account for optimizing the trajectory avoiding obstacles. But with a multisystem setup, there are other elements in the workspace which move along the time. Therefore, the calculation of the trajectory should be in real time, taking into account the position of the other systems in every moment. For a good trajectory optimization, the algorithm has to be able to recalculate the trajectory to prevent collisions.
· Constraints-When a piece is transported by two arms, some constraints should be considered. For instance, both the distance between grippers and the orientation are indispensable to ensure a successful task so that during the whole trajectory the robot holds the piece.
In this case the research has been carried out with an ABB YuMi robot, collaborative robots that are capable of haptic HRC without add-on sensors or skins [28]. The YuMi is a dual arm collaborative manipulator with 7 Degrees Of Freedom (DOF) in each arm, hence it is a redundant robot and it has 14 DOF in total. Both arms are joined by a body, in [12] a kinematic analysis for the ABB YuMi is presented.

B. OPTIMAL TIME FOR PICK-AND-PLACE TASKS
One feature which has been thoroughly studied is the time motion planning optimization for robotic manipulator. In the literature, multiple solutions for this problems are proposed. [13] suggested a phase plane analysis method to obtain a minimum time trajectory with confined torque. In [14], [15] and [16] a similar problem is solved with a greedy search algorithm and convex optimization approaches. A solution can be presented by solving the optimal time problem as presented in [17] based on the travelling salesman problem (TSP) with a genetic algorithm studied in [18] and [19]. Another research such as [20] studies the use of decision trees in planning optimal time trajectories. The vast majority of researches presented above focus on a single arm setup and a group of points where the sequence is not so important like in [21], where a TSP resolution is presented taking into account a constant speed. But in this article a different system configuration is presented; the task to be performed will be a pick-and-place. This implies a pair of points to execute compulsorily one point after the other. Furthermore, the system is based on dual arm manipulator. The optimization time will have to take into account the coordination between arms that are moving and the possibility to work in the same task with two arms at the same time or two arms working together for transporting the same piece.
The current problem resolution is a variation of TSP and Taxi Driver Problem, this last kind of problem studied in a deep way in [22] and [23] where a set of vehicle starting and ending routes is designed and in which objects or people have to be collected and distributed. Where the pursuit of the optimal time is based on the points of pick-and-place tasks, the sequence of points and which arm goes to each point or if it is necessary two arms so as to transport a piece.

C. MANUFACTURING PROCESS
The final goal is to do an industrial task which incorporates the minimum time in the pick-and-place task with a dual arm manipulator and a pattern recognition system in order to identify the pieces positions to do the picking. In some researches some of these features are presented. In [1] a process of assembly with a dual arm manipulator is developed.
[24] presents a manufacturing system of a robot arm with object recognition. In [25] and [26], the path generation for the task of industrial robots so as to optimize the industrial VOLUME 8, 2020 process is established. In [27], a tree-based decision method is developed in order to decide which configuration machine tool is the optimal one in each case.
Currently, industry automation is following several lines of development: use of collaborative robots in robot-operator interaction tasks [28], use of clouds of robots in the industry [27] or a reinforcement based on learning robot handling skills [28].
Nowadays, the production of fashion footwear is mainly handcrafted. Some manufacturing procedures are assisted by specialized machinery (last production, injection of soles, cutting of upper pieces, etc.) and there are few automated lines in some types of footwear in massive demand. But most of the production continues to be handcrafted, especially in case of the mid-priced footwear market.
This article is organized as follows. The following section introduces some preliminaries, while Section II first presents the Methodology used and the Procedure and guidelines required in subsections A, B and C. The case study is presented in subsections E, F and G of Section II. The feasibility and robustness of the proposed approach is supported by the experimental results in Section III. In this section, the overall results obtained are presented and discussed in subsections A and B. In Section III C, decision trees are outlined. In Section III D, these results are tested in the real system. Finally, some conclusions are given in Section IV.

II. METHODOLOGY AND MATERIALS
This research is based on a sequence of essential data collection tasks for subsequent analysis. With this analysis, a decision tree will guide the best decision in each case presented. Firstly, the methodology consists of the generation of sequences, the analysis of the results of the sequences and the strategy of the decision tree. Secondly, the study of the case and its environment (robot, system description and image processing) is presented. Finally, the summary of the main results and conclusions is developed.

A. SEQUENCE GENERATION
For a preliminary study an ensemble of different array of images full of pieces is generated. Pieces used in this study are from two different models of shoes, manufactured by two different companies, these models are presented in Figure 1. These two models are, the Manufacturer 1 model and the Manufacturer 2 model. Both models consist of four pieces, three of them are grabbed with one arm (the two lateral pieces and the heel piece) and one of them (the frontal piece) is grabbed with two arms. The difference between them lies on the size of the pieces and their final position.
The array of trays is shown in Figure 1c, each one with the random position and orientation of the four pieces, and for each tray, a total of 30 sequences of the robot movements have been selected in order to study which is the optimum sequence in each case.

B. ANALYSIS OF SEQUENCE RESULTS
In this research, two models have been considered to analyze the data in the development of the decision tree. The Manufacturer 1 shoe model will be used as a training model to develop the mathematical model. The most important data that will be collected to develop the decision tree correctly will be the average and minimum sequence time and the correct execution of the trajectories of each arm to ensure that the task has been carried out successfully. This data will be collected for each sequence and each tray. The Manufacturer 2 shoe model will be used to verify the mathematical model developed with the Manufacturer 1 shoe model, using the same data as in the previous one.

C. STATISTICAL ANALYSIS AND DECISION TREE STRATEGY
The assembly times have been summarized using averages, standard deviations, minimum, maximum and range. The classifications have been summarized by means of counts and percentages. To obtain the optimum assembly sequencing, conditional decision trees (CTREE-Conditional Tree) have been used.
In the conditional inference trees, the selection of the variables is made in two phases. First, a global hypothesis of independence is formulated in terms of partial hypotheses. In other words, it is evaluated if there is dependence between the response variable and each of the explanatory variables. If the null hypothesis of independence cannot be rejected, the recursive process is stopped. In contrast, if the global hypothesis of independence is rejected, the next step is to measure the level of association between the dependent variable and each one of the explanatory variables, which allows to generate new divisions of the tree in a sequential way, [29]. Some of the advantages of this tool are: the easy interpretation of the models due to the structure of the resulting tree [30], and its versatility in case of non-linear relationships and in the handling of numerical and categorical variables simultaneously.

D. ROBOTICS SOLUTION
The robot used in this research is the ABB IRB 14000, coomnly called YuMi. It is a collaborative robot designed to assembly of small parts, with a double arm that includes twofingers. YuMi can collaborate hand-in-hand with people in a common manufacturing environment. One of YuMi's distinctive features is its intrinsic safety, as it can work alongside with people. Each arm has 559mm range and can lift 0.5kg.
The robot also monitors the efforts that are being made, but not through a sensor in the base, but through the electrical consumption of its motors. This robot has features that will have to be taken into account when making the programming. ABB has an own software (RobotStudio) and programming language (RAPID) to develop the code, this environment is used to do the programation.
The ABB RobotStudio program is a software that allows the creation, programming and simulation of industrial robot stations, designed and patented by ABB company. Robot-Studio has a virtual controller, an exact copy of the real software used by robots in production. This makes it possible to program an offline robot on a PC, exporting the results obtained in simulation to the real station.
The program also provides tools to increase the profitability of the robotic system through tasks such as training, programming and optimization. Version 6.08 has been used to create and program the stations for this work. The combination of these tools has been used to develop a program capable of reading a file with different trays.
The use of a bimanual robot implies having to make a more precise control of each trajectory of the arms, because both arms work in the same workspace and there could be collisions if it is not done properly. For synchronous execution of tasks, a real time control must be done controlling in each instant of time the position of both arms to be able to correct the trajectory in case of predicting a collision between them.
The ABB RobotStudio provides a functions group to do this task in the best way. Both arms are synchronized in some references points to make sure the trajectory of each arm, avoiding collisions. Not only synchronization is established but also the distance between both arms must be defined to transport one piece with both arms. These tasks have been developed with a ABB function, called SyncMove, which coordinates multisytems with synchronization points. It is essential in order to transport the piece four with two arms in the proper way.

E. IMAGE PROCESSING SYSTEM AND PIECE FEATURES
The image processing is a complex task in industrial environments. In pick-and-place operation, the calibration is an important parameter. In [31] a task-oriented marker less hand-eye calibration method is developed by using non-linear iterative optimization in order to improve the calibration task The image processing system is based on an external ABB camera connected with a PC, where a RobotStudio program is developed. To cover as many possibilities as possible, 250 samples are collected in random positions, Figure 1c, due to in the industrial process, the pieces could get in any position. All possibilities should be verified and tried to distinguish a pattern in the sequence of the pick-and-place.
The computer vision is developed in order to recognize the different kinds of pieces. When the piece is detected, the gripping point (x,y) is preprogrammed and the orientation in z-axis is saved to be used by the simulation program.
Each piece is classified according to its position and orientation for searching a relation when the pieces are in a nearby area and reducing the number of different trajectories which should be programmed. The vision workspace is divided into four quadrants (1,2,3,4) and the orientation is divided in four as well ([0, π 2 ]; [ π 2 , π]; [π, 3π 2 ]; [ 3π 2 , 0]).

F. SYSTEM SIMULATION DESCRIPTION
The simulation system is developed with a RobotStudio software and RAPID language program. The robot which has been worked on is the ABB dual arm manipulator, YuMi. The program receives the information about the position and orientation of the pieces. With this information, the collection points that are established will change from one sequence to another, since the placement points will be the same for each piece in all sequences. Figure 2 shows the system in RobotStudio.    For each piece position there will be thirty sequences, combining each arm in order to be able to pick the pieces and changing the order to pick them. In Table 1 the combination between the arms in the pick are shown. The number represents the order in the picking and the letter is the arm: R = Right arm, L = Left arm and B = Both arms.
These sequences give as a result the total time in the pickand-place task. Therefore, for each piece position there will be 30 sequences and more than 200 different positions will have been taken. Altogether, more than 7000 sequences are calculated to find a pattern in the configuration of the picking order of pieces.
A filter is established to simulate the main cases. The piece 4 which is picked by two arms is always positioned in the last picking position because it uses both arms and it is an addition in the time no matter what position it is in. In cases of two arms picking two pieces at same time, the third piece is picked always by the right arm because it is the arm which is close to the pick area and while the right arm is going to the pick, the left arm places the second piece.

III. RESULTS AND DISCUSSION
In this section, a discussion about the results obtained with the two different shoes is presented: Manufacturer 1 and Manufacturer 2. Manufacturer 1 is safety shoe models and Manufacturer 2 is a Mexican manufacturer of casual shoes. For each case, time require for the pick-and-place task is calculated by simulation in RobotStudio.
First of all, the times for each sequence by the simulation are presented in order to search a relation between the two different models. Thirty different sequences or procedures have been chosen. The fourth piece, which is always picked up with both arms, is situated in the last position to optimize the time. The first twenty four sequences are sequential, but in the last six sequences some pieces are transported simultaneously. The problem of these last sequences is the high number of collisions produced during the task, having a low percentage of success. For this reason, these sequences are discarded from the study. With the analysis of the information, a decision tree is made. The first shoe model (Manufacturer 1) is used as a training in order to develop the decision tree algorithm and the second shoe model (Manufacturer 2) is used to validate the decision tree algorithm.

A. MISSING VALUES, COLLISIONS AND ASSOCIATIONS
The thirty combinations analyzed, specify which arm will be used to pick up each piece and its order, leaving always pieces that must be grabbed with two arms at last. For the preliminary study, the missing values of the two models (Manufacturer 1 and Manufacturer 2) have been analyzed. These missing values are the combination of the position of the parts where there is no way to perform the task (outside the working range, collision, etc.).
In case of Manufacturer 1 model, the group analyzed consists of 243 trays, obtaining a large number of collisions in the last six sequences because they develop the pick-and-place task moving both arms too close. The other case analyzed is the Manufacturer 2 model shoe.
This model has 222 trays and it presents similar results to Manufacturer 1 model. The number of collisions values in the last six sequences is the same, therefore there is a similar behaviour in both models. This behaviour could indicate that a common model could be found. See Figure 3.
Several groups of sequences are observed, for instance, the sequence combination that procedure 5 does not solved are the same ones that procedure 1 does not solved, neither the reason for the failure is the same. While comparing sequence 1 and sequence 2, no correlation is found, in Figure 4 a correlation between sequences 1, 21 and 22 are shown as well as the no correlation between sequence 1 and 2. These patterns are fulfilled in the two models analyzed. In summary, the analysis of missing values and collisions has led  · Group 1-S1, S5, S21 and S22 · Group 2-S2, S6, S14 and S17 · Group 3-S3, S11, S23 and S24 · Group 4-S4, 12, S13 and S16 · Group 5-S7, S9, S19 and S20 · Group 6-S8, S10, S15 and S18

B. SEQUENCES RESULTS
Analyzing the assembly times from the two models, in the Manufacturer 1 model the average times per tray required to carry out the assembly which goes from 15.18 sec. (S18) to 17.03 sec. (S9), without any sequence completing the total number of trays (197 to 228).
If the sequence that solves it fastly could be selected for each tray, an average time per tray of 15.025 sec. would have been obtained, this value is the mean of the best sequence for each tray. Figure 5a shows the average assembly times of the trays for the Manufacturer 1 model. Table 2 shows the main parameters associated with the assembly time of the Manufacturer 1 model shoe. Table 3 shows the same parameters for the Manufacturer 2 model shoe. In the model Manufacturer 2 that the average times per tray required to carry out the assembly range from VOLUME 8, 2020    Figure 5b shows the average assembly times of the trays for the Manufacturer 2 model. In Manufacturer 2 shoe model the difference between the correct sequence and the slowest one is 1.35 sec. per task, in terms of average.
It can be seen in which sequences the best time is obtained for each tray. Figure 6a and Figure 6b show the result obtained when considering all the sequences. It can be appreciated that the sequences that most frequently appear as the best are S13, S14, S15, S16, S17 and S18. Being the importance of the S20, S22 and S24 sequences residual. The rest of the sequences do not show better results in any tray.
In Figure 6c and Figure 6d, the range of possibilities has been closed to the sequences that have been considered more frequent and the quickest selection only taking into account these six sequences.

C. DECISION TREE
In summary, the best sequence is necessary to be able to select, from the 6 indicated (S13, S14, S15, S16, S17 and S18) to carry out the assembly of each tray of pieces. To do this, we use a conditional decision tree.
The construction of the decision tree will be focused on trying to predict the optimal sequence selection for each tray. In table 4, the columns represent which sequence has been the fastest and the rows, the sequence chosen for the adjusted model. The diagonal of the table indicates that the fastest sequence is the choice. The 58.43% of the trays are correctly classified, which means that 28.38% of the trays in the fitting sample (Manufacturer 2) being correctly classified. The assembly would be completed in 237 of the 243 trays of Manufacturer 1 shoe model with an average time per tray of 15.14 sec. While in the Manufacturer 2 shoe model the assembly of 216 of the 222 trays  with an average time per tray of 15.51 sec. would be completed.
The Figure 7 represents the tree decision with the six best sequences model. In this tree, the variables which will be   Secondly, a decision tree for the three best sequences is developed in order to optimize the sequence. Figure 6e and Figure 6f represent the best three options in the Manufacturer 1 model and the Manufacturer 2 model that have been chosen. It is important to remark that S13 and S16 follow the same pattern, as did S14 and S17, and on the other hand, S15 and S18. This means that there is often not much difference between the times obtained by each pair. This is why the possible sequences are reduced to 3, which would be S13, S17 and S18. The construction of the decision tree will be focused on trying to predict the optimal sequence selection for each tray. Table 5 shows that 69.55% of the trays are correctly classified, which means that 42.34% of the trays in the fitting sample (Manufacturer 1) being correctly classified.
With this last decision tree, the following features are obtained: in the adjustment 236 trays with a mean time of 15.14 sec. are placed, a standard deviation of 1.7 sec. In the validation, the values are 217 trays placed with a mean time of 15.48 sec., a standard deviation of 1.3 sec. Figure 8, represents the tree decision with the three best sequences model. In this tree, the parameters which will bear in mind are the quadrant where the right piece has been placed (c_d), the euclidean distance of right and left pieces (dis_d and dis_i) and the angle of right and heel pieces (a_d and a_t). This tree is simpler compared to the six sequences tree. When the pieces are placed, the first feature to be taken into account is the quadrant where right piece has been placed, if it has been placed in 3rd or 4th quadrant only the right piece euclidean distance will have to be analyzed in order to select the best sequence.
In the other case, first the angle of the heel piece will have been taken into account and then the angle of the right piece and the Euclidean distance of the left piece will have to be analyzed.

D. REAL SYSTEM TEST
The pick-and-place task has been programmed in a real system (ABB YuMi) in RAPID language. In Figure 9, the real VOLUME 8, 2020 FIGURE 10. Pick-and-place task sequence in the real system. setup is presented. It consists of the ABB YuMi robot, the tray, the artificial vision which is consist of Industrial Camera placed in a structure for taking the photo and the shoe mould where the pieces are placed.
The program has been developed to pick-and-place the first model (Manufacturer 1), in Figure 10 the sequence of the task can be seen. Times in the real system are similar compared to simulated times. In the real system, times are a little higher because the velocity of the robot slightly is reduced in order to do it in a best way. The real system has a mean time of 18.26 sec.

IV. CONCLUSION
This article presents the results of the optimization of pickand-place times performed by a two-arm robot. A model has been successfully achieved. It simplifies the complex scenario presented in the robotic solution section. With the information analyzed in the Manufacturer 1 shoe model, the slowest average time sequence is 18.78 sec. While the fastest average time sequence is 15.20, a difference of 3.58 sec. This difference makes necessary to study the best sequence that minimize the time. With all data collected, the random positions and rotations of the trays and the times of pick-and-place task, the best sequences are selected from the thirty different sequences.
Having two different models of four pieces, the Manufacturer 1 model is used as a training model and the second model, Manufacturer 2 is used to verify the decision model. Analyzing the first model, six sequences are obtained with a vast majority of faster sequences, the value of the rest of them are negligible. There are six sequences which are discarded because there are a lot of missing values due to collisions between both arms.
In the first study performed with a six sequences tree, a 58.43% in adjustment and 28.28% in predictions of success in selecting the correct sequence is obtained with a mean time of 15.14 sec. for adjustment and a 15.51 sec. for the predictions. With data and the groups observed in Table 2 whose behaviour is the same, the number of sequences have been reduced from six to three. With this new criteria a new tree has been done and better results have been obtained. The percentage of selecting the quickest option rising to 69.55% in adjustment and to 42.34% in predictions. The mean time is similar 15.14 sec. for adjustment and a 15.48 sec. for the predictions, the mean time goes down to 30 msec. This great difference between the first decision tree and the second one, lies on the increase of 14% in selecting the best option.
If the best option is selected, the average time is 15.025 sec. in the Manufacturer 1 model and 15.190 sec. in the Manufacturer 2 model. Therefore, in case of the three sequence decision tree model, between the Manufacturer 2 model and the Manufacturer 1 model, there is only a difference of 115 msec. and 290 msec. respectively with according to the best average time, but the processing cost is lower since it is reduced from 30 possible options to 3. The last decision tree, only programming three different trajectories for the robot is able to complete each situation which is presented, focusing on the rotation and the position of each piece.