Pre-grasp Manipulation Planning to Secure Space for Power Grasping

An object can be gripped firmly through power grasping, in which the gripper fingers and palm are wrapped around the object. However, it is difficult to power-grasp an object if it is placed on a support surface and the grasping point is near the support surface. Because there is no gap between the object and the support surface, the gripper fingers and the support surface will collide when the gripper attempts to power-grasp the object. To address this, we propose a pre-grasp manipulation planning method that uses two robot arms, whereby space can be secured for power grasping by rotating the object while being supported against the support surface. The objects considered in this study are appropriately shaped for a power grasp, but power grasping cannot be performed directly because the desired power-grasping location is close to the support surface on which the object is placed. First, to power-grasp the object, candidate rotation axes on the object and in contact with the support surface are derived based on a mesh model of the object. Then, for each such axis, the object pose that allows power grasping is obtained. Finally, according to the obtained object pose, the paths for rotating and power-grasping the object are planned. We evaluate the proposed approach through simulations and experiments using two UR5e robot arms with a 2F-85 gripper.

surrounding environment. Here, the term "appropriate shape" 18 implies that the size or shape of the object at the location 19 where power grasping is to be performed is such that the 20 whole or part of the object can be placed between the gripper 21 fingers (i.e., object size is smaller than the maximum distance 22 between the gripper fingers). Assuming a free-floating object, 23 power grasping can be performed on any part of the object 24 as long as its shape is suitable. However, if the object is 25 placed on a horizontal and flat support surface, such as a 26 table, a power grasp may not be possible because there is no 27 space between the object and the support surface, and thus 28 the gripper fingers cannot be wrapped around the object. If 29 an object is voluminous, such as a cuboid, it may be rotated 30 and placed stably so that another side of the object is in 31 contact with the support surface. Thereby, it is possible to 32 secure space for power grasping at a desired point of the 33 object. However, if an object is not voluminous, that is, one 34 of its dimensions is significantly smaller than the others, 35 it is difficult to create space for power grasping using this stably.

43
In this paper, we propose pre-grasp manipulation planning 44 so that space for power grasping may be secured ( Figure   45 1). Pre-grasp manipulation is the process of determining 46 preliminary manipulations for reorienting an object to an appropriate pose (position and orientation) so that a given 48 task can be performed [5]. In the present study, it refers to parts of the surrounding environment, such as the edges 65 of the support surface or the walls, can be used for 66 manipulation planning, except for the support surface 67 itself.

68
(c4) The object is not voluminous. This implies that one di-69 mension of the object is assumed very small compared 70 with the others. As shown in Figure 2, an object may 71 be placed on the support surface in different poses, but 72 it is not stable because it is not voluminous.

73
(c5) Because the gripping force of the precision grasp using 74 only one gripper is not sufficient to lift the target object, 75 grasping the object in an arbitrary position may cause 76 slippage.

77
Because of (c3) and (c4), it is not possible to power-78 grasp the object directly if a grasp planning that does not 79 allow collision with the surrounding environment is used. 80 Therefore, it is necessary to determine an object pose that 81 can secure space for two grippers to perform power grasping. 82 In this situation, because, initially, the target object can 83 only be precision-grasped, and condition (c5) is assumed, it 84 should be rotated while being supported against the surface 85 so that its pose can be changed without slipping. To this end, 86 we use a mesh model of the object to determine rotation-87 axis candidates based on the position of all vertices in the 88 model. For each candidate axis, a target rotation angle is 89 determined to secure space for two grippers to power-grasp 90 the object. Subsequently, planning is performed to rotate the 91 object using one robot arm, power-grasp it by the other arm, 92 and finally regrasp it in a power-grasp manner by the robot 93 arm used for the rotation. All grasping points for rotating 94  with the upper part of the object, and finally gripping the 127 object [6]. A thin nail made of smooth plastic and rubber was 128 attached to each finger to increase the grasping success rate.

129
In [7], a gripper was proposed that could grasp a flat object 130 by attaching a crawler to one fingertip. If the force applied to 131 the object while the gripper finger is closed exceeds a certain 132 threshold, the crawler starts to move, and the object rotates 133 in the direction of the gripper palm. Through this in-hand 134 manipulation, the gripper can power-grasp flat objects. 135 Grippers have also been proposed for power-grasping gen-136 eral objects. In [8], a three-fingered gripper was presented, on 137 one finger of which a small two-fingered gripper is mounted.

138
It is thus possible to power-grasp an object by using a gripper 139 selected according to the size of the target object, thereby 140 separating the problem of maintaining force closure and ma-141 nipulating the object. In another study, a plate was attached 142 to each finger of an existing gripper for power grasping [9].  Effective power grasping can also be achieved by grasp 148 planning. In [10], the power-grasping strategy was selected 149 depending on the size of the object. Specifically, for small 150 target objects, parallel fingers (index, middle, ring, and little) 151 of the robot hand were used, but the thumb and palm were 152 not used. For larger objects, the thumb and palm of the robot 153 hand were also used. Another grasp planning method was 154 based on hand-object geometric fitting [11]. Considering the 155 surface of an object, target contacts suitable for the finger or 156 palm of the robot hand were located for grasp planning. In 157 the case of precision grasping, as many target contacts were 158 located as the number of fingers. To power-grasp an object, 159 both the finger and the palm of the robot hand were associated 160 with the target contacts. However, the actual number of 161 contacts was larger than that of the target contacts because 162 the proximal links of the finger were encouraged to touch the 163 object for more stable gripping. In another study, gripping 164 was performed by reshaping the fitting weights to points on 165 the finger and palm of the gripper [12]. To precision-grasp 166 an object, more weight was applied to the fingertip area 167 than elsewhere. In the case of a power grasp, the object was 168 gripped by assigning more weight to the center of the finger 169 and palm.

170
However, the gripper is not always possible to approach 171 the grasp pose obtained by the aforementioned methods. 172 For example, if an object is placed on a flat surface, or if 173 there are obstacles, a collision may occur when the gripper 174 moves to a grasp pose that would be valid if the surrounding 175 environment were not considered. However, if pre-grasp ma-176 nipulation is performed to relocate an object to an appropriate 177 pose before final grasping, the gripper can grasp the object by 178 attaining the obtained grasp pose. In [5], preparatory object 179 rotation was presented, where a handled object was rotated 180 about an axis perpendicular to the support surface so that the 181 handle was in front of the robot before the object was lifted. 182 This was motivated by the fact that a person usually rotates 183 a handled object so that the person could hold the handle 184 and lift the object conveniently. The authors in [13] also pre-185 rotated an object about an axis vertical to the support surface, 186 but the target rotation angle was determined according to the 187 manipulator payload calculated based on the load limit of 188 each joint.

189
Another approach is to slide an object to render grasp-190 ing possible. In [14] and [15], objects were classified into 191 categories according to appearance, and each category had 192 a pre-shape data structure comprising a robot hand, a set 193 of starting poses, etc. When a target object was input, after 194 the category in which the object belonged was determined, 195 the robot hand slid and moved the object to the final region 196 so that it could be grasped using the corresponding pre-197 shape. In [16], a combination of pre-grasp manipulation and 198 transport task planning was proposed. The trajectory was 199 optimized using the functional gradient optimization method. 200 In addition, to increase the planning success rate, objects 201 were released after pre-grasp manipulation, and were grasped 202 again to perform the transport task. In [17], the configurations 203 of the start and goal regions were sampled to determine 204 the appropriate motion for grasping a thin object by pairing 205 each configuration. In this strategy, the object was grasped 206 after it was slid to the edge of the support surface. In [18], 207 not only the edge of the support surface but also various 208 parts of the surrounding environment (such as surface and 209 walls) were used to grip an object. Specifically, a graph was 210 constructed using environmental constraint exploitation and 211 valid transitions, and manipulation planning was performed 212 VOLUME 4, 2016 by regarding gripping motions as goal nodes. 213 Certain studies have been concerned with securing space 214 for grasping by rotating an object around a axis parallel to 215 the support surface. A method for rotating a cuboid object 216 using hybrid force-velocity control was proposed in [19]. 217 Grasp planning for a cuboid object in the corner of a box, 218 is difficult. However, using this method, the cuboid could be 219 rotated so that space for grasping was secured. Two robot 220 arms were used in [20] to grasp an object. One arm pushed 221 and lifted the object using a half-sphere end-effector to secure  The regrasp method can be used to grasp an object in 227 a desired pose. In [21], a pin perpendicular to the support 228 surface was installed to increase the success rate of pick-  Another strategy proposed in [22] was to reorient an object to 235 an appropriate pose for performing the next task by decom- the object could be grasped in this pose. In [24], pre-grasp 245 planning was performed using multimodal motion planning.

246
Using the regrasp graph, it was automatically determined 247 whether to use only one robot arm or both arms to perform 248 a given task, including pre-grasp manipulation. It was also 249 determined whether to include handover motion if two robot 250 arms were used.

251
The main difference between the present study and the 252 aforementioned studies is the target object, which is too 253 heavy or too large to stably lift using only one gripper (c5).

254
Therefore, handover or in-hand manipulation is not an appro-255 priate method to secure space for grasping. To address this, 256 when an object is rotated to secure gripping space, such as 257 space for power grasping in this study, the support surface is

271
Our task is to power-grasp an object in a situation where 272 power grasping cannot be performed directly. The proposed 273 method requires two robot arms: a supportive and a master 274 robot arm. The role of the latter is to power-grasp an object 275 after the former rotates the object. Then, after the supportive 276 arm regrasps it in a power-grasp manner, the task is com-277 pleted.

278
For each object, a support-grasping pose and two power-279 grasping poses are heuristically predefined (Figure 1 (a)). The 280 support grasping pose is associated with rotating the object, 281 and the power grasping pose for each robot arm is the target 282 pose for power grasping the object.

283
The pipeline of the proposed method is shown in Figure 284 3. First, in the simulation, the mesh model of the target 285 object is loaded in the initial pose. To find cases in which 286 the target object and any flat surface can be in an edge-face 287 or two vertex-face contacts, the convex hull of the object 288 is derived. Among the edges of the convex hull, those 289 that are not short and are close to the support surface are 290 designated as rotation-axis candidates that may be used to 291 sustain the object while being rotated. For each candidate, 292 the goal angle is determined for securing space so that power 293 grasping may be performed using two grippers. When the 294 target object is rotated by the goal angle around the candidate 295 rotation axis, inverse kinematics solutions for reaching the 296 support-grasping pose and power-grasping poses are calcu-297 lated. Based on these solutions, integrated paths for rotating 298 and finally power-grasping the object are planned. After the 299 integrated paths are obtained, one of them is executed by 300 the robot arms. In the next section, we provide a detailed 301 explanation of the proposed method. 303 Under condition (c5), the target object should be rotated 304 while being sustained on the support surface so that space 305 may secured for power grasping by the grippers. For this 306 reason, the rotation axis must be on the object surface and in 307 contact with the support surface while the object is rotated. 308 Algorithm 1 is proposed to determine candidate rotation axes 309 satisfying this requirement.

311
Various types of contact can occur between two objects [25]. 312 In order that an object be rotated around an axis while being 313 supported by the support surface, an edge-face contact must 314 occur. In this study, the edge in this contact is the rotation axis 315 that lies on the object surface. In addition, the support surface 316 corresponds to the face of the contact. Rotation is possible 317 even when an object is supported by a single vertex-face 318 contact. However, the axis of rotation may be distorted be-319 cause the part supporting the object is small. Therefore, in 320 this study, rotation is assumed to occur only in the case of 321  if d e < ∆d s and l e > ∆l then 8:

324
To determine the rotation axis that induces an edge-face or 325 two vertex-face contacts, the convex hull C obj of the object 326 O obj is calculated, and the edges E obj of C obj are obtained 327 using the ConvexHull(·) and Edges(·) functions, re-328 spectively (Figure 3 (b)). Among these edges, one that is not 329 short and is close to the support surface is designated as a 330 rotation axis candidate (Figure 3 (c)). To find the candidate 331 rotation axes, in the simulation, we use the position of the 332 endpoints for each edge e obj ∈ E obj and that of the support 333 surface parallel to the xy-plane in the world frame. These 334 points can be derived from the mesh model of the object and 335 the predefined height of the support surface in the simulation. 336 Based on these positions, the DistEdgeSupport(·) and 337 EdgeLength(·) functions return the maximum distance 338 d e between the endpoints of the edge e obj and the support 339 surface, and the length l e of the edge e obj , respectively. If d e 340 is within a certain value ∆d s , and l e is larger than a thresh-341 old ∆l, the corresponding edge e obj becomes a candidate 342 rotation axis. Because a very short edge e obj is similar to a 343 vertex-face contact, the threshold ∆l is used to exclude it. In 344 addition, the value ∆d s is used to find edges that can make 345 contact with the support surface. In fact, the rotation-axis 346 candidates obtained in the simulation should not be in contact 347 with the support surface, but only close to it. If the rotation-348 axis candidates are in contact, then the object already collides 349 with the support surface because the candidates are on the 350 object surface. Therefore, the maximum rotation θ max cannot 351 be determined by checking the collisions between the object 352 and the support surface. To resolve this, in the simulation, the 353 object is slightly separated from the support surface ( Figure 354  4 (a)). The detailed derivation of the maximum rotation θ max 355 VOLUME 4, 2016  is described in Section IV-B.

357
To determine the maximum rotation θ max for each rotation-358 axis candidate λ and rotation direction r dir , the object is ro-359 tated repeatedly by ∆φ until a collision occurs (Figure 4 (b)).

360
The rotation direction r dir for each rotation-axis candidate 361 may be clockwise (r cw ) or counterclockwise (r ccw ). Based 362 on λ, r dir , and ∆φ, the TFRotObj(·) function returns 363 the transformation matrix T ∆φ for rotating the object by 364 ∆φ with respect to the rotation axis candidate λ and in 365 the direction r dir . If the object collides with the support 366 surface before it is rotated to the limit angle φ obj , that is, 367 the CheckCollision(·) function returns true and θ max 368 is less than φ obj , the current rotation angle returned from 369 the RotObj(·) function is the maximum rotation θ max ; 370 otherwise, the limit angle φ obj is the maximum rotation θ max .

V. OBTAINING OBJECT POSES FOR POWER GRASPING 389
In the previous section, the rotation-axis candidates and their 390 suitability scores were obtained. Herein, we describe the 391 determination of rotation angles and object poses so that 392 space may obtained for power grasping (Algorithm 2). When 393 one robot arm attempts to rotate the target object, the robot 394 precision-grasps the object in the support-grasping pose G s . 395 Then, two robot arms power-grasp the object in the power-396 grasping poses G p1 , G p2 . As noted earlier, G s , G p1 , and G p2 397 are heuristically predefined with respect to the object; G p1 398 corresponds to robot1, G p2 to robot2, and G s to both robot1 399 and robot2 (Figure 1 (a)). has rotated ( Figure 5 (b)). That is, this pose represents the 428 case in which space is secured so that two grippers can 429 power-grasp the object in G p1 and G p2 , when the object is 430 rotated around λ.

432
Although the previously obtained object pose secures space 433 for the grippers to power-grasp the object, it does not ensure 434 that the gripper attached to each robot arm can approach 435 G p1 or G p2 . To determine whether this is feasible, the 436 IKSolPow(·) function yields inverse kinematics solutions 437 C p IK for the two robot arms reaching G p1 and G p2 when the 438 object is in P obj . Specifically, the solutions induce G p1 and 439 G p2 to coincide with the power-point frame that is defined 440 with respect to each gripper base (Figure 5 (a) and (c)). The 441 gripper has a mechanism that enables power-grasping the 442 object when the power-grasping pose G pi (i = 1, 2) coincides 443 with the power-point frame (c1).

444
Because of (c2), to rotate the object from the initial pose to 445 P obj , the gripper should precision-grap the object. Therefore, 446 it is necessary to determine whether the gripper can reach the 447 support-grasping pose G s when the object is in P obj . First, 448 one of the two robot arms is selected to rotate the object. If 449 d g1 > d g2 , then robot1 and robot2 become the master robot 450 arm R mas and the supportive robot arm R sup , respectively; 451 otherwise, R mas and R sup are assigned reversely. Subse-452 quently, inverse kinematics solutions C s IK are calculated for 453 the supportive robot arm R sup reaching G s when the rotated 454 object is located in P obj . That is, we obtain the solutions 455 whereby G s coincides with the precision point frame defined 456 with respect to the gripper base of the supportive robot 457 arm R sup . The IKSolSup(·) function performs the above 458 process and returns C s IK , R mas , and R sup . If both C p IK and 459 C s IK are valid, the corresponding T ∆φ , S rot , P obj , C p IK , C s IK , 460 R mas , and R sup are coupled and added to the set Ω so that 461 they can be used in the next algorithm.

463
In the previous section, P obj was obtained for each rotation 464 axis candidate, and inverse kinematics solutions were derived 465 so that the two robot arms could reach the power-grasping 466 points. Based on the set Ω, herein, the integrated paths for 467 pre-grasp manipulation are planned (Algorithm 3). When the 468 robot arm attempts to reach the gripping location, namely, 469 G p1 , G p2 , or G s , it initially approaches a position slightly 470 away from the gripping location, and then the robot reaches 471 the gripping location. Thereby, collisions between the gripper 472 and the object, which could occur if the gripper moved di-473 rectly to the gripping location, are avoided. If valid integrated 474 paths are obtained, pre-grasp manipulation can be performed 475 using real robots.

5:
if µ rot = ∅ then end foreach 9: end foreach 10: foreach (µ rot , S rot , P obj , C p IK , R mas , R sup ) ∈ M sup do 11: µ rot,first ← FirstPath(µ sup ) 12: µ rot,last ← LastPath(µ sup ) 13: if µ sup1 = ∅ and µ mas1 = ∅ and µ sup2 = ∅ then 18: end if 20: end foreach 21: end foreach path for rotating the object from P obj to P init . In fact, as the 483 supportive robot arm R sup should rotate the object from P init 484 to P obj , the order of the points in the obtained path should be 485 reversed. In Figures 5 (b) and (c), the points of the path µ rot 486 corresponding to P obj and P init are µ rot,last and µ rot,first , 487 respectively.

488
To determine the path µ rot , inverse kinematics solutions 489 corresponding to all the support-grasping poses shown in 490 Figure 5 (c) should be obtained. These poses are specified 491 whenever the object is rotated by ∆φ. To check whether the 492 obtained path µ rot is valid, it is necessary to observe the 493 change for each joint of the supportive robot arm R sup with 494 respect to adjacent points on the path µ rot . If the change in 495 all joint values is sufficiently small, µ rot is valid. An example 496 of path µ rot is shown in Figure 6 (a).

498
If the path µ rot is valid, the first and last configurations of 499 µ rot should be used so that a path can be obtained, whereby 500 the master robot arm R mas can power-grasp the object, and 501 the supportive robot arm R sup can regrasp the object in a 502 power-grasp manner. First, the FirstPath(·) function is used 503 to return µ rot,first , which is the first configuration of the 504 path µ rot . That is, the supportive robot arm R sup precision-505 grasps the object in P init . Using µ rot,first , the ReachPath1(·) 506 function returns the path µ sup1 planned to connect the initial 507 configuration and µ rot,first of R sup (Figure 6 (b)). Next, to 508 power-grasp the object using the master robot arm R mas , 509 the ReachPath2(·) function with c p IK ∈ C p IK is used to 510 obtain the path µ mas1 of R mas that connects the initial 511 configuration and c p IK of R mas (Figure 6 (c)). Then, the path 512 µ sup2 of the supportive robot arm R sup is obtained from the 513 ReachPath3(·) function using c p IK and µ rot,last (Figure 6 (d)); 514 µ rot,last returned from the LastPath(·) function is the last 515 configuration of the path µ rot , that is, the supportive robot 516 arm R sup precision-grasps the object located in P obj .

517
Based on the valid µ sup1 , µ rot , µ mas1 , and µ sup2 , the 518 integrated path for pre-grasp manipulation can be obtained, 519 and finally, power-grasping can be performed by securing 520 space for the grippers (Figure 6 (e)). The integrated path power-grasp these objects firmly, we used thick gripper tips 574 so that sufficient force may be applied against the palm to 575 hold the object even if the volume is small (Figure 8).

577
The proposed method secures space so that two grippers may 578 power-grasp an object in such a way that it is rotated while 579 touching the support surface, which sustains the weight of the 580 object. The rotation axis was not distorted in the simulation; 581 however, in experiments with real robot arms, there are cases 582 in which the rotation axis is distorted as the object is rotated. 583 That is, the contact between the support surface and part 584 of the rotation axis is broken while the object is rotated. 585 This is because the mesh model of the object is manually 586 constructed, and thus the shape of the real object is not 587 accurately represented. In addition, the real object must be 588 placed in the same pose as in the simulation so that it can 589 be rotated using the path derived in the simulation. However, 590 because the actual object is placed manually, an error in the 591 pose may occur. In addition, when the real robot arm power-592 grasps the object, it may move arbitrarily, and thus distortion 593 of the rotation axis can also occur. These factors cause 594 inconsistencies between the rotation axis in the simulation 595 and that in the real world, so that the derived path may not be 596 appropriate for manipulating the object as planned.

597
To address this, the gap between the simulation and the 598 real world should be reduced. To this end, A vision system 599 and the non-rigid registration and pose refinement methods 600 in [23] can be used. In addition, the in-hand pose estimation 601 used in [23] can be applied when the object moves arbitrarily 602 while the robot arm power-grasps the object. Based on this 603 estimation, the actual pose of the object can be calculated 604 such that the modified pose for power grasping can be deter-605 mined. Therefore, the success rate of power-grasping can be 606 improved. 608 We proposed pre-grasp manipulation planning involving two 609 robot arms to secure space for power grasping. One of the 610 conditions is that the gripping force of the precision grasp 611 using only one gripper is insufficient to lift the object stably. 612 Therefore, to obtain space for power grasping, the object was 613 rotated while being supported against the support surface. 614 To this end, rotation-axis candidates that allow the object 615 to rotate while touching the support surface were derived 616 from a mesh model of the object, and the rotation required 617 for the gripper to perform power grasping was calculated for 618 each rotation axis. The robot arm for rotating the object was 619 determined based on the distance between the rotation axis 620 and the power-grasping point of each arm. After the object 621 pose allowing power grasp by two grippers was obtained, 622 the path for rotating the object in this pose using one robot 623 arm was determined. Finally, the path for power-grasping 624 the object using two robot arms was derived. The proposed 625 method was verified through simulations and experiments 626 with two UR5e robot arms attached to a 2F-85 gripper. pre-grasp manipulation planning. Currently, the object is only 633 rotated for power grasping, but the proposed method can be 634 applied to any task in which the robot arm cannot grip an 635 object at a given location.

636
In the future, we intend to automatically, rather than manu-637 ally, determine power-grasping and support-grasping points. 638 Thereby, it is expected that pre-grasp manipulation planning 639 may be more effective, and the proposed method can be 640 easily applied to other objects. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. ERIC DEMEESTER (Member, IEEE) graduated in 1999 as Mechanical Engineer, option mechatronics, at KU Leuven. His PhD research targeted user-adapted plan recognition and shared control for wheelchair driver assistance. Since 2018, he is associate professor at KU Leuven. He is one of the coordinators of the ACRO research group, which focuses on automation, computer vision and robotics in various sectors such as manufacturing, logistics, care, pharma, nuclear, and agriculture. His research interests include probabilistic state estimation, computer vision, collision-free path planning and decision-making under uncertainty, and increasingly also machine learning.