Visual Servoing based object pick and place manipulation system

: Object pick and place manipulation is essential both in the industrial and service robot field. In this work, we proposed a whole scheme utilizing eye-in-hand visual servoing for object pick and place manipulation system. The proposed paper makes the following three contributions over existing research, 1) a feasible scheme of object pick and place manipulation system is proposed in this work. 2) An efficient structured edge detector is used for contour extraction and object detection. 3) We use the feedback of force sensor to check whether the pick operation is successful.


Introduction
Robotic pick and place automation accelerates the procedure of picking workpieces up and placing them in new locations, which increasing production rates [1,2].Traditional industrial manipulators that able to perform such operations often need to be programmed offline, and the workpieces are placed in the fixed location.Utilizing machine vision and computer vision to enhance the flexibility is critical to improve industrial profitability.Visual servoing is an approach to control the motion of a robot by visual feedback, involving many cutting-edge areas, such as computer vision, robotics, and control theory [3][4][5].Integrating visual servo control system will enable better cooperation between humans and robots, which may enhance the man-machine collaboration.Numerous schemes have been proposed, which can be grouped into position-based methods and image-based methods.The image-based scheme is considered the most effective due to its simplicity of implementation [6][7][8].
To demonstrate the validity of the proposed unified dynamic control strategy, experiments were implemented by using a Baxter robot [9].Baxter Research Robot is a humanoid robot platform with two 7-axis arms, integrated cameras, sonar, torque sensors, and direct programming access via a standard ROS interface [10].Studies on Baxter Research Robot cover human-robot interaction, collaborative robotics, planning, manipulation, control and perception in all applications.
This paper utilize the image-based visual servo system to assist the object pick and place manipulations.This work makes the following three contributions over existing research, 1) a feasible scheme of object pick and place manipulation system is pro-posed in this work.2) An efficient structured edge detector is used for contour extraction and object detection, 3) we use the feedback of force sensor to check whether the pick operation is successful.The remainder of this paper is organized as follows.Section 2 detailed the whole object pick-and-place manipulation framework.In Section 3, edge detection results of two algorithms are illustrated.The conclusion of this work is drawn in Section 4.

The proposed pick and place manipulation system
Figure 1 describes the framework of visual servo system.Visual servo system usually consists of the visual system, control strategy and the robot system.Through image acquisition and visual processing, the visual system obtains appropriate visual feedback.Then the controller gains control input for robot.In specific applications, implementation of the visual servo system strategy needs to be designed according to practical task demand.The flowchart of object pick-and-place manipulation system is illustrated in Figure 2. To place scattered golf balls from the table into a box with separate compartments which we called ball tray.After image are obtained with eye-in-hand camera, image processing and detection algorithms are used for ball tray detection.Edge detection operator and Hough circle methods and visual servoing are employed to detect and locate balls on the table.And then basic planning is used to choose a ball and plan a grip.Finally, an inverse kinematic solution of 7-DOF robot is made to conduct path planning, motion generation and control.Finally, force feedback is employed to check if the ball is successfully grabbed until there is no ball found outside the ball tray.

Camera Calibration
In the process of realizing the pick and place task, the positions of the balls and ball tray must be obtained with sufficient accuracy.With the range sensor integrated on Baxter arm, the distance of the arm to the table can be easily obtained.Combined with hand image acquired from hand camera, the object's position relative to robot can be determined.The calibration board was placed at varying distances from a camera and images analyzed to find the camera calibration factor.The camera calibration factor is calculated as 2.5mm per pixel at 1 meter.

Coordinate frame transformation
The coordinate transformation of eye-in-hand visual system is achieved from the image plane to the robot reference coordinate frame [11].We assume all arm movements are performed with Baxter's arm pointing vertically down at the table, which simplifies the coordinates transformation.With robot's arm pose, the camera calibration factor and the height above the table pixel values can be converted to Baxter coordinates using the following formula: where B represents the Baxter coordinates, p P is the pixel coordinates, and c P are the center pixel coordinates,

B
Pose represents the Baxter's pose, and g O is the gripper offset, cc is the camera calibration factor, and d describes the distance from the table, which can be obtained with the infra-red range sensor on Baxter's wrist.

Locating the Segmented Tray in the workspace
The robot carries out pick and place task under unstructured environment, which is sensitive to illumination and shadow.Edge extraction of ball tray is the prerequisite for locating.Canny edge detector [12] is proved to be the optimal edge detector due to due to good localization and detection results, however it has some drawbacks.More false edges are detected caused by using Gaussian filtering and not adaptive in thresholding.Canny operator use two thresholds, and high and low threshold values need to be manually adjusted with many experiments.Recently Dollar and Zitnick [13] take advantages of the structure present in local image patches to learn both an accurate and computationally efficient edge detector.This novel structured edge (SE) detector achieves high-quality results while being orders of magnitude faster than most competing state-of-the-art methods.In the proposed scheme, we utilize SE detector to find the ball tray.As the tray is the largest bounded region if the bounded regions can be isolated and the largest area will be extracted.The gripper fingers obscure part of the tray.To find the bounded regions the outer region of the edge image is flood filled with white pixels and the areas of the remaining bounded regions calculated.All small areas and all but the largest bounded region are flooded with white pixels.If there is a large bounded area left it is assumed to represent the segmented tray.If there is no bounded area, the arm position is dithered and a new image evaluated.Upon finding a bounded area, the center of the area is found through averaging the pixel coordinates and the arm is moved towards the center of this area.This is performed iteratively until the displacement is below a threshold value.This improves the accuracy of the calculated tray position.The camera gets a clear view of the tray so the gripper does not obscure part of the tray and the camera calibration is more accurate in the center of the image.
Once the ball tray is found in the image, the orientation of the tray and accurate locations of the ball to be dropped need to be determined.A good indication of the position of the bottom left corner of the ball tray is given by the left most pixel in the bounded area and the position of the bottom right corner is given by the lowest pixel in the bounded area.The coordinates of the two corners and the coordinates of the center of the tray can be used to calculate the position of the other two corners and the orientation of the tray.
Given two corners A 1 and A 2 and the center O of a rectangle the other two corners are given by the vector equations: The segmented tray can contain 6 balls.The lengths of the sides of the tray will give the orientation of the tray: Meantime, the accurate locations of each ball can be calculated.

Locating the balls
Shape detection in an image is highly important in practical industrial applications.Many algorithms have been proposed to detect circles.Among these algorithms, Circular Hough transform has been widely used for rapidity and robustness.In this paper, the circular Hough transform is employed to find circles of a given size in an image.
The transform is computed by taking the gradient of in gray scale images.The potential circles with approximately the right size are detected.The selecting strategy is always selecting the left most ball in the image with priority which is on the edge of the cluster.However, to avoid false detection of circles inside the tray due to tray structure, an exclusion area is considered during ball selection procedure.Baxter's arm is iteratively moved over the selected ball and the image analyzed until the arm is above the selected ball to within a given tolerance.

Pick the Balls
There are practical problems when perform the function of picking balls.When gripper pick the selected ball, the balls near the target ball may be touched and moved sideways.Thus Baxter's arm movements should be slowed down when close to the balls.Besides, analyze the neighborhood of the selected ball from the image and rotate the gripper to the best angle to minimize disruption of neighboring balls.Rotating the gripper and moving slowly results in minimal disturbance of neighboring balls.The robot grip a ball and lift it vertically, and then transfer to a position a little height above the free space in the tray and dropped it.

Experiments
We tested our proposed whole system on Baxter research robot.For pick and place operation, we use the stock two fingered parallel gripper with a maximum width (open state) of 75mm and a minimum width (close state) of 37mm. Figure 3 illustrates some intermediate results of eye-in-hand visual servoing.Figure 3a displays the experimental Baxter research robot platform.Figure 3b shows one frame image obtained from eye-in-hand camera.Figure 3c shows the edge detection result using Canny operator, while Figure 3d illustrates edge detection result using SE detector.Figure 3e shows ball tray detection results.Therefore by comparing the edge detection results of two algorithms, we conclude that SE detector is more robust to noises and no need of tuning the parameters.In addition, with the processing speed up to 60 fps, SE detector adequately meets the real-time requirement.

Conclusion
In this paper, a visual servoing based object pick-and-place manipulation system is designed and tested.The structured forest based edge detector achieves accurate results in real-time.Effectiveness of the eye-in-hand visual servoing system has been evaluated through Baxter research robot.However, there are several aspects will be discussed in our further studies: first, employing machine learning algorithms to classify various shapes of workpieces; second, adopting advanced methods to solve the inverse kinematics problem; third, improve the image-based visual servo control scheme.

Figure 1 .
Figure 1.Framework of visual servo control system

Figure 2 .
Figure 2. The flowchart of object pick-and-place manipulation system

Figure 3 .
Figure 3. Eye-in-hand camera based visual servoing.a) the experimental Baxter research robot platform; b) one frame image obtained with eye-in-hand camera; c) the edge detection result using Canny operator ; d) edge detection result using SE detector; e) ball tray detection results