Visual Servo Algorithm of Robot Arm for Pick And Place Application

The robot arm is a device consisting of a moving chain of links connected by joints. Electrical motors are frequently used to move each robot arm joint. An end-effector that can move freely in space is usually attached to one end of the robot platform, which is fixed. Robot arms can do repetitive operations at rates and precision far exceeding human operators. Nowadays, robot arm systems are widely used worldwide to increase the quality and efficiency of the manufacturing process in the industry. Typical applications of the robot arm system are assembly, painting, welding, pick and place operation, and others. Besides, many industries employ robot arms for various jobs such as selecting and putting, painting, and material handling. However, one of the most challenging issues in completing these jobs is determining the target location of the robot arm’s end-effector. There are two different methods for analyzing the robot arm’s movement: forward and inverse kinematic analysis. Based on the visual servo algorithm, this study uses inverse kinematics to execute the pick and place operation. First, an object recognition algorithm is implemented to identify the object to be grasped. Then, an algorithm to avoid any obstacles is done. The study’s findings show that good system performance has been obtained in all three algorithms: first, object recognition algorithm, second, obstacle avoidance algorithm, and lastly, visual servo-based pick and place operation. Thus, it can be concluded that the robot arm’s visual servo algorithm is suitable for pick-and-place applications.


INTRODUCTION
The COVID-19 pandemic has severely impacted factories worldwide, including factories that have been closed to prevent the spread of COVID-19.As a result, the need to ensure the long-term operation of a factory during the COVID-19 pandemic becomes a critical and challenging task (Chen & Lin 2020).
Applying intelligent technology and automation has been considered practical (Chen & Lin 2020).In this regard, industrial robot arms are widely used in manufacturing.Moreover, this robot arm can efficiently perform repetitive tasks or tasks in dangerous and dirty environments.Occasionally, industrial robot arms are activated through teaching tasks (Tsuchida et al. 2021).
Therefore, intelligent mobile robot arms have been widely used in various industries and have capabilities and potential in other sectors, such as the agricultural and plantation sectors (Onishi et al. 2019;Vaghefi et al. 2022).In addition, to recognize objects from a distance from the robot arm's end effector as the target object automatically, sensors such as Lidar or stereo cameras are usually used to obtain 3-dimensional position and rotation by calculating point cloud data (point cloud) or RGB-D (red-green-bluedepth) (Tsuchida et al. 2021;Zheng et al. 2020;Ajmal et al. 2019).
In the industrial field, the performance of detecting the position of a target object through a visual servo algorithm in a robot arm system based on color features is very important (Andhare & Rawat 2017;Shaw & Chi 2018;Suppramaniam et al. 2022).Therefore, the question arises about the best method for a robot arm in a pick-and-place application.Currently, several vision schemes are used to identify the target object through color characteristics and the target object's position.Some vision schemes are (1) monocular camera, (2) binocular stereo vision, (3) laser active visual, (4) thermal imaging, and (5) spectral imaging scheme.In addition, visual servo systems can be divided into two main classes, namely (1) image-based visual servo (IBVS) and (2) position-based visual servo (PBVS).However, each vision scheme has its own advantages and limitations.Thus, the selection of the visual scheme is based on what application is required.In addition, while the robot arm is moving, any other object that becomes an obstacle must be avoided using an obstacle avoidance algorithm (Shi et al. 2019).This algorithm is based on images obtained from commonly used 3D cameras.However, further research needs to be done on integrating those algorithms for pick-and-place applications.
The primary purpose of this study is to (1) develop algorithms to recognize objects for pick and place applications, (2) formulate an algorithm to avoid obstacles for the robot arm while it is moving, and (3) develop algorithms for pick and place applications based on visual servo algorithms.
This study focuses on the simulation of two types of robots, namely the RB-Kairos robot and the Fetch robot used in pick and place applications based on the visual servo algorithm.The software used in this study is Robot Operating System (ROS) software and Python.In addition, the degree of freedom (DOF) of the RB-Khairos robot arm is 6-DOF, while the DOF for the Fetch robot is 7-DOF.In addition, color is a feature used to detect target objects through a visual servo algorithm in this study.Furthermore, only a binocular stereo vision scheme is used, a 3D depth camera that can obtain the 3D position of the detected object (Huang & Cheng 2013;Zhao et al. 2016;Lv et al. 2019).

METHODOLOGY
Figure 1 shows the operation flow chart of this system.This study has several steps to implement the robot arm operating system.Regarding Figure 1, this simulation procedure is carried out by systematically following the flow chart sequence.Based on Figure 1, phase 1 is the flow to build an object detection algorithm using HSV color encoding.After object detection, the position of the targeted object is determined.Phase 2 is the flow to build an obstacle avoidance algorithm to perform motions on the robot arm without colliding with obstacles and their links.Finally, using inverse kinematics, pick and place applications are executed.
In phase 1, color segmentation is required in color classification to detect the specific color of the targeted object.So, HSV encoding is used to encode based on values between 0 to 255.By using HSV color encoding, the image is filtered in the HSV color space.After the filter, the parameter file is obtained and used in the object recognition algorithm.

OBJECT DETECTION ALGORITHM
In Phase 1, a blob detector is used to test the object recognition algorithm.Blob detection methods aim to detect areas in a digital image that differ in properties, such as brightness or color, compared to the surrounding area.A blob is an image region where some property is constant or approximately constant.All points in a blob can be considered similar in some respect.The most common method for blob detection is convolution.Figure 2 is a picture of the Mira robot and the red cricket ball in front of it, and Figure 3 is a preview of the Mira robot looking at the red cricket ball with its camera.Phase 2 is the phase involving the obstacle avoidance algorithm.Motion planning is a technique to find the optimal path that moves the robot gradually from the "start" position to the "home" position.At the same time, it does not touch any obstacles in the environment it passes, nor does it collide with the link of the robot itself.This algorithm uses a 3D depth camera to generate the real world in 3D form and help move toward dynamic obstacles instead of programming objects in the robot's configuration file, i.e., the URDF file.

PICK AND PLACE APPLICATION
In phase 3, which is the pick and place application, a program is used to detect objects from the 3D camera (PointCloud) and generate the grasp sequence required to pick up the object, as shown in Figure 6.In this study, two type of mobile manipulator robot have been used, i.e.Fetch Mobile Manipulator and RB-Kairos.Inverse kinematics, with visual feedback, is used to perform the pick and place task (Ting et al. 2021;Zhang et al. 2018)).First, the object's position can be obtained by reading the robot's sensor data.Then, the vision node provides the position of the object.After the robot detects the object's position (x, y, z), the system performs inverse kinematic calculations.Thus, the robot knows the motion it needs to perform to reach the object's position.

RESULTS AND DISCUSSION
In the object recognition algorithm section, the b lob detection method is used to test the HSV color coding used.
Next, the obstacle avoidance algorithm section shows the algorithm's performance in avoiding obstacles in front of the robot while moving the robot from one position to another.Meanwhile, the pick and place application section displays the robot's performance in picking up and placing boxes on the table without colliding with the table.

OBJECT DETECTION ALGORITHM
In phase 1 of the study, which is to build an object recognition algorithm, the blob detection method was used using the Mira robot to test HSV color coding.Figure 7 shows how Mira tracks a red cricket ball.A red cricket ball can be tracked in the image's center.When the robot is planned to move from the 'start' to the 'home' position, the robot arm may collides with the object in front of it.The collision occurred because the robot did not detect the new object, as shown in Figure 9.To ensure that the robot's RGBD camera can detect new objects placed in front of the robot's arm, RViz is launched, and the point cloud obtained from the camera is visualized, as shown in Figure 10.Once MoveIt is fully loaded, the robot arm's trajectory is planned to move from the 'start' to the 'home' position, where the robot arm did not move, as shown in Figure 12.
The red color indicates that the red joint collides with another object.Therefore, perception is critical in planning the robot arm's trajectory because it allow the robot to know the environment and avoid colliding with other objects.Next, in phase 3 of the study, to develop a pick and place application, objects on the table can be detected through HSV color encoding.Additionally, an obstacle avoidance algorithm must be implemented in the pick-and-place application to avoid collisions with the table when planning trajectories to pick up and place blocks on the table.
Figure 13 shows the movement flow of the Fetch Mobile Manipulator robot, starting from detecting the blue block on the table, then selecting the blue block, and finally placing it in another position on the table.The findings show that the three object recognition algorithms, obstacle avoidance and pick and place applications, have been successfully implemented in this study.

CONCLUSION
In this study, a robot arm with a visual servo algorithm for pick and place applications has been successfully developed.First, an object recognition algorithm is built using HSV color detection.Additionally, the obstacle avoidance algorithm works by using 3D perception.
Finally, the robot arm can pick up and place objects without colliding with any obstacles.
The method developed in this study is carried out through simulation, which may not be suitable for use in the real world.Furthermore, the performance of object detection algorithms, obstacle avoidance algorithms, and pick-and-place applications in experiments using real robots should be carefully considered.
As a suggestion and improvement in the future, the method proposed in this study can be implemented in real hardware.Furthermore, image processing methods need to be studied in the conversion from image distances to real-world distances.Finally, field testing should be done to test the performance of the application pick and place real objects in real environments.

FIGURE 1 .
FIGURE 1. Flow chart of the pick and place application system

FIGURE 4 .
FIGURE 4. Preview of Mira seeing a red cricket ball (before HSV filter)

FIGURE 6 .
FIGURE 6. Fetch Mobile Manipulator and blue box on the table

FIGURE 7 .
FIGURE 7. Preview of red cricket ball detection by Mira robot

FIGURE 8 .
FIGURE 8.A new object is placed in front of the robot arm

FIGURE 9 .
FIGURE 9. Collision of a robot arm with an object

FIGURE 10 .
FIGURE 10.Visualization of cloud points from the camera After the robot's RGBD camera detects a new object, the real environment is created in Rviz, as shown in Figure 11.

FIGURE 11 .
FIGURE 11.New objects detected by depicting the camera's PountCloud

FIGURE 12 .
FIGURE 12. RViz preview when plotting a trajectory to the home position

FIGURE 13 .
FIGURE 13.The simulated movement of the robot arm to pick up and put down in sequence