Fruit Detachment and Classification Method for Strawberry Harvesting Robot

Fruit detachment and on-line classification is important for the development of harvesting robot. With the specific requriements of robot used for harvesting strawberries growing on the ground, a fruit detachment and classification method is introduced in this paper. OHTA color spaces based image segmentation algorithm is utilized to extract strawberry from background; Principal inertia axis of binary strawberry blob is calculated to give the pose information of fruit. Strawberry is picked selectively according to its ripeness and classified according to its shape feature. Histogram matching based method for fruit shape judgment is introduced firstly. Experiment results show that this method can achieve 93% accuracy of strawberry's stem detection, 90% above accuracy of ripeness and shape quality judgment on black and white background. With the improvement of harvesting mechanism design, this method has application potential in the field operation.


Introduction
Robot is regarded as a potential replacement to release human beings from monotonous fruit harvest operation. The study of harvesting robot has started 20 years ago, (Naoshi Kondo and KC Ting, 1999) had developed many fruit harvesting robots for tomato, eggplant, lettuce, etc, while the developed prototypes still can not meet the challenging requirements of practical application: speed, accuracy, low-cost, etc. (A.R. Jimenez & R.Ceres,2000) had presented 3 problems to be solved in developing fruit harvesting robot: (a) the guidance of the robot through the crop; (b) the location and characterization of the fruit on the tree; (c) the grasping and detachment of each piece. Some successful sample machines have been reported recently: (Peter P. Ling& Reza Ehsani,2004), (Yu-Tseh Chi & Peter P. Ling,2004)had developed a Robotic Tomato Harvester for continuous, selective picking of mature tomatoes; (Jun Qiao & Akira Sasao,2005) had introduced a mobile fruit grading robot which could realize the harvest and on-line classification of sweet pepper and generate the yield map of fruit. As a variety of delicious and nutritious fruit, strawberry is very popular to Japanese consumers, Kyushu is one of the major strawberry planting areas. While the harvest and classification of strawberry is done manually in Japan, there exists large market requirements for strawberry harvesting robot because of extreme shortage of labor and aging population engaged in agriculture. (Seiichi Arima & Naoshi Kondo,2004), (Naoshi Kondo & Kazunori Ninomiya,2005) had developed some protopotypes of robot that could harvest strawberries grown on tabletop culture. A typical sample machine consisted of a 4 DOF manipulator, an end-effector using sucking force and a visual sensor. As its manipulator, a Cartesian coordinate type was adopted and it was suspended under the planting bed of strawberry. The end-effector could suck a fruit using a vacuum device. The visual sensor gave the robot two dimensional information based on an acquired image In the laboratory of Agriculture Engineering, Image Processing, Faculty of Agriculture, Miyazaki University Japan, some studies related with strawberry quality judgment and sorting system had been performed (Nagata Masteru, 1996,1997. The robot introduced in this paper could harvest strawberries growing on the annual hill top, and ccould be installed on a vehicle and execute mobile harvest task in the green house. In order to avoid any impact to the fruit during harvest process, specific stem-cutting mechanism was designed for fruit detachment : the end-effector consisted of a pneumatic scissor and two pneumatic fingers. In the harvest operation, scissor was used to cut the stem fristly, then pneumatic fingers picked the stem and put the fruit onto the sroting line. In order to make the end-effector work correctly, an effective fruit location and quality judgment method is critical. It should avoid damage to fruit during detachment process and execute selective harvest task according to fruit's appearance. A stem detection and fruit classification method was introduced in this paper in detail: OHTA color space based segmentation algorithm could detect fruit on black and white plastic sheets; Principal inertia axis feature was utilized to define the stem position; Strawberries were selectively harvested according to their ripeness and classified into 2 groups according to their shape features. Ripeness judgment was done under HSI color space, shape histograms were utilized to represent shape feature. The performance of this method was tested in a simulated field environment and experiment results showed that it had application potential of field operation.

Materials 2.1.1 Strawberry Samples
The strawberry samples were selected in the greenhouse of Miyazaki-city's KIBANA area, Japan, its variety was AKIHIME. It could be classified into 5 grades according to its ripeness (immaturity, 40%, 60%, 80%, 100%), and could be classified into 2 grades according to its shape (normal shape fruit and ill-shaped fruit). The strawberry harvesting robot (shown in Fig.1) was developed by Laboratory of Agriculture Engineering, Image Processing, Faculty of Agriculture, University of Miyazaki (Yongjie Cui & M Nagata, 2006, 2007. This robot consisted of XYZ three-axial moving stage (1); pneumatic end effector (2); global camera (3); local camera (4) and controller PC (5). In order to improve the speed of operation, two-camera imaging system was designed for strawberry recognition.

Strawberry Harvesting Robot
One camera called global camera(SONY DXC-151A, 640 480)was installed on the top of robot frame, which could capture the image consisting of 8~10 strawberries; the other camera (called local camera: ELMO EC-202 II, 640 480)was installed on the end effector and could move with the XYZ moving stage. The image taken by local camera often consisted of one or two strawberries. The global camera had a larger view field than local camera and was responsible for the general location of fruits, while the local camera could capture the image of strawberries with high resolution and was responsible for fruit stem detection and quality judgment. This imaging system could decrease fruit searching time and improve harvest speed effectively. A frame grabbing board (PHOTRON FDM-PCI MULTI) was responsible for the image capturing from these two cameras. A PC (SHARP PC-SJ145) was utilized to execuate image processing task. Four fluorescent tubes (TOSHIBA FL20S. D-EDL-D65) were installed around the robot frame as lighting source, which could decrease the brightness disturbance caused by the natural light from outside. A ring-shaped fluorescent lamp (NATIONAL FCL9EX-N) was installed around the local camera to provide the stable lighting envrionment for fruit location and quality judgement. The procedure of strawberry harvest was listed below: 1) The global camera located the general position of every strawberry in its image; 2) With the position coordinates provided by global camera, controller PC sent instruction to XYZ moving stage and made it move to the related position. Then local camera installed on the end effector captured the image of strawberries; 3) Blob based algorithm was utilized to locate the strawberry in the central part of local image; 4) Inertia principal axis of singular strawberry blob was calculated to provide the pose information of fruit. Then quality judgment algorithm would evaluate the ripeness of fruit; 5) If fruit met the harvest standard, robot would execute strawberry harvest operation according to the extracted stem position coordinates.

2 Stem Detection Method for Harvesting Robot 2.2.1 OHTA Color Space based Image Segmentation Algorithm
Segmentation of fruit form image was the foundation of stem detection. In order to extract red strawberries in image, OHTA color space was selected. This color space was introduced by Ohta (Ohta Yu-Ichi & Kanade Takeo,1980), who analyzed more than 100 color features which were thus obtained during segmenting eight kinds of color pictures and found a set of orthogonal color features. Compared with other traditional color spaces, the conversion from RGB to OHTA color space is linear and computation inexpensive. To the imaging system of strawberry harvesting robot, the transfomation time from RGB to OHTA color space was about 50ms, which make it suitable for real-time image processing. OHTA color space has two different kinds of expression as shown in equation (1). In this paper, Fruit segmentation was achieved by a threshold based algorithm with the 2 The image taken by global camera was transformed from RGB color space to OHTA color space firstly, then threshold based method was selected for image segmentation. The threshold value could be set with software interface, which would make the segmentation algorithm work well even if the illumination intensity was different. The approximate center coordinates of every strawberry could be achieved by calculating the average coordinate of the pixels in row and column. Because farmers usually utilize white or black thin sheets to cover the planting bed, so fruit segmentation experiment was performed on white and black background. Planting bed was covered with white and black plastic sheets, images of 20 samples were taken on different background with local camera. The pixels in the image were classified into four groups: red, unripe, green and background. The average value of R-B for each part was calculated, calculation results of 20 samples on different background were shown in Fig.2. Fig.2 (a) showed average R-B value of four parts of 20 sample images on white background, Fig.2 (b) showed the average R-B value of four parts of 20 sample images on black background. As shown in Fig.2

Blob based Singular Strawberry Detection Algorithm
Because of the randomness of strawberry distribution, there often existed more than one strawberries in the image of local camera, so a detection algorithm to extract complete singular strawberry from image background was necessary. A blob based singular strawberry recognition algorithm was designed. The image taken by local camera was transformed from RGB color space into OHTA color space, and fruit parts were segmented from the image; an R-B value R-B value image morphology algorithm was introduced to erase image noises, then blob algorithm was utilized to detect fruit parts in the image; the central blob was defined as strawberry that could be harvested, consequent image processing was executed only within the range of central blob. The procedure and result of strawberry detection was shown in Fig.4.

Inertia Principal Axis based Pose Judgment Algorithm
Because the inertia principal axis and mass center were inherent properties of object, and the direction of its principal axis often represented the orientation information, so the pose of strawberry was approximately represented with the inertia principal axis of strawberry blob. The principal axis was a line that passed through the mass center of the object, around which the first order moment of this object was minimum. The segmented binary image was recognized as a two dimensional inertia object, and two order central moments of binary image were calculated to get the inertia principal axis of fruit. The stem part could be found along the principal axis. The arc tangent value of inertia principal axis' orientation angle could be calculated with equation (2) The head and rear of fruit were defined according to the length and color feature along the longer principal axis. The ends of principal axis with green pixels were potential head parts. According to the fruit shape investigation, the distance from head to fruit mass center was shorter than that from rear to fruit mass center. So the head part of fruit was defined as the shorter semi-axis of principal axis with green pixels on it. The opposite end of principal axis was defined as the rear of fruit. trapezoid shaped range was selected to search more space than rectangle area. The trapezoid range was classified into two parts along the direction of principal axis, the mass center of green pixels in every part was calculated separately, the connected line of two centers represented the position information of stem. The procedure and result of stem detection algorithm was shown in Fig.5. To a fruit harvesting robot, the measurement of distance from detected fruit to the robot's end effector was difficult. While this robot was developed for harvesting strawberries growing on the ground, the height of planting bed was relatively fixed in a green house, so the distance from fruit to the robot was relatively constant, and this parameter could be defined beforehand in the system calibration process. With the extracted image X-Y coordinates, robot could control the manipulator to execute harvest operation.

HSI Color Space based Fruit Ripeness Judgment Method
Ripeness judgment method was developed for selective harvest of fruit, in other words, the robot only picked strawberry whose ripeness was 80% above. Fruit's ripeness was defined as the color ratio between the red distribution area and the whole fruit surface area, which could be calculated with equation (6): in which C represented color ratio, R represented the area of red pixels in fruit image and A represented the area of strawberry in the image, which could be calculated with the sum of segmented unripe part and red part. Because it was difficult to distinguish the unripe part and green leaf in OHTA color space, so HSI color space was selected. This color space was claimed to be the closet approximation to human interpretation of color, the hue feature was independent of illumination variety.
In HSI space, color is represented by hue angle from 0 to 360 , so threshold method could be used to segment red pixels in the image. Red pixels were marked blue in the range of central blob, the detection result was shown in Fig.6.

Histogram Matching based Fruit Shape Judgment Method
Histogram matching method had been proved robust to changes of the object's orientation, scale, or the viewing position by (Bernt Schiele, 2000), and was utilized for strawberry shape judgment in this paper: Shape judgment result was given by calculating and comparing the similarity between the inspected sample's shape histogram and standard template's shape histogram for every grade, inspected sample would be assigned to the grade whose templates had the biggest similarity with it. There existed two major factors influencing strawberry's shape judgment result: side symmetry and rear roundness of fruit. In order to evaluate shape feature comprehensively, two varieties of shape histogram representing fruit's side and rear feature were built respectively. Side symmetry feature was represented by normalized side diameter histogram and rear roundness feature was represented by normalized rear radius histogram. In the image, the central axis of fruit could be defined as the line from fruit's mass center to the mass center of lower part of trapezoid area. In the polar coordinate system defined by central axis and mass center of fruit area, the coordinates of fruit's edge points could be given with the values of angle and related radius. The angles lied in the set [120 , 240 ] were defined as rear range and the angles distributed in the set (60 ,120 ), (240 ,300 ) were defined as side range. The shape histograms extraction process was shown in Fig.7. Histogram similarity measure was a function defined on pairs of histograms to indicate the degree of resemblance of related fruit features. In the calculation of similarity, histogram features were regarded as multidimensional space vectors and the simliarity could be represented with the distance between them. In order to reduce calculation consumption, histogram intersection method was selected (Richard O. Duda & Peter E. Hart, 2000).
The similarity ( ) j i S , between two histograms i h , j h could be calculated using equation (7): in which m was the number of histogram bins; i h was the histogram of inspected sample j h was the histogram of template sample ( ) j i S , was the similarity between two histograms i h , j h .
It ranged from 0 to 1. In the evaluation of shape grade, the side similarity s S and rear similarity r S were compared separately, fruit sample was assigned to the grade whose side similarity and rear similarity were both larger than those of the other grades. This judgment method could avoid the samples with good side symmetry and poor rear roundness being misclassified as normal shaped fruit.

Strawberry Harvest Performance Experimentt
In order to test the harvest performance, growing environment of strawberry was simulated in laboratory.
With 100 strawberries harvested from farmers, the performance of strawberry harvest was tested in the simulated growing environment. The average weight of fruit was 15.7 g; the average diameter of stem was 1.7mm; the average length of fruit was 42.6mm; the average width of fruit was 29.4mm. These strawberries (ten samples a time) were placed on the planting bed (1.3m 0.5m). Five~Six strawberries were placed along the long side of planting bed, the fruits were distributed randomly to simulate the real growing environment (the distance between plants was about 0.1m, the distance between branches was about 0.2m). The stem recognition process (from image capturation to stem detection) was timed with the library function GetTickCount() of WindowsAPI, the fruit harvesting process was timed manually. Experiment result was shown in table.1. It only took about 1 second for the controller PC to give the stem location information, while the developed trial mechanism slowed the harvest process. With optimized harvesting mechanism being able to improve the speed of moving stage and harvesting end-effector, the harvest speed could achieve 10 samples per minute. The average stem length was 8mm, and only 5% of inspected samples were injured during detachment process. The injury was mainly caused by ill-shaped fruit that made principal axis algorithm failed to judge the head of fruit. The stem detection algorithm failed to locate 7% of strawberries because of the occlusion caused by leaves or the strawberries were so close that the blob algorithm could not separate them. So in the future, some artificial intelligence algorithms should be added to improve the harvest performance. In order to test the selective harvest performance, 50 strawberries (40 above 80% maturity and 10 below 80% maturity) were selected manually. These samples were placed specifically so that the stem detection method could work correctly. The experiment proceeded on both black and white background, experiment result was shown in table 2. The result showed that the robot could harvest mature strawberries on black background without mistakes, while missed two samples on white background. The white plastic sheet reflected more light than black sheet, and caused some bright spots appearing on the fruit surface, which would influence the color ratio calculation result. While robot did not pick samples whose ripeness below 80%, which met the requirements of farmers. 60 strawberries (45 Grade one (normal shaped)and 15 Grade two (ill-shaped)) were selected manually for fruit shape judgment experiment. 5 normal shaped samples and 5 ill-shaped samples were utilized as templates, average similarity between inspected sample histogram and 5 templates histograms from each grade was calculated for fruit classification. The samples were placed specifically so that the stem detection method could work correctly. The experiment proceeded on the black and white plastic sheet, experiment result was shown in table 3. From the experiment result, we could see that the histogram matching based shape judgment method could achieve 90% for grade one strawberries and 100% for grade two strawberries on white background, while 92.5% for grade one strawberries and 90% for grade two strawberries on black background. The misclassification error was mainly caused by image segmentation error and there existed no obvious difference on the black and white background. Experiment result proved that this method had high accuracy in shape judgment and was appropriate for field application.

Conclusion
A fruit detachment and quality judgment method for strawberry harvesting robot is presented in this paper. The robot can locate fruit with principal inertia axis based stem detection method; can achieve selective harvest of fruit with the on-line ripeness calculation result and can evaluate the shape quality with histogram matching method. Experiment result shows that this robot can achieve 93% accuracy of strawberry's stem detection, 90% above accuracy of ripeness and shape quality judgment and satisfy farmer's requirements.
The study of harvesting robot is still in its initial stage, future work is focused on improving the robustness of image processing algorithm and designing specific mechanism which can sweep small obstacles on planting bed and redistribute strawberries to reduce the difficulty of fruit detection task.