Development of a Tomato Harvesting Robot

Most of commercialized robots are for industry, and the robots for agriculture, forestry and fisheries are under developing and not commercialized yet. The reasons for the difficulty are caused by cost-efficiency of the robotization, safety ensuring of the works using robots, outdoor operations, and knowledge transfer problem from farmers to computer so on. Tomato is one of important fruit vegetables and most tomatoes are produced in the greenhouses, or large-scale farms, where the high temperature and humidity, and long harvest age force the farmers heavy works. With an aim to promote the automation of tomato harvesting, we have organized the tomato harvesting robot competition and developed a tomato harvesting robot. In this paper, we propose the system of tomato harvesting robot.


Introduction
According to the statistical information of the Ministry of Agriculture, Forestry and Fisheries, Japan's food selfsufficiency rate (calorie base) is about 39%, which is the lowest among major developed countries.In the agricultural field, the aging of the producers and depopulation are progressing, and the labor shortage due to the lack of successors is a big problem."Smart agriculture" is proposed as a future image that will solve these problems by introduction of robot as the tools to realize the smart agriculture, is expected 1 .Japan is one of the countries where robots are being introduced to production sites, but many of the robots being put into practical use are automation robots for secondary industries, and the ratio of robots in P -408 agriculture, forestry and fishery fields is few.If we can disseminate agricultural robots, we can expect labor saving, automation, improvement of productivity, etc.Moreover, it becomes possible to convert information such as management of agricultural crop environment, cultivation, state and quality.From the viewpoint of safety, it is possible to provide accurate information to consumers.There are major challenges to overcome, but there is a possibility that a new robot market will be created in the agricultural field.Among various fruits and vegetables, tomato is one of the major fruit and vegetables consuming a lot.Many of the shipped tomatoes are cultivated in facilities such as houses, but due to the high temperature and humid work environment and long harvest period, labor load is large and labor saving is desired.In the Netherlands exporting tomato, various techniques are introduced to cultivate tomatoes and realize higher productivity than Japan.In this paper, we describe our tomato-harvesting robot.
Fig. 1 The tomato-harvesting Robot

Tomato-Harvesting Robot
The system configuration of the tomato-harvesting robot used in this research is shown in Fig. 1 (a), and its external appearance is shown in (b).The robot assumes the house environment of a large-scale tomato production facility.In the house, there are tomato seedlings with multiple bunches, with shelves aligned in line at a certain interval.And along the shelf rails for the bogies, the workers move and harvest tomatoes.This system consists of a Kinect v.2, a USB camera, a sixaxis serial link manipulator, an end effector, a computer, and a moving mechanism that carries them and moves on the rails.As the strategy, 1) Move to the edge of the shelf with a bogie and generate an environmental map of the entire shelf from the acquired image.
2) By analyzing the environmental map, we extract information such as tomato condition to be harvested, its fruit position and total harvest time.
3) Move the dolly to the front of the harvested bunches and harvest the fruits in order.

Tomato Detection Method Using Infrared
Image and Specular Reflection

IR images of Tomato
From the spectral reflection characteristics, it is known that tomatoes have high reflectance for both stems and leaves against infrared light, so in this study IR images were taken in order to detect the fruit of tomatoes .
As a previous study, Fujiura and colleagues focused on this characteristic and developed a two-wavelength 3D vision sensor.In this sensor, the distance is measured by placing near-infrared rays and red lasers coaxially and imaging the reflected light on a PSD (Position Sensitive Device) with a lens.Three-dimensional shape is measured by spatially scanning this.When the laser is incident near the fruit center, the sensor captures specularly reflected light and responds strongly.In this system, the pixel whose spatial response is the maximum is detected as the fruit position.Also, the stem and fruit are distinguished by the ratio of response to red light and near infrared light.This sensor blinks the beam and extracts only the component of the blinking frequency at the light receiving portion, thereby removing the sunlight component which is the stationary light.

P -409
Fig. 2 Dichromatic vision sensor as the previous method.

Proposed Method
With reference to the method in Section 3.1, we propose a method for detecting fruits using IR images and specular reflection.In order to acquire IR images and RGB-D images, Kinect v.2 was adopted as a visual sensor.Since Kinect v2 acquires depth data by irradiating with instantaneous infrared light, the IR image can reduce the influence of sunlight.
Fig. 3 shows a flow of three-dimensional position measurement of the fruit.Kinect gets RGB-D image and IR image capturing the stem of tomato at 30 fps (Fig. 3  (a)).Next, the IR image is processed to detect the position of the fruit (Figs. 3 (b) to 3 (d)).From the pixel information of the RGB-D image corresponding to the detection position, the fruit color determination and the three-dimensional position measurement are performed.
In this report, we propose a fruit detection and discrimination method in the room.The fruit on the IR image shows a concentric reaction with a strong center and a weak perimeter.The gradient orientation (8 azimuth angle) of the IR image is calculated not to depend on the absolute value of the response intensity, and the gradient orientation image (Fig. 3 (b), each orientation is represented by eight color) fruit pattern template Match.We extract only the region with high coincidence rate by the binarization process (Fig. 3 (c)) and label it to detect the position of the fruit.Finally, the correspondence between the red region extracted image obtained from the RGB image and the pixel position of the IR image is taken, and the coloring judgment is carried out (Fig. 3 (d)).Fig. 3 The proposed tomato detection method.

Evaluation of proposed method using real environment image
At the house of Hibikinada Saien Co., Ltd., a largescale production facility, RGB-D images and IR images were acquired using Kinect, and experiments were conducted.47 pairs of images were used for evaluation of the proposed method.The range that the arm reaches, that is, 260 tufts whose distance to Kinect is within 60 cm to 80 cm was targeted.The number of recognition subjects divided by the number of successful fruit detection and coloring judgment in all the fruits was evaluated as the correct answer rate.
As an example, Fig. 4 (a) shows the RGB image in the house and IR image.The box surrounded by the dotted line in the figure is the object to be recognized.Figures 4 (b) and 4 (c) show examples of bunches that succeeded and failed in fruit detection (red in the figure ).The correct answer rate of this method was 88.1%.An example of failure was divided into two, fruits (14 cases) shielding the majority of fruits of the main stem and fruits (3 cases) facing the front.

Conclusions
In this research, in order to realize a tomato fruit harvesting robot, we proposed an infrared image and fruit harvesting technique using specular reflection.In this method, fruit detection in the chamber is carried out by detecting the center of the fruit which causes specular reflection in the infrared image and strongly responds.As a result of evaluation using the real environment image, the correct answer rate was 88.1%.
The main reason for the failure was shielding of the fruit by the main stem, so we plan to add a detection function for covering objects.In addition, we plan to analyze the information such as generation of environmental map of the entire shelf, selection of the harvesting target bunches and estimation of harvest time as an issue within the action strategy.