Design and experiment of visual navigated UGV for orchard based on Hough matrix and RANSAC

The objective of this study was to develop a visual navigation system capable of navigating an unmanned ground vehicle (UGV) travelling between tree rows in the outdoor orchard. Thus far, while most research has developed algorithms that deal with ground structures in the orchard, this study focused on the background of canopy plus sky to eliminate the interference factors such as inconsistent lighting, shadows, and color similarities in features. Aiming at the problem that the traditional Hough transform and the least square method are difficult to be applied under outdoor conditions, an algorithm combining Hough matrix and random sample consensus (RANSAC) was proposed to extract the navigation path. In the image segmentation stage, this study used an H-component that was adopted to extract the target path of the canopy plus sky. Then, after denoising and smoothing the image by morphological operation, line scanning was used to determine the midpoint of the target path. For navigation path extraction, this study extracted the feature points through Hough matrix to eliminate the redundant points, and RANSAC was used to reduce the impact of the noise points caused by different canopy shapes and fit the navigation path. The path acquisition experiment proved that the accuracy of Hough matrix and RANSAC method was 90.36%-96.81% and the time consumption of the program was within 0.55 s under different sunlight intensities. This method was superior to the traditional Hough transform in real-time and accuracy, and had higher accuracy, slightly worse real-time compared with the least square method. Furthermore, the OPENMV was used to capture the ground information of the orchard. The experiment proved that the recognition rate of OPENMV for identifying turning information was 100%, and the program running time was 0.17-0.19 s. Field experiments showed that the UGV could autonomously navigate the rows with a maximum lateral error of 0.118 m and realize the automatic turning of the UGV. The algorithm satisfied the practical operation requirements of autonomous vehicles in the orchard. So the UGV has the potential to guide multipurpose agricultural vehicles in outdoor orchards in the future.

1 Introduction  At present, agricultural autonomous navigation research mainly includes three methods, light detection and ranging (LiDAR) [1][2][3] , global navigation satellite system (GPS) [4,5] , and machine vision [6,7] . The problem with 2D lidar is that it has a small angle of view and has high requirements for the symmetry of crop rows and tree rows. The manufacturing process of 3D lidar is complicated and the cost is high [8] . The dense canopy of the orchard blocks the satellite signal, so there are some problems in GNSS navigation, which limits its application in complex farmland environments [9] . Agricultural machinery navigation technology, based on machine vision, has drawn increasing attention in the last decades due to the advantages of wide signal detection ranges, complete information, noncontact measurements, and high-cost performance. Navigation path acquisition, coupled with the fitting method of navigation path, is one of the research hotspots. The method of image segmentation is one of the key points of path acquisition in visual navigation [10,11] . In the study of Yao et al. [12] , the mean image of the pixel value in five channels (R, G, B, S , and V) were treated as the inspected image and the feature points of the inspected image were extracted by the Canny algorithm, which would provide a reliable path for visual navigation. Sumesh et al. [6] used a Red-Green-Blue (RGB) camera for sugarcane yield estimation with minimal field dataset. A region of interest was detected by horizontal projection method to extract tree trunks. Meng et al. [7] selected Cg component of YCrCg color model for subsequent image processing to reduce the adverse effects of light change on image segmentation and navigation line extraction. The fuzzy C-means clustering method (FCM) based on two-dimensional histogram is used for Cg component segmentation, so as to identify the green crop. Chang et al. [13] presented an improved HSV-based image segmentation algorithm to decompose the captured image, where morphological operation is employed for foreground target extraction. It is important to fit navigation paths for accurate real-time control of visual navigation, and accurate acquisition of navigation paths is critical for fit navigation paths. Compared with other color spaces, HSV space can directly express the brightness, hue, and brightness of colors, which is convenient for color comparison. After image segmentation, path fitting is performed to get the final navigation path. Ma et al. [14] used the least square method to detect the navigation line to improve the real-time performance of the system. In order to extract the root row lines, Yang et al. [15] used the least square method to fit the positioning points which were obtained by calculation. In other different environments, the accuracy of the navigation centerline extracted by this algorithm is above 92%. The least square method has a fast detection speed for crop line fitting, but it is sensitive to noise, so it is difficult to extract crop lines accurately when the environment noise is high [16,17] . The outstanding advantage of Hough transform is robustness [18][19][20][21] . Barawid et al. [22] used a Hough transform as the algorithm to recognize the tree row. It was possible to navigate the robot tractor autonomously between the orchard row crops. In order to improve the adaptability of the crop row recognition algorithm for different kinds and growth periods of vegetables with machine vision, Chen et al. [23] studied on a method based on automatic accumulation threshold of Hough Transformation performed to fit straight lines for all feature points in the image coordinate system. Chen et al. [24] proposed prediction point Hough transform fitting the navigation paths to solve the problem of poor real-time performance caused by the wide range of the traversal angle and intersection point of traditional Hough transform. Random Sample Consensus (RANSAC) can improve the performance of computational efficiency, reliability, and accuracy in high-noise environments [25] . Zhou et al. [26] used the RANSAC algorithm to fitting the DBH of the trees in the plane. The DBH of the 71 trees in the forest area was fitted and extracted using RANSAC algorithm. The overall error satisfied the requirements of forestry mapping. Zhu et al. [27] got 151 pairs of refined cotton matches by using the RANSAC algorithm, and there were no wrong matches in the refined matches. which helped make the results of cotton three-dimensional (3D) reconstruction more accurate. To improve the robustness and accuracy of the feature extraction, Sun et al. [28] studied on an improved Features from Accelerated Segment Test (FAST) feature extraction based on RANSAC method to remove the mismatched points. The results showed that the improved FAST feature extractor based on RANSAC method can not only reduce the mismatched points significantly, but also enhance the positioning accuracy of the vision/SINS integrated navigation system at the cost of little computation load.
Although the color model and random sampling algorithm have been respectively used to acquire and fit the navigation path, the improvement of real-time performance and matching accuracy is in great demand. Previous studies mainly focused on the use of visual navigation techniques to identify ground structures in the orchard [29][30][31][32] , while less attention has been paid to the combination of canopy plus sky. In order to achieve the purpose of protecting fruit trees in the process of driving, this study designed a method based on the sky background and took the canopy as the navigation standard. The objective of this study was to develop the unmanned ground vehicle (UGV) based on visual navigation that is less vulnerable to interference from the outdoor environment simultaneously, which satisfies the practical operation requirements of outdoor orchard vehicles in different sunlight intensities.

Platform of the UGV
In order to solve the problem of weeds and soft soil in the outdoor orchard, the UGV adopted crawler chassis. The diagram of the UGV (Figure 1) has six main components. The laptop (Intel i7, 3.20 GHz) used Matlab 2019a to design GUI and process images. The USB WebCam and OpenMV were used to estimate the current vehicle position. The STM32 controlled the DC motor drive unit by controlling the DC motor controller to correct the position of the UGV. This UGV is driven by two twelve-volt DC motors connected directly to each of the two drive wheels. Figure  1 shows the UGV platform with its components.

Figure 1 UGV platform and its six components
As the vehicle moved between trees, the USB WebCam captured the navigation path in the background of the sky and canopy. The laptop calculated the control angle by navigation path information. Pulse-Width Modulation (PWM) signal was transmitted by STM32 to control the DC motor driver to correct the position of the UGV. When OpenMV obtained the turning sign at the end of the orchard, PC controlled the vehicle to perform the turning operation which was designed in advance.

Control design of the UGV
The motion characteristic of the UGV is a two-wheel differential drive. The rotation of two isomorphic driving wheels at the rear end provides power for the UGV. In the driving diagram of the UGV (Figure 2), v l and v r are defined as the linear speed of these two points in the vehicle body coordinate system moving in the inertial coordinate system. The values can be obtained from the angular speeds ϕ l and ϕ r output from the motor drive interface and radius r of the driving wheel, as expressed in Equation (1). (1) where, v l is the linear speed of the left track, m/s; v r is the linear speed of the right track, m/s; ϕ l is the angular speed of the left driving wheel, rad/s; ϕ r is the angular speed of the right driving wheel, rad/s; r is the radius of the driving wheel, m.
Taking the midpoint of the central line of the two driving wheels as the reference point C(x, y) of the UGV, the instantaneous angular velocity ω c of UGV can be calculated as: where, l is the distance between the centers of the two tracks, m.
Note: vl and vr define the linear speed of these two points in the vehicle body coordinate system moving in the inertial coordinate system, m/s; C(x, y) is the reference point at the midpoint of the central line of the two tracks of the UGV; l is the distance between the centers of the two tracks, m. Figure 2 Driving diagram of the UGV The operating system of the UGV consists of four main units: the detection unit, the control unit, the processing unit, and the execution unit ( Figure 3). The detection unit obtains the left and right wheels' speeds through the speed sensor, OpenMV gains the left and right turning information, and the camera acquires the navigation path information. The control unit has the following functions: estimate the Course Angle based on the left and right track speeds, conduct path planning according to the acquired path, determine the turning operation according to the turning information, and determine the drive output according to the actual motion trajectory and the offset angle of the target trajectory. The processing unit conducts the hierarchical control function using STM32. Finally, STM32 sends the PWM signal to the execution unit. The execution unit controls the motor for navigation. In this study, the image formed by canopy plus sky was used for visual navigation recognition, and the impact of environmental factors can be greatly reduced.
The difficulties such as inconsistent lighting, shadows, and color similarities in features were eliminated by using the sky-based approach where the image was reduced to canopy plus sky, thus simplifying the segmentation process [33] . For visual navigation in outdoor orchards, in terms of hardware selection, USB WebCam was used to obtain images in order to improve the universality and economy of the system. With a 135° view in the outdoor orchard sky, it was impossible to accurately identify the turning sign at the end of outdoor orchards. OpenMV was added as a secondary sensor to capture the turning information of the navigation system. The overall software system of the UGV is shown in Figure 4. First, it is judged whether the turning sign is obtained from the images that are collected by OPENMV. And then the left/right turning program drives the UGV to perform a turning operation. When the turning information is not obtained, the outdoor orchard path image is collected by Matlab. The image is preprocessed by image segmentation operation and morphological operation. Next, the center point of the path in the image is obtained and the redundant points are removed by Hough matrix. Optimal feature points are obtained by removing outliers by the RANSAC algorithm. Finally, the UGV performs the path tracking operation through the fitted path line.

Methods of navigation path acquisition 2.2.1 Image acquisition
Because of the multiple influencing factors such as seasonal weeds and irregular potholes in the orchard, this study used the image that was formed by canopy plus sky, as illustrated by the yellow line in Figure 5, as the basis for the extraction of the baseline road. The image acquisition mode was triggered by the Matlab software, and when an image was acquired, the next image was immediately acquired after the navigation baseline was extracted.
Note: The two yellow lines are the ideal dividing line between the sky and the canopy. In order to extract the information of the canopy accurately, this study compared three color models (RGB, Lab and HSV). As shown in Figure 6, for the component that determines the green Since there were many small black noises, white voids, and irregularly shaped crown edges in the binary image after image segmentation, the opening and closing operations were performed by the octagon with a side length of 3 pixels. Finally, a smooth binary image was obtained from morphological image processing. 2.2.3 Determination of path reference point 1) Acquisition of path reference points Because of the irregular growth of outdoor fruit trees, this study used the vertical coordinates of the picture to extract the center points of fruit trees and the edge of the sky as the reference points of the navigation path. Assuming that the original image size is U×V, V and U represent row pixels and column pixels of the image, respectively. Then the number of horizontal bars is V, and the pixel coordinate point P(i, j) of the feature edge acquired in line i is the pixel value here, as expressed in Equation (4).
The feature point of the final middle path is the midpoint of the edge feature point in each row, as expressed in Equation (6). This study used these feature points as reference points for navigation paths, which can effectively avoid damage to fruit trees in the process of driving. The result is shown in Figure 9. 2) Reference points are optimized by Hough matrix Hough transform is a method to find and link line segments in an image based on Hough matrix. The basic idea is to concentrate the points on the line in the image to a point in the parameter model to form a local peak. Hough transform makes use of the global characteristics of the image, which is less affected by the boundary fracture, having strong robustness and anti-noise ability.
In this study, Hough matrix was used to optimize the feature points. Firstly, a two-dimensional accumulation array A(a, b) was established. The range of the first dimension is the possible range of the slope of the line in the image coordinate model, and the range of the second dimension is the possible range of the cut moment of the line in the image coordinate model. In the beginning, array A(a, b) is initialized to 0. Then for each former scenic spot (x i , y i ) in the image coordinate model, the corresponding value of b can be expressed as: where, a is the discrete value in the parameter model. Every time a pair of (a, b) is calculated, the corresponding array element A(a, b) is added with 1. After all the calculations, the maximum peak value of A(a, b) is found in the voting result of the parameter calculation. The corresponding a 0 and b 0 are the parameters of the linear equation with the largest number of collinear points (a, b) in the original image. And save the key feature points (line_point1 and line_point2) that determine the straight line. Next, it is can continue to find the second peak value, the third peak value, and the fourth peak value, etc., which correspond to the lines with slightly fewer collinear points in the original image. The final Hough matrix is calculated as shown in Figure 10 When the value of 50 peaks is calculated, the value of the elements in the neighborhood is set to 0 so that other interference points and the duplicate points near key peaks are removed. The peak value less than half of the maximum peak value is removed so that only the key feature points are retained. Finally, the coordinate points marked with red "+" are left as shown in Figure 11. Note: The red "+" are the key feature points of the navigation path, the same as below. Figure 11 Key feature points of the orchard path This method retained the features of path feature points. However, there were interference areas caused by irregular tree growth as shown in Figure 11, resulting in wrong key peaks.

The fitting method of navigation path
In order to solve the influence of interference areas caused by different tree growth, this study used the RANSAC algorithm to estimate the parameters of the mathematical model from a group of observed data iteratively containing outliers. This algorithm assumes the data containing correct data and abnormal data. The correct data is recorded as inliers and the abnormal as outliers. The core idea of the algorithm is randomness and hypothesis. Randomness is to randomly select sampling data according to the probability of correct data occurrence. According to the law of large numbers, randomness simulation can approximate the correct results. A hypothesis is to assume that the selected sample data are all correct data, and then use these correct data to calculate other points through the problem satisfaction model, and then make a score for this result. The procedure for this method is as follows. First of all, to get a linear model, two points are needed to uniquely determine a linear equation. So in the first step, two points are selected randomly. Through these two points, the model equation can be calculated. Then all the data points are put into this model to calculate the error and determine the number of points that satisfy the error. Finally, repeat the previous process until a certain number of iterations is reached, and then select the most supported model as the solution to the problem. The inliers are points for supporting the most supported model and the outliers are deleted by the most supported model.
In this study, t is assumed to be the probability of the inliers in the whole data set, as expressed in Equation (8). where, n inliers is the number of inliers of the most supported model; n outliers is the number of outliers of the most supported model; t n is the probability that n points are inliers. P is the probability of the correct solution, as expressed in Equation (9).
where, n is the number of points needed by the model, m is the number of iterations. t is a prior value that can be set. It can also be seen that P can be guaranteed by increasing m, it can be calculated as: After determining the values of n, t, and m, the key feature points are brought into the calculation to obtain the model. But the straight line of the obtained path is the path formed by canopy plus sky. So the angle θ 0 of the navigation path can be calculated as: where, k 0 is the slope of the model calculated by using RANSAC.

Turning information recognition
Since the path separates from the sky background, it is impossible to accurately determine the turning information at the end of the road. Based on the OpenMV visual processing chip, the OpenMV IDE tool was used to determine the threshold and recognize turning information. When the size of the identified area (blue area) was greater than a certain value, the navigation system was at the turning point. Then four equally green boxes were used to segment the center position of the different areas, as shown in Figure 12. The final turning information is determined by its positive and negative values.

Evaluation of navigation path acquisition
Images of the outdoor orchard were acquired for validating the effectiveness of the proposed algorithm in this study. The images in the outdoor orchard were acquired at different times in one day. In the morning, noon, afternoon, and late in the afternoon, the illuminance and color temperature are different, so this study used the trial and error method to get the appropriate threshold of the H component is 0.15-0.42. Navigation path information measurement was conducted in normal sunlight intensity, which was further used to compare with the experimental results for validating the performance of the proposed method in this study. As shown in Figure 13, because Hough transform method was more concerned about the uniform collinearity of local feature points in detecting the line features, some regional points in the path become lines separately, so the path obtained through Hough transform method had a large error. Compared with Hough transform, the least square method could solve the problem of incomplete line detection. Moreover, based on the principle of minimum residual error, the detection accuracy was guaranteed. The least square method was to find the global minimum directly for derivation, which had better global property, but it was also easy to be affected by error points. In this study, Hough matrix was used to eliminate the redundant points, and then the RANSAC algorithm was used to eliminate the outliers (wrong points). Finally, it could be observed from figure 13 that the path fitting by Hough matrix and RANSAC could more fully reflect the information between rows of orchard trees.
This method improved the robustness of road fitting. The experimental results of different sunlight intensities and backgrounds showed that the proposed method has good universality. Under three different sunlight intensities, Hough transform continuously searched for the peak value through the two-dimensional accumulation array established by slope and intercept and finally obtained the line with the most collinear points. The least square method determined the linear equation by calculating the least square sum of errors from all points to the line. Table 1 shows the equations and time-consuming of fitting lines of different methods for navigation path fitting. The average time of traditional Hough transform to fit the navigation path was 0.73 s. The average time of the least square method to fit the navigation path was 0.32 s. The average time of Hough matrix and RANSAC method proposed in this paper to fit the navigation path was 0.49 s. Compared with the traditional Hough transform [21] , Hough matrix and RANSAC method saved 0.24 s. In summary, Hough matrix and RANSAC method was superior to the traditional Hough transform in real-time and accuracy, and had higher accuracy, slightly worse real-time compared with the least square method. 60 pictures under different backgrounds and sunlight intensities were used as experiment samples. Four parameters such as target pixel ratio, time-consuming of navigation path acquisition, average time-consuming of navigation path acquisition and average recognition accuracy rate were calculated under different pictures. The recognition accurate rate was defined according to the degree of coincidence between the slope of the obtained line and the slope of the line under normal sunlight intensity. The target pixel ratio was the percentage of the number of orchard pixels and the total number of image pixels. The statistical results of the method for 20 samples under each sunlight intensity were summarized, as shown in Table 2. It was observed that the time-consuming of navigation path acquisition is within 0.55 s. And in the case of strong sunlight intensity, the average time of the method was almost the same. The average recognition accuracy rate of this method was 95.73%-96.81% and 90.36%-90.36% in strong sunlight intensity and low sunlight intensity, respectively.
However, Low sunlight intensity increased the difficulty in acquiring high-quality images, thus reducing the overall matching accuracy. These results showed that Hough matrix and RANSAC method still has certain stability under different background and sunlight intensities.

Evaluation of turning information recognition
In order to experiment with the effectiveness of the turning recognition method, 40 pairs of turning information images were acquired by the OpenMV. Through the recognition of the turning information, it could be seen that this method can recognize the turning sign, as shown in Figure 14.
This study summarized the recognition results of the method for different angles. It is observed from Table 3 that the average time-consuming of recognition and Average recognition accurate rate for Turning information recognition were 0.17-0.19 s and 100%, respectively. These results showed that the proposed method could accurately recognize the turning information, which provided a fundamental basis for the visual navigation system.  Figure 14 Turning information recognition

Evaluation of the UGV
Before the experiment, the coordinates of the starting and ending points of the driving path in the orchard were obtained through GPS produced by Shanghai Sinan Satellite Navigation Technology Co., Ltd., and the equation of the standard driving path was determined by the coordinates. Figure 15 shows the Graphical User Interface (GUI) of the visual navigation system developed by Matlab 2019a. The orchard path and its slope were shown in the navigation GUI. When the "Start" button was clicked, the visual navigation system started to obtain the navigation path and outputted control signals. During navigation, the UGV drove at speed of 0.5 m/s. As shown in Figure 16a, GPS recorded the trajectory of the UGV in three-dimensional coordinates. From the trajectory mapped in the X-Y plane, it could be seen that the initial visual navigation process was relatively stable, but it was difficult to continue the straight-line navigation quickly after the turning operation. Figure 16b shows the lateral error of the UGV from the standard driving path. It indicated that the visual navigation system was effective in guiding the UGV in the outdoor orchard. For the orchard experiment, the UGV had a maximum lateral error of 0.118 m from the standard driving path where the width of the whole row was 3.20 m. It could be seen that the relative error of the width of the row was very small. At 19.42 s, OpenMV recognized the turning information and controlled the UGV to perform the steering operation. Experiments show that the vision navigation system designed in this study can realize the UGV's autonomous full coverage walking in the orchard without damaging the fruit trees.

Discussion
In this study, different path fitting methods affected the accuracy of the UGV navigation path. Although the robustness, accuracy, and delay of different algorithms affect the visual navigation effect in varying degrees, the appropriate algorithm can still meet the requirements of outdoor orchard navigation. Based on the experiment results, it was determined that Hough matrix and RANSAC method for the visual navigation system was adequate for guiding the UGV down the orchard row. The interference factors such as inconsistent lighting, shadows, and color similarities in features were eliminated by using the image that is formed by canopy plus sky, thus simplifying the segmentation process.
The proposed method based on Hough matrix and RANSAC for orchard navigation demonstrated the potential of guiding the UGV. However, there were obvious limitations to the proposed method. First of all, the sky-based method could only be used in the spring and summer seasons when trees have leaves or canopies year-round such as citrus. Secondly, because the path fitting method based on Hough matrix and RANSAC only worked well in standard orchards which are planted neatly, this method had some limitations for orchards. Finally, this study used OpenMV to identify the turning sign, which increased the complexity of the software program and the instability of the navigation system. In the future, something can be done to optimize the method for these problems, and to constantly improve the performance of the system.

Conclusions
This study developed the visual navigated UGV for orchard based on Hough matrix and RANSAC. The UGV was controlled by two DC motors to control the crawler chassis, and it used the ordinary USB WebCam and the low-cost OPENMV to obtain the outdoor orchard environmental information. The following issues have been resolved.
1) For orchard trees are composed of green, the HSV color model was proposed to do image segmentation. Experiments showed that the valid threshold can be used to accurately extract the target path of canopy plus sky under full-day.
2) In view of the dense weeds in the outdoor orchard and the complex and changing environment, the natural path of identifying the canopy of the outdoor orchard was used as the basis for path identification. Experiments showed that this method reduces the impacts such as inconsistent lighting, shadows, and color similarities of the outdoor environment.
3) According to the path recognition error that is caused by the unevenness of the canopy, an improved method combined with Hough matrix and the RANSAC algorithm was proposed. This method eliminated the influence of the outliers (wrong points) caused by the abnormal crown. Through the experiment and comparison, it is concluded that Hough matrix and RANSAC method is superior to the traditional Hough transform in real-time and accuracy, and has higher accuracy, slightly worse real-time compared with the least square method. 4) We incorporated the auxiliary OPENMV image recognition chip to make the UGV autonomously turn during operations and improve the integrity and applicability of the navigation system. The outdoor orchard field experiment showed the UGV achieves autonomous navigation in outdoor orchards, which satisfies the practical operation requirements of orchard management machinery. So the visual navigation system that was developed in this study has the potential to guide multipurpose agricultural vehicles in outdoor orchards in the future.