Detection navigation baseline in row-following operation of maize weeder based on axis extraction

: Detection navigation baseline is primary for the automation of maize weeder in seedling. In the navigation technology based on machine vision, maize seeding or weed near the camera is photographed as a discrete area, while a plant far away from the camera is photographed as a strip area along with other plants in the same row. The two problems cannot be solved by one method. However, in this paper, an algorithm of detection navigation baseline in the row-following operation of maize weeder based on axis extraction was proposed to solve the both problems. Firstly, plants are distinguished from the background based on color feature, and the binary image is acquired. Secondly, plants are described as a set of connected components with numbers after connected components labeling and noise clearing. Thirdly, the axes of all connected components are extracted according to the calculation method of rotary inertia in physics. Next, the abnormal connected components with axes are deleted because the angles between the axes and X -axis are above angle threshold. Then, the judgment model is built based on angle tolerance and distance tolerance, the connected components in a same row based on this model through two-step traversal are merged, and a new axis is re-extracted as the axis of the plant row. Finally, the navigation baselines are detected based on the axes of the plant row. The experimental results show that the accuracy of this algorithm is more than 93%, and the computing time is less than 1.6 s, which can meet the accuracy and real-time performance requirements of maize weeder.


Introduction 
Maize is the largest grain crop in China. Maize crops usually suffer serious damage by weeds in the seedling stage, so the key stage of weeding is in maize seedling [1] . The detection navigation baseline is primary for maize weeder. Since the maize is seeded in rows, it is convenient to detect the navigation baseline of maize weeder based on machine vision, which collects agricultural image information with a camera and extracts navigation baseline with image processing method [2] .
Currently, many research achievements have been acquired in the methods of detection navigation baseline based on machine vision, and the methods are as follows: using K-means clustering algorithm to cluster different plants in the region of interest (ROI) into two rows [3] ; using Hough transformation to detect the boundary frame of continuous row region [4][5][6][7] ; extracting feature pointers of different plants and using the least square method to fit these pointers into lines [8][9][10][11] ; extracting peak pointers of different horizontals lines in super-green feature image and using robust regression method to fit these pointers into lines [12] , and so on. However, for maize seedling, the plant near the camera is photographed as a discrete area, while the plant far away from the camera is photographed as a strip area along with other plants in a same row. The current methods cannot solve the problem where both issues are involved. Therefore, this paper will propose an algorithm of detection navigation baseline in the row-following operation of maize weeder based on axis extraction, and the flowchart of the algorithm is as Figure 1.

Image acquisition
Images were acquired through a color camera Z30C (Daying digital industrial camera, made in Changsha, Hunan, China) with a resolution of 640×480 pixels, a maximum frame rate of 60 per second and a field of view by 46°. The camera's installation height was 1.2-1.5 m from the ground, and its installation angle was 45° to the horizontal direction. The camera was connected to a microcomputer with Intel Core (TM) i5-4590 CPU through a USB 2.0 cable. Image processing was performed on MATLAB R2015a (USA). Figure 2 shows the image captured by the camera.

Image binarization
As is shown in Figure 2, to detect the navigation baseline, it is necessary to take the plant region as the foreground and the other regions as the background, that is, to binary the image. In farmland environment, the plant region especially the maize region in the RGB image has higher green component than red component and blue component regardless of whether the natural light changes, so the color feature can be used to binary the image, and the specific processing method can be shown as follows: where, (x, y) is the coordinate of a pixel; R, G and B represent the red, green and blue components of an RGB image respectively; I BW is the binary image. Figure 3 is the binary image of Figure 2.

Connected components labeling and noise cleaning
Detection navigation baseline is to obtain the position information for agricultural machinery running between different plant rows so the plant regions should be marked with different numbers to facilitate the analysis of plant rows. Connected components labeling is a very common operation for image processing. This paper adopts the method proposed by Robert M. Haralick and Linda G. Shapiro in their book "Computer and Robot Vision" volume I [13] , and the method can be concluded as follows: 1) Encode run-length to the input image; 2) Scan the runs, assign preliminary labels and record label equivalences in a local equivalence table; 3) Resolve the equivalence classes; 4) Relabel the runs based on the resolved equivalence classes.
Based on connected components labeling, a set of connected components with different labels can be used to analyze the plant row information.
However, the image after connected components labeling directly is perturbed by noise, and the area of noise is very small, therefore, in order to clean noise, the area of each connected component should be calculated and compared with area threshold, if the area is less than area threshold, the connected component will be labeled 0 and regarded as background.
In order to show better effects of connected components labeling and noise cleaning, different colors are filled in the connected components with different labels as shown in Figure 4.

Axis measurement based on regional axis theory
In order to detect the navigation baseline, we can calculate the axis of each plant row firstly, and then detect the navigation baseline based on the axis of adjacent plant rows. The axis of each plant row can be calculated according to connected components acquired by section 2.3 in this paper.

Measuring axis of each connected component
All connected components have been labeled in section 2.3 of this paper. Assuming that each point in the connected component is a point of equal mass, the axis of the connected component can be measured by calculating the rotary inertia of the rigid body with a planar mass distribution [14] .
According to the calculation method of rotary inertia in physics, the rotary inertia formula of a rigid body with a planar mass distribution is showed as follows: where, J is the rotary inertia; ds is the area element; r is the distance from any area element to the centroid in the connected component; σ is the areal density. σ can be assigned to 1 because each point in the connected component is assumed as the point of equal mass. In addition, the coordinate of each point in the connected component is known, so Equation (2) can be further represented as follows: where, D is the scope of the connected component; α is the angle between the rotary inertia and X-axis, (°); x and y respectively represent X-coordinate and Y-coordinate of the centroid in the connected component, and the formula of centroid coordinate is given by Equation (4): 10 00 01 00 where, m pq is the original moment of the Hu invariant moment [15] in September, 2020 Feng J H, et al. Detection navigation baseline in row-following operation of maize weeder based on axis extraction Vol. 13 No.5 183 the connected component and the formula is as follows: x y f x y dxdy   (5) where, f(x, y) is the pixel value of the point (x, y) in the image, and the image is the binary image in this paper, so f(x, y) is a Boolean value. For Equation (3), when the rotary inertia J is minimized, the corresponding line which not only passes the centroid but also is at the angle of α to X-axis would be the axis of the connected component. By mathematical derivation, the formula of α is given by 11 20 02 where, μ pq is the central moment of the Hu invariant moment in the connected component, and the formula is defined by Since the centroid coordinate ( x , y ) and the angle  can be calculated, the formula of the axis which not only passes the centroid but also is at the angle of α to X-axis can be calculated, and it is as follows: () where, k is the slope, and k = 1/tanα. If Equation (6) is put to calculate, the formula of the slope k can be derived as follows: where, 11 20 02 calculate, the axis of each connected component can be measured. In order to display the calculation result visually, Figure   The axis of each connected component has been measured in subsection 2.4.1, and the result is showed in Figure 5. As known from prior knowledge, in the image taken by the navigation camera, the angle between the axis of each plant row and the horizontal direction is at most 90°, but may not be too small. However, as known from Figure 5, a plant near the camera is photographed as a discrete area (that is a connected component), while the growth direction of the plant is random, which results in the axis direction of the connected component is random. If the axis direction of the connected component is close to the horizontal direction, it would differ greatly from the axis direction of the plant row, so the connected component is abnormal, and it should be deleted.
The method of deleting abnormal connected components with axes can be concluded as follows: traverse each connected component to judge whether the angle α between the axis of the connected component and X-axis is in the range of the angle threshold if the angle α is in the range, the corresponding connected component would be considered valid and be preserved, otherwise, it would be considered abnormal and be deleted.
As is shown in Figure 6, the connected components are valid after abnormal connected components with axes deleted, and the axes are still white lines with corresponding numbers marked nearby. However, navigation baselines in the row-following operation of maize weeder are detected based on the axes of the plant rows, not the connected components. As is analyzed in the introduction of this paper, when the navigation camera is fixed, a plant near the camera is photographed as a discrete area, while the plant far away from the camera is photographed as a strip area along with other plants in the same row, which results in that the axis of each connected component is not the axis of each plant row. In order to solve this problem, it is necessary to merge the axes of the connected components in a same row.
Firstly, define two concepts, angle tolerance and distance tolerance, where, angle tolerance is the threshold of the angle between the axes of two connected components, and distance tolerance is the threshold of the distance between two connected components. Angle tolerance is a constant, while distance tolerance is a variable, which is in the actual measurement. The adjacent plant rows are approximately parallel, while after imaging by navigation camera, the adjacent plant rows in the image are narrow on the top and wide on the bottom. Therefore, distance tolerance is a variable whose value is a function of Y-coordinate in the image. The case is similar to the problem of thin prism distortion in camera calibration [16,17] , so the function between distance tolerance and Y-coordinate can be built according to the model of thin prism distortion. After repeated derivation and experiments, the function can be represented by where, dis_t is distance tolerance; y is Y-coordinate; k_t is the Y-coordinate squared coefficient, and dis_0 is the initial value of distance tolerance, that is distance tolerance at y is 0. Calibration should be done before the operation of the maize weeder to adjust the parameters k_t and dis_0.
Since angle tolerance and distance tolerance have been defined, a judgment model based on the two concepts can be established. In this judgment model, if the angle between the axes of two connected components is less than angle tolerance, and the distance between two connected components is less than distance tolerance, then the two connected components are thought in a same row, and can be merged. The angle between the axes of the two connected components can be calculated simply. However, the distance between two connected components is the nearest distance between the two connected components. Strictly speaking, the distance calculation needs to traverse all points of two connected components to calculate the nearest distance by comparison, but the computation is too much. In order to simplify the calculation, the sketch of calculating the nearest distance between two connected components is showed in Figure 7. In Figure 7, the closed curves c 1 and c 2 are two connected components, line segments AB and CD are axes of the two connected components, and the Y-coordinates of points A, B, C and D are the maximum or minimum Y-coordinates of the two connected components respectively. Since the Y-coordinates of points A and B are within the Y-coordinates of points C and D, the horizontal projection is made through points A and B respectively, which intersect the points E and F with line segment CD. The lengths of line segments AE and BF are calculated respectively, and the minimum value of them is considered as the closest distance between connected components c1 and c2. Although the approximate calculation is not very accurate, it can be used as the judgment basis for the distance between two connected components, and the calculation amount is greatly reduced.
Therefore, it is achievable to judge whether two connected components should be merged in-row based on the judgment model. However, there may be more than two connected components that satisfy the condition of in-row merging. As shown in Figure 6, the connected components labeled 8, 11, 5, 7, 4, 9 and 10 are possible to be merged in-row. So it is necessary to traversal all the connected components find the connected components that satisfy the condition of in-row merging. A two-step traversal method is proposed as follows: 1) Build an array FLAG, and its size is n × n, where, n is the number of the connected components. The array FLAG is used to mark whether any two connected components satisfy the condition of in-row merging or not, such as the connected components i and j, if they satisfy the condition of in-row merging, FLAG(i, j) and FLAG(j, i) are both assigned to 1, otherwise, they are both assigned to 0. The assignment of the array FLAG is the judgment, which is the first step of two-step traversal, and it is done by two loops; 2) The second step of the two-step traversal is traversing the array FLAG to find all groups of the connected components to be merged (In the following paper, a group of connected components will be abbreviated to a group). The relationship between any two connected components obtained from the array FLAG is assumed to be represented by Figure 8a. As is shown, all connected components are divided into two groups, each group can be approximated as a tree structure, but not as a strict tree structure [18] , for example, node 3 has two parent nodes, that is node 3 has two paths to root node 1, and node 5 and 6 have the same problem too. So the original data structure is improved as Figure 8b, where, each group is a standard tree, and all groups make up the forest. In the second step of two-step traversal, preorder traversal is adopted, and the visited root nodes (including the root nodes of subtrees) are marked so that they are no longer traversed, such as nodes 7 and 6. a. Original data structure b. Improved data structure Figure 8 Data structure in the second step of two-step traversal By two-step traversal, all connected components are divided into M groups, where, M is the number of trees in the second step, and it is also the number of axes after all the connected components merged in-row later. In addition, in each group, all the connected components are the computing region of the new axis. As known in subsection 2.4.1 of this paper, the calculation of the centroid coordinate and the slope is related to the origin moment and the central moment. In the calculation of the origin moment and the central moment, the integral region (Actually in digital image processing, it is the cumulative summation region) should be updated to all the connected components in each group.
After merging and calculating, the new axes are the axes of the plant rows. In the following Figure 9, the white lines are the axes of the plant rows, and there are corresponding numbers marked nearby. Figure 9 Results of main plant rows with marked axes

Detection navigation baseline based on axis extraction
The axes of the plant rows have been extracted in section 2.4 of this paper, so navigation baseline can be detected based on axis extraction. Since the navigation baseline is located in the middle line of two adjacent rows, detection navigation baselines can be described with the sketch of Figure 10. In Figure 10, assuming that r 1 , r 2 and r 3 are three axes of the plant rows, and two horizontal lines l 1 and l 2 which can intersect with all axes are plotted, and the intersection points are marked as A, B, C, D, E and F respectively. After this, the midpoints of line segments AB, BC, DE and EF are measured and marked as G, H, I and J respectively, then the navigation baseline between r 1 and r 2 is the line GI, and the navigation baseline between r 2 and r 3 is the line HJ. Therefore, a detection navigation baseline based on axis extraction can be reached. The result of detection navigation baselines to Figure 2 is shown in Figure 11, where, the navigation baselines are blue lines, and the axes of plant rows are red lines. Figure 11 Results of navigation baselines detected

Results and discussion
In the whole algorithm of detection navigation baseline in the row-following operation of maize weeder based on axis extraction, some thresholds need to be set, and the reasonability of the thresholds directly affects the accuracy of the algorithm, so it is necessary to discuss in following.
In the process of cleaning noise in section 2.3 of this paper, area threshold is set to compare with the area of each connected component, and the connected component whose area is less than area threshold is cleaned as noise. The setting of area threshold is related to the size of the image, and in this paper, area threshold is set to (row/20) × (column/20), where, row and column are the number of rows and columns in the image respectively.
In the process of deleting abnormal connected components with axes in subsection 2.4.2 of this paper, angle threshold is set to compare with the angle between the axis of each connected component and X-axis, if the angle is out of the angle threshold range, the corresponding connected component would be considered abnormal and be deleted. Numerous experiments have shown that it is reasonable to set the minimum value of the angle threshold to π/6 and the maximum to 5π/6.
In the process of merging axes of the connected components in a same row in subsection 2.4.3 of this paper, angle tolerance and distance tolerance are set to be used in the judgment model. If the angle between the axes of two connected components is less than angle tolerance, and the distance between two connected components is less than distance tolerance, then the two connected components are thought in a same row, and can be merged. Numerous experiments have shown that it is reasonable to set angle tolerance to π/4, while distance tolerance is a variable whose function is as Equation (10), where, the parameters k_t and dis_0 should be set before one operation of a maize weeder. The method of undetermined coefficients is used to set k_t and dis_0. The distance between the two adjacent rows at any two Y-coordinates in the image are calculated respectively, and the two groups of data (a group of data is made up by Y-coordinate and the distance) are put into equation (10), then k_t and dis_0 can be calculated. In the experiment shown in chapter 2 of this paper, dis_0 is 39, and k_t is 1/1226. It should be noted that distance tolerance is strictly related to not only Y-coordinate but also X-coordinate. In Figure 11, the navigation baselines and the axes of plant rows in the upper right corner of the image have not been detected, and there are two reasons: 1) The region is far away from the navigation camera, and the different plant rows are linked together as a connected component so that it is impossible to identify these plant rows after imaging; 2) Some adjacent plant rows are not in a same connected component, but in the process of merging axes of the connected components in a same row, since the functional relationship between distance tolerance and X-coordinate is not considered, these plant rows are merged. However, the purpose of detection navigation baseline is to provide agricultural machinery with suitable running lines, so the navigation baselines in Figure 11 are valid and do not affect agricultural machinery operation.
In this study, 5 maize plots in the seedling stage were tested, each plot was tested 10 times, and 100 frames of images were taken successively in each test. The shooting interval of every two frames was 1.6 s. According to the experimental statistics, the computing time of the whole algorithm is 1.4-1.6 s, which is the factual basis that the shooting interval is 1.6 s. The process of merging axes of the connected components accounts for 48%-50% of the total computing time of the whole algorithm. In addition, the accuracy of the algorithm is above 93%, and the average accuracy is 94.67%.
As is presented by section 2.1 in this paper, the camera's field of view is 46°, its installation height is 1.2-1.5 m from the ground, and its installation angle is 45° to the horizontal direction. The installation sketch is shown in Figure 12.

Figure 12 Sketch of camera installation
As shown in Figure 12, in the image captured by the camera, the Y-coordinate range corresponds to the actual photograph distance from point A to B, so, where, θ is the installation angle, θ=45°, and ω is the half of the field of view, ω=23°. If the installation height is 1.2 m, then AB equals 2.4853 m, that is, the navigation baseline that can be detected by taking and processing a single image is 2.4853 m. Furthermore, if the installation height of the navigation camera is 1.2 m, the maximum speed of the agricultural machinery is 5.5919 km/h, while if the installation height is 1.5 m, the maximum speed is 6.9898 km/h. Therefore, our algorithm can match the speed of the maize weeder's accurate operating [19,20] .

Conclusions
In order to detect navigation baselines in the row-following operation of maize weeder in the seedling stage, the procedure of the algorithm in this paper is as follows: 1) Binary the image based on color feature; 2) Label connected components based on the binary image; 3) Clean noise based on the area of each connected component; 4) Extract axes of the connected components according to the calculation method of rotary inertia in physics; 5) Delete abnormal connected components with axes based on the angle threshold; 6) Merge axes of the connected components in a same row based on the judgment model through two-step traversal; 7) Detect navigation baselines based on the axes of the plant rows.
In the maize seedling stage, the plant near the camera is photographed as a discrete area, while a plant far away from the camera is photographed as a strip area along with other plants in the same row. This paper proposed a method based on the regional axis theory to extract the axes of the plant rows in two cases. The novelties of the method are as follows: 1) Rotary inertia in physics is introduced to measure the axes of the connected components; 2) The function expression is built to calculate distance tolerance for a given Y-coordinate; 3) The judgment model based on angle tolerance and distance tolerance is established to determine whether two connected components would be merged in-row; 4) Two-step traversal is proposed to find all connected components that satisfy the condition of in-row merging.
To improve efficiency and robustness, our ongoing and future research is as follows: 1) In the whole algorithm, the process of merging axes of the connected components accounts for 48%-50% of the total computing time, so further optimization is necessary to reduce the computing time; 2) In the function of distance tolerance, the parameters need to be calculated manually before one operation of a maize weeder, therefore, the adaptive setting of parameters is necessary to be further studied.