Vision system of apple picking robot based on twin support vector machine

The key technology of picking robot is visual system, whose accuracy and stability directly and indirectly affect the efficiency of the system. Aiming at the intelligent sorting and grading problem of fruit tree picking robot, a vision system of apple picking robot based on twin support vector machine was proposed. First, the system divided the apples into mature and immature according to the color attributes of the collected samples, and segmented the lesion areas in the image using Gabor wavelet transform to identify the defects in the fruit. Secondly, according to the defect vector constructed by additional color and geometric features, the advanced twin support vector machine is used as a classifier to identify three types of defects. Finally, the performance of the system is evaluated with the indexes of accuracy, specificity, sensitivity and precision. Experimental results show that, compared with other classifier models, the proposed system has the best performance on all performance indexes, can greatly improve the accuracy of the picking robot, and promote the practical progress of the fruit picking robot.


Introduction
With the rapid development of our country's economy and science and technology, the contradiction between the increasing demand for agricultural products and the slow development of modern agricultural productivity has become increasingly acute. The backward agricultural production has seriously hindered the economic development. At present, the domestic fruit harvesting is completed by human resources. With the aging of the population, gradually reducing human resources and increasing production costs, these basic national conditions greatly reduce the advantages of China's fruit market. In this context, agricultural robots came into being.
Because of the convenience and high efficiency of fruit picking robot, it has become an important research direction for scholars. It can not only liberate farmers' labor force, but also ensure the effectiveness of fruit picking. Of course, the advantages of fruit picking robot are not limited to these two aspects, and there are many undiscovered advantages to be developed and utilized. With the rapid development of domestic fruit industry, the market demand of intelligent fruit picking robots has greatly increased, which makes the demand of agricultural productivity improve the application value and practical significance of intelligent fruit picking robots.
The vision system of robot is the key of intelligent picking robot, which is closely related to the significance of eyes to human beings, and how to improve the recognition accuracy and positioning accuracy of picking robot. For example, Jhawar et al. [6] proposed a pattern recognition method based on nearest neighbor and linear regression to determine the maturity of oranges. Ramos et al. [7] proposed a non-destructive method based on linear estimation to classify the number and harvestable fruits of coffee trees. Sofu et al. [8] put forward an automatic system for apple variety sorting and quality evaluation. In this method, two industrial cameras fixed on the conveyor belt capture four photos of a single apple in real time, and classify them according to color, size and weight by classifier algorithm.
However, most of the above methods can only classify the maturity or size, and cannot detect the defective apples affected by staining, scab and decay from the images. Therefore, this paper proposes an intelligent sorting and grading method for processing apples. Accurate classification of mature and immature apples with digital images is a completely nonlinear problem. Support Vector Machine (SVM) [9][10] has been successfully used to realize nonlinear systems characterized by complex decision boundaries, which proves its effectiveness in various pattern classification problems.
Twin support vector machines (TWSVM) is a new machine learning method based on statistical learning theory. As a variant algorithm of traditional SVM, TWSVM inherits its excellent learning ability, but its running efficiency is increased by 4 times. Therefore, this paper uses basic SVM classifier to classify mature apples and immature apples, and uses TWSVM to detect defects. Using the collected apple images and performance evaluation indicators, a specific system experiment was carried out, and the results verified the effectiveness of the system in maturity detection and disease classification.

Apple image acquisition and preprocessing
In this paper, a hybrid contrast stretching method is used to improve the image contrast by using a Top Hat [11][12] filter and Gaussian function, and to capture the target on the image surface to classify apples. Top-hat transform is an advanced morphological operation, which has great potential in extracting micro-structures and fine features from images. The input images were respectively transformed by white top hat and filtered by Gaussian. The Gaussian removes noise and smoothes the image. Let the original RGB image with a size of 256×256×3 be represented, and the white-topped hat filtering image hat ( , ) T s t is given by Equation (1).
Where M is the structure element, which is initialized to the value 9. Image hat ( , ) T s t contains the fine objects needed to classify the apple image. The Gaussian function given in Equation (2) is applied to smooth the image.
Where I is the input RGB image, µ is the average value of the input image, and σ is the scaling parameter.
A differential image n ( , ) T s t is obtained by equation (3).
Use image n ( , ) T s t to calculate the value of average factor. The average factor α is used to further smooth the differential images to eliminate false positives in the classification. This factor is added to the top-hat filtered image to generate new ( , ) T s t as shown in Equation (5), which is contrast enhanced without artifacts.
Finally, the contrast-improved image new ( , ) T s t is added to the Gaussian image to generate f I as shown in equation (6) where the visual contrast of the infected area is enhanced compared to the rest of the image.

Feature extraction
The classification process of the system is divided into two stages, including maturity detection and disease classification, and adopts 24 features, including color, texture, shape and geometric attributes. Subsets of these features are used in different classification stages. The red, green and blue components of the image are represented as R, G and B, respectively, and 18 color features from 1 f to 18 f are extracted from the R, G and B components. The RGB images were converted into gray images and the gray co-occurrence matrix texture features 19 f~2 2 f , i. e. contrast, correlation, energy and uniformity, were extracted. The feature x is extracted from the first moment of the Hu invariant moment of the candidate image. The number of white pixels 24 f is the number of white pixels extracted from the original image by Gabor wavelet transform.

Gabor wavelet transform
Gabor wavelets have been widely applied to classification problems such as feature extraction and texture analysis of digital images [13][14]. The Gabor function is exponentially distributed near 0 y = , as shown in Equation (7).   In the spatial domain, the two-dimensional Gabor function is the product of the complex plane wave and the elliptic Gaussian function. In the frequency domain, the Gabor function is a two-dimensional elliptic gaussian function. In this system, the candidate images undergo multi-directional and multi-scale Gabor wavelet transform, which is sufficient to extract features from the apple surface for disease detection.

Twin support vector machines
As an improved version of traditional machine learning, TWSVM is looking for a pair of non-parallel hyperplanes, so it has better classification ability and is very suitable to solve the problem of sample classification of approximate type. In addition, compared with the traditional SVM, TWSVM performs two SVM-type problem solving, so the computational efficiency is higher. The time complexity of the standard SVM is about , and m represents the number of samples. It can be seen that the computational cost is about 1/4 of that of the standard SVM.
In actual application cases, most data samples are not simple binary classification. If simple linear TWSVM classification is used in the feature space of genomic data studied in this paper, satisfactory classification results can not be obtained. Therefore, for the nonlinear classification problem, that is, the linear non-time-sharing, we need to introduce a kernel function to solve the problem. The training sample set in the dimensional real number space is assumed to be ( , ) m was the number of negative sample points. Then the way to find the nonlinear TWSVM hyperplane is: SVM is basically a binary classifier, which is a powerful tool for nonlinear classification, regression and multivariate function estimation. Its goal is to maximize the distance between the hyperplane and the boundary. The optimal boundary is defined as follows: By solving the optimization problem represented, the values of the parameter w and the parameter m can be obtained.
Where c is the penalty parameter and ε is the error, Implementations of TWSVM are available in the LIBSVM open source library for machine learning. In the proposed system, the apple maturity is distinguished by the basic SVM algorithm, and the disease classification is realized by TWSVM algorithm.

System implementation steps
After the image acquisition processing, the classification processing is performed. The implementation process of the proposed system is divided into two stages, as follows: 1) Maturity detection: this process is performed using a basic binary SVM classifier. A feature vector consisting of 14 features { 1 f~1 4 f , 18 f , 21 f } was constructed from mature/immature apples and was specifically used to determine the maturity of the fruit.
2) Disease identification: this process was used to detect the black spot, ulcer and melanin infection in the defective apple sample. The candidate images are segmented in multi-scale and multi-direction by the Gabor filter to separate the infected area. The extent of the infected area is determined according to the number of white pixels in the segmented area. A feature vector, i.e., a defect vector, is constructed f } to clearly distinguish defects. In this stage, TWSVM is used to train the defect vector and classify the samples.

Experimental data set
The experimental system uses 1394 image acquisition card, as shown in Fig. 1.

Fig. 1 1394 image acquisition card
The 1394 card interface occupies little resources of the host processor when transmitting data, can accurately control the transmission packet length, has no signal loss, is more suitable for multi-channel image acquisition and transmission at the same time, improves the real-time performance of experiments, and well meets the requirements of experiments on a visual system. The camera model is DH-HV1300FC with a measured maximum distance of 0.6m and a resolution of 12801024.
Images of 266 apple fruits, including ripe, immature, dark spots, canker and melanosis fruits, were collected. The training and test sets were constructed in a 1: 1 ratio using these images. The parameter settings for the training and test sets are shown in table 1. The candidate images were subjected to Gabor wavelet transform in four different scales and four directions to segment the infected area without overlapping with the healthy area. Taking healthy and mature images as reference, each test image is subtracted from the reference image to generate a difference image, and the difference image is subjected to Gabor wavelet transform in four scales and four directions.

Evaluation Indicators
For specific analysis, quantified performance metrics [15] were used: Accuracy, Specificity, Sensitivity, and Precision. These metrics are defined as follows: Where TP is true positive, TN is true negative, FP is false positive, and FN is false negative.

Analysis of results
To make a fair comparison, we evaluated the performance of the other three classifier models using MATLAB 2017. The overall performance index comparison of the classifiers is shown in Fig. 2. It can be seen that compared with other classifier models, the proposed system has the best performance in all performance indicators.

Conclusion
This paper presents a vision system of apple picking robot based on twin support vector machine. The classification process of the system is divided into two stages, including maturity detection and disease classification, and 24 features are used for image feature extraction. The basic SVM algorithm was used to distinguish the maturity of apple and TWSVM algorithm was used to achieve disease classification. The test results of four quantified performance indexes verify the effectiveness of the system. The system has the advantages of simple setup and high reliability, and is suitable for a variety of application scenarios in the field of apple picking.