Abstract
In unstructured agricultural environments, accurate fruit detection and picking point localization are important for automatic robotic harvesting. In order to solve the machine vision problem of selective mechanized picking of clustered fruit, this study proposed a novel vision algorithm for target recognition and picking point localization based on the shape and growth characteristics of clustered tomatoes. The picking point has to be determined according to its position and posture relative to the fruit or fruit cluster. The shape, texture and color features of tomatoes were extracted and combined to realize accurate tomato recognition. The precision, recall and accuracy of the recognition model were as high as 100%, and the recognition time was less than 1 s. After the recognition of tomato fruit area, this study further developed a two-step picking point positioning algorithm. Firstly, the co-ordinates and radius of the fruit center of mass was acquired, and then the contour line of the whole bunch of fruits were fitted based on Hough circle detection. Secondly, spatially symmetrical spline interpolation method and geometric analysis were applied for peduncle estimation, contour fitting and picking point location. Experimental results showed that located picking points were distributed on the branches, and the position deviation was small, which met the requirements within a certain positioning accuracy range. This indicated that the proposed method can achieve satisfactory picking-points location effect of clustered fruit in complex environments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Azarmdel, H., Jahanbakhshi, A., Mohtasebi, S. S., & Muñoz, A. R. (2020). Evaluation of image processing technique as an expert system in mulberry fruit grading based on ripeness level using artificial neural networks (ANNs) and support vector machine (SVM). Postharvest Biology and Technology, 166, 111201. https://doi.org/10.1016/j.postharvbio.2020.111201
Bachche, S. (2015). Deliberation on design strategies of automatic harvesting systems: A survey. Robotics, 4, 194–222. https://doi.org/10.3390/robotics4020194
Bai, Y., Guo, Y., Zhang, Q., Cao, B., & Zhang, B. (2022). Multi-network fusion algorithm with transfer learning for green cucumber segmentation and recognition under complex natural environment. Computers and Electronics in Agriculture, 194, 106789. https://doi.org/10.1016/j.compag.2022.106789
Benavides, M., Cantón-Garbín, M., Sánchez-Molina, J. A., & Rodríguez, F. (2020). Automatic tomato and peduncle location system based on computer vision for use in robotized harvesting. Applied Sciences, 10(17), 5887. https://doi.org/10.3390/app10175887
Chaivivatrakul, S., & Dailey, M. N. (2014). Texture-based fruit detection. Precision Agriculture, 15(6), 662–683. https://doi.org/10.1007/s11119-014-9361-x
Cortes, C., & Vapnik, V. (1995). Support vector network. Machine Learning, 20, 273–297. https://doi.org/10.1007/BF00994018
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society conference on computer vision and pattern recognition (CVPR’05) (Vol. 1, pp. 886–893). IEEE. https://doi.org/10.1109/CVPR.2005.177
Dias, P. A., Tabb, A., & Medeiros, H. (2018). Multispecies fruit flower detection using a refined semantic segmentation network. IEEE Robotics and Automation Letters, 3(4), 3003–3010. https://doi.org/10.1109/LRA.2018.2849498
Feng, J., Zeng, L., & He, L. (2019). Apple fruit recognition algorithm based on multi-spectral dynamic image analysis. Sensors, 19(4), 949. https://doi.org/10.3390/s19040949
Fu, L., Duan, J., Zou, X., Lin, G., Song, S., Ji, B., et al. (2019). Banana detection based on color and texture features in the natural environment. Computers and Electronics in Agriculture, 167, 105057. https://doi.org/10.1016/j.compag.2019.105057
Gao, Z., Shao, Y., Xuan, G., Wang, Y., Liu, Y., & Han, X. (2020). Real-time hyperspectral imaging for the in-field estimation of strawberry ripeness with deep learning. Artificial Intelligence in Agriculture, 4, 31–38. https://doi.org/10.1016/j.aiia.2020.04.003
Huang, Z., Wane, S., & Parsons, S. (2017). Towards automated strawberry harvesting: Identifying the picking point. In Annual conference towards autonomous robotic systems (pp. 222–236). Springer. https://doi.org/10.1007/978-3-319-64107-2_18
Hong, X., Zhao, G., Pietikäinen, M., & Chen, X. (2014). Combining LBP difference and feature correlation for texture description. IEEE Transactions on Image Processing, 23(6), 2557–2568. https://doi.org/10.1109/TIP.2014.2316640
Iqbal, Z., Khan, M. A., Sharif, M., Shah, J. H., ur Rehman, M. H., & Javed, K. (2018). An automated detection and classification of citrus plant diseases using image processing techniques: A review. Computers and Electronics in Agriculture, 153, 12–32. https://doi.org/10.1016/j.compag.2018.07.032
Jiang, B., He, J., Yang, S., Fu, H., Li, T., Song, H., et al. (2019). Fusion of machine vision technology and AlexNet-CNNs deep learning network for the detection of postharvest apple pesticide residues. Artificial Intelligence in Agriculture, 1, 1–8. https://doi.org/10.1016/j.aiia.2019.02.001
Jin, Y., Liu, J., Wang, J., Xu, Z., & Yuan, Y. (2022). Far-near combined positioning of picking-point based on depth data features for horizontal-trellis cultivated grape. Computers and Electronics in Agriculture, 194, 106791. https://doi.org/10.1016/j.compag.2022.106791
Liu, S., & Whitty, M. (2015). Automatic grape bunch detection in vineyards with an SVM classifier. Journal of Applied Logic, 13(4), 643–653. https://doi.org/10.1016/j.jal.2015.06.001
Liu, X., Zhao, D., Jia, W., Ji, W., & Sun, Y. (2019). A detection method for apple fruits based on color and shape features. IEEE Access, 7, 67923–67933. https://doi.org/10.1109/ACCESS.2019.2918313
Mao, S., Li, Y., Ma, Y., Zhang, B., Zhou, J., & Wang, K. (2020). Automatic cucumber recognition algorithm for harvesting robots in the natural environment using deep learning and multi-feature fusion. Computers and Electronics in Agriculture, 170, 105254. https://doi.org/10.1016/j.compag.2020.105254
Mehta, S. S., & Burks, T. F. (2014). Vision-based control of robotic manipulator for citrus harvesting. Computers and Electronics in Agriculture, 102, 146–158. https://doi.org/10.1016/j.compag.2014.01.003
Nishad, P. M., & Manicka Chezian, R. (2013). Various colour spaces and colour space conversion. Journal of Global Research in Computer Science, 4(1), 44–48.
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002). Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 971–987. https://doi.org/10.1109/TPAMI.2002.1017623
Otsu, N. (1979). A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 62–66.
Pathan, M., Patel, N., Yagnik, H., & Shah, M. (2020). Artificial cognition for applications in smart agriculture: A comprehensive review. Artificial Intelligence in Agriculture, 4, 81–95. https://doi.org/10.1016/j.aiia.2020.06.001
Raghavendra, A., Guru, D. S., & Rao, M. K. (2021). Mango internal defect detection based on optimal wavelength selection method using NIR spectroscopy. Artificial Intelligence in Agriculture, 5, 43–51. https://doi.org/10.1016/j.aiia.2021.01.005
Saravanan, G., Yamuna, G., & Nandhini, S. (2016). Real time implementation of RGB to HSV/HSI/HSL and its reverse color space models. In 2016 international conference on communication and signal processing (ICCSP) (pp. 0462–0466). IEEE. https://doi.org/10.1109/ICCSP.2016.7754179
Si, Y., Liu, G., & Feng, J. (2015). Location of apples in trees using stereoscopic vision. Computers and Electronics in Agriculture, 112, 68–74. https://doi.org/10.1016/j.compag.2015.01.010
Sun, Q., Chai, X., Zeng, Z., Zhou, G., & Sun, T. (2021). Multi-level feature fusion for fruit bearing branch keypoint detection. Computers and Electronics in Agriculture, 191, 106479. https://doi.org/10.1016/j.compag.2021.106479
Tarel, J. P., & Hautiere, N. (2009). Fast visibility restoration from a single color or gray level image. In IEEE 12th international conference on computer vision (pp. 2201–2208). IEEE. https://doi.org/10.1109/ICCV.2009.5459251
Tang, Y., Chen, M., Wang, C., Luo, L., Li, J., Lian, G., et al. (2020). Recognition and localization methods for vision-based fruit picking robots: A review. Frontiers in Plant Science, 11, 510. https://doi.org/10.3389/fpls.2020.00510
Vougioukas, S. G. (2019). Agricultural robotics. Annual Review of Control, Robotics, and Autonomous Systems, 2(1), 365–392. https://doi.org/10.1146/annurev-control-053018-023617
Wang, D., Song, H., Yu, X., Zhang, W., Qu, W., & Xu, Y. (2015). An improved contour symmetry axes extraction algorithm and its application in the location of picking points of apples. Spanish Journal of Agricultural Research, 13(1), 205.
Wu, J., Zhang, B., Zhou, J., Xiong, Y., Gu, B., & Yang, X. (2019). Automatic recognition of ripening tomatoes by combining multi-feature fusion with a bi-layer classification strategy for harvesting robots. Sensors, 19(3), 612. https://doi.org/10.3390/s19030612
Xiang, R., Jiang, H., & Ying, Y. (2014). Recognition of clustered tomatoes based on binocular stereo vision. In Computers and Electronics in Agriculture, 106, 75–90. https://doi.org/10.1016/j.compag.2014.05.006
Xiaomei, H., Bowen, N., & Jianfei, C. (2019). Research on the location of citrus picking point based on structured light camera. In IEEE 4th international conference on image, vision and computing (ICIVC) (pp. 567–571). IEEE. https://doi.org/10.1109/ICIVC47709.2019.8980938
Xie, Y., Zhang, B., Zhou, J., Bai, Y., & Zhang, M. (2020). An integrated multi-sensor network for adaptive grasping of fragile fruits: Design and feasibility tests. Sensors, 20(17), 4973. https://doi.org/10.3390/s20174973
Xiong, J., Lin, R., Liu, Z., He, Z., Tang, L., Yang, Z., et al. (2018). The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment. Biosystems Engineering, 166, 44–57. https://doi.org/10.1016/j.biosystemseng.2017.11.005
Yan, J., Wang, P., Wang, T., Zhu, G., Zhou, X., & Yang, Z. (2021). Identification and localization of optimal picking point for truss tomato based on mask R-CNN and depth threshold segmentation. In IEEE 11th annual international conference on CYBER technology in automation, control, and intelligent systems (CYBER) (pp. 899–903). IEEE. https://doi.org/10.1109/CYBER53097.2021.9588274
Yu, Y., Velastin, S. A., & Yin, F. (2020). Automatic grading of apples based on multi-features and weighted K-means clustering algorithm. Information Processing in Agriculture, 7(4), 556–565. https://doi.org/10.1016/j.inpa.2019.11.003
Yu, L., Xiong, J., Fang, X., Yang, Z., Chen, Y., Lin, X., et al. (2021). A litchi fruit recognition method in a natural environment using RGB-D images. Biosystems Engineering, 204, 50–63. https://doi.org/10.1016/j.biosystemseng.2021.01.015
Zhao, Y., Gong, L., Huang, Y., & Liu, C. (2016). A review of key techniques of vision-based control for harvesting robot. Computers and Electronics in Agriculture, 127, 311–323. https://doi.org/10.1016/j.compag.2016.06.022
Zhang, B., Gu, B., Tian, G., Zhou, J., Huang, J., & Xiong, Y. (2018). Challenges and solutions of optical-based nondestructive quality inspection for robotic fruit and vegetable grading systems: A technical review. Trends in Food Science & Technology, 81, 213–231. https://doi.org/10.1016/j.tifs.2018.09.018
Zhang, B., Xie, Y., Zhou, J., Wang, K., & Zhang, Z. (2020). State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: A review. Computers and Electronics in Agriculture, 177, 105694. https://doi.org/10.1016/j.compag.2020.105694
Zheng, C., Chen, P., Pang, J., Yang, X., Chen, C., Tu, S., et al. (2021). A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard. Biosystems Engineering, 206, 32–54. https://doi.org/10.1016/j.biosystemseng.2021.03.012
Zheng, W., Guo, N., Zhang, B., Zhou, J., Tian, G., & Xiong, Y. (2022). Human grasp mechanism understanding, human-inspired grasp control and robotic grasping planning for agricultural robots. Sensors, 22(14), 5240. https://doi.org/10.3390/s22145240
Zhu, Y., Zhang, T., Liu, L., Liu, P., & Li, X. (2022). Fast location of table grapes picking point based on infrared tube. Inventions, 7(1), 27. https://doi.org/10.3390/inventions7010027
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Project No. 31901415), and the Jiangsu Agricultural Science and Technology Innovation Fund (JASTIF) [Grant No. CX (21) 3146].
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no know competing financial interests or personal relationships that could have appeared of influence the reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bai, Y., Mao, S., Zhou, J. et al. Clustered tomato detection and picking point location using machine learning-aided image analysis for automatic robotic harvesting. Precision Agric 24, 727–743 (2023). https://doi.org/10.1007/s11119-022-09972-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-022-09972-6