Skip to main content
Log in

Low and high-level visual feature-based apple detection from multi-modal images

  • Published:
Precision Agriculture Aims and scope Submit manuscript

Abstract

Automated harvesting requires accurate detection and recognition of the fruit within a tree canopy in real-time in uncontrolled environments. However, occlusion, variable illumination, variable appearance and texture make this task a complex challenge. Our research discusses the development of a machine vision system, capable of recognizing occluded green apples within a tree canopy. This involves the detection of “green” apples within scenes of “green leaves”, shadow patterns, branches and other objects found in natural tree canopies. The system uses both thermal infra-red and color image modalities in order to achieve improved performance. Maximization of mutual information is used to find the optimal registration parameters between images from the two modalities. We use two approaches for apple detection based on low and high-level visual features. High-level features are global attributes captured by image processing operations, while low-level features are strong responses to primitive parts-based filters (such as Haar wavelets). These features are then applied separately to color and thermal infra-red images to detect apples from the background. These two approaches are compared and it is shown that the low-level feature-based approach is superior (74% recognition accuracy) over the high-level visual feature approach (53.16% recognition accuracy). Finally, a voting scheme is used to improve the detection results, which drops the false alarms with little effect on the recognition rate. The resulting classifiers acting independently can partially recognize the on-tree apples, however, when combined the recognition accuracy is increased.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. The total number of images used is 146 (training) + 34 (testing) = 180 images. This set of images was described in the beginning of the “Materials and methods” section.

  2. We found that using 146 IR images for training did not include enough apples. The number of apples in the IR images is much lower than the RGB, since the resolution is about 10 times lower. Therefore, a detector trained with approximately 10 times less apples would have had a disadvantage with respect to the RGB detector. Therefore we decided to increase the dataset to another 106 images to have a comparable number of apples between the two detectors.

References

  • Annamalai, P., & Lee, W. S. (2003). Citrus Yield Mapping System Using Machine Vision. Paper number 031002, ASAE, St Joseph, MI, USA.

  • Baeten, J., Donné, K., Boedrij, S., Beckers, W., & Claesen, E. (2008). Autonomous fruit picking machine: A robotic apple harvester. Springer Tracks in Advanced Robotics, 42, 531–539.

    Article  Google Scholar 

  • Brown, L. G. (1992). A survey of image registration techniques. ACM Computing Surveys (CSUR) archive, 24(4), 325–376.

    Article  Google Scholar 

  • Cha, S.-H., & Srihari, S. N. (2002). On measuring the distance between histograms. Pattern Recognition, 35, 1355–1370.

    Article  Google Scholar 

  • Fernandez-Maloigne, C., Laugier, D., & Boscolo, C. (1993). Detection of apples with texture analyses for an apple picker robot. Intelligent Vehicles ‘93 Symposium: pp. 323–328. ISBN: 0-7803-1370-4.

  • Freund, Y., & Shapire, R. E. (1996). Experiments with a new boosting algorithm. In Machine learning: Proceedings of the 13th international conference, July 3–6, 1996 (pp. 148–156). Bari, Italy: Morgan Kaufmann.

  • Hunter, R. S. (1948). Photoelectric color-difference meter. JOSA 38 (7): 661. In Proceedings of the Winter Meeting of the Optical Society of America.

  • Jiménez, A. R., Ceres, R., & Pons, J. L. (2000a). A vision system based on a laser range-finder applied to robotic fruit harvesting. Machine Vision and Applications, 11(6), 321–329.

    Article  Google Scholar 

  • Jiménez, A. R., Ceres, R., & Pons, J. L. (2000b). A survey of computer vision methods for locating fruit on trees. Transactions of ASAE, 43(6), 1911–1920.

    Google Scholar 

  • Lienhart, R., & Maydt, J. (2002). An extended set of haar-like features for rapid object detection. In Proceedings of the IEEE Conference on Image Processing (ICIP ‘02) (pp. 155–162).

  • Mao, W., Jia, B., Zhang, X., & Hub, X. (2009). Detection and position method of apple tree image. Computer and Computing Technologies in Agriculture II, 2.

  • Sapina, R. (2001). Computing textural features based on co-occurrence matrix for infrared images. In S. Loncaric & H. Babic (Eds.), Proceedings of 2nd international symposium on image and signal processing and analysis, 2001. ISPA 2001 (pp. 373–376). Croatia: University Computing Center, University of Zagreb.

  • Sonka, M., Hlavac, V., & Boyle, R. (1999). Image processing, analysis, and machine vision (2nd Edn., pp. 256–260), CA, USA: Brooks/Cole Publishing.

  • Stanjnko, D., Lakota, M., & Hocevar, M. (2004). Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Computers and Electronics in Agriculture, 42, 31–42.

    Article  Google Scholar 

  • Tabb, A. L., Peterson, D. L., & Park, J. (2006). Segmentation of apple fruit from video via background modeling. Paper number 063060, ASABE, St Joseph, MI, USA.

  • Timofeev, R. (2004). Classification and regression trees (cart) theory and applications. Master’s thesis, Humboldt University Berlin.

  • Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137–154.

    Article  Google Scholar 

  • Viola, P., & Wells, W. M. III (1995). Alignment by maximization of mutual information. In Proceedings of 5th international conference on computer vision, June 20–23, 1995 (pp. 16–22). Cambridge, MA, USA: Massachusetts Institute of Technology, IEEE Computer Society.

  • Wachs, J., Stern, H., Burks, T., & Alchanatis, V. (2009). Multi-modal registration using a combined similarity measure. In E. Avineri, M. Köppen, K. Dahal, Y. Sunitiyoso, R. Roy (Eds.), Applications of soft computing: Updating the state of the art series: Advances in intelligent and soft computing (Vol. 52, pp. 159–168). Springer, Warsaw, Poland.

  • Werbos, P. J. (1990). Backpropagation through time: what it does and how to do it. In Proceedings of IEEE, 78, pp. 1550–1560.

  • Zhao, J., Tow, J., & Katupitiya, J. (2005). On-tree fruit recognition using texture properties and color data. In IEEE/RSJ International Conference on Intelligent Robots and Systems, Aug 2–6, 2005 (pp. 263–268). Alberta, Canada: IEEE Robotics and Automation Society.

Download references

Acknowledgments

This research was supported by Research Grant No US-3715-05 from BARD, The United States—Israel Binational Agricultural Research and Development Fund, and by the Paul Ivanier Center for Robotics Research and Production Management, Ben-Gurion University of the Negev.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to V. Alchanatis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wachs, J.P., Stern, H.I., Burks, T. et al. Low and high-level visual feature-based apple detection from multi-modal images. Precision Agric 11, 717–735 (2010). https://doi.org/10.1007/s11119-010-9198-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11119-010-9198-x

Keywords

Navigation