Skip to main content
SpringerLink
Log in

Stereo-based real-time 6-DoF work tool tracking for robot programing by demonstration

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

This contribution presents a new system for fast and intuitive industrial robot reprogramming. It is based on a luminous marker built with high-intensity LEDs, which are captured by a set of industrial cameras. Using stereoscopy, the marker supplies 6-DoF human wrist tracking with both position and orientation data. This marker can be efficiently attached to any working tool which then provides a way to capture human skills without further intrusion in the tasks. The acquisition technique makes the tracking very robust against lighting conditions so no environment preparation is needed. The robot is automatically programmed from the demonstrated task which delivers complete abstraction of programming concepts. The system is able to perform in real time, and is low-cost starting with a single pair of industrial cameras though more can be used for improved effectiveness and accuracy. The real-time feature means that the robot is ready to perform as soon as the demonstration is over which carries no overhead of reprogramming times. Also, there is no interference with the task itself since the marker is attached to the work tool and the tracking is contactless; the human operator can then perform naturally. The test bed is a real industrial environment: a spray painting application. A prototype has been developed and installed, and is currently in operation. The tests show that the proposed system enables transferring to the machine the human ability of manipulating a spray gun.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ferreira M, Paulo Moreira A, Neto P (2012) A low-cost laser scanning solution for flexible robotic cells: spray coating. Int J Adv Manuf Technol 58(9–12):1031–1041

    Article  Google Scholar 

  2. Chen H, Weihua S, Xi N, Song M, Chen Y (2002) Automated robot trajectory planning for spray painting of free-form surfaces in automotive manufacturing. Robotics and Automation, IEEE Int Conf 1:450–455

    Google Scholar 

  3. Chen H, Weihua S, Xi N, Chen Y, Roche A, Dahl J (2003) A general framework for automatic CAD-guided tool planning for surface manufacturing. Robotics and Automation, IEEE Int Conf 3:3504–3509

    Google Scholar 

  4. Chen H, Xi N (2008) Automated tool trajectory planning of industrial robots for painting composite surfaces. Int J Adv Manuf Technol 35(7–8):680–696

    Article  Google Scholar 

  5. Chen H, Fuhlbrigge T, Li X (2008) Automated industrial robot path planning for spray painting process: A review. Automation Science and Engineering, 2008. CASE 2008. IEEE International Conference on, pages 522–527

  6. Norberto Pires J, Godinho T, Ferreira P (2004) CAD interface for automatic robot welding programming. Ind Robot: Int J 31(1):71–76

    Article  Google Scholar 

  7. Neto P, Mendes N, Araujo R, Pires JN, Moreira AP (2012) High-level robot programming based on CAD: Dealing with unpredictable environments. Ind Robot 39(3):294–303

    Article  Google Scholar 

  8. Neto P, Mendes N (2013) Direct off-line robot programming via a common CAD package. Robot Auton Syst 61(8):896–910

    Article  Google Scholar 

  9. Waldherr S, Romero R, Thrun S (2000) A gesture based interface for human-robot interaction. Auton Robot 9:151–173

    Article  Google Scholar 

  10. Kumar P, Verma J, Prasad S (2012) Hand Data Glove: A Wearable Real-Time Device for Human-Computer Interaction. Hand, 43

  11. Dillmann R, Rogalla O, Ehrenmann M et al. (2000) Learning robot behaviour and skills based on human demonstration and advice: the machine learning paradigm. In Robotics Rresearch, International Symposium, 9, pages 229–238

  12. Zollner R, Rogalla O, Dillmann R (2001) Integration of tactile sensors in a programming by demonstration system. Robot Autom, IEEE Int Conf 3:2578–2583

    Google Scholar 

  13. Zollner R, Rogalla O, Dillmann R, Zollner M (2002) Understanding users intention: programming fine manipulation tasks by demonstration. Intell Robot Syst, Int Conf 2:1114–1119

    Google Scholar 

  14. Rogalla O, Ehrenmann M, Dillmann R (1998) A sensor fusion approach for PbD. IEEE/RSJ Int Conf 2:1040–1045, Intelligent Robots and Systems

    Google Scholar 

  15. Hein B, Worn H (2009) Intuitive and model-based on-line programming of industrial robots: New input devices. In Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on, pages 3064–3069

  16. Hein B, Hensel M, Worn H (2008) Intuitive and model-based on-line programming of industrial robots: A modular on-line programming environment. In Robotics and Automation, 2008. ICRA 2008. IEEE International Conference on, pages 3952–3957

  17. Field M, Pan Z, Stirling D, Naghdy F (2011) Human motion capture sensors and analysis in robotics. Ind Robot 38(2):163–171

    Article  Google Scholar 

  18. Elgammal A, Lee C (2009) Tracking people on a torus. Pattern Anal Mach Intell, IEEE Transactions on 31(3):520–538

    Article  Google Scholar 

  19. Shon AP, Grochow K, Rao RPN (2005) Robotic imitation from human motion capture using Gaussian processes. In Humanoid Robots, 2005 5th IEEE-RAS International Conference on, pages 129–134

  20. Naksuk N, Lee CSG, Rietdyk S (2005) Whole-body human-to-humanoid motion transfer. In Humanoid Robots, 2005 5th IEEE-RAS International Conference on, pages 104–109

  21. Sigal L, Balan A, Black MJ (2010) HumanEva: synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int J Comput Vis 87(1–2):4–27

    Article  Google Scholar 

  22. Azad P, Asfour T, Dillmann R (2008) Robust real-time stereo-based markerless human motion capture. In Humanoid Robots, 2008. Humanoids 2008. 8th IEEE-RAS International Conference on, pages 700–707

  23. Azad P, Ude A, Asfour T, Dillmann R (2007) Stereo-based Markerless Human Motion Capture for Humanoid Robot Systems. In Robotics and Automation, 2007 I.E. International Conference on, pages 3951–3956

  24. Cutti AG, Giovanardi A, Rocchi L, Davalli A (2006) A simple test to assess the static and dynamic accuracy of an inertial sensors system for human movement analysis. In Engineering in Medicine and Biology Society, 2006. EMBS’06. 28th Annual International Conference of the IEEE, pages 5912–5915

  25. Yamamoto T, Fujinami T (2008) Hierarchical organization of the coordinative structure of the skill of clay kneading. Hum Mov Sci 27(5):812–22

    Article  Google Scholar 

  26. Benbasat AY, Paradiso JA (2002) An Inertial Measurement Framework for Gesture Recognition and Applications. In Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in HumanComputer Interaction, GW’01, pages 9–20, London, UK, UK, SpringerVerlag

  27. Malheiros P, Costa P, Moreira AP Robust 3D motion capture and object positioning system using light emitting markers synchronized with stereoscopic camera system. UPIN NPat.77/Pat. 41, Int.Patent PCT/IB2009/007186

  28. Hartley RI, Zisserman A (2004) Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition

  29. Pratt V, Point P (1987) Direct Least-Squares Fitting of Algebraic Surfaces, Technical report

  30. Lukács G, Marshall AD, Martin RR (1997) Geometric least-squares fitting of spheres, cylinders, cones and tori. Technical report

  31. Kabsch W (1976) A solution for the best rotation to relate two sets of vectors. Acta Crystallogr A 32(5):922–923

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Engineering, Institute for Systems and Computer Engineering, University of Porto and INESC-TEC, R. Dr. Roberto Frias 378, 4200-465, Porto, Portugal

    Marcos Ferreira, Paulo Costa, Luís Rocha & A. Paulo Moreira

Authors
  1. Marcos Ferreira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paulo Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luís Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. Paulo Moreira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcos Ferreira.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(MPG 19172 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ferreira, M., Costa, P., Rocha, L. et al. Stereo-based real-time 6-DoF work tool tracking for robot programing by demonstration. Int J Adv Manuf Technol 85, 57–69 (2016). https://doi.org/10.1007/s00170-014-6026-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-014-6026-x

Keywords

Search

Navigation