Skip to main content

ROS and Stereovision Collaborative System

  • Chapter
  • First Online:
Optoelectronic Devices in Robotic Systems

Abstract

In this chapter, a collaborative work between stereo and a laser scanning system is presented in order to obtain an enriched matrix of rectangular coordinates of elements in a scene using SAD for stereovision and real-time dynamic triangulation in ROS as processing algorithms. Dense data volume and precise depth estimation are the complementary parameters for enrichment and the main reasons for taking these systems to work together on applications as robot navigation, structural health, and body scanning. This combination improves weak points of each separate system, thanks to the use of output data of each technique for calibration and control of the other technique, making it redundant when data confirmation is required and increasing of data volume. Systems link is done through ROIs and known relative position between cameras and ROS positioner and aperture, achieving a collaborative work even when there is displacement. For future work, sensors comparison and motor control optimization is expected for scanning time reduction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abbreviations

CCD:

Charge-coupled device

CMOS:

Complementary metal-oxide-semiconductor

DC:

Direct current

DLT:

Direct linear transformation

FOV:

Field of view

FPGA:

Field-programmable gate array

NCC:

Normalized cross-correlation

RADAR:

Radio detection and ranging

ROI:

Region of interest

ROS:

Rotational optical scanner

SAD:

Sum of absolute differences

SONAR:

Sound navigation and ranging

SSD:

Sum of squared differences

TOF:

Time of flight

References

  1. Vongbunyong S, Chen WH (2015) Vision System. Springer, Cham, pp. 55–93

    Google Scholar 

  2. Chen X (2007) Stereo vision based motion identification. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer Verlag, pp. 575–586

    Google Scholar 

  3. He Z, Ren Q, Yang T, Li J, Zhang Y (2016) Multi-object detection based on binocular stereo vision. In: Communications in Computer and Information Science. Springer Verlag, pp. 114–121

    Google Scholar 

  4. Mo H, Luo C, Liu K (2016) Robot indoor navigation based on computer vision and machine learning. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer Verlag, pp. 528–534

    Google Scholar 

  5. Panigrahi PK, Tripathy HK (2015) Analysis on intelligent based navigation and path finding of autonomous mobile robot. In: Advances in Intelligent Systems and Computing. Springer Verlag, pp. 219–232

    Google Scholar 

  6. Hart S, Mikhailova E, Post C, McMillan P, Sharp J, Bridges W (2017) Spatio-temporal analysis of flowering using LiDAR topography. J Geogr Sci 27:62–78. https://doi.org/10.1007/s11442-017-1364-x

    Article  Google Scholar 

  7. Broggi A, Medici P, Porta PP (2007) StereoBox: A robust and efficient solution for automotive short-range obstacle detection. Eurasip J Embed Syst 2007:1–7. https://doi.org/10.1155/2007/70256

    Article  Google Scholar 

  8. Kim S, Kim H Bin (2010) High resolution mobile robot obstacle detection using low directivity ultrasonic sensor ring. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer, Berlin, Heidelberg, pp. 426–433

    Google Scholar 

  9. Zarandy A, Nagy Z, Vanek B, Zsedrovits T, Kiss A, Nemeth M (2013) A five-camera vision system for UAV visual attitude calculation and collision warning. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer, Berlin, Heidelberg, pp. 11–20

    Google Scholar 

  10. Sergiyenko O, Tyrsa V (2020) 3D optical machine vision sensors with intelligent data management for robotic swarm navigation improvement. IEEE Sens J. https://doi.org/10.1109/JSEN.2020.3007856

  11. Flores-Fuentes W, Rivas-Lopez M, Sergiyenko O, Rodríguez-Quiñonez JC, Hernández-Balbuena D, Rivera-Castillo J (2014) Energy center detection in light scanning sensors for structural health monitoring accuracy enhancement. IEEE Sens J 14:2355–2361. https://doi.org/10.1109/JSEN.2014.2310224

    Article  Google Scholar 

  12. C. J, Sergiyenko O, Tyrsa V, C. L, Rivas-Lopez M, Hernndez-Balbuena D, Pea-Cabrer M (2011) 3D Body & Medical Scanners’ Technologies: Methodology and Spatial Discriminations. In: Optoelectronic Devices and Properties. InTech

    Google Scholar 

  13. Lindner L, Sergiyenko O, Rivas-López M, Hernández-Balbuena D, Flores-Fuentes W, Rodríguez-Quiñonez JC, Murrieta-Rico FN, Ivanov M, Tyrsa V, Básaca-Preciado LC (2017) Exact laser beam positioning for measurement of vegetation vitality. Ind Rob 44:532–541. https://doi.org/10.1108/IR-11-2016-0297

    Article  Google Scholar 

  14. Chung SH, Lee SW, Lee SK, Park JH (2019) LIDAR system with electromagnetic two-axis scanning micromirror based on indirect time-of-flight method. Micro Nano Syst Lett 7:1–5. https://doi.org/10.1186/s40486-019-0082-9

    Article  Google Scholar 

  15. Chung S-H, Lee S-W, Lee S-K, Park J-H (2019) LIDAR system with electromagnetic two-axis scanning micromirror based on indirect time-of-flight method. Micro Nano Syst Lett 2019 71 7:1–5. https://doi.org/10.1186/S40486-019-0082-9

    Article  Google Scholar 

  16. Szeliski R (2011) Stereo correspondence. Springer, London, pp. 467–503

    Google Scholar 

  17. Vilaça JL, Fonseca JC, Pinho AM (2009) Non-contact 3D acquisition system based on stereo vision and laser triangulation. Mach Vis Appl 2008 213 21:341–350. https://doi.org/10.1007/S00138-008-0166-7

    Article  Google Scholar 

  18. Denker K, Lehner B, Umlauf G (2010) Real-time triangulation of point streams. Eng with Comput 2010 271 27:67–80. https://doi.org/10.1007/S00366-010-0181-Y

    Article  Google Scholar 

  19. Mikhaylichenko AA, Kleshchenkov AB (2018) Approach to Non-Contact Measurement of Geometric Parameters of Large-Sized Objects. Program Comput Softw 2018 444 44:271–277. https://doi.org/10.1134/S0361768818040096

    Article  Google Scholar 

  20. Yoo H-S, Kim Y-S, Kwon S-W (2014) A comparative study of noise elimination algorithms for a 3D terrain model through object clustering and the differential method. KSCE J Civ Eng 2015 193 19:498–509. https://doi.org/10.1007/S12205-013-0370-5

    Article  Google Scholar 

  21. Madeo S, Pelliccia R, Salvadori C, del Rincon JM, Nebel JC (2016) An optimized stereo vision implementation for embedded systems: application to RGB and infra-red images. In: Journal of Real-Time Image Processing. Springer Verlag, pp. 725–746

    Google Scholar 

  22. Lu R, Lai J, Xie X (2018) Asymmetric Two-Stream Networks for RGB-Disparity Based Object Detection. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 11259 LNCS:3–15. https://doi.org/10.1007/978-3-030-03341-5_1

  23. Heng J, Xu Z, Zheng Y, Liu Y (2017) Disparity Refinement Using Merged Super-Pixels for Stereo Matching. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 10666 LNCS:295–305. https://doi.org/10.1007/978-3-319-71607-7_26

  24. Popielski P, Wróbel Z, Koprowski R (2014) Object Detail Correspondence Problem in Stereovision. Adv Intell Syst Comput 283:209–222. https://doi.org/10.1007/978-3-319-06593-9_19

    Article  Google Scholar 

  25. Yousfi J, Lahouar S, Ben Amara A (2017) Strategy of Image Capture and Its Impact on Correspondence Error in Reconstructed 3D-Images-Based Point. Lect Notes Mech Eng 115–126. https://doi.org/10.1007/978-3-319-66697-6_12

  26. Lazaros N, Sirakoulis GC, Gasteratos A (2008) Review of stereo vision algorithms: From software to hardware. Int J Optomechatronics 2:435–462. https://doi.org/10.1080/15599610802438680

    Article  Google Scholar 

  27. Hirschmüller H, Scharstein D (2007) Evaluation of cost functions for stereo matching. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2007.383248

  28. Bae K ryeol, Moon B (2017) An accurate and cost-effective stereo matching algorithm and processor for real-time embedded multimedia systems. Multimed Tools Appl 76:17907–17922. https://doi.org/10.1007/s11042-016-3248-y

  29. Patil S, Nadar JS, Gada J, Motghare S, Nair SS (2013) Comparison of Various Stereo Vision Cost Aggregation Methods. Int J Eng Innov Technol 2:222–226

    Google Scholar 

  30. Shen Y (2011) Efficient normalized cross correlation calculation method for stereo vision based robot navigation. Front Comput Sci China 5:227–235. https://doi.org/10.1007/s11704-011-9190-2

    Article  MathSciNet  MATH  Google Scholar 

  31. Dong Q, Feng J (2018) Adaptive disparity computation using local and non-local cost aggregations. Multimed Tools Appl 2018 7724 77:31647–31663. https://doi.org/10.1007/S11042-018-6236-6

    Article  Google Scholar 

  32. Middlebury (2014) Adirondack. https://vision.middlebury.edu/stereo/data/scenes2014/

  33. Hisham MB, Yaakob SN, Raof RAA, Nazren ABA, Embedded NMW (2015) Template Matching using Sum of Squared Difference and Normalized Cross Correlation. In: 2015 IEEE Student Conference on Research and Development, SCOReD 2015. Institute of Electrical and Electronics Engineers Inc., pp. 100–104

    Google Scholar 

  34. Ilmenau TU, Kuhl A (2005) Comparison of Stereo Matching Algorithms for Mobile Robots

    Google Scholar 

  35. Michalik S, Michalik S, Naghmouchi J, Berekovic M (2017) Real-Time smart stereo camera based on FPGA-SoC. In: IEEE-RAS International Conference on Humanoid Robots. IEEE Computer Society, pp. 311–317

    Google Scholar 

  36. Cambuim LFS, Oliveira LA, Barros ENS, Ferreira APA (2020) An FPGA-based real-time occlusion robust stereo vision system using semi-global matching. J Real-Time Image Process 17:1447–1468. https://doi.org/10.1007/s11554-019-00902-w

    Article  Google Scholar 

  37. Masmoudi MBM, Jerad C, Attia R (2016) On-the-Fly Architecture Design and Implementation of a Real-Time Stereovision System. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 10016 LNCS:711–722. https://doi.org/10.1007/978-3-319-48680-2_62

  38. Li Y, Huang K, Claesen L (2016) A Novel Hardware-Oriented Stereo Matching Algorithm and Its Architecture Design in FPGA. IFIP Adv Inf Commun Technol 508:213–232. https://doi.org/10.1007/978-3-319-67104-8_11

    Article  Google Scholar 

  39. Jin S, Yuanzhi W, Yining S (2018) Design and implementation of wireless multimedia sensor network node based on FPGA and binocular vision. EURASIP J Wirel Commun Netw 2018 20181 2018:1–8. https://doi.org/10.1186/S13638-018-1172-8

    Article  Google Scholar 

  40. Rodriguez-Quinonez JC, Sergiyenko O, Gonzalez-Navarro FF, Basaca-Preciado L, Tyrsa V (2013) Surface recognition improvement in 3D medical laser scanner using Levenberg-Marquardt method. Signal Processing 93:378–386. https://doi.org/10.1016/j.sigpro.2012.07.001

    Article  Google Scholar 

  41. Sankowski W, Włodarczyk M, Kacperski D, Grabowski K (2017) Estimation of measurement uncertainty in stereo vision system. Image Vis Comput 61:70–81. https://doi.org/10.1016/j.imavis.2017.02.005

    Article  Google Scholar 

  42. Rajeshkannan S, Korah R (2016) Improved CRC Based Disparity Estimation of Vision System Using Local Adaptive Hue Census and Mean Shift Clustering. Natl Acad Sci Lett 2016 391 39:35–38. https://doi.org/10.1007/S40009-015-0412-2

    Article  Google Scholar 

  43. Pratt P, Bergeles C, Darzi A, Yang G-Z (2014) Practical Intraoperative Stereo Camera Calibration. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 8674 LNCS:667–675. https://doi.org/10.1007/978-3-319-10470-6_83

  44. Yang S, Liu M, Song J, Yin S, Guo Y, Ren Y, Zhu J (2017) Projector calibration method based on stereo vision system. Opt Rev 2017 246 24:727–733. https://doi.org/10.1007/S10043-017-0370-7

    Article  Google Scholar 

  45. Taryudi, Wang M-S (2017) Eye to hand calibration using ANFIS for stereo vision-based object manipulation system. Microsyst Technol 2017 241 24:305–317. https://doi.org/10.1007/S00542-017-3315-Y

  46. Hyun J, Moon B (2016) A simplified rectification method and its hardware architecture for embedded multimedia systems. Multimed Tools Appl 2016 7619 76:19761–19779. https://doi.org/10.1007/S11042-016-3517-9

    Article  Google Scholar 

  47. Hansen P, Alismail H, Rander P, Browning B (2012) Online continuous stereo extrinsic parameter estimation. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 1059–1066. https://doi.org/10.1109/CVPR.2012.6247784

  48. Taketomi T, Okada K, Yamamoto G, Miyazaki J, Kato H (2014) Camera pose estimation under dynamic intrinsic parameter change for augmented reality. Comput Graph 44:11–19. https://doi.org/10.1016/j.cag.2014.07.003

    Article  Google Scholar 

  49. Zhao Z, Ye D, Zhang X, Chen G, Zhang B (2016) Improved Direct Linear Transformation for Parameter Decoupling in Camera Calibration. Algorithms 9:31. https://doi.org/10.3390/a9020031

    Article  MathSciNet  MATH  Google Scholar 

  50. Monasse P, Morel JM, Tang Z (2010) Three-step image rectification. Br Mach Vis Conf BMVC 2010 – Proc. https://doi.org/10.5244/C.24.89

  51. Dinh VQ, Nguyen TP, Jeon JW (2019) Rectification Using Different Types of Cameras Attached to a Vehicle. IEEE Trans Image Process 28:815–826. https://doi.org/10.1109/TIP.2018.2870930

    Article  MathSciNet  MATH  Google Scholar 

  52. Abraham S, Förstner W (2005) Fish-eye-stereo calibration and epipolar rectification. ISPRS J Photogramm Remote Sens 59:278–288. https://doi.org/10.1016/j.isprsjprs.2005.03.001

    Article  Google Scholar 

  53. Kumar S, Micheloni C, Piciarelli C, Foresti GL (2010) Stereo rectification of uncalibrated and heterogeneous images. Pattern Recognit Lett 31:1445–1452. https://doi.org/10.1016/j.patrec.2010.03.019

    Article  Google Scholar 

  54. Ramírez-Hernández LR, Rodríguez-Quiñonez JC, Castro-Toscano MJ, Hernández-Balbuena D, Flores-Fuentes W, Rascón-Carmona R, Lindner L, Sergiyenko O (2020) Improve three-dimensional point localization accuracy in stereo vision systems using a novel camera calibration method. Int J Adv Robot Syst 17:172988141989671. https://doi.org/10.1177/1729881419896717

    Article  Google Scholar 

  55. Barrena N, Sánchez JR, Ugarte RJ, Alonso AG (2018) Proving the efficiency of template matching-based markerless tracking methods which consider the camera perspective deformations. Mach Vis Appl 29:573–584. https://doi.org/10.1007/s00138-018-0914-2

    Article  Google Scholar 

  56. Lindner L, Sergiyenko O, Rodríguez-Quiñonez JC, Rivas-Lopez M, Hernandez-Balbuena D, Flores-Fuentes W, Murrieta-Rico FN, Tyrsa V (2016) Mobile robot vision system using continuous laser scanning for industrial application. Ind Rob 43:360–369. https://doi.org/10.1108/IR-01-2016-0048

    Article  Google Scholar 

  57. Sohn K, Jang G (2019) Ground Vehicle Driving by Full Sized Humanoid. J Intell Robot Syst 2019 992 99:407–425. https://doi.org/10.1007/S10846-019-01130-X

    Article  Google Scholar 

  58. Everett MF (2017) Robot designed for socially acceptable navigation

    Google Scholar 

  59. Belbachir A (2017) An embedded testbed architecture to evaluate autonomous car driving. Intell Serv Robot 2017 102 10:109–119. https://doi.org/10.1007/S11370-016-0213-6

    Article  Google Scholar 

  60. Ellery A (2016) Autonomous Navigation—Self-localization and Mapping (SLAM). Planet Rovers 331–374. https://doi.org/10.1007/978-3-642-03259-2_9

Download references

Acknowledgments

Authors have special thanks to Mexican organization CONACyT for funding of PhD students participating as co-authors of this chapter, as well as our universities who provide our group with facilities, laboratory and equipment for herein presented research.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ruben Alaniz-Plata .

Editor information

Editors and Affiliations

Appendices

Appendix A

Code used to send stepper motors’ sequence usable for Arduino. This code considers a serial input with format “DX SY,” where D is a symbolic reference to the direction where Y can take a value from 1 to 4, being 1–2 both directions of one stepper motor (horizontal for ROS) and 3–4 for the second stepper motor (vertical for ROS). S refers to the number of steps Y.

#define m1_1 22 #define m1_2 23 #define m1_3 24 #define m1_4 25 #define m2_1 26 #define m2_2 27 #define m2_3 28 #define m2_4 29 int dir = 0, lastdirH = 0, lastdirV = 0, steps = 0, binH = 1, binV = 1; bool enc = false; float vMotor = 0; void setup() { pinMode(m1_1,OUTPUT); pinMode(m1_2,OUTPUT); pinMode(m1_3,OUTPUT); pinMode(m1_4,OUTPUT); pinMode(m2_1,OUTPUT); pinMode(m2_2,OUTPUT); pinMode(m2_3,OUTPUT); pinMode(m2_4,OUTPUT); digitalWrite(m1_1,0); digitalWrite(m1_2,0); digitalWrite(m1_3,0); digitalWrite(m1_4,0); digitalWrite(m2_1,0); digitalWrite(m2_2,0); digitalWrite(m2_3,0); digitalWrite(m2_4,0); Serial.begin(9600); } void loop(){ if (Serial.available() > 0){ String dataSt = Serial.readString(); int str_len = dataSt.length() + 1; char data[str_len]; dataSt.toCharArray(data, str_len); sscanf(data,"D%d S%d",&dir,&steps); if (lastdirH == 0 && dir < 3){ lastdirH = dir; } if (lastdirV == 0 && dir > 2){ lastdirV = dir; } if (dir == 1 || dir == 2){ if (lastdirH != dir){ binH = -(binH-5); lastdirH = dir; } } if (dir == 3 || dir == 4){ if (lastdirV != dir){ binV = -(binV-5); lastdirV = dir; } } int i = 0; for (i = 1;i <= steps;i++){ delay(500); if(dir == 2 || dir == 4){ if (dir == 2){digitalWrite(m1_1+binH-1,0);} if (dir == 4){digitalWrite(m2_1+binV-1,0);} } if(dir == 1 || dir == 3){ if (dir == 1){digitalWrite(m1_4-binH+1,0);} if (dir == 3){digitalWrite(m2_4-binV+1,0);} } if (dir == 1 || dir == 2){ if (binH == 4){binH = 0;} binH++; }else{ if (binV == 4){binV = 0;} binV++;} if (dir == 1 || dir == 2){ if (dir == 2){digitalWrite(m1_1+binH-1,1);} if (dir == 1){digitalWrite(m1_4-binH+1,1);} } if (dir == 3 || dir == 4){ if (dir == 4){digitalWrite(m2_1+binV-1,1);} if (dir == 3){digitalWrite(m2_4-binV+1,1);} } } Serial.println((String) 5); } }

Appendix B

Code used for beta angle estimation in ROS usable for Arduino

float vmaxO = 0,vminO = 999,vrefO = 0; //Reference volages int capture = 0,c = 0,pos = 0, pos_ant = 0,change = 0; //Flags float opto = 0,foto = 0; //Data int picoO[2] = {0},picoF = 0, maxF = -9999; //Peaks int cpicosO = 0,diffO = 0,diffF = 0; //Peaks; void setup() { pinMode(A0,INPUT); pinMode(A1,INPUT); Serial.begin(9600); } void loop() { if (c>50000){ vmaxO = vminO * 2; } opto = analogRead(A0); foto = analogRead(A1); foto = - foto; if (opto < vminO){vminO = opto;} if (opto > vmaxO || vrefO > vmaxO){vmaxO = opto;} vrefO = (vminO + vmaxO) / 2; if (opto<=vrefO){capture=1;} if (capture==1){ if (foto > maxF){ maxF = foto; picoF = c; } c++; if (opto > vrefO * 1.1){ pos = 1; if (pos_ant == 0){ change = 1; if (cpicosO == 1){ picoO[0] = c; } else{ picoO[1] = c; } } pos_ant = 1; } else if (opto < vrefO * .9){ pos = 0; if (pos_ant == 1){ change = 2; if (cpicosO == 0){ cpicosO++; } else{ cpicosO=2; } } pos_ant = 0; } if (cpicosO == 2 && change == 1){ diffO=picoO[1]-picoO[0]; diffF=picoF-picoO[0]; Serial.println((String) "O" + diffO + " F" + diffF); maxF = -9999; c = 0; cpicosO = 1; picoO[0] = 0; picoO[1] = 0; } } }

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Alaniz-Plata, R. et al. (2022). ROS and Stereovision Collaborative System. In: Sergiyenko, O. (eds) Optoelectronic Devices in Robotic Systems. Springer, Cham. https://doi.org/10.1007/978-3-031-09791-1_4

Download citation

Publish with us

Policies and ethics