Skip to main content
SpringerLink
Log in

Approach to 3D SLAM for Mobile Robot Based on RGB-D Image with Semantic Feature in Dynamic Environment

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Aiming at the problems of low localization accuracy and poor mapping consistency for mobile robots in dynamic environments, this paper presents a 3D simultaneous localization and mapping (SLAM) algorithm based on RGB-D (Red Green Blue and Depth) image with semantic features. In the front-end, semantic segmentation is performed using Mask R-CNN (Region-based Convolutional Neural Network), also, combining the depth and distance information of feature points with random sampling to obtain static feature points with higher confidence. Then, the relative static point is obtained by matching the dynamic feature points of the adjacent frames, which is added to the tracking and optimization of the current frame to obtain the minimized re-projection error and the optimal pose of robot. In order to adapt the uncertainty of dynamic feature point changes, a high-performance keyframe selection method is developed for different situations. In the back-end, a closed loop detection strategy is constructed by combining semantic information with bag-of-words (BOW) model. Further, according to whether a closed loop is formed or not, a combinatorial optimization strategy based on graph optimization and factor graph optimization is adopted to improve the real time performance of the algorithm. Finally, these images with semantic tags obtained after semantic cutting are employed to build a semantic octree map of the environment. A series of simulations and experiments demonstrate the superior performance of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author Dr. Jingwen Luo upon reasonable request

Code Availability

Our source code is not public

References

  1. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: Real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136 (2011). https://doi.org/10.1109/ISMAR.2011.6092378

  2. Engel, J., Schps, T., Cremers, D.: Lsd-slam: Large-scale direct monocular slam. In: European Conference on Computer Vision, pp. 834–849 (2014). Springer

  3. Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans. Robot. 33(5), 1255–1262 (2017). https://doi.org/10.1109/TRO.2017.2705103

    Article  Google Scholar 

  4. Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry. IEEE Trans. Pattern Anal. Mach. Intell. 40(3), 611–625 (2018). https://doi.org/10. 1109/TPAMI.2017.2658577

  5. Zhao, L., Liu, Z., Chen, J., Cai, W., Wang, W., Zeng, L.: A compatible framework for rgb-d slam in dynamic scenes. IEEE Access 7, 75604–75614 (2019). https://doi.org/10.1109/ACCESS.2019.2922733

    Article  Google Scholar 

  6. Xie, W., Liu, P.X., Zheng, M.: Moving object segmentation and detection for robust rgbd-slam in dynamic environments. IEEE Trans. Instrum. Meas. 70, 1–8 (2021). https://doi.org/10.1109/TIM.2020.3026803

    Article  Google Scholar 

  7. Yuan, X., Chen, S.: Sad-slam: a visual slam based on semantic and depth information. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 4930–4935 (2020). IEEE

  8. Li, A., Wang, J., Xu, M., Chen, Z.: Dpslam: A visual slam with moving probability towards dynamic environments. Inf. Sci. 556, 128–142 (2021)

    Article  Google Scholar 

  9. Wen, S., Li, P., Zhao, Y., Zhang, H., Sun, F., Wang, Z.: Semantic visual slam in dynamic environment. Auton. Robots 45(4), 493–504 (2021)

    Article  Google Scholar 

  10. Cheng, J., Wang, C., Mai, X., Min, Z., Meng, M.Q.-H.: Improving dense mapping for mobile robots in dynamic environments based on semantic information. IEEE Sens. J. 21(10), 11740–11747 (2021). https://doi.org/10.1109/JSEN.2020.3023696

    Article  Google Scholar 

  11. Yu, C., Liu, Z., Liu, X.-J., Xie, F., Yang, Y., Wei, Q., Fei, Q.: Ds-slam: A semantic visual slam towards dynamic environments. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1168–1174 (2018). https://doi.org/10.1109/IROS.2018.8593691

  12. Vincent, J., Labbé, M., Lauzon, J.-S., Grondin, F., Comtois-Rivet, P.-M., Michaud, F.: Dynamic object tracking and masking for visual slam. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4974–4979 (2020). https://doi.org/10.1109/IROS45743.2020.9340958

  13. Bolya, D., Zhou, C., Xiao, F., Lee, Y.J.: Yolact: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9156–9165 (2019). https://doi.org/10.1109/ICCV.2019.00925

  14. Liu, Y., Miura, J.: Rdmo-slam: Real-time visual slam for dynamic environments using semantic label prediction with optical flow. IEEE Access 9, 106981–106997 (2021). https://doi.org/10.1109/ACCESS.2021.3100426

    Article  Google Scholar 

  15. Zhang, C., Huang, T., Zhang, R., Yi, X.: Pld-slam: a new rgb-d slam method with point and line features for indoor dynamic scene. ISPRS Int. J. Geo-Inf. 10(3), 163 (2021)

    Article  Google Scholar 

  16. Yan, L., Hu, X., Zhao, L., Chen, Y., Wei, P., Xie, H.: Dgs-slam: A fast and robust rgbd slam in dynamic environments combined by geometric and semantic information. Remote Sens. 14(3), 795 (2022)

    Article  Google Scholar 

  17. Li, S., Lee, D.: Rgb-d slam in dynamic environments using static point weighting. IEEE Robot. Autom. Lett. 2(4), 2263–2270 (2017). https://doi.org/10.1109/LRA.2017.2724759

    Article  Google Scholar 

  18. Cheng, J., Wang, C., Meng, M.Q.-H.: Robust visual localization in dynamic environments based on sparse motion removal. IEEE Trans. Autom. Sci. Eng. 17(2), 658–669 (2020). https://doi.org/10.1109/TASE.2019.2940543

    Article  Google Scholar 

  19. Cheng, J., Zhang, H., Meng, M.Q.-H.: Improving visual localization accuracy in dynamic environments based on dynamic region removal. IEEE Trans. Autom. Sci. Eng. 17(3), 1585–1596 (2020). https://doi.org/10.1109/TASE.2020.2964938

    Article  Google Scholar 

  20. Liu, Y., Wu, Y., Pan, W.: Dynamic rgb-d slam based on static probability and observation number. IEEE Trans. Instrum. and Meas. 70, 1–11 (2021). https://doi.org/10.1109/TIM.2021.3089228

    Article  Google Scholar 

  21. Liu, Y., Miura, J.: Kmop-vslam: Dynamic visual slam for rgb-d cameras using k-means and openpose. In: 2021 IEEE/SICE International Symposium on System Integration (SII), pp. 415–420 (2021). https://doi.org/10.1109/IEEECONF49454.2021.9382724

  22. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017). https://doi.org/10.1109/ICCV.2017.322

  23. Ren, S., He, K., Girshick, R., Sun, J.: Faster rcnn: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031

    Article  Google Scholar 

  24. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment–a modern synthesis. In: Vision Algorithms: Theory and Practice: InternationalWorkshop on Vision Algorithms Corfu, Greece, September 21–22, 1999 Proceedings, pp. 298–372 (2000). Springer

  25. Gálvez-López, D., Tardos, J.D.: Bags of binary words for fast place recognition in image sequences. IEEE Trans. Robot. 28(5), 1188–1197 (2012)

    Article  Google Scholar 

  26. Kaess, M., Ranganathan, A., Dellaert, F.: isam: Incremental smoothing and mapping. IEEE Trans. Robot. 24(6), 1365–1378 (2008)

    Article  Google Scholar 

  27. Kaess, M., Johannsson, H., Roberts, R., Ila, V., Leonard, J.J., Dellaert, F.: isam2: Incremental smoothing and mapping using the bayes tree. Int. J. Robot. Res. 31(2), 216–235 (2012)

    Article  Google Scholar 

  28. Davis, T.A., Gilbert, J.R., Larimore, S.I., Ng, E.G.: A column approximate minimum degree ordering algorithm. ACM Trans. Math. Softw. (TOMS) 30(3), 353–376 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  29. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: Proc. of the International Conference on Intelligent Robot Systems (IROS) (2012)

  30. Zhang, J., Liu, Y., Guo, C., Zhan, J.: Optimized segmentation with image inpainting for semantic mapping in dynamic scenes Applied Intelligence, 1–16 (2022)

  31. Chen, H., Zhang, J., Chen, B., Li, B.: Global practical stabilization for non-holonomic mobile robots with uncalibrated visual parameters by using a switching controller. IMA J. Math. Control Inf. 30(4), 543–557 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  32. Chen, H.: Robust stabilization for a class of dynamic feedback uncertain nonholonomic mobile robots with input saturation. Int. J. Control Autom. Syst. 12(6), 1216–1224 (2014)

    Article  Google Scholar 

  33. Chen, H., Chen, Y., Wang, M.: Trajectory tracking for underactuated surface vessels with time delays and unknown control directions. IET Control Theory & Appl. 16(6), 587–599 (2022)

Download references

Funding

This work was supported by National Nature Science Foundation of China (Grant No. 62063036) and Research Foundation for Doctor of Yunnan Normal University (Grant No. 01000205020503115)

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Yunnan Normal University, no.768 Juxian Street, Chenggong District, Kunming, Yunnan, 650500, China

    Jiajie Li & Jingwen Luo

Authors
  1. Jiajie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingwen Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jiajie Li and Jingwen luo wrote the main manuscript text, prepared figures and contributed equally to this work. All authors reviewed the manuscript

Corresponding author

Correspondence to Jingwen Luo.

Ethics declarations

Conflicts of interest

No conflict of interest exists in the submission of this manuscript

Ethics approval

Not applicable (this article does not contain any studies with human participants or animals performed by any of the authors)

Consent to participate

All authors have participated in conception and design, or analysis and interpretation of the data; drafting the article or revising it critically for important intellectual content, and approval of the final version

Consent for publication

The authors declare no interest conflict regarding the publication of this paper

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Jiajie Li and Jingwen Luo these authors contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Luo, J. Approach to 3D SLAM for Mobile Robot Based on RGB-D Image with Semantic Feature in Dynamic Environment. J Intell Robot Syst 109, 15 (2023). https://doi.org/10.1007/s10846-023-01922-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01922-2

Keywords

Search

Navigation