Skip to main content
SpringerLink
Log in

Online static point cloud map construction based on 3D point clouds and 2D images

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

With the development of science and technology, robots have been applied to many fields to free people’s hands. Environment perception and map construction are one of the key technologies for robots to achieve autonomy. In this paper, a system based on 3D point cloud and 2D image fusion is proposed to solve the problem of dynamic object segmentation and static map construction during robot motion. Different from the existing methods, the current relatively mature target detection method is used to design the extrinsic parameters between the two coordinate systems of the images and the 3D point cloud, and the probabilistic method is used to reduce the error. The above calibration results are applied to map the image detection results to the 3D point cloud to improve the segmentation accuracy of the targets. At the same time, target tracking and filtering methods are used to classify 3D points as static and dynamic. The segmented dynamic points can be applied to obstacle avoidance, while the static points are applied to the construction of a 3D point cloud map. Finally, the open-source datasets KITTI and DAIR-V2X are used to verify the proposed method, and the results show that the method is feasible and superior.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Algorithm 2
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data are not available.

References

  1. Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: Orb-slam: a versatile and accurate monocular slam system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)

    Article  Google Scholar 

  2. Mur-Artal, R., Tardós, J.D.: Orb-slam2: an open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans. Robot. 33(5), 1255–1262 (2017)

    Article  Google Scholar 

  3. Xu, W., Cai, Y., He, D., Lin, J., Zhang, F.: Fast-lio2: fast direct lidad-inertial odometry. IEEE Trans. Robot. 38(4), 2053–2073 (2022)

    Article  Google Scholar 

  4. Cattaneo, D., Vaghi, M., Valada, A.: Lcdnet: deep loop closure detection and point cloud registration for lidar slam. IEEE Trans. Robot. 38(4), 2074–2093 (2022)

    Article  Google Scholar 

  5. Ramachandran, S., Sahin, F.: Smart walker v: Implementation of rtab-map algorithm. In: 2019 14th Annual Conference System of Systems Engineering (SOSE), pp. 340–345 (2019)

  6. Qin, T., Li, P., Shen, S.: Vins-mono: a robust and versatile monocular visual-inertial state estimator. IEEE Trans. Robot. 34(4), 1004–1020 (2018)

    Article  Google Scholar 

  7. Biber, P.: The normal distributions transform: a new approach to laser scan matching. In: IROS 2003: Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, vols. 1–4, pp. 2743–2748 (2003)

  8. Pomerleau, F., Krusi, P., Colas, F., Furgale, P., Siegwart, R.: Long-term 3d map maintenance in dynamic environments. In: 2014 IEEE International Conference On Robotics and Automation (ICRA). IEEE International Conference on Robotics and Automation ICRA, pp. 3712–3719 (2014)

  9. Lim, H., Hwang, S., Myung, H.: Erasor: egocentric ratio of pseudo occupancy-based dynamic object removal for static 3d point cloud map building. IEEE Robot. Autom. Lett. 6(2), 2272–2279 (2021)

    Article  Google Scholar 

  10. Pagad, S., Agarwal, D., Narayanan, S., Rangan, K., Kim, H., Yalla, G.: Robust method for removing dynamic objects from point clouds. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10765–71 (2020)

  11. Kim, G., Kim, A.: Remove, then revert: Static point cloud map construction using multiresolution range images. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE International Conference on Intelligent Robots and Systems, pp. 10758–10765 (2020)

  12. Zhang, Q., Pless, R.: Extrinsic calibration of a camera and laser range finder (2003)

  13. Pandey, G., Mcbride, J., Savarese, S., Eustice, R.: Extrinsic calibration of a 3d laser scanner and an omnidirectional camera. IFAC Proc. Vol. 43(16), 336–341 (2010)

    Article  Google Scholar 

  14. Dhall, A., Chelani, K., Radhakrishnan, V., Krishna, K.M.: Lidar-camera calibration using 3d-3d point correspondences (2017)

  15. Wang, W., Ken, S., Nobuo, K.: Reflectance intensity assisted automatic and accurate extrinsic calibration of 3d lidar and panoramic camera using a printed chessboard. Remote Sens. 9(8), 851 (2017)

    Article  Google Scholar 

  16. Tamas, L., Kato, Z.: Targetless calibration of a lidar - perspective camera pair. In: 2013 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 668–675 (2013)

  17. Yuan, C., Liu, X., Hong, X., Zhang, F.: Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments. IEEE Robot. Autom. Lett. 6(4), 7517–7524 (2021)

    Article  Google Scholar 

  18. Weng, X., Wang, J., Held, D., Kitani, K.: Ab3dmot: A baseline for 3d multi-object tracking and new evaluation metrics. arXiv e-prints (2020)

  19. Kim, A., Osep, A., Leal-Taixe, L.: Eagermot: 3d multi-object tracking via sensor fusion. In: 2021 IEEE International Conference on Robotics And Automation (ICRA 2021), pp. 11315–11321 (2021)

  20. Arora, M., Wiesmann, L., Chen, X., Stachniss, C.: Static map generation from 3d lidar point clouds exploiting ground segmentation. Robot. And Autonom. Syst. 159, 104287 (2023)

  21. Lee, S., Kim, C., Cho, S., Myoungho, S., Jo, K.: Robust 3-dimension point cloud mapping in dynamic environment using point-wise static probability-based ndt scan-matching. IEEE Access 8, 175563–175575 (2020)

    Article  Google Scholar 

  22. Yao, Z., Chen, X., Xu, N., Gao, N., Ge, M.: Lidar-based simultaneous multi-object tracking and static mapping in nearshore scenario. Ocean Eng. 272, 113939 (2023)

  23. Zou, C., He, B., Zhang, L., Zhang, J.: Static map reconstruction and dynamic object tracking for a camera and laser scanner system. IET Comput. Vis. 12(4), 384–392 (2018)

    Article  Google Scholar 

  24. Pandey, G., Mcbride, J.R., Savarese, S., Eustice, R.M.: Automatic extrinsic calibration of vision and lidar by maximizing mutual information. J. Field Robot. 32(5), 696–722 (2015)

  25. Fu, B., Wang, Y., Ding, X., Jiao, Y., Xiong, R.: Lidar-camera calibration under arbitrary configurations: observability and methods. IEEE Trans. Instrum. Meas. PP(99), 1–1 (2019)

    Google Scholar 

  26. Iyer, G., Ram, R.K., Murthy, J.K., Krishna, K.M.: Calibnet: Geometrically supervised extrinsic calibration using 3d spatial transformer networks. In: Maciejewski, A., Okamura, A., Bicchi, A., Stachniss, C., Song, D., Lee, D., Chaumette, F., Ding, H., Li, J., Wen, J., Roberts, J., Masamune, K., Chong, N., Amato, N., Tsagwarakis, N., Rocco, P., Asfour, T., Chung, W., Yasuyoshi, Y., Sun, Y., Maciekeski, T., Althoefer, K., AndradeCetto, J., Chung, W., Demircan, E., Dias, J., Fraisse, P., Gross, R., Harada, H., Hasegawa, Y., Hayashibe, M., Kiguchi, K., Kim, K., Kroeger, T., Li, Y., Ma, S., Mochiyama, H., Monje, C., Rekleitis, I., Roberts, R., Stulp, F., Tsai, C., Zollo, L. (eds.) 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1110–1117 (2018)

  27. Sun, Y., Zuo, W., Huang, H., Cai, P., Liu, M.: Pointmoseg: Sparse tensor-based end-to-end moving-obstacle segmentation in 3-d lidar point clouds for autonomous driving. IEEE Robot. Autom. Lett. PP(99), 1–1 (2020)

    Google Scholar 

  28. Kim, J., Woo, J., Im, S.: Rvmos: range-view moving object segmentation leveraged by semantic and motion features. IEEE Robot. Autom. Lett. 7(3), 8044–8051 (2022)

    Article  Google Scholar 

  29. Charles, R.Q., Su, H., Kaichun, M., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017)

  30. Milioto, A., Vizzo, I., Behley, J., Stachniss, C.: Rangenet ++: Fast and accurate lidar semantic segmentation. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4213–4220 (2019)

  31. Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4490–4499 (2018)

  32. Lang, A.H., Vora, S., Caesar, H., Zhou, L., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12689–12697 (2019)

  33. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C.: SSD: Single shot MultiBox detector. In: Computer Vision—ECCV 2016 (2016)

  34. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017)

  35. Gao, Y., Qi, Z., Zhao, D.: Edge-enhanced instance segmentation by grid regions of interest. Vis. Comput. 39(3), 1137–1148 (2023)

    Article  Google Scholar 

  36. Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: Yolov4: optimal speed and accuracy of object detection. arXiv (2020)

  37. Wang, C.Y., Bochkovskiy, A., Liao, H.: Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv e-prints (2022)

  38. Pang, Z., Li, Z., Wang, N.: Simpletrack: Understanding and rethinking 3d multi-object tracking (2021)

  39. Oussalah, M., Schutter, J.D.: Hybrid fuzzy probabilistic data association filter and joint probabilistic data association filter. Inf. Sci. 142(1–4), 195–226 (2002)

    Article  Google Scholar 

  40. Blackman, S.S.: Multiple hypothesis tracking for multiple target tracking. IEEE Aerosp. Electron. Syst. Mag. 19(1), 5–18 (2009)

    Article  Google Scholar 

  41. Patel, A.S., Vyas, R., Vyas, O.P., Ojha, M., Tiwari, V.: Motion-compensated online object tracking for activity detection and crowd behavior analysis. Vis. Comput. 39(5), 2127–2147 (2023)

    Article  Google Scholar 

  42. Gao, X.S., Hou, X.R., Tang, J., Cheng, H.F.: Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 25(8), 930–943 (2003)

    Article  Google Scholar 

  43. Lepetit, V., Moreno-Noguer, F., Fua, P.: Epnp: an accurate o(n) solution to the pnp problem. Int. J. Comput. Vis. 81(2), 155–166 (2009)

    Article  Google Scholar 

  44. Li, Y., Fan, S., Sun, Y., Qiang, W., Sun, S.: Bundle adjustment method using sparse bfgs solution. Remote Sens. Lett. 9(8), 789–798 (2018)

    Article  Google Scholar 

  45. Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric [arxiv]. arXiv (2017)

  46. Zhang, J., Singh, S.: LOAM: Lidar Odometry and Mapping in Real-time. In: Proceedings of Robotics: Science and Systems (RSS’14) (2014)

  47. Shan, T., Englot, B.: Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain. IEEE (2019)

  48. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: IEEE Conference on Computer Vision & Pattern Recognition (2012)

  49. Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J.: Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection (2022)

  50. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.: Ros: an open-source robot operating system, vol. 3 (2009)

  51. Wang, W., Nobuhara, S., Nakamura, R., Sakurada, K.: SOIC: semantic online initialization and calibration for lidar and camera. arXiv (2020)

  52. Everingham, M., Gool, L.V., Williams, C., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 2, 88 (2010)

    Google Scholar 

  53. Sualeh, M., Kim, G.W.: Visual-lidar based 3d object detection and tracking for embedded systems. IEEE Access PP(99), 1–1 (2020)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the editorial department and the reviewers.

Funding

This work was supported by Guangdong Provincial Science and Technology Plan Project (Grant Number 2021B1515420006, Grant Number 2021B1515120026); Guangdong Province Marine Economic Development Special Fund Project (Six Major Marine Industries) (GDNRC [2021]46); National Natural Science Foundation of China (Grant Number U2141216, Grant Number 51875212); Shenzhen Technology Research Project (JSGG20201201100401005, JSGG20201 201100400001).

Author information

Authors and Affiliations

  1. School of Mechanical & Automotive Engineering, South China University of Technology, WuShan Street, Guangzhou, 510641, Guangdong, China

    Peng Chi, Haipeng Liao, Xiangmiao Wu, Jiyu Tian & Zhenmin Wang

  2. School of Computer Science & Engineering, South China University of Technology, Xiaoguwei Street, Guangzhou, 511400, Guangdong, China

    Qin Zhang

Authors
  1. Peng Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haipeng Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiangmiao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiyu Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhenmin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jiyu Tian or Zhenmin Wang.

Ethics declarations

Conflict of interest

No potential conflict of interest was reported by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chi, P., Liao, H., Zhang, Q. et al. Online static point cloud map construction based on 3D point clouds and 2D images. Vis Comput 40, 2889–2904 (2024). https://doi.org/10.1007/s00371-023-02992-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02992-x

Keywords

Search

Navigation