Skip to main content
SpringerLink
Log in

Overview of Data Fusion in Autonomous Driving Perception

  • Chapter
  • First Online:
Multi-sensor Fusion for Autonomous Driving

  • 399 Accesses

Abstract

In autonomous driving, research on data fusion has influential academic and application value. This chapter is proposed to summarize the data fusion methods of autonomous driving in recent years. Firstly, the development of deep object detection and data fusion in autonomous driving is introduced, as well as existing reviews. From three aspects of multimodal object detection, fusion levels, and calculation methods, the cutting-edge progress in this field is comprehensively shown. Finally, open issues are discussed, and the performance, challenges, and prospects are summarized.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Atzmon, M., Maron, H., Lipman, Y.: Point convolutional neural networks by extension operators. Preprint (2018). arXiv:1803.10091

    Google Scholar 

  2. Bai, M., Mattyus, G., Homayounfar, N., Wang, S., Lakshmikanth, S.K., Urtasun, R.: Deep multi-sensor lane detection. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3102–3109. IEEE (2018)

    Google Scholar 

  3. Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1907–1915 (2017)

    Google Scholar 

  4. Cheng, X., Zhong, Y., Dai, Y., Ji, P., Li, H.: Noise-aware unsupervised deep lidar-stereo fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6339–6348 (2019)

    Google Scholar 

  5. Frossard, D., Urtasun, R.: End-to-end learning of multi-sensor 3d tracking by detection. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 635–642. IEEE (2018)

    Google Scholar 

  6. Guan, H., Yan, W., Yu, Y., Zhong, L., Li, D.: Robust traffic-sign detection and classification using mobile lidar data with digital images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11(5), 1715–1724 (2018)

    Article  Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  8. Hua, B.S., Tran, M.K., Yeung, S.K.: Pointwise convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 984–993 (2018)

    Google Scholar 

  9. Klokov, R., Lempitsky, V.: Escape from cells: deep kd-networks for the recognition of 3d point cloud models. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 863–872 (2017)

    Google Scholar 

  10. Ku, J., Mozifian, M., Lee, J., Harakeh, A., Waslander, S.L.: Joint 3d proposal generation and object detection from view aggregation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–8. IEEE (2018)

    Google Scholar 

  11. Lei, H., Akhtar, N., Mian, A.: Octree guided cnn with spherical kernels for 3d point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9631–9640 (2019)

    Google Scholar 

  12. Liang, M., Yang, B., Chen, Y., Hu, R., Urtasun, R.: Multi-task multi-sensor fusion for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7345–7353 (2019)

    Google Scholar 

  13. Lv, X., Liu, Z., Xin, J., Zheng, N.: A novel approach for detecting road based on two-stream fusion fully convolutional network. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1464–1469. IEEE (2018)

    Google Scholar 

  14. Ma, F., Karaman, S.: Sparse-to-dense: depth prediction from sparse depth samples and a single image. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4796–4803. IEEE (2018)

    Google Scholar 

  15. Ma, L., Li, Y., Li, J., Wang, C., Wang, R., Chapman, M.A.: Mobile laser scanned point-clouds for road object detection and extraction: a review. Remote Sens. 10(10), 1531 (2018)

    Article  Google Scholar 

  16. Mogelmose, A., Trivedi, M.M., Moeslund, T.B.: Vision-based traffic sign detection and analysis for intelligent driver assistance systems: perspectives and survey. IEEE Trans. Intell. Transp. Syst. 13(4), 1484–1497 (2012)

    Article  Google Scholar 

  17. Narote, S.P., Bhujbal, P.N., Narote, A.S., Dhane, D.M.: A review of recent advances in lane detection and departure warning system. Pattern Recogn. 73, 216–234 (2018)

    Article  Google Scholar 

  18. Park, K., Kim, S., Sohn, K.: High-precision depth estimation using uncalibrated lidar and stereo fusion. IEEE Trans. Intell. Transp. Syst. 21(1), 321–335 (2019)

    Article  Google Scholar 

  19. Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3d object detection from RGB-D data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 918–927 (2018)

    Google Scholar 

  20. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)

    Google Scholar 

  21. Qi, C.R., Su, H., Nießner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view CNNs for object classification on 3d data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5648–5656 (2016)

    Google Scholar 

  22. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. Preprint (2017). arXiv:1706.02413

    Google Scholar 

  23. Tang, J., Tian, F.P., Feng, W., Li, J., Tan, P.: Learning guided convolutional network for depth completion. IEEE Trans. Image Process. 30, 1116–1129 (2020)

    Article  Google Scholar 

  24. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6411–6420 (2019)

    Google Scholar 

  25. Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4604–4612 (2020)

    Google Scholar 

  26. Wu, W., Qi, Z., Fuxin, L.: Pointconv: deep convolutional networks on 3d point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9621–9630 (2019)

    Google Scholar 

  27. Xiao, L., Dai, B., Liu, D., Hu, T., Wu, T.: Crf based road detection with multi-sensor fusion. In: 2015 IEEE Intelligent Vehicles Symposium (IV), pp. 192–198. IEEE (2015)

    Google Scholar 

  28. Xiao, L., Wang, R., Dai, B., Fang, Y., Liu, D., Wu, T.: Hybrid conditional random field based camera-lidar fusion for road detection. Inf. Sci. 432, 543–558 (2018)

    Article  MathSciNet  Google Scholar 

  29. Xu, D., Anguelov, D., Jain, A.: Pointfusion: deep sensor fusion for 3d bounding box estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 244–253 (2018)

    Google Scholar 

  30. Yang, Z., Sun, Y., Liu, S., Shen, X., Jia, J.: Ipod: intensive point-based object detector for point cloud. Preprint (2018). arXiv:1812.05276

    Google Scholar 

  31. Zhou, Y., Tuzel, O.: Voxelnet: end-to-end learning for point cloud based 3d object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490–4499 (2018)

    Google Scholar 

  32. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The School of Vehicle and Mobility, Tsinghua University, Beijing, China

    Xinyu Zhang

  2. School of Vehicle and Mobility, Tsinghua University, Beijing, China

    Jun Li, Mo Zhou & Li Wang

  3. College of Information Science and Technology, Beijing University of Chemical Technology, Beijing, China

    Zhiwei Li

  4. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Huaping Liu

  5. The School of Vehicle and Mobility, Tsinghua University, Beijing, China

    Zhenhong Zou

Authors
  1. Xinyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiwei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huaping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mo Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Li Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhenhong Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Zhang, X. et al. (2023). Overview of Data Fusion in Autonomous Driving Perception. In: Multi-sensor Fusion for Autonomous Driving. Springer, Singapore. https://doi.org/10.1007/978-981-99-3280-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-3280-1_2

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-3279-5

  • Online ISBN: 978-981-99-3280-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation