Abstract
Transparent objects and surfaces are pervasive in man-made environments and need to be considered in any vision system. Accurate depth data is a key factor for such systems reliability, requiring transparency to be inferred, due to the sensing challenges. However, the current state-of-the-art methods to predict the depth of such objects are not reliable enough to ensure safe operation of robots in arbitrary complex scenes. In order to better understand and improve upon existing solutions, we evaluate the performance of a variety of depth estimation methods. Doing so, we disentangle the different factors impacting their performance. Among our findings, neural radiance fields offer the best accuracy, but are very sensitive to the number of images used to understand the scene, and do not benefit from any level of object understanding to help them fill in the gaps.
Supported by the EU-program EC Horizon 2020 for Research and Innovation under grant agreement No. 101017089, project TraceBot.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Chen, W., Fu, Z., Yang, D., Deng, J.: Single-image depth perception in the wild. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS 2016, Red Hook, NY, USA, pp. 730–738. Curran Associates Inc. (2016)
Chen, X., Zhang, H., Yu, Z., Opipari, A., Jenkins, O.C.: ClearPose: large-scale transparent object dataset and benchmark. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13668, pp. 381–396. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20074-8_22
Fang, H., Fang, H.S., Xu, S., Lu, C.: TransCG: a large-scale real-world dataset for transparent object depth completion and a grasping baseline. IEEE Robot. Autom. Lett. 7(3), 7383–7390 (2022). https://doi.org/10.1109/LRA.2022.3183256
Gupta, A., Dollar, P., Girshick, R.: LVIS: a dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
Ichnowski, J., Avigal, Y., Kerr, J., Goldberg, K.: Dex-NeRF: using a neural radiance field to grasp transparent objects. In: Conference on Robot Learning (CoRL) (2021)
Jung, H., et al.: On the importance of accurate geometry data for dense 3D vision tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 780–791 (2023)
Kerr, J., et al.: Evo-NeRF: evolving nerf for sequential robot grasping of transparent objects. In: Liu, K., Kulic, D., Ichnowski, J. (eds.) Proceedings of the 6th Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 205, pp. 353–367. PMLR, 14–18 December 2023. https://proceedings.mlr.press/v205/kerr23a.html
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24
Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41(4), 1–15 (2022). https://doi.org/10.1145/3528223.3530127
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Sajjan, S., et al.: Clear grasp: 3D shape estimation of transparent objects for manipulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3634–3642 (2020). https://doi.org/10.1109/ICRA40945.2020.9197518
Suchi, M., Neuberger, B., Salykov, A., Weibel, J.B., Patten, T., Vincze, M.: 3D-DAT: 3D-dataset annotation toolkit for robotic vision. In: 2023 IEEE International Conference on Robotics and Automation (ICRA) (2023)
Tian, Z., Shen, C., Chen, H., He, T.: FCOS: fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627–9636 (2019)
Yin, W., et al.: Learning to recover 3D scene shape from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Zhu, L., et al.: RGB-D local implicit function for depth completion of transparent objects. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4647–4656 (2021). https://doi.org/10.1109/CVPR46437.2021.00462
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Weibel, JB., Sebeto, P., Thalhammer, S., Vincze, M. (2023). Challenges of Depth Estimation for Transparent Objects. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2023. Lecture Notes in Computer Science, vol 14361. Springer, Cham. https://doi.org/10.1007/978-3-031-47969-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-47969-4_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47968-7
Online ISBN: 978-3-031-47969-4
eBook Packages: Computer ScienceComputer Science (R0)