Abstract
We propose an efficient multi-view stereo (MVS) network for inferring depth value from multiple RGB images. Recent studies use the cost volume to encode the matching correspondence between different views, but this structure can still be optimized from the perspective of image features. First of all, to fully aggregate the dominant interrelationship from input images, we introduce a self-attention mechanism to our feature extractor, which can accurately model long-range dependencies between adjacent pixels. Secondly, to unify the extracted feature maps into the MVS problem, we further design an efficient feature-wise loss function, which constrains the corresponding feature vectors more spatially distinctive during training. The robustness and accuracy of the reconstructed point cloud are improved by enhancing the reliability of correspondence matches. Finally, to reduce the extra memory burden caused by the above methods, we follow the coarse to fine strategy. The group-wise correlation and uncertainty estimates are combined to construct a lightweight cost volume. This can improve the efficiency and generalization performance of the network while ensuring the reconstruction effect. We further combine the previous steps to get what we called attention thin volume. Quantitative and qualitative experiments are presented to demonstrate the performance of our model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Ji, M., Gall, J., Zheng, H., Liu, Y., Fang, L.: Surfacenet: an end-to-end 3d neural network for multiview stereopsis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2307–2315 (2017)
Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: MVSNet: depth inference for unstructured multi-view stereo. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 785–801. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_47
Cheng, S., et al.: Deep stereo using adaptive thin volume representation with uncertainty awareness. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2524–2534 (2020)
Guo, C., Szemenyei, M., Yi, Y., Wang, W., Chen, B., Fan, C.: Sa-unet: spatial attention u-net for retinal vessel segmentation. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 1236–1242. IEEE, January 2021
Huang, P.H., Matzen, K., Kopf, J., Ahuja, N., Huang, J.B.: Deepmvs: learning multi-view stereopsis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2821–2830 (2018)
Luo, K., Guan, T., Ju, L., Huang, H., Luo, Y.: P-mvsnet: learning patch-wise matching confidence aggregation for multi-view stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10452–10461 (2019)
Zhang, X., Hu, Y., Wang, H., Cao, X., Zhang, B.: Long-range attention network for multi-view stereo. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3782–3791 (2021)
Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., Quan, L.: Recurrent mvsnet for high-resolution multi-view stereo depth inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5525–5534 (2019)
Zhang, K., Liu, M., Zhang, J., Dong, Z.: Pa-mvsnet: sparse-to-dense multi-view stereo with pyramid attention. IEEE Access 9, 27908–27915 (2021)
Yang, J., Mao, W., Alvarez, J.M., Liu, M.: Cost volume pyramid based depth inference for multi-view stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4877–4886 (2020)
Gu, X., Fan, Z., Zhu, S., Dai, Z., Tan, F., Tan, P.: Cascade cost volume for high-resolution multi-view stereo and stereo matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2495–2504 (2020)
Yi, Hongwei, Wei, Zizhuang, Ding, Mingyu, Zhang, Runze, Chen, Yisong, Wang, Guoping, Tai, Yu-Wing.: Pyramid multi-view stereo net with self-adaptive view aggregation. In: Vedaldi, Andrea, Bischof, Horst, Brox, Thomas, Frahm, Jan-Michael. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 766–782. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_44
Chen, R., Han, S., Xu, J., Su, H.: Point-based multi-view stereo network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1538–1547 (2019)
Xue, Y., et al.: Mvscrf: learning multi-view stereo with conditional random fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4312–4321 (2019)
Luo, K., Guan, T., Ju, L., Wang, Y., Chen, Z., Luo, Y.: Attention-aware multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1590–1599 (2020)
Yu, A., Guo, W., Liu, B., Chen, X., Wang, X., Cao, X., Jiang, B.: Attention aware cost volume pyramid based multi-view stereo network for 3d reconstruction. ISPRS J. Photogrammetry Remote Sens. 175, 448–460 (2021)
Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H.: Large scale multi-view stereopsis evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 406–413 (2014)
Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)
Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 873–881 (2015)
Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, vol. 7, June 2006
Guo, X., Yang, K., Yang, W., Wang, X., Li, H.: Group-wise correlation stereo network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3273–3282 (2019)
Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., Shlens, J.: Stand-alone self-attention in vision models. In: Advances in Neural Information Processing Systems, 32 (2019)
Acknowledgments
The authors would like to thank all anonymous reviewers. This work was supported by the National Key Research and Development Program of China [grant number 2021YFF0901203].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wan, Z., Xu, C., Hu, J., Xiao, J., Meng, Z., Chen, J. (2022). Multi-view Stereo Network with Attention Thin Volume. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13631. Springer, Cham. https://doi.org/10.1007/978-3-031-20868-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-20868-3_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20867-6
Online ISBN: 978-3-031-20868-3
eBook Packages: Computer ScienceComputer Science (R0)