Abstract
Until recently, convolutional neural networks have dominated various machine vision fields-including stereo disparity estimation-with little to no competition. Vision transformers have shaken up this domination with the introduction of multiple models achieving state of art results in fields such as semantic segmentation and object detection. In this paper, we explore the viability of stereo transformers, which are attention-based models inspired from NLP applications, by designing a transformer-based stereo disparity estimation as well as an end-to-end transformer architectures for both feature extraction and feature matching. Our solution is not limited by a pre-set maximum disparity and manages to achieve state of the art on SceneFlow dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dzmitry, B., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR arXiv:1409.0473 (2015). n. pag
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. CoRR, arXiv (2020)
Chang, J., Chen, Y.: Pyramid stereo matching network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, pp. 5410–5418 (2018)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Yang, G., Manela, J., Happold, M., Ramanan, D.: Hierarchical deep stereo matching on high-resolution images. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 5510–5519 (2019)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), pp. 6000–6010. Curran Associates Inc., Red Hook (2017)
Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., Chen, L.-C.: Axial-DeepLab: stand-alone axial-attention for panoptic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 108–126. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_7
Hirschmuller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 807–814 (2005)
Herb, M.: Computing the Stereo Matching Cost with a Convolutional Neural Network Seminar Recent Trends in 3D Computer Vision (2015)
Khamis, S., Fanello, S., Rhemann, C., Kowdle, A., Valentin, J.P.C., Izadi, S.: StereoNet: guided hierarchical refinement for real-time edge-aware depth prediction. In: ECCV (2018)
Li, Z., et al.: Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers. In: IEEE/CVF International Conference on Computer Vision (ICCV), vol. 2021, pp. 6177–6186 (2021). https://doi.org/10.1109/ICCV48922.2021.00614
Ohta, Y., Kanade, T.: Stereo by intra- and inter-scanline search using dynamic programming. IEEE Trans. Pattern Anal. Mach. Intell. 7(2), 139–154 (1985)
Cheng, X., et al.: Hierarchical neural architecture search for deep stereo matching. In: Lorochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems (NeurIPS 2020), December 2020, vol. 33. Neural Information Processing Systems (NIPS) (2020)
Zheng, S., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6877–6886. IEEE (2021)
Zhou, Y., Barnes, C., Shechtman, E., Amirghodsi, S.: TransFill: reference-guided image inpainting by merging multiple color and spatial transformations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2266–2276. IEEE (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. NAACL (2019)
Brown, T., et al.: Language Models are Few-Shot Learners (2020)
Fuzhi, Y., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5790–5799 (2020)
Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: transformer for semantic segmentation. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, pp. 7242–7252 (2021)
Dai, J., et al.: Deformable convolutional networks. In: IEEE International Conference on Computer Vision (ICCV), vol. 2017, pp. 764–773 (2017)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, vol. 2018, pp. 7132–7141 (2018)
Hu, H., Zhang, Z., Xie, Z., Lin, S.: Local relation networks for image recognition. In: IEEE/CVF International Conference on Computer Vision (ICCV), vol. 2019, pp. 3463–3472 (2019)
Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 4040–4048 (2016)
Moritz, M., Heipke, C., Geiger, A.: Joint 3D estimation of vehicles and scene flow. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2, 427–434 (2015)
Wang, L., et al.: Parallax attention for unsupervised stereo correspondence learning. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020.3026899
Xu, H., Zhang, J.: AANet: adaptive aggregation network for efficient stereo matching. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 1956–1965 (2020)
Tankovich, V., Häne, C., Zhang, Y., Kowdle, A., Fanello, S., Bouaziz, S.: HITNet: hierarchical iterative tile refinement network for real-time stereo matching. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2021, pp. 14357–14367 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Bengana, N., Mustaniemi, J., Heikkilä, J. (2022). Seeking Attention: Using Full Context Transformers for Better Disparity Estimation. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-09037-0_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09036-3
Online ISBN: 978-3-031-09037-0
eBook Packages: Computer ScienceComputer Science (R0)