Abstract
Depth recovery from light field is an essential part of many light field applications. However, conventional methods usually suffers from two challenges: sub-pixel displacements and occlusions. In this paper, we propose an effective convolutional neural network (CNN) framework to perform the depth estimation on 4-dimensional (4D) light field. Based on the orientation-depth relationship of epipolar images (EPIs), we firstly build a training set by extracting a group of valid EPI-patch pairs with balanced depth distribution, and then an EPI-patch based CNN architecture is designed and trained to estimate the disparity of each pixel. Finally, a post-processing with global constrains is applied to the whole images to refine the output of CNN. Experimental results demonstrate the effectiveness and robustness of our method.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Jeon, H., Park, J., Choe, G., Park, J., Bok, Y., Tai, Y., Kweon, I.: Accurate depth map estimation from a lenslet light field camera. In: CVPR (2015)
Wang, T., Efros, A., Ramamoorthi, R.: Depth estimation with occlusion modeling using light-field cameras. IEEE TPAMI 38(11), 2170–2181 (2016)
Williem and Park, I.: Robust light field depth estimation for noisy scene with occlusion. In: CVPR (2016)
Johannsen, O., Sulc, A., Goldluecke, B.: What sparse light field coding reveals about scene structure. In: CVPR, pp. 3262–3270 (2016)
Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: ACCV (2016)
4D Light Field Benchmark Dataset and Evaluation. http://hci-lightfield.iwr.uni-heidelberg.de/
Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: Vision, Modelling and Visualization (2013)
Georgiev, T., Lumsdaine, A.: Reducing plenoptic camera artifacts. Comput. Graph. Forum 29(6), 1955–1968 (2010)
Yu, Z., Guo, X., Ling, H., Lumsdaine, A., Yu, J.: Line assisted light field triangulation and stereo matching. In: ICCV (2013)
Lin, H., Chen, C., Kang, S.B., Yu, J.: Depth recovery from light field using focal stack symmetry. In: ICCV (2015)
Tao, M., Srinivasan, P., Hadap, S., Rusinkiewicz, S., Malik, J., Ramamoorthi, R.: Shape estimation from shading, defocus, and correspondence using light-field angular coherence. In: IEEE TPAMI (2015)
Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.: Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. (Proc. SIGGRAPH) 32(4) (2013)
Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE TPAMI 36(3), 606–619 (2014)
Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D light fields. In: CVPR, pp. 41–48 (2012)
Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Trans. Graph. (TOG) 24, 765–776 (2005)
Marwah, K., Wetzstein, G., Bando, Y., Raskar, R.: Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Trans. Graph. (Proc. SIGGRAPH) 32, 1–11 (2013)
Mousnier, A., Vural, E., Guillemot, C.: Partial light field tomographic reconstruction from a fixed-camera focal stack. arXiv preprint (2015). arXiv:1503.01903
Rerabek, M., Ebrahimi, T.: New light field image dataset. In: 8th International Conference on Quality of Multimedia Experience (QoMEX) (2016)
Wang, T.-C., Zhu, J.-Y., Hiroaki, E., Chandraker, M., Efros, A.A., Ramamoorthi, R.: A 4D light-field dataset and CNN architectures for material recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 121–138. Springer, Cham (2016). doi:10.1007/978-3-319-46487-9_8
Fischer, P., Dosovitskiy, A., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V.: FlowNet: Learning optical flow with convolutional networks. In: ICCV (2015)
Luo, W., Schwing, A.G., Urtasun, R.: Efficient deep learning for stereo matching,. In: CVPR, pp. 5695–5703 (2016)
Pulgar, F., Riveragar, A., Charte, F., del Jesus, M.: On the impact of imbalanced data in convolutional neural networks performance. In: de Pisón, F.M., Urraca, R., Quintián, H., Corchado, E. (eds.) HAIS 2017. LNCS, vol. 10334, pp. 220–232. Springer, Cham (2017). doi:10.1007/978-3-319-59650-1_19
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint (2015). arXiv:1502.03167
Delong, A., Osokin, A., Isack, H., Boykov, Y.: Fast approximate energy minimization with label costs. IJCV 96(1), 1–27 (2012)
Zhang, S., Sheng, H., Li, C., Zhang, J., Xiong, Z.: Robust depth estimation for light field via spinning parallelogram operator. Comput. Vis. Image Underst. 145, 148–159 (2016)
Strecke, M., Alperovich, A., Goldluecke, B.: Accurate depth and normal maps from occlusion-aware focal stack symmetry. In: CVPR (2017)
Acknowledgments
This work is supported in part by the National High-tech R&D Program of China (863 Program, 2015AA015901), Key Program of Zhejiang Provincial Natural Science Foundation of China (No. LZ14F020003), and International Cooperation and Exchange of the National Natural Science Foundation of China (No. 2014DFA12040).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Luo, Y., Zhou, W., Fang, J., Liang, L., Zhang, H., Dai, G. (2017). EPI-Patch Based Convolutional Neural Network for Depth Estimation on 4D Light Field. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10636. Springer, Cham. https://doi.org/10.1007/978-3-319-70090-8_65
Download citation
DOI: https://doi.org/10.1007/978-3-319-70090-8_65
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70089-2
Online ISBN: 978-3-319-70090-8
eBook Packages: Computer ScienceComputer Science (R0)