Nils Plath
ABSTRACT
Many 2D-to-3D conversion techniques rely on image-based rendering methods in order to synthesize 3D views from monoscopic images. This leads to holes in the generated views due to previously occluded objects becoming visible for which no texture information is available. Approaches attempting to alleviate the effects of these artifacts are referred to as hole-filling. This paper proposes a method which determines a non-uniform deformation of the stereoscopic view such that no holes are visible. Additionally, an energy term is devised, which prevents straight lines in the input image from being bent due to the non-uniform image warp. This is achieved by constructing a triangle mesh, which approximates the depth map of the input image and by integrating a set of detected lines into it. The line information is incorporated into the underlying optimization problem in order to prevent bending of the lines. The evaluation of the proposed algorithm on a comprehensive dataset with a variety of scenes shows that holes are efficiently filled without obvious background distortions.
Supplemental Material
Available for Download
Supplemental material.
- F. Bornemann and T. März. Fast image inpainting based on coherence transport. Journal of Mathematical Imaging and Vision, 28(3):259--278, jul 2007. Google Scholar
Digital Library
- D. Doria and R. J. Radke. Filling large holes in lidar data by inpainting depth gradients. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on, pages 65--72. IEEE, 2012.Google ScholarCross Ref
- D. Farin, R. Peerlings, and P. de With. Depth-image representation employing meshes for intermediate-view rendering and coding. In 3DTV Conference, 2007, pages 1--4, May 2007.Google Scholar
- M. Farre, O. Wang, M. Lang, N. Stefanoski, A. Hornung, and A. Smolic. Automatic content creation for multiview autostereoscopic displays using image domain warping. In Multimedia and Expo (ICME), 2011 IEEE International Conference on, pages 1--6, 2011. Google ScholarDigital Library
- C. Fehn. Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv. In Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems, volume 5291, pages 93--104, 2004.Google Scholar
- R. Gal, O. Sorkine, and D. Cohen-Or. Feature-aware texturing. In Proceedings of EUROGRAPHICS Symposium on Rendering, volume 11, pages 297--303, 2006. Google ScholarDigital Library
- R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004. Google ScholarDigital Library
- K. He, H. Chang, and J. Sun. Content-aware rotation. Computer Vision, IEEE International Conference on, 0:553--560, 2013. Google ScholarDigital Library
- L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 20(11):1254--1259, Nov 1998. Google ScholarDigital Library
- M. Lang, A. Hornung, O. Wang, S. Poulakos, A. Smolic, and M. Gross. Nonlinear disparity mapping for stereoscopic 3d. ACM Transactions on Graphics (proceedings of ACM SIGGRAPH), 29(3):1, 2010. Google ScholarDigital Library
- Y. Liu and V. Caselles. Exemplar-based image inpainting using multiscale graph cuts. Image Processing, IEEE Transactions on, 22(5):1699--1711, May 2013.Google ScholarDigital Library
- H. Oh and Y.-S. Ho. H.264-based depth map sequence coding using motion information of corresponding texture video. In L.-W. Chang and W.-N. Lie, editors, Advances in Image and Video Technology, volume 4319 of Lecture Notes in Computer Science, pages 898--907. Springer Berlin Heidelberg, 2006. Google ScholarDigital Library
- N. Plath, S. Knorr, L. Goldmann, and T. Sikora. Adaptive image warping for hole prevention in 3d view synthesis. Image Processing, IEEE Transactions on, 22(9):3420--3432, 2013.Google ScholarDigital Library
- M. Rubinstein, D. Gutierrez, O. Sorkine, and A. Shamir. A comparative study of image retargeting. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia), 29(5):160:1--160:10, 2010. Google ScholarDigital Library
- M. Sarkis, W. Zia, and K. Diepold. Fast depth map compression and meshing with compressed tritree. In H. Zha, R.-i. Taniguchi, and S. Maybank, editors, Computer Vision -- ACCV 2009, volume 5995 of Lecture Notes in Computer Science, pages 44--55. Springer Berlin Heidelberg, 2010. Google ScholarDigital Library
- D. Scharstein and C. Pal. Learning conditional random fields for stereo. In Computer Vision and Pattern Recognition, 2007. CVPR '07. IEEE Conference on, pages 1--8, June 2007.Google ScholarCross Ref
- H. Shum and S. B. Kang. Review of image-based rendering techniques. In Proceedings of SPIE Visual Communications and Image Processing, volume 4067, pages 2--13, 2000.Google ScholarCross Ref
- A. Smolic, S. Poulakos, S. Heinzle, P. Greisen, M. Lang, A. Hornung, M. Farre, N. Stefanoski, O. Wang, L. Schnyder, R. Monroy, and M. Gross. Disparity-aware stereo 3d production tools. In Visual Media Production (CVMP), 2011 Conference for, pages 165--173, Nov 2011. Google ScholarDigital Library
- J. Sun and H. Ling. Scale and object aware image thumbnailing. International Journal of Computer Vision, 104(2):135--153, 2013. Google ScholarDigital Library
- R. Tanger, M. Müller, P. Kauff, and R. Schäfer. Depth/disparity creation for trifocal hybrid 3d system. In SMPTE Conferences, volume 2013, pages 1--8. Society of Motion Picture and Television Engineers, 2013.Google ScholarCross Ref
- C. Vazquez, L. Zhang, F. Speranza, N. Plath, and S. Knorr. 2D-to-3D Video Conversion: Overview and Perspectives, pages 37--61. John Wiley & Sons, Ltd, 2013.Google Scholar
- R. G. von Gioi, J. Jakubowicz, J.-M. Morel, and G. Randall. Lsd: a line segment detector. Pattern Analysis and Machine Intelligence (PAMI), IEEE Transactions on, 32(4):722--732, 2012.Google Scholar
- O. Wang, M. Lang, M. Frei, A. Hornung, A. Smolic, and M. Gross. Stereobrush: Interactive 2d to 3d conversion using discontinuous warps. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling, SBIM '11, pages 47--54, New York, NY, USA, 2011. ACM. Google ScholarDigital Library
- T. Yan, W. Lau, Rynson, Y. Xu, and L. Huang. Depth mapping for stereoscopic videos. International Journal of Computer Vision, 102(1--3):293--307, 2013. Google ScholarDigital Library
- M. Zwicker, H. Pfister, J. van Baar, and M. Gross. Ewa splatting. IEEE Transactions on Visualization and Computer Graphics, 8:223--238, 2002. Google ScholarDigital Library
Index Terms
- Line-preserving hole-filling for 2D-to-3D conversion
Recommendations
Development of Semi-Automatic Methodology for Extraction of Depth for 2D-to-3D Conversion
ICMLC '17: Proceedings of the 9th International Conference on Machine Learning and ComputingRecently, stereoscopic image/video conversion has been most demanding as it has a great stereo video experience. This paper presents the development of the methodology for generation of the depth map used for stereoscopic view. Normally, Stereoscopic ...
Depth image based hole-filling algorithm for 2D to 3D conversion
SA '14: SIGGRAPH Asia 2014 Posters2D to 3D conversion technologies had been researching with developing of 3D display. Existing 2D movies can be converted to 3D stereoscopic movies easily with low cost using conversion technology. Depth image-based rendering (DIBR) technique is applied ...
An Efficient Hole-Filling Approach Using Adaptive Rendering
ICGEC '12: Proceedings of the 2012 Sixth International Conference on Genetic and Evolutionary ComputingBased on depth image based rendering (DIBR), this paper presents an efficient hole-filling approach using adaptive rendering for the 2D-to-3D conversion system, i.e., converting monocular image to multi-view virtual images. the traditional processing of ...
Comments