Kevin Karsch
David Forsyth
ABSTRACT
We propose a method to realistically insert synthetic objects into existing photographs without requiring access to the scene or any additional scene measurements. With a single image and a small amount of annotation, our method creates a physical model of the scene that is suitable for realistically rendering synthetic objects with diffuse, specular, and even glowing materials while accounting for lighting interactions between the objects and the scene. We demonstrate in a user study that synthetic images produced by our method are confusable with real scenes, even for people who believe they are good at telling the difference. Further, our study shows that our method is competitive with other insertion methods while requiring less scene information. We also collected new illumination and reflectance datasets; renderings produced by our system compare well to ground truth. Our system has applications in the movie and gaming industry, as well as home decorating and user content creation, among others.
Supplemental Material
- Alnasser, M., and Foroosh, H. 2006. Image-based rendering of synthetic diffuse objects in natural scenes. In ICPR, 787--790. Google Scholar
Digital Library
- Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. In Comp. Vision Sys., 3--26.Google Scholar
- Blake, A. 1985. Boundary conditions for lightness computation in mondrian world. Computer Vision, Graphics and Image Processing 32, 314--327.Google ScholarCross Ref
- Boivin, S., and Gagalowicz, A. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. ACM SIGGRAPH, 107--116. Google ScholarDigital Library
- Brelstaff, G., and Blake, A. 1987. Computing lightness. Pattern Recognition Letters 5, 2, 129--138. Google ScholarDigital Library
- Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. 30 (August), 43:1--43:10. Google ScholarDigital Library
- Cossairt, O., Nayar, S., and Ramamoorthi, R. 2008. Light field transfer: global illumination between real and synthetic objects. ACM Trans. Graph. 27 (August), 57:1--57:6. Google ScholarDigital Library
- Criminisi, A., Reid, I., and Zisserman, A. 2000. Single view metrology. Int. J. Comput. Vision 40 (November), 123--148. Google ScholarDigital Library
- Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, SIGGRAPH '98, 189--198. Google ScholarDigital Library
- Farenzena, M., and Fusiello, A. 2007. Recovering intrinsic images using an illumination invariant image. In ICIP, 485--488.Google Scholar
- Fournier, A., Gunawan, A. S., and Romanzin, C. 1993. Common illumination between real and computer generated scenes. In Proceedings of Graphics Interface '93, 254--262.Google Scholar
- Funt, B. V., Drew, M. S., and Brockington, M. 1992. Recovering shading from color images. In ECCV, 124--132. Google ScholarDigital Library
- Furukawa, Y., and Ponce, J. 2010. Accurate, dense, and robust multiview stereopsis. IEEE PAMI 32 (August), 1362--1376. Google ScholarDigital Library
- Greger, G., Shirley, P., Hubbard, P. M., and Greenberg, D. P. 1998. The irradiance volume. IEEE Computer Graphics and Applications 18, 32--43. Google ScholarDigital Library
- Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, 2335--2342.Google Scholar
- Guo, R., Dai, Q., and Hoiem, D. 2011. Single-image shadow detection and removal using paired regions. In CVPR, 2033--2040. Google ScholarDigital Library
- Hartley, R., and Zisserman, A. 2003. Multiple View Geometry in Computer Vision, 2 ed. Cambridge University Press, New York, NY, USA. Google ScholarDigital Library
- Hedau, V., Hoiem, D., and Forsyth, D. 2009. Recovering the spatial layout of cluttered rooms. In ICCV, 1849--1856.Google Scholar
- Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM Trans. Graph. 24 (July), 577--584. Google ScholarDigital Library
- Horn, B. K. P. 1974. Determining lightness from an image. Computer Vision, Graphics and Image Processing 3, 277--299.Google ScholarCross Ref
- Horry, Y., Anjyo, K.-I., and Arai, K. 1997. Tour into the picture: using a spidery mesh interface to make animation from a single image. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, SIGGRAPH '97, 225--232. Google ScholarDigital Library
- Kang, H. W., Pyo, S. H., Anjyo, K., and Shin, S. Y. 2001. Tour into the picture using a vanishing line and its extension to panoramic images. Computer Graphics Forum 20, 3, 132--141.Google ScholarCross Ref
- Kee, E., and Farid, H. 2010. Exposing digital forgeries from 3-d lighting environments. In WIFS, 1--6.Google Scholar
- Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25 (July), 654--663. Google ScholarDigital Library
- Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In ICCV, 1--8.Google Scholar
- Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. Graph. 26 (July). Google ScholarDigital Library
- Lalonde, J.-F., Efros, A. A., and Narasimhan, S. G. 2009. Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. 28 (December), 131:1--131:10. Google ScholarDigital Library
- Land, E., and McCann, J. 1971. Lightness and retinex theory. J. Opt. Soc. Am. 61, 1, 1--11.Google ScholarCross Ref
- Lee, D. C., Hebert, M., and Kanade, T. 2009. Geometric reasoning for single image structure recovery. In CVPR, 2136--2143.Google Scholar
- Lee, D. C., Gupta, A., Hebert, M., and Kanade, T. 2010. Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. Advances in Neural Information Processing Systems (NIPS) 24 (November), 1288--1296.Google Scholar
- Levin, A., Rav-Acha, A., and Lischinski, D. 2008. Spectral matting. IEEE PAMI 30 (October), 1699--1712. Google ScholarDigital Library
- Liebowitz, D., Criminisi, A., and Zisserman, A. 1999. Creating architectural models from images. In Eurographics, vol. 18, 39--50.Google ScholarCross Ref
- Lopez-Moreno, J., Hadap, S., Reinhard, E., and Gutierrez, D. 2010. Compositing images through light source detection. Computers & Graphics 34, 6, 698--707. Google ScholarDigital Library
- Merrell, P., Schkufza, E., Li, Z., Agrawala, M., and Koltun, V. 2011. Interactive furniture layout using interior design guidelines. ACM Trans. Graph. 30 (August), 87:1--87:10. Google ScholarDigital Library
- Mury, A. A., Pont, S. C., and Koenderink, J. J. 2009. Representing the light field in finite three-dimensional spaces from sparse discrete samples. Applied Optics 48, 3 (Jan), 450--457.Google ScholarCross Ref
- Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, SIGGRAPH '01, 433--442. Google ScholarDigital Library
- Rother, C. 2002. A new approach to vanishing point detection in architectural environments. IVC 20, 9--10 (August), 647--655.Google ScholarCross Ref
- Sato, I., Sato, Y., and Ikeuchi, K. 2003. Illumination from shadows. IEEE PAMI 25, 3, 290--300. Google ScholarDigital Library
- Saxena, A., Sun, M., and Ng, A. Y. 2008. Make3d: depth perception from a single still image. In Proceedings of the 23rd national conference on Artificial intelligence - Volume 3, AAAI Press, 1571--1576. Google ScholarDigital Library
- Sinha, S. N., Steedly, D., Szeliski, R., Agrawala, M., and Pollefeys, M. 2008. Interactive 3d architectural modeling from unordered photo collections. ACM Trans. Graph. 27 (December), 159:1--159:10. Google ScholarDigital Library
- Tappen, M. F., Freeman, W. T., and Adelson, E. H. 2005. Recovering intrinsic images from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 27 (September), 1459--1472. Google ScholarDigital Library
- Tappen, M. F., Adelson, E. H., and Freeman, W. T. 2006. Estimating intrinsic component images using non-linear regression. In CVPR, vol. 2, 1992--1999. Google ScholarDigital Library
- Wang, Y., and Samaras, D. 2003. Estimation of multiple directional light sources for synthesis of augmented reality images. Graphical Models 65, 4, 185--205. Google ScholarDigital Library
- Weiss, Y. 2001. Deriving intrinsic images from image sequences. In ICCV, II: 68--75.Google Scholar
- Yeung, S.-K., Tang, C.-K., Brown, M. S., and Kang, S. B. 2011. Matting and compositing of transparent and refractive objects. ACM Trans. Graph. 30 (February), 2:1--2:13. Google ScholarDigital Library
- Yu, Y., Debevec, P., Malik, J., and Hawkins, T. 1999. Inverse global illumination: recovering reflectance models of real scenes from photographs. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, SIGGRAPH '99, 215--224. Google ScholarDigital Library
- Yu, L.-F., Yeung, S.-K., Tang, C.-K., Terzopoulos, D., Chan, T. F., and Osher, S. J. 2011. Make it home: automatic optimization of furniture arrangement. ACM Trans. Graph. 30 (August), 86:1--86:12. Google ScholarDigital Library
- Zhang, L., Dugas-Phocion, G., Samson, J., and Seitz, S. 2001. Single view modeling of free-form scenes. In CVPR, 990--997.Google Scholar
Index Terms
- Rendering synthetic objects into legacy photographs
Recommendations
Rendering synthetic objects into legacy photographs
We propose a method to realistically insert synthetic objects into existing photographs without requiring access to the scene or any additional scene measurements. With a single image and a small amount of annotation, our method creates a physical model ...
Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography
SIGGRAPH '08: ACM SIGGRAPH 2008 classesWe present a method that uses measured scene radiance and global illumination in order to add new objects to light-based models with correct lighting. The method uses a high dynamic range image-based model of the scene, rather than synthetic light ...
Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniquesIn this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
Comments