Ben Mildenhall
Abstract
We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully connected (nonconvolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x, y, z) and viewing direction (θ, ϕ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis.
- Buehler, C., Bosse, M., McMillan, L., Gortler S., Cohen, M. Unstructured lumigraph rendering. In SIGGRAPH (2001).Google Scholar
Digital Library
- Chang, A.X., Fhnkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. ShapeNet: An information-rich 3D model repository. arXiv:1512.03012 (2015).Google Scholar
- Curless, B., Levoy, M. A volumetric method for building complex models from range images. In SIGGRAPH (1996).Google ScholarDigital Library
- Debevec, P., Taylor, C.J., Malik, J. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In SIGGRAPH (1996).Google ScholarDigital Library
- Kajiya, J.T., Herzen, B.P.V. Ray tracing volume densities. Comput. Graph. (SIGGRAPH) (1984).Google Scholar
- Kingma, D.P., Ba, J. Adam: A method for stochastic optimization. In ICLR (2015).Google Scholar
- Li, T.-M., Aittala, M., Durand, F., Lehtinen, J. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (SIGGRAPH Asia) (2018).Google Scholar
- Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph. (SIGGRAPH) (2019).Google Scholar
- Loper, M.M., Black, M.J. OpenDR: An approximate differentiable renderer. In ECCV (2014).Google ScholarCross Ref
- Max, N. Optical models for direct volume rendering. IEEE Trans. Visual. Comput. Graph. (1995).Google Scholar
- Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A. Occupancy networks: Learning 3D reconstruction in function space. In CVPR (2019).Google ScholarCross Ref
- Mildenhall, B., Srinivasan, P.P., Ortiz-Cayon, R., Kalantari, N.K., Ramamoorthi, R., Ng, R., Kar, A. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (SIGGRAPH) (2019).Google Scholar
- Mildenhall, B., Srinivasan, P.P, Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV (2020).Google ScholarDigital Library
- Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR (2019).Google Scholar
- Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR (2019).Google ScholarCross Ref
- Porter, T., Duff, T. Compositing digital images. Comput. Graph. (SIGGRAPH) (1984).Google Scholar
- Rahaman, N., Baratin, A., Arpit, D., Dräxler, F., Lin, M., Hamprecht, F.A., Bengio, Y., Courville, A.C. On the spectral bias of neural networks. In ICML (2018).Google Scholar
- Schönberger, J.L., Frahm, J.-M. Structure-from-motion revisited. In CVPR (2016).Google ScholarCross Ref
- Seitz, S.M., Dyer, C.R. Photorealistic scene reconstruction by voxel coloring. Int. J. Comput. Vision (1999).Google Scholar
- Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., Zollhöfer, M. Deepvoxels: Learning persistent 3D feature embeddings. In CVPR (2019).Google ScholarCross Ref
- Sitzmann, V., Zollhoefer, M., Wetzstein, G. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In NeurIPS (2019).Google Scholar
- Tancik, M., Srinivasan, P.P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J.T., Ng, R. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS (2020).Google Scholar
- Wood, D.N., Azuma, D.I., Aldinger, K., Curless, B., Duchamp, T., Salesin, D.H., Stuetzle, W. Surface light fields for 3D photography. In SIGGRAPH (2000).Google ScholarDigital Library
- Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR (2018).Google ScholarCross Ref
- Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (SIGGRAPH) (2018).Google Scholar
Index Terms
- NeRF: representing scenes as neural radiance fields for view synthesis
Recommendations
S3-NeRF: neural reflectance field from shading and shadow under a single viewpoint
NIPS '22: Proceedings of the 36th International Conference on Neural Information Processing SystemsIn this paper, we address the "dual problem" of multi-view scene reconstruction in which we utilize single-view images captured under different point lights to learn a neural scene representation. Different from existing single-view methods which can ...
EGRA-NeRF: Edge-Guided Ray Allocation for Neural Radiance Fields
Highlights- Novel ray allocation strategy enhances textures and edges in scenes.
- Canny edge detector guides dynamic ray allocation.
- Improves performance of NeRF-based algorithms quantitatively and qualitatively.
AbstractRecently, Neural Radiance Fields (NeRF) has demonstrated great potential in synthesizing novel views for realistic video generation. However, renderings from NeRF appear excessively blurred and contain aliasing artifacts in some textures or ...
NeRF for Outdoor Scene Relighting
Computer Vision – ECCV 2022AbstractPhotorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can ...
Comments