Jack Oliver Hilliard
ABSTRACT
In this paper we present a method for single-view illumination estimation of indoor scenes, using image-based lighting, that incorporates state-of-the-art outpainting methods. Recent advancements in illumination estimation have focused on improving the detail of the generated environment map so it can realistically light mirror reflective surfaces. These generated maps often include artefacts at the borders of the image where the panorama wraps around. In this work we make the key observation that inferring the panoramic HDR illumination of a scene from a limited field of view LDR input can be framed as an outpainting problem (whereby the original image must be expanded beyond its original borders). We incorporate two key techniques used in outpainting tasks: i) separating the generation into multiple networks (a diffuse lighting network and a high-frequency detail network) to reduce the amount to be learnt by a single network, ii) utilising an inside-out method of processing the input image to reduce the border artefacts. Further to incorporating these outpainting methods we also introduce circular padding before the network to help remove the border artefacts. Results show the proposed approach is able to relight diffuse, specular and mirror surfaces more accurately than existing methods in terms of the position of the light sources and pixelwise accuracy, whilst also reducing the artefacts produced at the borders of the panorama.
- Jiayang Bai, Jie Guo, Chenchen Wang, Zhenyu Chen, Zhen He, Shan Yang, Piaopiao Yu, Yan Zhang, and Yanwen Guo. 2023. Deep graph learning for spatially-varying indoor lighting prediction. Science China Information Sciences 66, 3 (feb 2023). https://doi.org/10.1007/s11432-022-3576-9 arxiv:2202.06300Google Scholar
Cross Ref
- Dachuan Cheng, Jian Shi, Yanyun Chen, Xiaoming Deng, and Xiaopeng Zhang. 2018. Learning Scene Illumination by Pairwise Photos from Rear and Front Mobile Cameras. Computer Graphics Forum 37, 7 (oct 2018), 213–221. https://doi.org/10.1111/cgf.13561Google ScholarCross Ref
- Paul Debevec. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1998. 189–198. https://doi.org/10.1145/280814.280864Google ScholarDigital Library
- Marc Andre Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagne, and Jean Francois Lalonde. 2019. Deep parametric indoor lighting estimation. Proceedings of the IEEE International Conference on Computer Vision 2019-October (oct 2019), 7174–7182. https://doi.org/10.1109/ICCV.2019.00727 arxiv:1910.08812Google ScholarCross Ref
- Marc André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean François Lalonde. 2017. Learning to predict indoor illumination from a single image. ACM Transactions on Graphics 36, 6 (mar 2017), 1–14. https://doi.org/10.1145/3130800.3130891 arxiv:1704.00090Google ScholarDigital Library
- Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean Francois Lalonde. 2019. Fast spatially-varying indoor lighting estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2019-June. 6901–6910. https://doi.org/10.1109/CVPR.2019.00707 arxiv:1906.03799Google ScholarCross Ref
- Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2016-December (dec 2016), 2414–2423. https://doi.org/10.1109/CVPR.2016.265Google ScholarCross Ref
- Vasileios Gkitsas, Nikolaos Zioulis, Federico Alvarez, Dimitrios Zarpalas, and Petros Daras. 2020. Deep lighting environment map estimation from spherical panoramas. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Vol. 2020-June. 2719–2728. https://doi.org/10.1109/CVPRW50498.2020.00328 arxiv:2005.08000Google ScholarCross Ref
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (oct 2020), 139–144. https://doi.org/10.1145/3422622 arxiv:1406.2661Google ScholarDigital Library
- Robin Green. 2003. Spherical Harmonic Lighting: The Gritty Details. Technical Report. 4 pages.Google Scholar
- Takayuki Hara and Tatsuya Harada. 2020. Spherical Image Generation from a Single Normal Field of View Image by Considering Scene Symmetry. Thirty-Fifth {AAAI} Conference on Artificial Intelligence, {AAAI} 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, {IAAI} 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, {EAAI} 2021, Vir (jan 2020), 1513–1521. arxiv:2001.02993https://arxiv.org/abs/2001.02993v1Google Scholar
- Changho Jo, Woobin Im, and Sung-Eui Yoon. 2021. In-N-Out: Towards Good Initialization for Inpainting and Outpainting. In BMVC. https://doi.org/10.48550/arxiv.2106.13953 arxiv:2106.13953Google ScholarCross Ref
- Tero Karras, Samuli Laine, and Timo Aila. 2021. A Style-Based Generator Architecture for Generative Adversarial Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 12 (dec 2021), 4217–4228. https://doi.org/10.1109/TPAMI.2020.2970919 arxiv:1812.04948Google ScholarDigital Library
- Kyunghun Kim, Yeohun Yun, Keon Woo Kang, Kyeongbo Kong, Siyeong Lee, and Suk Ju Kang. 2021. Painting outside as inside: Edge guided image outpainting via bidirectional rearrangement with progressive step learning. In Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021. 2121–2129. https://doi.org/10.1109/WACV48630.2021.00217 arxiv:2010.01810Google ScholarCross Ref
- Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings (dec 2015). arxiv:1412.6980https://arxiv.org/abs/1412.6980v9Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (may 2017), 84–90. https://doi.org/10.1145/3065386Google ScholarDigital Library
- Chloe Legendre, Wan Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul Debevec. 2019. Deeplight: Learning illumination for unconstrained mobile mixed reality. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2019-June. 5911–5921. https://doi.org/10.1109/CVPR.2019.00607 arxiv:1904.01175Google ScholarDigital Library
- Mengtian Li, Jie Guo, Xiufen Cui, Rui Pan, Yanwen Guo, Chenchen Wang, Piaopiao Yu, and Fei Pan. 2019. Deep spherical Gaussian illumination estimation for indoor. 1st ACM International Conference on Multimedia in Asia, MMAsia 2019 19 (dec 2019). https://doi.org/10.1145/3338533.3366562Google ScholarDigital Library
- Han Lin, Maurice Pagnucco, and Yang Song. 2021. Edge guided progressively generative image outpainting. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (jun 2021), 806–815. https://doi.org/10.1109/CVPRW53098.2021.00090Google ScholarCross Ref
- Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z. Qureshi, and Mehran Ebrahimi. 2019. EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning. (jan 2019). https://doi.org/10.48550/arxiv.1901.00212 arxiv:1901.00212Google ScholarCross Ref
- Gowri Somanath and Daniel Kurz. 2021. HDR Environment Map Estimation for Real-Time Augmented Reality. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (nov 2021), 11293–11301. https://doi.org/10.1109/CVPR46437.2021.01114 arxiv:2011.10687Google ScholarCross Ref
- Yu Ting Tsai and Zen Chung Shih. 2006. All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Transactions on Graphics 25, 3 (jul 2006), 967–976. https://doi.org/10.1145/1141911.1141981Google ScholarDigital Library
- Guangcong Wang, Yinuo Yang, Chen Change Loy, and Ziwei Liu. 2022. StyleLight: HDR Panorama Generation for Lighting Estimation and Editing. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 13675 LNCS (jul 2022), 477–492. https://doi.org/10.1007/978-3-031-19784-0_28 arxiv:2207.14811Google ScholarDigital Library
- Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019. Wide-context semantic image extrapolation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June (jun 2019), 1399–1408. https://doi.org/10.1109/CVPR.2019.00149Google ScholarCross Ref
- Gang Wei, Jing Guo, Yongzhen Ke, Kai Wang, Shuai Yang, and Nan Sheng. 2023. A three-stage GAN model based on edge and color prediction for image outpainting. Expert Systems with Applications 214 (mar 2023), 119136. https://doi.org/10.1016/j.eswa.2022.119136Google ScholarDigital Library
- Weihao Weng and Xin Zhu. 2021. INet: Convolutional Networks for Biomedical Image Segmentation. IEEE Access 9 (may 2021), 16591–16603. https://doi.org/10.1109/ACCESS.2021.3053408 arxiv:1505.04597Google ScholarCross Ref
- Fangneng Zhan, Changgong Zhang, Wenbo Hu, Shijian Lu, Feiying Ma, Xuansong Xie, and Ling Shao. 2021a. Sparse Needlets for Lighting Estimation with Spherical Transport Loss. In Proceedings of the IEEE International Conference on Computer Vision. 12810–12819. https://doi.org/10.1109/ICCV48922.2021.01259 arxiv:2106.13090Google ScholarCross Ref
- Fangneng Zhan, Changgong Zhang, Yingchen Yu, Yuan Chang, Shijian Lu, Feiying Ma, and Xuansong Xie. 2021b. EMLight: Lighting Estimation via Spherical Distribution Approximation. 35th AAAI Conference on Artificial Intelligence, AAAI 2021 4B (dec 2021), 3287–3295. https://doi.org/10.1609/aaai.v35i4.16440 arxiv:2012.11116Google ScholarCross Ref
- Richard Zhang. 2019. Making convolutional networks shift-invariant again. 36th International Conference on Machine Learning, ICML 2019 2019-June (apr 2019), 12712–12722. https://doi.org/10.48550/arxiv.1904.11486 arxiv:1904.11486Google ScholarCross Ref
- Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (jan 2018), 586–595. https://doi.org/10.1109/CVPR.2018.00068 arxiv:1801.03924Google ScholarCross Ref
- Xiaofeng Zhang, Feng Chen, Cailing Wang, Ming Tao, and Guo Ping Jiang. 2020. SiENet: Siamese Expansion Network for Image Extrapolation. IEEE Signal Processing Letters 27 (2020), 1590–1594. https://doi.org/10.1109/LSP.2020.3019705 arxiv:2007.03851Google ScholarCross Ref
Index Terms
- HDR Illumination Outpainting with a Two-Stage GAN Model
Recommendations
A new approach to outdoor illumination estimation based on statistical analysis for augmented reality
CASA' 2010 Special IssueIllumination consistency plays an important role in realistic rendering of virtual characters which are integrated into a live video of real scene. This paper proposes a novel method for estimating the illumination conditions of outdoor videos captured ...
Illumination estimation and cast shadow detection through a higher-order graphical model
CVPR '11: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern RecognitionIn this paper, we propose a novel framework to jointly recover the illumination environment and an estimate of the cast shadows in a scene from a single image, given coarse 3D geometry. We describe a higher-order Markov Random Field (MRF) illumination ...
Adaptive Illumination Sampling for Direct Volume Rendering
Advances in Computer GraphicsAbstractDirect volume rendering is used to visualize data from sources such as tomographic imaging devices. The perception of certain structures depends very much on visual cues such as lighting and shadowing. According illumination techniques have been ...
Comments