Samuel W. Hasinoff
Abstract
Cell phone cameras have small apertures, which limits the number of photons they can gather, leading to noisy images in low light. They also have small sensor pixels, which limits the number of electrons each pixel can store, leading to limited dynamic range. We describe a computational photography pipeline that captures, aligns, and merges a burst of frames to reduce noise and increase dynamic range. Our system has several key features that help make it robust and efficient. First, we do not use bracketed exposures. Instead, we capture frames of constant exposure, which makes alignment more robust, and we set this exposure low enough to avoid blowing out highlights. The resulting merged image has clean shadows and high bit depth, allowing us to apply standard HDR tone mapping methods. Second, we begin from Bayer raw frames rather than the demosaicked RGB (or YUV) frames produced by hardware Image Signal Processors (ISPs) common on mobile platforms. This gives us more bits per pixel and allows us to circumvent the ISP's unwanted tone mapping and spatial denoising. Third, we use a novel FFT-based alignment algorithm and a hybrid 2D/3D Wiener filter to denoise and merge the frames in a burst. Our implementation is built atop Android's Camera2 API, which provides per-frame camera control and access to raw imagery, and is written in the Halide domain-specific language (DSL). It runs in 4 seconds on device (for a 12 Mpix image), requires no user intervention, and ships on several mass-produced cell phones.
Supplemental Material
Available for Download
Supplemental file.
- Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D. E., Ajdin, B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J., Tico, M., Lensch, H. P. A., Matusik, W., Pulli, K., Horowitz, M., and Levoy, M. 2010. The Frankencamera: an experimental platform for computational photography. SIGGRAPH. Google Scholar
Digital Library
- Adams, A. 1981. The Print, The Ansel Adams Photography Series 3. New York Graphic Society.Google Scholar
- Adobe Inc., 2016. Photoshop CC 2015.1.2, http://www.adobe.com/creativecloud.html.Google Scholar
- Aubry, M., Paris, S., Hasinoff, S. W., Kautz, J., and Du-rand, F. 2014. Fast local laplacian filters: Theory and applications. TOG. Google ScholarDigital Library
- Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2011. A database and evaluation methodology for optical flow. IJCV. Google ScholarDigital Library
- Bennett, E. P., and McMillan, L. 2005. Video enhancement using per-pixel virtual exposures. SIGGRAPH. Google ScholarDigital Library
- Brox, T., and Malik, J. 2011. Large displacement optical flow: Descriptor matching in variational motion estimation. TPAMI. Google ScholarDigital Library
- Dabov, K., Foi, A., and Egiazarian, K. 2007. Video denoising by sparse 3D transform-domain collaborative filtering. EUSIPCO.Google Scholar
- Dabov, K., Foi, A., Katkovnik, V., and Egiazarian, K. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. TIP. Google ScholarDigital Library
- Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. SIGGRAPH. Google ScholarDigital Library
- Delbracio, M., and Sapiro, G. 2015. Hand-held video deblur-ring via efficient fourier aggregation. TCI.Google Scholar
- Donoho, D. L. 1995. De-noising by soft-thresholding. IEEE Transactions on Information Theory 41, 3, 613--627. Google ScholarDigital Library
- DxO Inc., 2015. Google Nexus 6P review, http://www.dxomark.com/Mobiles.Google Scholar
- Farbman, Z., Fattal, R., and Lischinski, D. 2011. Convolution pyramids. SIGGRAPH. Google ScholarDigital Library
- Farnebäck, G. 2002. Polynomial Expansion for Orientation and Motion Estimation. PhD thesis, Linköping University, Sweden.Google Scholar
- Farsiu, S., Elad, M., and Milanfar, P. 2006. Multi-frame demosaicing and super-resolution of color images. TIP. Google ScholarDigital Library
- Frigo, M., and Johnson, S. G. 2005. The design and implementation of FFTW3. Proc. IEEE.Google Scholar
- Gallo, O., and Sen, P. 2016. Stack-based algorithms for HDR capture and reconstruction. In High Dynamic Range Video: From Acquisition, to Display and Applications, F. Dufaux, P. L. Callet, R. K. Mantiuk, and M. Mrak, Eds. Academic Press, ch. 3, 85--119.Google Scholar
- Google Inc., 2016. Android Camera2 API, http://developer.android.com/reference/android/hardware/camera2/package-summary.html.Google Scholar
- Google Inc., 2016. HDR+ burst photography dataset, http://www.hdrplusdata.org.Google Scholar
- Gunturk, B., Glotzbach, J., Altunbasak, Y., Schafer, R., and Mersereau, R. 2005. Demosaicking: color filter array interpolation. IEEE Signal Processing Magazine.Google Scholar
- Hasinoff, S. W., Durand, F., and Freeman, W. T. 2010. Noise-optimal capture for high dynamic range photography. CVPR.Google Scholar
- Healey, G., and Kondepudy, R. 1994. Radiometric CCD camera calibration and noise estimation. TPAMI 16, 3, 267--276. Google ScholarDigital Library
- Heide, F., Steinberger, M., Tsai, Y.-T., Rouf, M., Pajk, D., Reddy, D., Gallo, O., Liu, J., Heidrich, W., Egiazarian, K., Kautz, J., and Pulli, K. 2014. FlexISP: A flexible camera image processing framework. SIGGRAPH Asia. Google ScholarDigital Library
- Horn, B. K. P., and Schunk, B. G. 1981. Determining optical flow. Artificial Intelligence. Google ScholarDigital Library
- Joshi, N., and Cohen, M. F. 2010. Seeing Mt. Rainier: Lucky imaging for multi-image denoising, sharpening, and haze removal. ICCP.Google Scholar
- Kim, S. J., Lin, H. T., Lu, Z., Süsstrunk, S., Lin, S., and Brown, M. S. 2012. A new in-camera imaging model for color computer vision and its application. TPAMI. Google ScholarDigital Library
- Kokaram, A. C. 1993. Motion picture restoration. PhD thesis, Churchill College, University of Cambridge. Section 8.1.Google Scholar
- Levoy, M. 2010. Experimental platforms for computational photography. IEEE CG&A 30. Google ScholarDigital Library
- Lewis, J. 1995. Fast normalized cross-correlation. Vision interface.Google Scholar
- Light, 2016. Light L16 camera, https://light.co/camera.Google Scholar
- Liu, C., Yuen, J., and Torralba, A. 2011. Sift flow: Dense correspondence across scenes and its applications. TPAMI. Google ScholarDigital Library
- Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., and Sun, J. 2014. Fast burst images denoising. SIGGRAPH Asia. Google ScholarDigital Library
- Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. IJCAI. Google ScholarDigital Library
- Mäkitalo, M., and Foi, A. 2013. Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise. TIP. Google ScholarDigital Library
- Martinec, E., 2008. Noise, dynamic range and bit depth in digital SLRs, http://theory.uchicago.edu/~ejm/pix/20d/tests/noise.Google Scholar
- Menze, M., and Geiger, A. 2015. Object scene flow for autonomous vehicles. CVPR.Google Scholar
- Mertens, T., Kautz, J., and Reeth, F. V. 2007. Exposure fusion. Pacific Graphics. Google ScholarDigital Library
- Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. SIGGRAPH. Google ScholarDigital Library
- Ragan-Kelley, J., Adams, A., Paris, S., Levoy, M., Amarasinghe, S., and Durand, F. 2012. Decoupling algorithms from schedules for easy optimization of image processing pipelines. SIGGRAPH. Google ScholarDigital Library
- Reinhard, E., Ward, G., Pattanaik, S. N., Debevec, P. E., and Heidrich, W. 2010. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Academic Press. Google ScholarDigital Library
- Stone, H. S., Orchard, M. T., Chang, E.-C., and Martucci, S. 2001. A fast direct Fourier-based algorithm for subpixel registration of images. TGRS.Google Scholar
- Tao, M. W., Bai, J., Kohli, P., and Paris, S. 2012. Simple-flow: A non-iterative, sublinear optical flow algorithm. Computer Graphics Forum (Eurographics 2012). Google ScholarDigital Library
- Telleen, J., Sullivan, A., Yee, J., Wang, O., Gunawardane, P., Collins, I., and Davis, J. 2007. Synthetic shutter speed imaging. Computer Graphics Forum.Google Scholar
- Wiegand, T., Sullivan, G. J., Bjøntegaard, G., and Luthra, A. 2003. Overview of the H.264/AVC video coding standard. TCSVT. Google ScholarDigital Library
- Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. SIGGRAPH. Google ScholarDigital Library
- Yamaguchi, K., McAllester, D., and Urtasun, R. 2014. Efficient joint segmentation, occlusion labeling, stereo and flow estimation. ECCV.Google Scholar
- Zhang, L., Deshpande, A., and Chen, X. 2010. Denoising vs. deblurring: HDR imaging techniques using moving cameras. CVPR.Google Scholar
Index Terms
- Burst photography for high dynamic range and low-light imaging on mobile cameras
Recommendations
Handheld mobile photography in very low light
Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, ...
Color Vision Based High Dynamic Range Images Rendering
CSIE '09: Proceedings of the 2009 WRI World Congress on Computer Science and Information Engineering - Volume 06The algorithms, mapping the high dynamic range images to the low dynamic range images, have been described in many literatures. Especially, tone mapping operators are the typical algorithms which are designed to produce visibility and the overall ...
Tone-mapping high dynamic range images by novel histogram adjustment
In this paper, we present novel histogram adjustment methods for displaying high dynamic range image. We first present a global histogram adjustment based tone mapping operator, which well reproduces global contrast for high dynamic range images. We ...
Comments