Dmitri Bitouk
ABSTRACT
In this paper, we present a complete system for automatic face replacement in images. Our system uses a large library of face images created automatically by downloading images from the internet, extracting faces using face detection software, and aligning each extracted face to a common coordinate system. This library is constructed off-line, once, and can be efficiently accessed during face replacement. Our replacement algorithm has three main stages. First, given an input image, we detect all faces that are present, align them to the coordinate system used by our face library, and select candidate face images from our face library that are similar to the input face in appearance and pose. Second, we adjust the pose, lighting, and color of the candidate face images to match the appearance of those in the input image, and seamlessly blend in the results. Third, we rank the blended candidate replacements by computing a match distance over the overlap region. Our approach requires no 3D model, is fully automatic, and generates highly plausible results across a wide range of skin tones, lighting conditions, and viewpoints. We show how our approach can be used for a variety of applications including face de-identification and the creation of appealing group photographs from a set of images. We conclude with a user study that validates the high quality of our replacement results, and a discussion on the current limitations of our system.
- Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive Digital Photomontage. ACM Transactions on Graphics 23, 294--302. Google Scholar
Digital Library
- Avidan, S., and Shamir, A. 2007. Seam carving for contentaware image resizing. ACM Transactions on Graphics 26. Google ScholarDigital Library
- Basri, R., and Jacobs, D. 2003. Lambertian reflectance and linear subspaces. IEEE TPAMI 25, 218--233. Google ScholarDigital Library
- Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H.-P. 2004. Exchanging Faces in Images. Computer Graphics Forum 23, 669--676.Google ScholarCross Ref
- Boyle, M., Edwards, C., and Greenberg, S. 2000. The Effects of Filtered Video on Awareness and Privacy. In ACM Conference on Computer Supported Cooperative Work. Google ScholarDigital Library
- Cootes, T., Edwards, G., and Taylor, C. 2001. Active Appearance Models. IEEE TPAMI 26, 681--685. Google ScholarDigital Library
- Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the Reflectance Field of a Human Face. In SIGGRAPH 00, 145--156. Google ScholarDigital Library
- Debevec, P. 1998. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH 98, 189--198. Google ScholarDigital Library
- Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In SIGGRAPH 01, 341--346. Google ScholarDigital Library
- Fergus, R., Singh, B., Hertzmann, A., Roweis, S., and Freeman, W. 2006. Removing camera shake from a single photograph. SIGGRAPH 06, 787--794. Google ScholarDigital Library
- Gross, R., Sweeney, L., de la Torre, F., and Baker, S. 2006. Model-Based Face De-Identification. 161--168. Google ScholarDigital Library
- Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics 26, 3. Google ScholarDigital Library
- Kundur, D., and Hatzinakos, D. 1996. Blind image deconvolution. IEEE Signal Processing Magazine, 3, 43--64.Google ScholarCross Ref
- Lanitis, I., Draganova, C., and Christodoulou, C. 2004. Comparing different classifiers for automatic age estimation. IEEE Trans. on Systems, Man, and Cybernetics, B 34, 621--628. Google ScholarDigital Library
- Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In SIGGRAPH 01: Proc. of the 28th CGIT, 271--276. Google ScholarDigital Library
- Malik, S., 2003. Digital face replacement in photographs. http://www.cs.toronto.edu/~smalik/2530/project/results.html.Google Scholar
- Moghaddam, B., and Yang, M.-H. 2002. Learning gender with support faces. IEEE TPAMI 24, 707--711. Google ScholarDigital Library
- Newton, E., Sweeney, L., and Malin, B. 2005. Preserving Privacy by De-Identifying Face Images. IEEE Trans. on Knowledge and Data Eng., 232--243. Google ScholarDigital Library
- Omron, 2007. OKAO vision. http://omron.com/rd/vision/01.html.Google Scholar
- Ramamoorthi, R., and Hanrahan, P. 2001. An Efficient Representation for Irradiance Environment Maps. In SIGGRAPH 01, 497--500. Google ScholarDigital Library
- Rubner, Y., Tomasi, C., and Guibas, L. J. 2000. The earth mover's distance as a metric for image retrieval. IJCV, 99--121. Google ScholarDigital Library
- Wang, H., Raskar, R., Xu, N., and Ahuja, N. 2007. Videoshop: A New Framework for Video Editing in Gradient Domain. Graphical Models 69, 57--70. Google ScholarDigital Library
- Wang, Y., Liu, Z., Hua, G., Wen, Z., Zhang, Z., and Samaras, D. 2007. Face re-lighting from a single image under harsh lighting conditions. CVPR '07.Google Scholar
- Wen, Z., Liu, Z., and Huang., T. S. 2003. Face Relighting with Radiance Environment Maps. In CVPR '03, 158--165.Google Scholar
Index Terms
- Face swapping: automatically replacing faces in photographs
Recommendations
SimSwap: An Efficient Framework For High Fidelity Face Swapping
MM '20: Proceedings of the 28th ACM International Conference on MultimediaWe propose an efficient framework, called Simple Swap (SimSwap), aiming for generalized and high fidelity face swapping. In contrast to previous approaches that either lack the ability to generalize to arbitrary identity or fail to preserve attributes ...
The Creation and Detection of Deepfakes: A Survey
Generative deep learning algorithms have progressed to a point where it is difficult to tell the difference between what is real and what is fake. In 2018, it was discovered how easy it is to use this technology for unethical and malicious applications, ...
Generative adversarial networks
Generative adversarial networks are a kind of artificial intelligence algorithm designed to solve the generative modeling problem. The goal of a generative model is to study a collection of training examples and learn the probability distribution that ...
Comments