Abstract
In this paper, we propose a novel manipulated face detection and localization approach, which simultaneously detect manipulated face images and videos and locate the manipulated regions at semantic-level. To do this, we design a multi-branch autoencoder composed of four types of modules, including feature encoder, shared decoder, semantic decoder, and classification network. The feature encoder extracts latent feature from the input face image. The shared decoder obtains structure feature from the latent feature. The four semantic decoders decode structure feature into four different semantic prediction masks, respectively. The classification network outputs the semantic prediction labels based on the latent feature. Finally, the manipulation prediction label and manipulation prediction mask of the input face image can be generated with the semantic prediction labels and semantic prediction masks. Extensive experiments show that our approach can effectively detect and locate manipulated face images and videos at semantic-level, even under cross-manipulation, cross-dataset, and cross-compression scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dang, H., Liu, F., Stehouwer, J., Liu, X., Jain, A.K.: On the detection of digital face manipulation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5780–5789 (2020). https://doi.org/10.1109/CVPR42600.2020.00582
Verdoliva, L.: Media Forensics and DeepFakes: an overview. IEEE J. Sel. Top. Signal Process. 14(5), 910–932 (2020). https://doi.org/10.1109/JSTSP.2020.3002101
Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. 38(4), 1–12 (2019). https://doi.org/10.1145/3306346.3323035. Article 66
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2Face: real-time face capture and reenactment of RGB videos. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2387–2395 (2016). https://doi.org/10.1109/CVPR.2016.262
Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: FaceForensics++: learning to detect manipulated facial images. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1–11 (2019). https://doi.org/10.1109/ICCV.2019.00009
Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for DeepFake forensics. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3204–3213 (2020). https://doi.org/10.1109/CVPR42600.2020.00327
Dolhansky, B., et al.: The deepfake detection challenge (DFDC) dataset. arXiv preprint arXiv:2006.07397 (2020)
Li, L., Bao, J., Yang, H., Chen, D., Wen, F.: Advancing high fidelity identity swapping for forgery detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5073–5082 (2020). https://doi.org/10.1109/CVPR42600.2020.00512
Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: MesoNet: a compact facial video forgery detection network. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–7 (2018). https://doi.org/10.1109/WIFS.2018.8630761
Güera, D., Delp, E.J.: Deepfake video detection using recurrent neural networks. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 (2018). https://doi.org/10.1109/AVSS.2018.8639163
Joseph, Z., Nyirenda, C.: Deepfake detection using a two-stream capsule network. In: 2021 IST-Africa Conference (IST-Africa), pp. 1–8 (2021)
Li, Y., Chang, M., Lyu, S.: In Ictu Oculi: exposing AI created fake videos by detecting eye blinking. In: 2008 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–7 (2018). https://doi.org/10.1109/WIFS.2018.8630787
Matern, F., Riess, C., Stamminger, M.: Exploiting visual artifacts to expose deepfakes and face manipulations. In: 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pp. 83–92 (2019). https://doi.org/10.1109/WACVW.2019.00020
Pan, X., Zhang, X., Lyu, S.: Exposing image splicing with inconsistent local noise variances. In: 2012 IEEE International Conference on Computational Photography (ICCP), pp. 1–10 (2012). https://doi.org/10.1109/ICCPhot.2012.6215223
Bappy, J.H., Simons, C., Nataraj, L., Manjunath, B.S., Roy-Chowdhury, A.K.: Hybrid LSTM and encoder-decoder architecture for detection of image forgeries. IEEE Trans. Image Process. 28(7), 3286–3300 (2019). https://doi.org/10.1109/TIP.2019.2895466
Zhou, P., Han, X., Morariu, V.I., Davis, L.S.: Learning rich features for image manipulation detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1053–1061 (2018). https://doi.org/10.1109/CVPR.2018.00116
Nguyen, H.H., Fang, F., Yamagishi, J., Echizen, I.: Multi-task learning for detecting and segmenting manipulated facial images and videos. In: 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1–8 (2019). https://doi.org/10.1109/BTAS46853.2019.9185974
Li, L., et al.: Face X-ray for more general face forgery detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5000–5009 (2020). https://doi.org/10.1109/CVPR42600.2020.00505
Huang, Y., Juefei-Xu, F., Guo, Q., Liu, Y., Pu, G.: FakeLocator: robust localization of GAN-based face manipulations. In: IEEE Transactions on Information Forensics and Security (2022). https://doi.org/10.1109/TIFS.2022.3141262
Songsri-in, K., Zafeiriou, S.: Complement face forensic detection and localization with faciallandmarks. arXiv preprint arXiv:1910.05455 (2019)
Karras, T., et al.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 43(12), 4217–4228 (2021). https://doi.org/10.1109/TPAMI.2020.2970919
Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017). https://doi.org/10.1109/ICCV.2017.244
Perarnau, G., et al.: Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355 (2016)
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J.: StarGAN: unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2018). https://doi.org/10.1109/CVPR.2018.00916
He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: AttGAN: facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28(11), 5464–5478 (2019). https://doi.org/10.1109/TIP.2019.2916751
Nirkin, Y., Keller, Y., Hassner, T.: FSGAN: subject agnostic face swapping and reenactment. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7183–7192 (2019). https://doi.org/10.1109/ICCV.2019.00728
Li, L., et al.: FaceShifter: towards high fidelity and occlusion aware face swapping. arXiv preprint arXiv:1912.13457 (2019)
Yang, X., Li, Y., Lyu, S.: Exposing deep fakes using inconsistent head poses. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8261–8265 (2019). https://doi.org/10.1109/ICASSP.2019.8683164
Li, G., Cao, Y., Zhao, X.: Exploiting facial symmetry to expose deepfakes. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 3587–3591 (2021). https://doi.org/10.1109/ICIP42928.2021.9506272
Li, Y., Lyu, S.: Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv:1811.00656 (2018)
Guarnera, L., Giudice, O., Battiato, S.: Deepfake detection by analyzing convolutional traces. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2841–2850 (2020). https://doi.org/10.1109/CVPRW50498.2020.00341
Zhao, H., Wei, T., Zhou, W., Zhang, W., Chen, D., Yu, N.: Multi-attentional deepfake detection. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2185–2194 (2021). https://doi.org/10.1109/CVPR46437.2021.00222
Amerini, I., Galteri, L., Caldelli, R., Del Bimbo, A.: Deepfake video detection through optical flow based CNN. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 1205–1207 (2019). https://doi.org/10.1109/ICCVW.2019.00152
Wu, X., Xie, Z., Gao, Y., Xiao, Y.: SSTNet: detecting manipulated faces through spatial, steganalysis and temporal features. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2952–2956 (2020). https://doi.org/10.1109/ICASSP40776.2020.9053969
Acknowledgements
This work was supported by National Key Technology Research and Development Program under 2020AAA0140000.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, G., Zhao, X., Cao, Y., Hu, C. (2023). Manipulated Face Detection and Localization Based on Semantic Segmentation. In: Zhao, X., Tang, Z., Comesaña-Alfaro, P., Piva, A. (eds) Digital Forensics and Watermarking. IWDW 2022. Lecture Notes in Computer Science, vol 13825. Springer, Cham. https://doi.org/10.1007/978-3-031-25115-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-25115-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25114-6
Online ISBN: 978-3-031-25115-3
eBook Packages: Computer ScienceComputer Science (R0)