FAPN: Face Alignment Propagation Network for Face Video Super-Resolution

Abstract

Face video super-resolution (FVSR) aims to use continuous low resolution (LR) video frames to reconstruct face and recover facial details under the premise of ensuring authenticity. The existing video super-resolution (VSR) technology usually uses inter-frame information to achieve better super-resolution (SR) performance. However, due to the complex temporal dependence between frames, as the number of input frames increases, the information cannot be fully utilized, and even wrong information is introduced, resulting in poor performance. In this work, we propose an alignment propagation network for accumulating facial prior information (FAPN). We design a neighborhood information coupling (NIC) module based on optical flow estimation and alignment, where the current frame, the adjacent frames and the SR results of the previous frame are locally fused. The coupled frames are sent to a unidirectional propagation (UP) structure for propagation. Meanwhile, in the UP structure, the facial prior information is filtered and accumulated in the face super-resolution cell (FSRC), and the high-dimensional hidden state is introduced to propagate effective temporal information between frames along the unidirectional structure. Extensive evaluations and comparisons validate the strengths of our approach, FAPN can accumulate more facial details while ensuring the authenticity of the face. And the experimental results demonstrated that the proposed framework achieves better performance on PSNR (up to 0.31 dB), SSIM (up to 0.15 dB) and face recognition accuracy (up to 1.99%) compared with state-of-the-art methods.

This work is supported by National Natural Science Foundation of China (grant number 62275046).