Structural similarity-based Bi-representation through true noise level for noise-robust face super-resolution

Abstract

In today’s real-world scenarios’ of computer vision applications, enhancing low-resolution (LR) facial images corrupted with unwanted noise effects is very challenging as the uneven noise distribution severely distorts these images’ local structure. This paper proposes a novel noise-robust face super-resolution (SR) method, namely structural similarity-based Bi-representation SR (SS-BRSR), to tackle this problem. It firstly estimates the true noise level in the corrupted LR face through the novel noise-level estimation algorithm. Afterward, it employs a robust deep-convolutional neural network, namely DnCNN, to separate the pixel-wise noise from the noisy LR face image. This network produces two outputs: (i) a residual image and (ii) a smooth LR face image. We utilize the first output for pixel-wise updating the entire LR training images, making the structural similarity between the test and the training LR images. Further, for SR reconstruction, the SS-BRSR consists of two patch representation components that individually reconstruct the HR faces corresponding to the initial noisy LR and smooth LR face images. Besides, in both the components, the Gradient and Laplacian features-based learning scheme is incorporated to preserve the discriminative facial features in the SR reconstruction. Here, the first component substantially minimizes the reconstruction error due to noise, and the second component compensates for the lost detail in the LR face image. The target HR face image is restored by taking the appropriate proportions of obtained HR face images from each component. The experimental results on different face datasets justify the SS-BRSR method’s superiority over the state-of-the-art face SR methods. For instance, the quantitative performance (in terms of PNSR and SSIM) of the proposed method over the state-of-the-art RLENR and DFDNet methods gained an improvement of [1%, 1.5%, 2.5%, 2.5%] under [10, 15, 20, 30] noise-level densities, and [1%, 1.5%, 2%, 1.5%] under [10, 15, 20, 30] noise-level densities, respectively, for the standard CelebA and FEI datasets.

Appendices

Appendix A: Algorithm for LR face representation through multi-feature learning

Algorithm 5

LR face representation through the optimal weights.

Appendix B: Optimization for noisy LR face representation through the residual updated LR training face images samples and multi-feature learning (Objection function (8))

The objective function given in (8) can be written as the following matrix arrangement.

$$ \begin{array}{@{}rcl@{}} \hat{w} &=& \underset{w}{\min} \bigg[ \left \| x_{LR_{n}}^{(p,\ q)}-w~y_{LR_{Res}}^{k(p,\ q)} \right \|_{2}^{2} + \tau_{1} \left \| \text{ } \mathcal{F}_{1}(x_{LR_{n}}^{(p,q)})- w~ \mathcal{F}_{1}(y_{LR_{Res}}^{k(p,\ q)}) \text{ } \right \|_{2}^{2} \\&&+ \tau_{2} \left \| \text{ } \mathcal{F}_{2}(x_{LR_{n}}^{(p,q)}) - w~ \mathcal{F}_{2}(y_{LR_{Res}}^{k(p,\ q)}) \text{ } \right \|_{2}^{2} + {\cdots} + \tau_{F} \left \| \text{ } \mathcal{F}_{F}(x_{LR_{n}}^{(p,q)}) - w~ \mathcal{F}_{F}(y_{LR_{Res}}^{k(p,\ q)}) \text{ } \right \|_{2}^{2} \\&&+ \lambda_{1}\left \|\mathcal{D}_{1} w\right \|_{2}^{2} \bigg], \text{where } w = \{w_{1},w_{2}, \dots,w_{K_{1}}\} \text{ and } \sum\limits_{k=1}^{K_{1}}w_{k} = 1 \end{array} $$

(B.1)

Further, for sake of simplicity, we assume (p,q)^th position patch as i^th patch; hence, (B.1) is reformulated as follows.

$$ \begin{array}{@{}rcl@{}} \hat{w} = \underset{w}{\min} \bigg[ \left \| x_{LR_{n}}^{i}-w~y_{LR_{Res}}^{k,i} \right \|_{2}^{2} &&+ \tau_{1} \left \| \text{ } \mathcal{F}_{1}(x_{LR_{n}}^{i})- w~ \mathcal{F}_{1}(y_{LR_{Res}}^{k,i}) \text{ } \right \|_{2}^{2} \\&&+ \tau_{2} \left \| \text{ } \mathcal{F}_{2}(x_{LR_{n}}^{i})- w~ \mathcal{F}_{2}(y_{LR_{Res}}^{k,i}) \text{ } \right \|_{2}^{2} + {\cdots} \\ &&+ \tau_{F} \left \| \text{ } \mathcal{F}_{F}(x_{LR_{n}}^{i})- w~ \mathcal{F}_{F}(y_{LR_{Res}}^{k,i}) \text{ } \right \|_{2}^{2} \\&&+ \lambda_{1}\left \|\mathcal{D}_{1} w\right \|_{2}^{2} \bigg] \end{array} $$

(B.2)

Here, $y_{LR_{Res}}^{k,i}$ is a matrix having column-wise K₁-NN training patches of i^th LR test patch, $\mathcal {F}_{j}(y_{LR_{Res}}^{k,i})$ denotes a matrix having column-wise features of respective K₁-NN training patches corresponding to feature index $j \in \{1,2,\dots ,F\}$, and $\mathcal {D}_{1}$ is the K₁ × K₁ diagonal matrix defined as follows.

$$ {\mathcal{D}_{1}}_{(k,k)} = {d^{i}_{k}}, 1 \le k \le K_{1} $$

(B.3)

By introducing the following auxiliary variables,

$$ Q^{i}_{LR} = \begin{Bmatrix} x^{i}_{LR_{n}}\\ \mathcal{F}_{1}x^{i}_{LR_{n}}\\ \mathcal{F}_{2}x^{i}_{LR_{n}}\\ \vdots\\ \mathcal{F}_{F}x^{i}_{LR_{n}}\\ \end{Bmatrix}, \text{ and } R^{k,i}_{LR} = \begin{Bmatrix} y_{LR_{Res}}^{k,i}\\ \mathcal{F}_{1}y_{LR_{Res}}^{k,i}\\ \mathcal{F}_{2}y_{LR_{Res}}^{k,i}\\ \vdots\\ \mathcal{F}_{F}y_{LR_{Res}}^{k,i}\\ \end{Bmatrix} $$

(B.4)

Equation (B.2) can be transformed into the following.

$$ \hat{w_{k}} = \underset{w}{\arg \min } \left \| Q^{i}_{LR}-wR^{k,i}_{LR} \right \|_{2}^{2}+ \lambda\left \|\mathcal{D}_{1} w\right \|_{2}^{2} $$

(B.5)

Equation (B.5) is an example of the regularized-least-square problem and its solution can be analytically derived as follows.

$$ \hat{w} = (Z_{i} + \lambda \mathcal{D}_{1})\backslash1 $$

(B.6)

Here, 1 is a N × 1 column vector of ones, Z_i denotes the covariance matrix for $Q^{i}_{LR}$ and it is expressed as follows.

$$ Z_{i} = G^{T}G, \text{ and } G = Q^{i}_{LR}1^{T}-R^{k,i}_{LR} $$

(B.7)

The final optimal weight is obtained by re-scaling it to satisfy the constraint ${\sum }_{k=1}^{K_{1}}w_{k}=1$.

Appendix C: Visual and quantitative results on CelebA dataset

Table 6 Average PSNR (dB) and SSIM of different SR methods for the test faces of CelebA dataset, which are corrupted by different AWGNs’ levels

Appendix D: Visual results on real-world ABV-IIITM and CMU-MIT surveillance datasets

Fig. 11

Hallucinated face images of LR test faces in CMU+MIT dataset for different methods: (a) LR input faces (from top to down: σ = 0 for first three faces, σ = 10 for next three faces, σ = 20 for further next two faces, and σ = 30 for last two faces), (b) NE [5], (c) LSR [40], (d) SR [25], (e) LcR [19], (f) TRNR [18], (g) SSR [21], (h) SRCNN [7], (i) WSRNet [16], (j) RLENR [43], (k) TLcRRL [22], (l) VDSR [26], (m) DFDNet [32], (n) SRNI [57], and (0) SS-BRSR