Learning stacking regression for no-reference super-resolution image quality assessment

Highlights

• We propose to employ a pre-trained VGGNet model to extract deep visual features to quantify the quality of SR images.

• The used features are propitious to reveal the fundamental artifacts of SR images than the traditional hand-crafted features.

• We develop a novel stacking regression-based framework to learn a coarse-to-fine metric for NR-SRIQA.

• The proposed NR-SRIQA metric can yield more accurate quality prediction on SR images than other-state-of-art predecessors.

Abstract

No-reference super-resolution (SR) image quality assessment (NR-SRIQA) aims to evaluate the quality of SR images without relying on any reference images. Currently, most previous methods usually utilize a certain handcrafted perceptual statistical features to quantify the degradation of SR images and a simple regression model to learn the mapping relationship from the features to the perceptual quality. Although these methods achieved promising performance, they still have some limitations: 1) the handcrafted features cannot accurately quantify the degradation of SR images; 2) the complex mapping relationship between the features and the quality scores cannot be well approximated by a simple regression model. To alleviate the above problems, we propose a novel stacking regression framework for NR-SRIQA. In the proposed method, we use a pre-trained VGGNet to extract the deep features for measuring the degradation of SR images, and then develop a stacking regression framework to establish the relationship between the learned deep features and the quality scores to achieve the NR-SRIQA. The stacking regression integrates two base regressors, namely Support Vector Regression (SVR) and K-Nearest Neighbor (K-NN) regression, and a simple linear regression as a meta-regressor. Thanks to the feature representation capability of deep neural networks (DNNs) and the complementary features of the two base regressors, the experimental results indicate that the proposed stacking regression framework is capable of yielding higher consistency with human visual judgments on the quality of SR images than other state-of-the-art SRIQA methods.

Introduction

The objective of image super-resolution (SR) reconstruction is to generate a high-resolution (HR) image with more details by using one or several low-resolution (LR) images from the same scene [1]. The SR technology has great potential applications in many fields such as computer vision, medical image analysis, remote sensing imaging, and life entertainment. In order to assess the quality of SR images and to further optimize the performance of SR algorithms, one of the most key tasks is to evaluate the quality of resultant SR images. There is no wonder that human’s opinion is the ultimate receiver of images, so subjective quality assessment is regarded as the most direct yet effective way to reflect the quality of SR images [2]. Nevertheless, the process of subjective quality assessment is time-consuming and energy-draining. As a result, this kind of methods cannot be easily integrated into an SR application system for real-world scenarios.

In contrast to subjective quality assessment on SR images, the other kind of SRIQA methods is objective quality assessment, which automatically evaluates the quality of SR images through a computational model. In general, these methods can be classified into three major categories [3], i.e., full reference image quality assessment (FRIQA) [4], reduced-reference image quality assessment (RRIQA) [5], and no reference image quality assessment (NRIQA) [6], [7]. When applying the FRIQA metrics such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [8], and information fidelity criterion (IFC) [9] to SRIQA, the corresponding original HR image is required as a reference to calculate the quality. However, the results obtained by these FRIQA metrics sometimes are not consistent with human’s perceptual quality. Moreover, the original HR images are not available in practice. Consequently, the conventional FRIQA metrics are not suitable for evaluating the quality of SR images and the capability of SR algorithms. The second kind of image quality assessment methods is called RRIQA metrics which only need reduced amount of reference images, so they are better for application in practice than FRIQA. Yeganeh et al. [10] quantified artifacts of SR images by distilling the natural scene statistical features from the frequency domain to the spatial domain. He et al. [11] operated a quality aware collection of local similarity features to predict the SR images degradation. Fang et al. [12] measured the quality of the SR images by virtue of the energy change and texture variation. Although this type of evaluation methods is more flexible than the FRIQA metrics, these methods still require partial information of the original HR images. The third subclass of IQA is named as NRIQA. This kind of methods does not need any information of original images, so they can overcome the shortcomings of the two aforementioned kinds of methods, leading to much attention in the literature.

To target NRIQA, most previous methods usually elaborate on using handcrafted perceptual statistical features to quantify the degradation of SR images. For example, Moorthy et al. [13] adopted natural statistical properties of images in the wavelet domain to characterize the quality of images. Observing that human visual mechanisms are more sensitive to the structural characteristics and the contrast of images, Saad et al. [14] combined the contrast, structural and anisotropic features of DCT statistical coefficients to represent the image distortion. Ma et al. [15] utilized local frequency features, global frequency features, and spatial domain features to quantify the degradation of SR images. Although these approaches are promising for IQA, the image quality cannot be well quantified by the used handcrafted features.

In contrast to the handcrafted features, learning high-level features from deep neural networks (DNNs) has gained much attention and shows successful applications in many fields such as computer vision, pattern recognition, and image processing. The DNNs show powerful representation capability and are able to automatically learn high-level feature representations to fully capture rich intrinsic information of image quality. Recently, a number of DNNs are used to address the task of NRIQA. For instance, Sun et al. [16] exploited the AlexNet architecture to extract semantic features implied in the global image content, and utilized saliency detection and Gabor filters to capture low-level features related to local image content. The overall image quality was estimated by combining these features. Li et al. [17] proposed to employ the ResNet architecture to represent the depth features of each overlapping image block in a statistical manner, by which the image quality is evaluated by a linear regression model. Gao et al. [18] utilized the VGGNet to extract image features of each layer, and the final image quality is estimated by averaging the predicted scores of multiple layers. Kang et al. [19] developed an IQA model by extracting features from Convolution Neural Network (CNN), where the feature extraction and the regression task are integrated into an optimization process. Bosse et al. [20] proposed to employ the CNN to extract high-level features from unpreprocessed image patches for quality prediction. Bianco et al. [21] elaborated on extracting deep features by fine-tuning a pre-trained network with an IQA dataset. With the obtained deep features, a support vector regression (SVR) model is trained to evaluate the quality of images. Experimental results suggested that these applied deep perceptual features perform better on IQA than many traditional handcrafted features due to their powerful capability of quantifying image quality. DNNs models are beneficial to capture the high-level semantics of images and the learned features are highly correlated with the quality degradation, which provides a potential application for NRIQA.

Besides perceptual statistical features, the other key component for IQA is how to build an accurate computational model to map the image features into the quality scores. In the literature of IQA, many predecessors tend to learn a single model, including SVR, for the quality prediction. However, in most cases only an individual model is insufficient enough to reveal the complicated relationship between the perceptual statistical features and the quality scores. To overcome this bottleneck, an alternative way is to introduce ensemble learning for IQA, by which multiple models such as different regression methods, are strategically generated and combined to amend the possible deviation on the quality estimate.

Stacking is an effective ensemble learning technique that builds a new model by combining the predictions from multiple models (e.g., decision tree, KNN or SVM) for a particular task. In principle, the method is the process of integrating different machine learning algorithms through holdout cross-validation [22]. Later Breiman [23] further improved stacking regression by replacing holdout cross-validation with k-fold cross-validation. Unlike bagging and boosting, the stacking regression is a useful ensemble learning technique that combines multiple diverse regression models via a meta-regressor. Previous works [24], [25] have proved that, as a heterogeneous ensemble approach, stacking regression can significantly boost up the prediction performance by maximizing the complementary merits of different models.

In this paper, inspired by the significant advantages of stacking learning, we propose a novel quality metric for NR-SRIQA. In the method, the perceptual statistical features extracted from off-the-shelf VGGNet model are used to quantify the degradation of SR images. And then an effective stacking framework, which employs SVR and KNN regression as the base regressors, is developed to learn a mapping model from the obtained deep features to the perceived quality scores. With the stacking regression model, an NR-SRIQA metric can be used to predict the quality of any an given SR image. In summary, the unique contributions of this paper are mainly two aspects:

• We propose to employ a pre-trained VGGNet model to extract deep visual features rather than hand-crafted statistical features, to quantify the quality of SR images. The used features are propitious to reveal the fundamental artifacts of SR images than the traditional hand-crafted features.

• We develop a novel stacking regression-based framework to learn a coarse-to-fine metric mapping from deep features to quality scores for NR-SRIQA. The proposed quality metric can yield more accurate quality prediction on SR images than other state-of-the-art predecessors.

The remainder of the paper is organized as follows. Section 2 provides the VGG deep features to measure the degradation of SR images and presents a two-layer stacking regression framework for NR-SRIQA. In Section 3, we evaluate the performance of the proposed method and experimentally compare it with the state-of-the-art IQA metrics. Finally, Section 4 concludes the paper and outlooks the future work.