Single image super resolution based on multiscale local similarity and neighbor embedding

Abstract

Image quality and algorithm efficiency are the two core problems of super resolution (SR) from a single image. In this paper, we propose a novel single image SR method by using multiscale local similarity and neighbor embedding method. The proposed algorithm utilizes the self similarity redundancy in the original input image, and does not depend on external example images or the whole input image to search and match patches. Instead, we search and match patches in a localized region of the image in each level, which can improve the algorithm efficiency. The neighbor embedding method is used to generate more accurate patches for reconstruction. Finally, we use the original image and filters we design to control the iterate errors which caused by layered reconstruction, which can further improve the quality of SR results. Experimental results demonstrate that our method can ensure the quality of SR images and improve the algorithm efficiency.

Introduction

The objective of super-resolution (SR) is to generate one or more high-resolution (HR) images from one or more low-resolution (LR) images. It has important applications in many fields, such as computer vision, remote sensing, medical imaging and entertainment. Therefore, it has attracted many attentions since the influential publication by Huang [1].

Existing single image SR techniques can be divided into three groups: interpolation based methods, reconstruction based methods, and learning based methods.

Interpolation based methods apply either a base function or some kernels to estimate the pixels in HR grid. These methods are easy to operate but liable to blur the details and cause unclear edges. Accordingly, the SR capability is poor of interpolation methods.

The reconstruction based SR approaches [2], [3], [23], [24], [28] usually assume that the high frequency details lost in an LR image split across a group LR images with sub-pixel misalignments from the same scene. Because many high frequency details are lost, an LR image can be corresponding to several HR images, and make the SR problem ill-posed. In order to solve this problem, many different priors were proposed, including edge-directed priors [21], [22], [4], Bayesian priors [5], [6], [7] and so on. These methods can generate sharper edges and suppress jaggy artifacts, but suffer from insufficient details of the SR results when the magnification ratio is larger.

Learning based methods [8], [9], [27], [29] usually obtain an SR image by learning the relationship of HR–LR image pairs from external database. These methods have stronger SR capability when the magnification factor is large, and they can produce accurate details and get good SR results. Markov network model [26] was used to predict lost details in LR image with the help of external HR–LR pairs [25]. But it was sensitive to different training images, and had a high time complexity. Neighbor embedding (NE) based method [10] is proposed to calculate SR patches by using the reconstruction coefficients and linearly combining the $k$-nearest neighbors ($k$-NN) to generate HR patches. Until now, lots of various methods over NE method has been proposed [12], [13]. Because NE method is overwhelmingly dependent on the training set, a selection algorithm of training images is proposed by using histogram matching, which can guide the SR process and obtain sharper details in SR results than that of NE method [12]. A partially supervised NE method was proposed [13]. This method used a Gaussian model to search the k-nearest neighbors. These methods do not need a large number of training images, but still suffer for long-time computation and blurring effects.

In recent years, a lot of researchers noted that there are huge amount of similar information in different scales of natural images [14], [15], [16], [17], [18], [30], [31]. Hou et al. [17] used the self-similarity property and propose a sparse domain selection method to reconstruct an LR image. Nonlocal self-similarity and local structural regularity were employed to reconstruct an LR image [18]. This method reconstructed the image pixel by pixel, the speed of the algorithm was very slow. Pan et al. [30] used the multiscale self-similarity in the original image to reconstruction, meanwhile controlled the iterate errors caused by layered reconstruction by using the original image. Zhang et al. [31] proposed a single image SR method by combining the self-similarity and neighbor embedding, and used nonlocal means method to further improve the quality of SR results. A single image SR algorithm is proposed by combining reconstruction based method and learning based methods, and employed self-similarity in different scales for details adding [14]. Freedman and Fattal [15] proposed an SR algorithm which used local self-similarity in image pyramid created by input LR image, and obtained good and efficient SR results.

The algorithm in [14], [15] reconstructed an LR image by employing the similar redundancy of the original LR image itself, which did not dependent on any external training set. However, the success of the two methods depended on the self-similarity redundancy in image pyramid generated by the original LR image. If there were not enough repetitive details, the two methods were prone to create blurred edges and false details.

In order to solve the quality reduction of SR results which caused by weak self- similarity, we propose a new algorithm by combining multiscale local self-similarity and NE method, and consider the control method of iterative errors caused by layered reconstruction. Novel filters are designed to generate the two degraded images of the input image in different scales. By searching and matching similar patches between the two degraded images, the HR image can be obtained. Specially, we do not use the image pyramid or the whole image in a scale to search and match image patches, and only search patches in a local region of the degraded version instead. Then repeating this process several times under a small magnification ratio, the final magnification ratio can be achieved. In order to avoid sharpened edges caused by less repetitive patterns in original image, we use NE based method to obtain more accurate HR patches for reconstruction. To further enhance the performance of proposed SR algorithm, we use the original LR image and filters we designed to control the iterative errors which caused by layered reconstruction.

In contrast to articles [14], [15], the main contributions are summarized below:

• We design a novel high-quality and efficient SR algorithm by using multiscale local self-similarity and NE based method, without the help of external example database. Multiscale local self-similarity can reduce the time cost involved in the nearest neighbor searching, and the NE based method can increase the similarity of patches, even if there are insufficient similar details in input image. So the algorithm can improve the efficiency and ensure the quality of SR results.

• In order to further improve the quality of the SR results, we use the input LR image and the filters we design to generate the two degraded images in different scales in each level, and control the iterative errors caused by layered reconstruction.

The remainder of this paper is organized as follows. Section 2 summarizes the related work which is based on NE method. In Section 3, we present proposed SR algorithm based on multiscale local self-similarity and NE based method. In Section 4, we experimentally test the new method and describe the results. The conclusion and future work are discussed in Section 4.2.