Advance Neighbor Embedding for Image Super Resolution

This paper presents the Advance Neighbor embedding (ANE) method for image super resolution. The assumption of the neighbor-embedding (NE) algorithm for single-image super-resolution Reconstruction is that the feature spaces are locally isometric of low-resolution and high-resolution Patches. But, this is not true for Super Resolution because of one to many mappings between Low Resolution and High Resolution patches. Advance NE method minimize the problem occurred in NE using combine learning technique used to train two projection matrices simultaneously and to map the original Low Resolution and High Resolution feature spaces onto a unified feature subspace. The Reconstruction weights of k-Nearest neighbour of Low Resolution image patches is found by performing operation on those Low Resolution patches in unified feature space. Combine learning use a coupled constraint by linking the LR–HR counterparts together with the k-nearest grouping patch pairs to handle a large number of samples. So, Advance neighbour embedding method gives better resolution than NE method.


INTRODUCTION
The Digital imaging system has lot of limitations, the imaging environment also important part of capturing an image, so it is not always easy to capture an image at a desired high-resolution (HR) level.However, in many practical applications such as medical imaging, video surveillance, computer vision, and entertainment, HR images are required usually to obtain a robust performance.Therefore, it becomes popular to synthesize a new HR image by using one or more lowresolution (LR) images [1].
The fast interpolation methods such as bilinear, bi-cubic, and other resampling methods [2]- [3] can increase the size of LR input, they are prone to blur high-frequency details and not super resolve the LR image, they only up-sample the image.Thus, a large number of super-resolution (SR) reconstruction techniques have been developed in recent years, which can be divided into two categories: multi image-based SR methods and example learning-based SR methods [4], [5].
There are two types of multi-image-based SR methods: The frequency-domain methods [6]- [7] and spatial-domain methods [8]- [9].It has been proved that the spatial representation of super resolution gives better performance than the frequency domain one.Representative spatial-domain methods include the regularized SR reconstruction approaches [10]- [11], the projection-onto-convex-sets approach [12], iterative back projection (IBP) [13], adaptive filtering [14], nonlocal means [15]- [16], and nonlocal kernel regression [9].But all the above mention methods require adequate number of LR images.Example learning based group of SR methods assumes that the high-frequency details lost in an LR image can be learned from a training set of LR and HR image pairs, i.e., the relationship between LR image patches and the corresponding HR patches can be used to estimate the missing HR frequency details in the given LR input.The example learning based method assumed that the counterparts of LR-HR image patch pairs (or their feature representations) are locally isometric.In contrast to above methods, this neighbor-embedding (NE)-based method does not require a large number of samples and achieves top level performance.

REVIEW OF NEIGHBOR EMBEDDING METHOD
Neighbor Embedding for SR reconstruction [19] is promising, except for its limitation of a locally isometric assumption in the Low Resolution and High Resolution feature spaces.To address this type of problem, the existing variations of the NE algorithm for image Super Resolution mainly concentrate on two aspects: one is to select more suitable features to characterize Low Resolution image patches such that the neighborhood relationship between LR-HR patch pairs can be preserved as consistently as possible and the other is to build a better reconstruction function by imposing some consistency constraints on HR-LR pairs.NE for SR reconstruction uses LLE [20] to estimate the HR patches corresponding to the LR inputs by assuming that the two manifolds in the HR and the LR image patch spaces are locally similar.Let


Training data sets The number of nearest neighbor -k Output: The neighbor embedding algorithm method can be summarized as follows: 1.For each patch c.Compute the high-resolution embedding using the appropriate high-resolution features of the k-nearest neighbors and the reconstruction weights.2. Construct the target high-resolution image by enforcing local compatibility and smoothness constraints between adjacent patches obtained in step 1(c).J u n e 2 0 , 2 0 1 3

ADVANCE NEIGHBOR EMBEDDING BASED SUPERRESOLUTION
In this section, a MAP reconstruction framework for SR is discussed.A coupled constraint on k-NNs of LR-HR counterparts, i.e., GPPs (groping patch pairs), is then established for the combine learning process.Thereafter, for each LR image patch to be super-resolved, its nearest GPPs (including k-NNs of LR-HR image patch pairs) is searched to perform combine learning for the unified feature subspace.Subsequently, the selection of k-NNs and the optimal weights for reconstruction are performed in the unified feature subspace for the initial HR output patches.Fig. 1 illustrates the reconstruction framework of Advance NE algorithm.
 The coupled constraint on GPPs consisting of the LR-HR image patches is applied to the learning process, which ensures a better consistency local LR and HR image patches.

 Matrices l
P and h P are the mapping matrices obtain from combine learning such that the difference between LR-HR counterparts is reduced as much as possible in the unified feature subspace.The optimal reconstruction weights for SR reconstruction are then estimated in the unified feature subspace rather than solely in the LR feature space.Taking the negative log likelihood E= -log (p (.)) in ( 2), we can get,

Coupled Constraint
In NE method [17], the training data set of the LR image patches, and that of their corresponding HR image patches where represents the index of each image patch and N is the number of image patches collected.In the traditional NE for SR i s x represents a d-dimensional feature vector of the ith LR image patch by concatenating the first-and second-order gradient features in horizontal and vertical directions, respectively.The Most existing NE algorithms perform SR reconstruction without considering the correlation between the LR and HR image patches.That means that they solely apply an isometric assumption to synthesize the expected HR estimates.However, this assumption does not hold perfectly for the SR problem.To solve or at least to reduce this problem, Advance NE considers the similarity between the LR image patch and the HR image patch with a coupled constraint as follows. Let .The existing NE-based SR [17] measures the similarity in the LR feature space.Generally, these algorithms make an assumption that the relationship between LR and HR feature spaces is locally isometric, i.e., the neighbor relationship between the LR and HR feature spaces is locally preserved.LR and HR features can share a unified feature subspace in which they are more closely associated with each other as in [18].Based on this consideration, the feature representations of LR and HR counterparts are projected onto a unified feature subspace by combine learning via a coupled constraint.Following this, the selection of k-NNs is conducted in this unified feature subspace instead of the original LR space.The combine learning on the GPP of each LR input locally also performed, rather than on all training samples, which is efficient and tractable for a training data set containing an enormous number of samples.
In unified feature space two mappinf matrices are measured one is for the LR feature space Suppose the two mapping functions  8) can modify as- From ( 13), the solution to h p can be obtained from, According to ( 14), eliminate h p in ( 12) and obtain Let eq. ( 15) reduces to ( 16) by replacing, U and V.
Once l P is obtained, substitute it into (13) for h P .By selecting the appropriate dimension of the p subspace (or the unified feature subspace), use the two derived projection matrices l P and h P to transform the original LR and HR feature spaces into a common subspace.In this way, the selection of k-NNs can be conducted within the unified feature subspace.Combine learning for the projection matrices is summarized in algorithm.

Algorithm 2(Combine Learning for Projection Matrices)
Input parameters:


Training data set x in the test data set


The size of LR image patch q*q ;  Neighborhood size k;  Dimension P of the unified feature subspace to be projected.J u n e 2 0 , 2 0 1 3

Output parameters:
 HR image Y Description: 1. Partition Y into q*q image patches with one or two pixels overlapped in raster-scan order to construct the test data set For each patch


Compute the transformed feature of


Compute the optimal weights by minimizing reconstruction error as follows:

EXPERIMENTAL RESULT
The PSNR obtained for the different images is given in table below.And the SR results of different images shown in fig.3,  4, and 5 respectively.Considering that the middle-frequency information of LR images has greater correlation with high frequency than low frequency, first magnify the original LR input by a factor of 3 with the bicubic interpolation and then perform SR on it.In this implementation, the combine learning technique on GPPs those are most relevant to each LR input patch to obtain the desired unified feature subspace.In principle, the Advance NE method has the potential to be extended to other SR applications such as face image hallucination.In addition, the construction of optimal GPPs rather than a fixed neighborhood size is challenging for this method.
When the magnification increase (e.g., more than *3 up-scaling), the SR problem becomes severely undetermined.It is also important to notice that NE methods takes bit more time to execute.Only single training image is used in NE to recover patches, but in Advance NE training dictionary means multiple training images used to recover the patches to obtain HR ouuput.
data set of the Corresponding HR image patch.And for Test image HR image patches.Here, N and M are the number of training image patches in the training data set and that of the test image patches in the test data set, respectively.
Compute the reconstruction weights of the neighbors that minimize the error of reconstructing j tx .

Fig. 1
Fig. 1 Illustration of neighbor embedding for Single image super-resolution HR images can be divided into three sub-bands, the high frequency band h I , the middle frequency band m I and the low frequency band l I .The middle-frequency components can be recovered based upon a simple interpolation algorithm (e.g., bilinear or bicubic).Thus, the goal of SR reconstruction is to estimate the missing high-frequency component with a single LR input.If prior knowledge, which is denoted as manifold MF, can be incorporated into the SR process, the problem of SR reconstruction can be written as, Where l f and h f denote two mapping functions used to convert two different feature vectors i s x and i s y into a unified feature space.In this way, use a common distance metric to measure the similarity between them.for the data cost of the reconstruction error of all input LR image patches with respect to their neighbourhood Ng (.) i.e. seen as the smoothness prior of the HR image, which can be approximated by overlapping one or several pixels within the adjacent image patches.Therefore, the key problem to solve is finding the two mapping functions l f and h f to transform the LR and HR feature spaces into a unified feature subspace.Once achieved, utilize the traditional NE algorithm to estimate the optimal weights and generate the desired HR image patch as a linear combination of the corresponding HR image patches from its neighbors.In addition, the problem of finding j M associated with each i s x remains unresolved.This problem is addressed by grouping the k-nearest group patch pairs of the LR and HR features in the training data set.
high frequency details of the i-th HR image patch.In order to apply combine learning to two different feature spaces of LR and HR patches, and augment another training data set the first-and second-order gradient features of HR image patches in horizontal and vertical directions, which is similar to the representation of each LR image patch.In such way i s z , denotes an m-dimensional feature vector of the ith HR image patch.
be a coupled set by concatenating each feature vector i s x and i s y .Thus, each column measurement i c in set C is a (d+m) dimensional feature vector, i.e. dynamic changes of the concatenated feature Vector, normalize it to a unit two-norm.For each vector i C in Set C, select the k-NNs associated with it and group them Together by

Fig. 2 .
Fig.2.Advance NE method reconstruction3.3Combine Learning on Patch PairsUnder the MAP reconstruction framework described in Section II-A, find the two mapping functions l f and

.
In this way, project the LR and HR feature spaces simultaneously onto a unified feature subspace and measure their similarity by function distance (Dist(.))represent Euclidean distance.For (7), the problem is converted to construct the mapping functions l f and h f such that distance ij d should be as close as possible in the unified feature subspace.If the similarity is measured by the Euclidean distance, then equation(11) obtain by assigning P, S, A to matrices shown in eq.(10) respectively.problem with respect to P can be obtained by the eigenvectors


Sum up mean values x and j t y together to generate the HR patch j t y and append it to

3 .
Produce the initial HR image X0 by merging all the HR patches in the set region between those adjacent patches, averaging fusion is applied to obtain the estimated pixels.

Fig. 3 (
Fig.3 (a) is down-sampled by *3 version of original image.Fig.3 (b) is High-Resolution image obtained from Bicubic Interpolation method.Fig.3 (c) is High-Resolution image obtained from Neighbor Embedding method.Fig.3 (d) is High-Resolution image obtained from Advance Neighbor Embedding Method and Fig.3 (e) is original High-Resolution image.

Fig. 5 (
Fig.5 (a) is down-sampled by *3 version of original image.Fig.5 (b) is High-Resolution image obtained from Bicubic Interpolation method.Fig.5 (c) is High-Resolution image obtained from Neighbor Embedding method.Fig.5 (d) is High-Resolution image obtained from Advance Neighbor Embedding Method and Fig.5 (e) is original High-Resolution image.CONCLUSIONThis paper represents Advance Neighbor Embedding and conventional NE-based single-image SR reconstruction.Thorough experimental results demonstrate that Advance NE method gives the best SR output.The distinction of the advance NE lies in the selection of k-NNs and the linear embedding being performed in the unified feature subspace rather than in the original LR space.In order to obtain the formation process of the LR image from its HR version, all the training HR images are downsampled by using the bicubic interpolation by a factor of 3 to obtain the corresponding LR images.Since the human visual system (HVS) is more sensitive to the luminance component than the chrominance components, Use of the YCbCr color space for color images and only perform SR reconstruction in the luminance component.Considering that the middle-frequency information of LR images has greater correlation with high frequency than low frequency, first magnify the original LR input by a factor of 3 with the bicubic interpolation and then perform SR on it.In this implementation, the combine learning technique on GPPs those are most relevant to each LR input patch to obtain the desired unified feature subspace.In principle, the Advance NE method has the potential to be extended to other SR applications such as face image hallucination.In addition, the construction of optimal GPPs rather than a fixed neighborhood size is challenging for this method.

3 .
Step 1(a) implement by using Euclidean distance to define neighborhood.Based on the k-nearest neighbors identified, step 1(b) seeks to find the best reconstruction weights for each patch