FFT multichannel interpolation and application to image super-resolution
Introduction
Interpolation by simple functions, such as trigonometric functions or rational functions, is an important mathematical technique used in physics and engineering sciences. It deals with the problem of reconstructing or approximating the continuous signals from a series of discrete points. An ideal interpolation usually tells us that a continuous signal can be exactly and uniquely reconstructed from the discrete points if it satisfies some suitable conditions, for instance, bandlimited in Fourier domain. Such an interpolation is also referred to as a sampling theorem in signal processing [1], [2]. In general, sampling formulas interpolate the given data even if the specific conditions for perfect reconstruction cannot be met. Therefore the error analysis of such sampling formulas is of great importance, because the recovered signal does not satisfy the conditions for ideal interpolation in most circumstances. Due to the wide range applications of interpolation, finding new interpolation or sampling formulas with error estimations as well as developing their fast algorithms for implementation have received considerable attentions in recently years, see for instance [3], [4], [5], [6], [7].
Most of the classical sampling formulas have centered around reconstructing a signal from its own samples [1], [8]. In fact, reconstructing a signal from data other than the samples of the original signal is possible. Papoulis [9] first proposed generalized sampling expansion (GSE) for bandlimited signals defined on real line . The GSE indicates that a σ-bandlimited signal f can be reconstructed from the samples of output signals of M linear systems. That is, one can reconstruct f from the samples of the output signalswhere F is Fourier transform (FT) of f and are the system functions and . The study of GSE has been extended in various directions. Cheung [10] introduced GSE for real-valued multidimensional signals associated with Fourier transform, while Wei et al. [11], [12] presented the GSE with generalized integral transformation, such as fractional Fourier transform (FrFT) and linear canonical transform (LCT). Some new sampling models in FrFT domain, such as shift-invariant spaces model [13] and multiple sampling rates model [14] were discussed. In [15], the authors studied GSE for quaternion-valued signals associated with quaternion Fourier transform (QFT). Importantly, the applications based on GSE as well as its generalizations have been widely conducted [3], [14], [16].
In a real application, the actual signal’s length is generally limited. The uncertainty principle states that a time-limited signal can not be bandlimited simultaneously in FT domain. However, it is known that the classical Shannon sampling theorem and GSE are aimed at reconstructing bandlimited signals in FT or FrFT domain. Therefore, a finite duration signal is commonly assumed to be a part of periodic signal in practice. Accordingly, there are certain studies concerning the interpolation theory of finite duration signals. A sampling theorem for trigonometric polynomials was first introduced by Goldman [17]. The author in [18] proposed pseudo-orthogonal bases for reconstructing signals defined on finite interval. In a series of papers [19], [20], [21], researchers extensively discussed the sinc interpolation of discrete cyclic signals and they also derived several equivalent interpolation formulas in distinct forms. As an extension of cyclic sinc interpolation, decomposing a finite duration signal in a basis of shifted and scaled versions of a generating function was studied in [22]. Moreover, they further presented an error analysis for their approximation method. Recently, the non-uniform sampling theorems for finite duration signals were also presented [23], [24].
In this paper, a multichannel interpolation for finite duration signals is studied. We derive a general interpolation formula that depends on the transfer functions of the linear systems. The formula bears a resemblance to the classical GSE defined on real line. Nevertheless, the recipe of derivation is different from the traditional one, and moreover, the proposed formula is given by a finite summation, as opposed to a infinite summation in the traditional case. Not only the theoretical error analysis but also the numerical simulations are provided to show the effectiveness of MCI in signal reconstruction. Since MCI is a novel interpolation method with high accuracy and efficiency, we also apply it to image super-resolution.
Single image super-resolution (SISR) is of importance in many personal, medical and industrial imaging applications [16], [25], [26], [27]. The existing SISR techniques roughly fall into three categories: the interpolation-based methods [14], [28], the learning-based methods [29], [30], [31], [32], [33], [34], [35] and the reconstruction-based methods [36], [37], [38], [39], [40]. The learning-based and reconstruction-based SISR techniques have been recognized as effective approaches to produce high-quality images with fine details from low-resolution inputs. It is known that, however, the performance of the learning-based methods depends much on the similarity between the images for training and for testing, while the performance of the reconstruction-based SISR methods relies on the reasonability of the incorporated prior information. Inspired by the success of the learning-based SISR methods, the authors in [41] proposed a novel reconstruction-based SISR method (called CRNS algorithm) which utilizes the complementary advantages of both the learning-based and the reconstruction-based methods. That is, CRNS algorithm takes both the external and internal priors into consideration to improve SISR results.
In spite of the remarkable performance of learning-based and reconstruction-based methods in SISR, the interpolation-based methods are also widely used to produce high resolution (HR) images for their computational efficiency. From Table 4, our MCI method holds over 300 times faster than the CRNS method. Besides producing HR images, the interpolation-based methods are commonly incorporated into other methods to improve HR results. In [42], the authors proposed a two-scale approach to recover HR images by interpolation and reconstruction respectively. The gradient profile model in [43] is trained from LR and HR image pairs, where the initial HR images are produced by interpolation-based methods. Therefore, it is convinced that a good interpolation-based method can be useful to SISR. Recently, the application of GSE associated with FrFT to SISR was investigated [14]. Compared with the classical image interpolation algorithms such as Lanczos and bicubic, the GSE-based algorithm tends to produce images with less blur and good contrast. However, the conventional Shannon sampling theorem [1] and GSE associated with FT or FrFT [9], [14] are involved in infinite number of sample values spreading over whole real line and the interpolation functions have infinite duration. Once they are applied to SISR, the truncation errors are inevitable. Fortunately, there is no problem with truncation error for MCI. Besides, the MCI-based SISR algorithm can preserve lots of information of original image for reshaped image, in view of the proposed MCI makes good use of multifaceted information such as first derivative (which may include edge information of image) and second derivative (which may include detail information of image). It will be shown that the MCI-based algorithm can produce better SISR results than GSE-based algorithm. Moreover, the FFT-based implementation of MCI makes the proposed algorithm very fast (see Table 4). To validate the performance of our SISR method, we further combine MCI with displacement field (FD) method such that the produced HR images can preserve sharp edges. The experiments show that the FD-based method achieves a significant improvement by introducing MCI.
In summary, the contributions of this paper are highlighted as follows:
- 1.
We propose a multichannel interpolation for finite duration signals (MCI). The proposed MCI is capable of generating various useful interpolation formulas by selecting suitable parameters according to the types and the amount of collected data. In addition, it would restore the original signal f and some integral transformations (such as Hilbert transform) of f.
- 2.
Based on FFT, a fast algorithm which brings high computational efficiency and reliability for MCI is also presented.
- 3.
Two questions naturally arise when using a sampling or interpolation formula to reconstruct a non-bandlimited signal. One is whether the set of original samples stays unchanged after reconstruction, namely, whether the interpolation consistency holds. The other one is whether the error of imperfect reconstruction can be estimated. To the authors’ knowledge, these two issues have not been addressed for the classical GSE in the literature. By contrast, error analysis arising in reconstructing non-bandlimited signals by the proposed MCI is studied. Moreover, the corresponding interpolation consistency is also proved.
- 4.
The proposed MCI is applied to single image super-resolution reconstruction (SISR). Its main advantage is making good use of multifaceted information of image, so that the reconstructed image retains lots of information of the original image. Moreover, the interpolation consistency and the untruncated implementation of MCI can reduce the reconstruction errors. The superior performance of the proposed algorithm in accuracy and speed of SISR is shown by several experimental simulations.
The rest of the paper is organized as follows: Section 2 recalls some preliminaries of Fourier series. Section 3 formulates MCI and presents some examples to illustrate how to use MCI flexibly. The error analysis and interpolation consistency are drawn in Section 4. In Section 5, the effectiveness of the proposed MCI for approximating signals is demonstrated by several numerical examples and the application of MCI to single image super-resolution reconstruction is also addressed. Finally, conclusions are made in Section 6.
Section snippets
Preliminaries
This part recalls some preparatory knowledge of Fourier series (see e.g. [44]). Throughout the paper, the set of real numbers, integers and positive integers are denoted by and respectively. Without loss of generality, we restrict attention to the signals defined on unit circle ≔ [0, 2π).
Let Lp() be the totality of functions f(t) such thatand lp be the sequence space defined byFor f ∈ L2() ⊂ L1(), it can be written as
Formulation of MCI
Let and assume that is divisible by M, namely, . We cut the set of integers into pieces for convenience. Let us setThen we have and .
For 1 ≤ m ≤ M, letIt follows from (2) thatwhere . We particularly mention that the series (5) may not be convergent in general. Nevertheless, gm(t) is well defined when {cm(n)} ∈ l
Interpolation consistency and error analysis
In practice, the signals (such as chirp signal and Gaussian signal) are not strictly bandlimited in general. As we know, if a signal f(t) is not bandlimited, the reconstructed signal given by any formula derived in examples of the previous section is not equal to f(t). In this section, f is merely assumed to be square integrable on unit circle and is not necessary to be bandlimited. We define the following approximation operator:There are two
Numerical examples
In this part, we shall demonstrate the effectiveness of for approximating signals experimentally. We compare the results by using the different formulas derived in Section 3 to reconstruct f (or ). LetFrom the theory of Hardy space, the imaginary part of ϕ(eit) is the Hilbert transform of its real part. In the following we select then its Hilbert transform is .
The relative mean square error (RMSE)
Conclusions
In this paper, we presented a novel multichannel interpolation for finite duration signals. We show that the reconstruction of a continuous signal using data other than the samples of original signal is feasible. Under suitable conditions, only μ(IN) total number of samples, no matter what their types, are needed to perfectly recover the signal of the form (4). Quantitative error analysis for reconstructing non-bandlimited signals is also studied. Both of the theoretical analysis and
Conflict of interest
None.
Acknowledgements
The authors acknowledge financial support from the Macao Science and Technology Development Fund (FDCT/031/2016/A1 and FDCT/085/2018/A2).
References (54)
- et al.
Papoulis-like generalized sampling expansions in fractional Fourier domains and their application to superresolution
Opt. Commun.
(2007) - et al.
New sampling formulae for non-bandlimited signals associated with linear canonical transform and nonlinear Fourier atoms
Signal Process.
(2010) Sampling theorem for the short-time linear canonical transform and its applications
Signal Process.
(2015)- et al.
Multichannel sampling expansion in the linear canonical transform domain and its application to superresolution
Opt. Commun.
(2011) - et al.
A generalized sampling model in shift-invariant spaces associated with fractional Fourier transform
Signal Process.
(2018) - et al.
Sampling theorems for signals periodic in the linear canonical transform domain
Opt. Commun.
(2013) - et al.
A locally adaptive zooming algorithm for digital images
Image Vis. Comput.
(2002) - et al.
Fast single image super-resolution using sparse gaussian process regression
Signal Process.
(2017) - et al.
Single image super-resolution using collaborative representation and non-local self-similarity
Signal Process.
(2018) - et al.
The AFD methods to compute Hilbert transform
Appl. Math. Lett.
(2015)
Advances in Shannon’s Sampling Theory
Sampling theorem for fractional bandlimited signals: a self-contained proof. application to digital holography
IEEE Signal Process. Lett.
Functionally weighted Lagrange interpolation of band-limited signals from nonuniform samples
IEEE Trans. Signal Process.
Novel sampling formulas associated with quaternionic prolate spheroidal wave functions
Adv. Appl. Clifford Algebr.
FFT Interpolation from nonuniform samples lying in a regular grid
IEEE Trans. Signal Process.
Generalized sampling expansion
IEEE Trans. Circuits Syst.
A multidimensional extension of Papoulis’ generalized sampling expansion with the application in minimum density sampling
Advanced Topics in Shannon Sampling and Interpolation Theory
Generalized sampling expansion for bandlimited signals associated with the fractional Fourier transform
IEEE Signal Process. Lett.
Generalized sampling expansions with multiple sampling rates for lowpass and bandpass signals in the fractional Fourier transform domain.
IEEE Trans. Signal Process.
Generalized sampling expansions associated with quaternion Fourier transform
Math. Meth. Appl. Sci.
Image scaling algorithm using multichannel sampling in the linear canonical transform domain
Signal Image Video Process.
Information Theory
A generalized sampling theorem
Electron. Commun. Jpn.
Sinc interpolation of discrete periodic signals
IEEE Trans. Signal Process.
Comments on “sinc interpolation of discrete periodic signals”
IEEE Trans. Signal Process.
Notes on the interpolation of discrete periodic signals using sinc function related approaches
IEEE Trans. Signal Process.
Sampling of periodic signals: a quantitative error analysis
IEEE Trans. Signal Process.
Cited by (19)
Fusing bi-directional global–local features for single image super-resolution
2024, Engineering Applications of Artificial IntelligenceHigh-order cross-scale attention network for single image super-resolution
2022, Digital Signal Processing: A Review JournalCitation Excerpt :However, it is an inherently ill-posed problem since infinite HR solutions can be downsampled to the same LR image. In the literature, a wide variety of SISR methods have been proposed to tackle this problem via interpolation filter [1,2], reconstruction optimization [3,4], and nonlinear mapping learning [5,6]. In recent years, benefiting from the strong learning ability of convolutional neural networks (CNN), deep learning-based SISR methods have achieved remarkable performance.
Signal reconstruction from noisy multichannel samples
2022, Digital Signal Processing: A Review JournalCitation Excerpt :Certain studies have been given to the sampling theorems for periodic bandlimited functions [10,11]. Moreover, the multichannel sampling approach has been extended to time-limited functions in [12]. The interpolation of noisy data introduces the undesirable error in the reconstructed signal.
Nonuniform reconstruction of periodic bandlimited signals without sampling points’ number restriction
2020, OptikCitation Excerpt :As is known to all, a bandlimited T-periodic signal has finite nonzero Fourier coefficients. Such signals are also known as finite order trigonometric polynomials [2]. To be specific, a T-periodic signal f(t) is said to be bandlimited to 2πK/T if ck = 0 for |k| > K, where K is a positive integer [24].
Multichannel interpolation of nonuniform samples with application to image recovery
2020, Journal of Computational and Applied MathematicsCitation Excerpt :We consider the image recovery problem from the point that some pixel values are missing in a digital image. We had also shown in [26] that the multichannel interpolation (uniform sampling case) has a good performance in single image super-resolution. Nevertheless, the proposed method has some limitations in image processing applications since it is only a 1D interpolation technique.
Sampling formulas for non-bandlimited quaternionic signals
2022, Signal, Image and Video Processing