Elsevier

Signal Processing

Volume 162, September 2019, Pages 21-34
Signal Processing

FFT multichannel interpolation and application to image super-resolution

https://doi.org/10.1016/j.sigpro.2019.03.025Get rights and content

Highlights

  • The paper presents the theory of multichannel interpolation which can be implemented fast and efficiently by FFT.

  • Theoretical error analysis and numerical simulations are provided to ensure its effectiveness in reconstructing signals from different types of samples.

  • The application of the proposed interpolation method to image super-resolution is also discussed.

Abstract

This paper presents an innovative set of tools to support a methodology for the multichannel interpolation (MCI) of a discrete signal. It is shown that a bandlimited signal f can be exactly reconstructed from finite samples of gk (1 ≤ k ≤ M) which are the responses of M linear systems with input f. The proposed interpolation can also be applied to approximate non-bandlimited signals. Quantitative error is analyzed to ensure its effectiveness in approximating non-bandlimited signals and its Hilbert transform. Based on the FFT technique, a fast algorithm which brings high computational efficiency and reliability for MCI is presented. The standout performance of MCI is illustrated by several simulations. Additionally, the proposed interpolation is applied to the single image super-resolution (SISR). Its superior performance in accuracy and speed of SISR is demonstrated by the experimental studies. Our results are compared qualitatively and quantitatively with the state-of-the-art methods in image upsampling and reconstruction by using the standard measurement criteria.

Introduction

Interpolation by simple functions, such as trigonometric functions or rational functions, is an important mathematical technique used in physics and engineering sciences. It deals with the problem of reconstructing or approximating the continuous signals from a series of discrete points. An ideal interpolation usually tells us that a continuous signal can be exactly and uniquely reconstructed from the discrete points if it satisfies some suitable conditions, for instance, bandlimited in Fourier domain. Such an interpolation is also referred to as a sampling theorem in signal processing [1], [2]. In general, sampling formulas interpolate the given data even if the specific conditions for perfect reconstruction cannot be met. Therefore the error analysis of such sampling formulas is of great importance, because the recovered signal does not satisfy the conditions for ideal interpolation in most circumstances. Due to the wide range applications of interpolation, finding new interpolation or sampling formulas with error estimations as well as developing their fast algorithms for implementation have received considerable attentions in recently years, see for instance [3], [4], [5], [6], [7].

Most of the classical sampling formulas have centered around reconstructing a signal from its own samples [1], [8]. In fact, reconstructing a signal from data other than the samples of the original signal is possible. Papoulis [9] first proposed generalized sampling expansion (GSE) for bandlimited signals defined on real line R. The GSE indicates that a σ-bandlimited signal f can be reconstructed from the samples of output signals of M linear systems. That is, one can reconstruct f from the samples g1(nT),,gM(nT) of the output signalsgm(t)=12πσσF(ω)Hm(ω)eitωdω,m=1,,Mwhere F is Fourier transform (FT) of f and H1,,HM are the system functions and T=Mπ/σ. The study of GSE has been extended in various directions. Cheung [10] introduced GSE for real-valued multidimensional signals associated with Fourier transform, while Wei et al. [11], [12] presented the GSE with generalized integral transformation, such as fractional Fourier transform (FrFT) and linear canonical transform (LCT). Some new sampling models in FrFT domain, such as shift-invariant spaces model [13] and multiple sampling rates model [14] were discussed. In [15], the authors studied GSE for quaternion-valued signals associated with quaternion Fourier transform (QFT). Importantly, the applications based on GSE as well as its generalizations have been widely conducted [3], [14], [16].

In a real application, the actual signal’s length is generally limited. The uncertainty principle states that a time-limited signal can not be bandlimited simultaneously in FT domain. However, it is known that the classical Shannon sampling theorem and GSE are aimed at reconstructing bandlimited signals in FT or FrFT domain. Therefore, a finite duration signal is commonly assumed to be a part of periodic signal in practice. Accordingly, there are certain studies concerning the interpolation theory of finite duration signals. A sampling theorem for trigonometric polynomials was first introduced by Goldman [17]. The author in [18] proposed pseudo-orthogonal bases for reconstructing signals defined on finite interval. In a series of papers [19], [20], [21], researchers extensively discussed the sinc interpolation of discrete cyclic signals and they also derived several equivalent interpolation formulas in distinct forms. As an extension of cyclic sinc interpolation, decomposing a finite duration signal in a basis of shifted and scaled versions of a generating function was studied in [22]. Moreover, they further presented an error analysis for their approximation method. Recently, the non-uniform sampling theorems for finite duration signals were also presented [23], [24].

In this paper, a multichannel interpolation for finite duration signals is studied. We derive a general interpolation formula that depends on the transfer functions of the linear systems. The formula bears a resemblance to the classical GSE defined on real line. Nevertheless, the recipe of derivation is different from the traditional one, and moreover, the proposed formula is given by a finite summation, as opposed to a infinite summation in the traditional case. Not only the theoretical error analysis but also the numerical simulations are provided to show the effectiveness of MCI in signal reconstruction. Since MCI is a novel interpolation method with high accuracy and efficiency, we also apply it to image super-resolution.

Single image super-resolution (SISR) is of importance in many personal, medical and industrial imaging applications [16], [25], [26], [27]. The existing SISR techniques roughly fall into three categories: the interpolation-based methods [14], [28], the learning-based methods [29], [30], [31], [32], [33], [34], [35] and the reconstruction-based methods [36], [37], [38], [39], [40]. The learning-based and reconstruction-based SISR techniques have been recognized as effective approaches to produce high-quality images with fine details from low-resolution inputs. It is known that, however, the performance of the learning-based methods depends much on the similarity between the images for training and for testing, while the performance of the reconstruction-based SISR methods relies on the reasonability of the incorporated prior information. Inspired by the success of the learning-based SISR methods, the authors in [41] proposed a novel reconstruction-based SISR method (called CRNS algorithm) which utilizes the complementary advantages of both the learning-based and the reconstruction-based methods. That is, CRNS algorithm takes both the external and internal priors into consideration to improve SISR results.

In spite of the remarkable performance of learning-based and reconstruction-based methods in SISR, the interpolation-based methods are also widely used to produce high resolution (HR) images for their computational efficiency. From Table 4, our MCI method holds over 300 times faster than the CRNS method. Besides producing HR images, the interpolation-based methods are commonly incorporated into other methods to improve HR results. In [42], the authors proposed a two-scale approach to recover HR images by interpolation and reconstruction respectively. The gradient profile model in [43] is trained from LR and HR image pairs, where the initial HR images are produced by interpolation-based methods. Therefore, it is convinced that a good interpolation-based method can be useful to SISR. Recently, the application of GSE associated with FrFT to SISR was investigated [14]. Compared with the classical image interpolation algorithms such as Lanczos and bicubic, the GSE-based algorithm tends to produce images with less blur and good contrast. However, the conventional Shannon sampling theorem [1] and GSE associated with FT or FrFT [9], [14] are involved in infinite number of sample values spreading over whole real line and the interpolation functions have infinite duration. Once they are applied to SISR, the truncation errors are inevitable. Fortunately, there is no problem with truncation error for MCI. Besides, the MCI-based SISR algorithm can preserve lots of information of original image for reshaped image, in view of the proposed MCI makes good use of multifaceted information such as first derivative (which may include edge information of image) and second derivative (which may include detail information of image). It will be shown that the MCI-based algorithm can produce better SISR results than GSE-based algorithm. Moreover, the FFT-based implementation of MCI makes the proposed algorithm very fast (see Table 4). To validate the performance of our SISR method, we further combine MCI with displacement field (FD) method such that the produced HR images can preserve sharp edges. The experiments show that the FD-based method achieves a significant improvement by introducing MCI.

In summary, the contributions of this paper are highlighted as follows:

  • 1.

    We propose a multichannel interpolation for finite duration signals (MCI). The proposed MCI is capable of generating various useful interpolation formulas by selecting suitable parameters according to the types and the amount of collected data. In addition, it would restore the original signal f and some integral transformations (such as Hilbert transform) of f.

  • 2.

    Based on FFT, a fast algorithm which brings high computational efficiency and reliability for MCI is also presented.

  • 3.

    Two questions naturally arise when using a sampling or interpolation formula to reconstruct a non-bandlimited signal. One is whether the set of original samples stays unchanged after reconstruction, namely, whether the interpolation consistency holds. The other one is whether the error of imperfect reconstruction can be estimated. To the authors’ knowledge, these two issues have not been addressed for the classical GSE in the literature. By contrast, error analysis arising in reconstructing non-bandlimited signals by the proposed MCI is studied. Moreover, the corresponding interpolation consistency is also proved.

  • 4.

    The proposed MCI is applied to single image super-resolution reconstruction (SISR). Its main advantage is making good use of multifaceted information of image, so that the reconstructed image retains lots of information of the original image. Moreover, the interpolation consistency and the untruncated implementation of MCI can reduce the reconstruction errors. The superior performance of the proposed algorithm in accuracy and speed of SISR is shown by several experimental simulations.

The rest of the paper is organized as follows: Section 2 recalls some preliminaries of Fourier series. Section 3 formulates MCI and presents some examples to illustrate how to use MCI flexibly. The error analysis and interpolation consistency are drawn in Section 4. In Section 5, the effectiveness of the proposed MCI for approximating signals is demonstrated by several numerical examples and the application of MCI to single image super-resolution reconstruction is also addressed. Finally, conclusions are made in Section 6.

Section snippets

Preliminaries

This part recalls some preparatory knowledge of Fourier series (see e.g. [44]). Throughout the paper, the set of real numbers, integers and positive integers are denoted by R, Z and Z+ respectively. Without loss of generality, we restrict attention to the signals defined on unit circle T ≔ [0, 2π).

Let Lp(T,) be the totality of functions f(t) such thatfp:=(12πT|f(t)|pdt)1p<,and lp be the sequence space defined bylp:={{x(n)}:nZ|x(n)|p<}For f ∈ L2(T,) ⊂ L1(T,), it can be written asf(t)=nZa

Formulation of MCI

Let N1,N2Z, MZ+ and assume that N2N1+1 is divisible by M, namely, N2N1+1M=LZ+. We cut the set of integers into pieces for convenience. Let us setIk={n:N1+(k1)LnN1+kL1},Jk=l=k+1M+kIl.Then we have IN=k=1MIk=J0 and Z=kIk.

For 1 ≤ m ≤ M, lethm(t)=nZbm(n)eint,gm(t)=(f*hm)(t)=12πTf(s)hm(ts)ds.It follows from (2) thatgm(t)=nZcm(n)eint,where cm(n)=a(n)bm(n). We particularly mention that the series (5) may not be convergent in general. Nevertheless, gm(t) is well defined when {cm(n)} ∈ l

Interpolation consistency and error analysis

In practice, the signals (such as chirp signal and Gaussian signal) are not strictly bandlimited in general. As we know, if a signal f(t) is not bandlimited, the reconstructed signal f˜(t) given by any formula derived in examples of the previous section is not equal to f(t). In this section, f is merely assumed to be square integrable on unit circle and is not necessary to be bandlimited. We define the following approximation operator:TNf(t):=1Lm=1Mp=0L1gm(2πpL)ym(t2πpL).There are two

Numerical examples

In this part, we shall demonstrate the effectiveness of TN for approximating signals experimentally. We compare the results by using the different formulas derived in Section 3 to reconstruct f (or Hf). Letϕ(z)=0.08z2+0.06z10(1.3z)(1.5z)+0.05z3+0.09z10(1.2+z)(1.3+z).From the theory of Hardy space, the imaginary part of ϕ(eit) is the Hilbert transform of its real part. In the following we select f(t)=[ϕ(eit)], then its Hilbert transform is Hf(t)=[ϕ(eit)].

The relative mean square error (RMSE)

Conclusions

In this paper, we presented a novel multichannel interpolation for finite duration signals. We show that the reconstruction of a continuous signal using data other than the samples of original signal is feasible. Under suitable conditions, only μ(IN) total number of samples, no matter what their types, are needed to perfectly recover the signal of the form (4). Quantitative error analysis for reconstructing non-bandlimited signals is also studied. Both of the theoretical analysis and

Conflict of interest

None.

Acknowledgements

The authors acknowledge financial support from the Macao Science and Technology Development Fund (FDCT/031/2016/A1 and FDCT/085/2018/A2).

References (54)

  • A.I. Zayed

    Advances in Shannon’s Sampling Theory

    (1993)
  • R. Torres et al.

    Sampling theorem for fractional bandlimited signals: a self-contained proof. application to digital holography

    IEEE Signal Process. Lett.

    (2006)
  • J. Selva

    Functionally weighted Lagrange interpolation of band-limited signals from nonuniform samples

    IEEE Trans. Signal Process.

    (2009)
  • D. Cheng et al.

    Novel sampling formulas associated with quaternionic prolate spheroidal wave functions

    Adv. Appl. Clifford Algebr.

    (2017)
  • J. Selva

    FFT Interpolation from nonuniform samples lying in a regular grid

    IEEE Trans. Signal Process.

    (2015)
  • A. Papoulis

    Generalized sampling expansion

    IEEE Trans. Circuits Syst.

    (1977)
  • K.F. Cheung

    A multidimensional extension of Papoulis’ generalized sampling expansion with the application in minimum density sampling

    Advanced Topics in Shannon Sampling and Interpolation Theory

    (1993)
  • D. Wei et al.

    Generalized sampling expansion for bandlimited signals associated with the fractional Fourier transform

    IEEE Signal Process. Lett.

    (2010)
  • D. Wei et al.

    Generalized sampling expansions with multiple sampling rates for lowpass and bandpass signals in the fractional Fourier transform domain.

    IEEE Trans. Signal Process.

    (2016)
  • D. Cheng et al.

    Generalized sampling expansions associated with quaternion Fourier transform

    Math. Meth. Appl. Sci.

    (2018)
  • Y. Li et al.

    Image scaling algorithm using multichannel sampling in the linear canonical transform domain

    Signal Image Video Process.

    (2014)
  • S. Goldman

    Information Theory

    (1953)
  • H. Ogawa

    A generalized sampling theorem

    Electron. Commun. Jpn.

    (1989)
  • T. Schanze

    Sinc interpolation of discrete periodic signals

    IEEE Trans. Signal Process.

    (1995)
  • F. Candocia et al.

    Comments on “sinc interpolation of discrete periodic signals”

    IEEE Trans. Signal Process.

    (1998)
  • S.R. Dooley et al.

    Notes on the interpolation of discrete periodic signals using sinc function related approaches

    IEEE Trans. Signal Process.

    (2000)
  • M. Jacob et al.

    Sampling of periodic signals: a quantitative error analysis

    IEEE Trans. Signal Process.

    (2002)
  • Cited by (19)

    • Fusing bi-directional global–local features for single image super-resolution

      2024, Engineering Applications of Artificial Intelligence
    • High-order cross-scale attention network for single image super-resolution

      2022, Digital Signal Processing: A Review Journal
      Citation Excerpt :

      However, it is an inherently ill-posed problem since infinite HR solutions can be downsampled to the same LR image. In the literature, a wide variety of SISR methods have been proposed to tackle this problem via interpolation filter [1,2], reconstruction optimization [3,4], and nonlinear mapping learning [5,6]. In recent years, benefiting from the strong learning ability of convolutional neural networks (CNN), deep learning-based SISR methods have achieved remarkable performance.

    • Signal reconstruction from noisy multichannel samples

      2022, Digital Signal Processing: A Review Journal
      Citation Excerpt :

      Certain studies have been given to the sampling theorems for periodic bandlimited functions [10,11]. Moreover, the multichannel sampling approach has been extended to time-limited functions in [12]. The interpolation of noisy data introduces the undesirable error in the reconstructed signal.

    • Nonuniform reconstruction of periodic bandlimited signals without sampling points’ number restriction

      2020, Optik
      Citation Excerpt :

      As is known to all, a bandlimited T-periodic signal has finite nonzero Fourier coefficients. Such signals are also known as finite order trigonometric polynomials [2]. To be specific, a T-periodic signal f(t) is said to be bandlimited to 2πK/T if ck = 0 for |k| > K, where K is a positive integer [24].

    • Multichannel interpolation of nonuniform samples with application to image recovery

      2020, Journal of Computational and Applied Mathematics
      Citation Excerpt :

      We consider the image recovery problem from the point that some pixel values are missing in a digital image. We had also shown in [26] that the multichannel interpolation (uniform sampling case) has a good performance in single image super-resolution. Nevertheless, the proposed method has some limitations in image processing applications since it is only a 1D interpolation technique.

    • Sampling formulas for non-bandlimited quaternionic signals

      2022, Signal, Image and Video Processing
    View all citing articles on Scopus
    View full text