1. Introduction

Degradation is usually unavoidable during image acquisition and initial construction. Thermal Infra-Red (TIR) image degradation is sourced from many factors, mainly: limitation of lens’s aperture, pixel pitch size, atmosphere attenuation and turbulence, distortion and blurring, and the non-uniform response of the TIR detector. With a staggered time delay integration (TDI) architecture, the TIR detector is able to offer better photosensitivity than a conventional Charge-Coupled Device (CCD). One primary advantage of this sensor design compared to the conventional line-scan sensor is the possibility of detecting low energy levels with a superior signal-to-noise ratio. TDI CCD is therefore widely used in remote sensing systems, thermal imaging in particular, for improving the low light level capability [1]. However, the discrete movement of the charge transfer in the TDI imaging mode can never entirely synchronize with the continuous image scanning velocity and this introduces image displacement leading to MTF (Modulation Transfer Function) loss. Another drawback is pixel response non-uniformity [2]. This situation becomes more severe for an imager mounted on a rapidly moving and vibrating platform such as an airborne thermal imager. Consequently, instantaneous geometric distortion and image degradation are introduced. Moreover, the thermal diffusion and non-uniformity radiometric response [2] between staggered time-division multiplexing scanning lines may degrade the image further. These degradations cannot be eliminated completely by a traditional optoelectronics or optics approach.

CI has shown that the integration of optics, optoelectronics and signal processing has a promising potential to improve imaging system performance [3]. It describes the emerging field of optical imaging method, in which a true underlying image f of an object is not simply formed through a lens and directly sampled by a photo-detector array, but rather the image formation and signal processing are combined together to improve the imaging system performance throughput. Over the last decade, a number of successful CI approaches have been presented for various tasks, such as motion de-blurring, high quality image reconstruction and 3D reconstruction from a thin observation module by a bound optics (TOMBO) compound-eye imaging system [4]. Additionally, various studies for radiometric non-uniformity and image de-noising were carried out for two-dimensional infrared focus panel array (IR FPA) staring cameras [2]. To improve Long-Wavelength IR (LWIR) image quality, a multi-channel sampling method with several sub-apertures on a stationary platform was studied and several image restoration methods were tested to attain higher spatial resolution images [5]. This imaging configuration is intrinsically similar to that of time-division multiplexing scanning lines mounted in a moving platform. Further, a time-division multiplexing imaging scheme with a moving lenslet array technique was presented in [6]; using a similar experimental configuration as [7,8 ], high resolution images were then reconstructed from the low-resolution images captured by many sub-apertures. In all these experiments, it was assumed that the precise relative positions between lenses or displacements between images were given. Most of the studies have focused on image de-blurring, restoration algorithms or some signal and noise models, and some are based on experiments in laboratory with relatively sophisticated setups or static platforms in visible and IR bands. However, in practical cases, prior to image restoration employing various de-convolution methods, the problem of displacement estimation between images must be resolved. Otherwise, strong artefacts will be introduced into the restored images. Such situations typically occur for imagers employing a time division multi-channel detector on rapidly moving platforms which are subject to mechanical vibration as well as discrete movement of the charges in TDI mode.

Inspired by the experimental configuration in [5–8 ], a CI image restoration approach is proposed for the staggered TDI line-scanning thermal infrared imager. By contrast, this study focuses on not only nonlinear instantaneous geometric distortions due to time-division multi-channel imaging but also spatial degradation of thermal infrared images. The contributions of the proposed CI method lie in the 1D high precision (0.01 pixel) displacement estimation and realignment method for non-uniform geometric distortion; a sparse-and-smooth constrained deconvolution CI method under the assumption that a thermal infrared image is 2D piecewise C² continuous.

2. Problems Formulation

2.1 TIR imager with staggered TDI-CCD liners

A TIR imager with staggered TDI CCD arrays usually operates with a relatively high scanning velocity imaging pattern as presented in Fig. 1(a) . In the imager, the TDI focal plane layout with staggered half-pitch offset imaging arrays is illustrated in Fig. 1(b). The TDI array is assembled with M rows with half-pitch offset between odd rows and even rows comprising N TDI stages each [1], in which the charge is clocked and transferred in phase with the motion of scene scanning [9]. These two linear detector arrays are arranged parallel with half-pixel pitch offset in the cross-scanning direction to improve the spatial sampling frequency. This is an advantage of the stagger TDI linear array camera.The main task of the camera is to capture a true underlying image of the scene line by line along the scanning direction at each instant of time. The staggered TDI imager is basically composed of a pair of spatially shifted linear imaging arrays, the red line and blue line shown in Fig. 1(a), which capture images separately in two spatially separated fields, as the scene passing through the field of view. These two time-division multiplexed channels collect two images of odd rows and even rows at different sampling instants, so as to synthesize a higher spatial resolution image. This imaging pattern is essentially equivalent to that of a moving lens and detector array at approximately the same focal plane in [6,8 ]. Each detector pixel has N integration stages as shown in Fig. 1(b) and thus the image SNR will be improved $\sqrt{N}$ times compared to the traditional CCD. Actually, it is a successful computational imaging method to improve image SNR and radiometric resolution in the case of high scanning velocity. The pixel digital number (DN) of a TDI line-scanning sensor, I_(r,c), is written as Eq. (1).

 

Fig. 1 Architecture of a staggered TDI imager. (a) Scanning Imaging with a staggered TDI detector. (b) TDI focal plane layout with offset imaging arrays.

Download Full Size | PDF

2.2 Non-uniform image displacement and degradation

However, this seemingly advantageous staggered TDI-CCD architecture will spoil image quality severely in some cases, mainly because of the inherent displacement of staggered photo-detector arrays along the scanning direction and non-uniform blurring. This type of displacement can be reduced by improving sensor design but residual motions always exist [10]. Firstly, although the average velocity of discrete charge movement in N stage of TDI can match the velocity of the mechanical scanning exactly, there are still relative motions between photosensitive pixels and imaged points [10,11 ]. The discrete movement of the charge transfer in TDI imaging mode is never in total synchrony with the continuous image scanning velocity; the motion of signal charge in TDI mode is always either lagging behind the image or ahead of the image by a sub-pixel amount, presented as a “jerky” motion [10]. Generally, nearly one pixel displacement is always introduced into the imaging system [10,11 ] leading to MTF loss, but it is considered as residual error and is ignored sometimes. In [12], it was shown that less than 1 pixel displacement would not lead to perceptible image quality degradation for a TDI imager without a staggered configuration. Secondly, the mechanical vibration of a platform, characterized by a sinusoidal type motion [13,14 ] existing particularly in imagers mounted on an airborne platforms, is unavoidable, which is coupled with image displacement and motion blur causing further MTF loss. The relative motion is more complicated between multiple TDI-CCD lines in practical cases. As a result, the “jerky” motion is usually too severe to be ignored for staggered TDI imagers (See Figs. 2(a) and 2(b) ). These two types of displacement are time-dependent. Additionally, for high spatial resolution imaging, the image displacement between two staggered detector arrays is entangled with 3D disparity in scenes of high relief. This case is scene-dependent; the larger the margin between two staggered TDI lines (See Fig. 1(b)), the larger displacement will occur. In the super-position procedure, these displacements along the scanning direction always introduce non-uniform image distortion. Therefore, despite the advantageous design of the staggered TDI-CCD imager, the image resolution often does not meet the design expectations due to motion and degradation. So the effort for high-resolution capability of the imager is often in vain [14].

 

Fig. 2 A TIR image sub-block taken by a staggered TDI airborne scanner. The image is dramtically degraded by instantaneous geometric distortion. Edge jitter is evident in the image. (a) The TIR image raw data. (b) The zoom-in sub-block of the area in the red box of the left image.

Download Full Size | PDF

In the scanning direction, as shown in Fig. 1(a), it is assumed that a relative motion velocity v(t) leads to a time-dependent displacement between two time-division multiplexing channels. Without loss generality, the displacement along the scanning direction can be formulated as Eq. (2).

For simplicity, we can define,

 

Fig. 3 MTF loss due to degradation of motion displacement

Download Full Size | PDF

CI can offer a flexible solution to this problem that is not achievable using a traditional optoelectronics or optics approach. In [13,14 ], the influence of platform vibration, which was considered as a type of slow sinusoidal vibration and high frequency vibration, on image quality, MTF and spatial resolution was analyzed and investigated. Then a TIR image restoration method for interlaced distortion and motion blurring was proposed with a block matching algorithm (BMA) and Wiener filter for motion deblurring, assuming linear uniform motion degradation in scanning mode [14,16 ]. In [13], a full search block-matching algorithm (FSBMA) was proposed for image displacement estimation. The basic limitation of the BMA-based algorithm is that this method can only achieve integer matching precision. Then Wiener deconvolution was utilized for restoration; but it tends to produce over-smoothing or ringing artifacts in the restored images, similar to the POCS method [1] and Tikhonov method. In this study, we assume that the displacement between two time-division channels is doubled and more complicated due to the 3D scene and platform vibration as shown in Fig. 2(b).

2.3 Diffraction degradation and thermal diffusion degradation

A thermal infrared imager is usually operated at the long-wavelength infrared band (8-12um), nearly 20 times the wavelength of visible light. The diffraction effect makes subtle textures in TIR images indistinguishable from noise, because the higher frequency signal components have been largely filtered out. Besides, the thermal diffusion effect, which can be described by a second order partial differential equation (PDE), dramatically degrades the spatial resolution of thermal infrared image as well [19]. This results in contrast reduction and loss of high-frequency components. These degradations are formulated as various Point Spread Functions (PSF), as a key priori parameter for restoration, making the TIR images smoother with MTF loss. A true latent TIR image therefore is assumed to be of piecewise C² continuity in this paper rather than local Lipschitz continuity, which is a suitable assumption for visible-band image de-noising using the Total Variation (TV) method [20,21 ]. Unlike the TV model, the latent image f in this study is only required to be in a Bounded Variation (BV) space. Under the assumption that thermal image is 2D piecewise C² smooth, a sparse-and-smooth de-convolution algorithm is developed for TIR image restoration.

3. Proposed method

Mathematically, a TIR image $g_{t_i}$yielded from staggered two time-division TDI linear arrays can be modeled as convolution of a latent true thermal infrared image $f(x,y)$ with the imager’s psf_sys. By considering non-uniform image displacements and various blurring operators, the degradation problem is formulated as Eq. (4):

3.1 Super-position with sub-pixel displacement vector estimation

In the case of multiple acquisitions, $g_{t_i}$ and $g_{t_{i+1}}$, should be perfectly spatially realigned in the superposition procedure. Here, the image displacement vector between images is denoted as M. The displacement component across the scanning direction M_x is assumed to be precisely known. According to the specific architecture of TDI scanning line, it is 1.0 pixel constant in this direction, which is utilized to increase horizontal sampling. The displacement component along scanning direction M_y is characterized by instantaneous geometric distortion. For one linear TDI detector, the image displacement is assumed to be about 1.0 pixel along the scanning direction. Thus between two staggered TDI lines, it is reasonable to assume a doubled or larger displacement along this direction. This is the reason why it can no longer be ignored in staggered TDI imaging mode for a high resolution imaging system onboard a high speed moving platform.

A 1D sub-pixel iterative realignment method is developed to estimate the sub-pixel displacement vector M_y for the non-uniform geometric distortion between adjacent scanning lines in the scanning direction. A shift in spatial domain will produce a phase difference in frequency domain. With Fourier inverse of the normalized cross-power spectrum of a pair of scanning lines, this method is able to measure the vector M_y pixel-by-pixel, directly from the phase shift according to Eq. (5). Then realign them pixel by pixel using the Fourier Shift Theorem as defined in Eq. (6). This algorithm is simple and efficient [15,22 ]. Let $g_i(n)$ and $g_{i+1}(n)$ be two adjacent scanning image-lines respectively. The phase-correlation method measures shift $△n$ with a moving window w by finding the peak in Eq. (5), which corresponds to the relative spatial shift.

One advantage of this method is that it is insensitive to the illumination variations resulting from the different responses of staggered TDI lines and produces a high accuracy solution, particularly for small displacements. Iteration of the algorithm can further improve the estimation accuracy of displacement vector. To assess the performance of the realignment method, a simulation was conducted with a overall shift between two image lines. Figure 4 is the comparison of realignment results by iterations. The red and black solid lines marked with ‘ + ’ are the target line and reference line respectively with 2.0 pixels displacement. After three iterations, the red line is aligned to the black line with very high accuracy up to 0.01 pixel as illustrated in Fig. 5 . As a result, the interlace noise due to the non-uniform displacement is removed completely, which ensures the reconstruction accuracy in the following de-convolution step. This is an experimentally validated state-of-the-art CI solution to this non-uniform image displacement.

 

Fig. 4 The displacement estimation and realignment of two adjacent scanning lines.

Download Full Size | PDF

 

Fig. 5 The realignment residual VS number of iterations of two adjacent scanning lines with 2.0 pixel displacement

Download Full Size | PDF

Another advantage of the method is that it does not employ any low-pass filter to remove interlace noise and thus, no loss of high-frequency components. The algorithm does not introduce unwanted artifacts and has low computational complexity.

3.2 Sparse-and-smooth deconvolution with C² continuity constraint

TIR images are generally of poor-contrast with fine details loss. Deconvolution is widely used to suppress degradation and improve image spatial resolution. However, as an ill-posed problem, it is non-trivial to retrieve a stable solution for practical applications.

There are various restoration algorithms discussed in published literature for thermal images [1,8,14 ]. However, infrared images are usually treated and processed similarly to visible light images. For instance, Wiener and Tikhonov [23,24 ] restoration algorithms are incapable of restoring high frequency components because of their linear nature. For noise-contaminated blurred images, these methods tend to produce ringing artifacts or an over-smoothing effect in the restored image. Another well-known model for image restoration is the Total Variation (TV) model, which falls into the category of deconvolution algorithms with some sparse priori knowledge. It is efficient to suppress blurring and ringing artifacts under the assumption that the desired image f is a piecewise constant or piecewise linear function. However, this assumption does not hold true for thermal images for which C² continuity is expected. As a result, TV restores sharp edges in thermal images at the cost of severe destruction of smoothness. Additionally, the TV model has a tendency to produce un-expected discontinuities as edges and ‘staircase’ artifacts [22] due to its basic assumption.

Enlightened by both the heat diffusion equation and the TV model [21,22,25 ], we assume that a TIR image (f ∈Ω) is of piecewise C² continuity rather than piecewise constant or piecewise linear, which are piecewise C⁰ or C¹ functions [26]. Incorporating a sparse constraint with a C² smooth prior, a Bounded Variation (BV) regularization term is proposed for TIR image restoration in Eq. (7). The L1 norm regularization term in Eq. (7), $\left|Df\right|$, has a tendency to suppress ringing and ‘staircase’ artifacts by imposing sparsity on the solution. The regularization term with a second-order differential operator, $\left|D^{2}f\right|$, is introduced in the proposed method as a smooth regularization term to ensure C² continuity. The key contribution of the BV regularization term is to curb the ‘staircase’ effect and other artifacts, so as to retrieve a sparse solution with good smoothness. As a result, the flawed edges and ‘staircase’ artifacts can be suppressed successfully to produce an edge-preserved smooth restoration image. Under the assumption of sparse prior and C² smooth prior, the solution of the proposed deconvolution method is equivalent to the unconstrained minimization of Eq. (7),

4. Experiment and validation

To validate the proposed method, both simulation images and airborne TIR images were used for experiments. With the 1D sub-pixel displacement estimation approach and sparse-and-smooth deconvolution algorithm, the proposed method successfully restored clear TIR images with higher spatial resolution.

4.1 Simulation experiment

Firstly, a 1951 USAF resolution target (AF target) was used for simulation and validation. The simulated degradation image, Fig. 6(a) , was generated by scanning from bottom to top with two parallel staggered scanning-lines, similar to the configuration illustrated in Fig. 1(b). The pixel-wise displacement vector of adjacent scanning lines is assumed to comply with a sinusoidal function varying from −2 pixels to + 2 pixels, shown as Fig. 6(e). An instantaneous geometric distortion due to non-uniform image displacement along the scanning direction was then introduced together with spatial blurring using a Gaussian function (r = 5, delta = 0.9) and Gaussian additive noise with noise level 0.005. As a result, the image suffered degradation dramatically from image displacement, blurring and noise as illustrated in Figs. 6(a) and 6(c) and in particular, the horizontal edge jitter is evident. Clearly, the bar elements groups n:2 in the region of interest (ROI) marked with a red box in Fig. 6(a) is almost indistinguishable as illustrated in a detailed version shown in Fig. 6(c).

 

Fig. 6 Simulation validation experiment with AF target image. (a) The degradation image with a sine pattern displacement along the scanning direction and blurring. (b) The restored result using the proposed CI method. (c) and (d) represent the zoom of ROI image blocks marked with a red box in (a) and (b), respectively. (e) The blue solid line is the true displacement vector in (a), and the red line is the displacement estimatedion result.

Download Full Size | PDF

Compared with Fig. 6(a), the restored image in Fig. 6(b) shows that the geometric distortion due to image displacement between adjacent scanning lines has been removed completely and the image texture has been sharply enhanced using the proposed CI algorithm. As an important parameter, the local displacement vector is very accurately estimated using the iterative pixel-wise phase correlation algorithm. Figure 6(e) shows that the estimation displacement vector coincides with the true value precisely.

4.2 Validation with airborne TIR images

Then raw TIR images were used for further validation, which were acquired by an airborne staggered TDI 480*6 scanning-liner imager in the architecture shown in Fig. 1(b). The linear TDI detector has 6 integration stages. The airplane flew at a velocity of 420kmph and altitude of about 6000m. The TIR imager of 8-12 μm IR spectral range was operated in a whisk broom scanning pattern. The focal length of the imager was 580mm, and the integration time about 22 μs. The TIR image was captured line by line as the scene passing through the field of view of the moving imager. Each raw image Digital Number (DN) of was recorded in 16 bits. Figure 7(a) shows a sub-block of 479*514 pixels raw image. Figure 7(b) is the restoration result, which is of high-contrast with well-defined edges. For fair comparison, the raw images and restored images are displayed in the same grayscale.

 

Fig. 7 An airborne TIR image and the restored result. (a) The staggered TDI thermal raw image. (b) The restoration result using the proposed CI method. (c) The zoom sub-image in (a) marked with red box #1. (d) The corresponding restored result in (b) marked with red box #2. (e) The difference map between (c) and (d). (f) The zoom sub-image in (a) marked with red box #3. (g) The corresponding restored result in (b) marked with red box #4. (h) The difference map between (f) and (g).

Download Full Size | PDF

The red boxes in Fig. 7(a) define two different ROIs of the raw image, image block #1 and image block #3, enlarged to show in Figs. 7(c) and 7(f). Clearly, the image was corrupted severely due to horizontal jitter stripes, image displacement and blurring resulting from optical diffraction, thermal diffusion, etc. The fine textures and edge discontinuities are smeared becoming hardly distinguishable. In contrast, Fig. 7(b) presents the restored result using the proposed sub-pixel realignment and sparse-and-smooth deconvolution method. The corresponding ROIs in the red boxes in Fig. 7(b) are denoted as image blocks #2 and #4 shown in Figs. 7(d) and 7(g). Compared with sub-images #1 and #3, not only are the edge jitters removed completely but also the fine textures and small objects are effectively restored. The results show that the C² continuity regularization term favors TIR image smoothness while the L1 norm enables us to find a sparse solution. Figures 7(e) and 7(h) are the residual maps between the raw image and restored image. Obviously the major difference between them is the vertical strips resulting from jittering. Figure 8 presents the estimated displacement vector M_y of the raw image in Fig. 7(a). It is approximately close to a sinusoidal periodical pattern due to platform vibration and the charges transfer velocity mismatching in TDI mode. This coincides perfectly with the assumption of geometric distortion mentioned above. Although the displacement varies just from about −1.0 pixels to about 1.0 pixel shown in Fig. 8, it can be measured precisely at sub-pixel accuracy using the proposed method; in this aspect, the algorithm outperforms the displacement estimation methods in [12,13,16 ] which can only achieve integer matching precision.

 

Fig. 8 The displacement estimation of the TIR raw image presented in Fig. 7(a)

Download Full Size | PDF

Figure 9(a) presents the image intensity profiles of lines #1 in the raw image (See Fig. 7(a)) and corresponding profile #1’ in the restored image (See Fig. 7(b)). In Fig. 9(a), the blue dashed line is the intensity profile of the raw image and the red line the profile of restored result. The black line is the corresponding intensity profile of the odd-rows image. Clearly, the red solid line presents higher peaks or lower troughs than the corresponding points in the raw image, which shows that the restoration image has more high-frequency components and less aliasing according the definition of MTF. From the aspect of diffusion effect, the backward diffusion and aliasing-suppress performance of the proposed CI method pointed with arrows 1,2,3,4 are evident.The restoration image therefore has fine textures with high-contrast. Figure 9(b) presents the image intensity profiles of lines #2 in the raw image (See Fig. 7(a)) and the corresponding profile #2’ in the restored image (See Fig. 7(b)). In Fig. 9(b), the blue solid line depicts the edge jitters of the raw image, which is the oscillation of interlace noise resulting from geometric distortion. After the 1D realignment and super-position step, the corresponding intensity profile of super-position image is shown as the black solid line. Thus, the geometric distortion is removed completely. Then, after the proposed sparse-and-smooth deconvolution method, the corresponding image intensity profile of the restoration image, line #2’ in Fig. 7(b), is shown as the red solid line in Fig. 9(b). Here, the higher peaks and lower troughs of the red line than those of black line indicate a successful restoration of higher frequency components, thanks to the backward-diffusion effect of this sparse-and-smooth deconvolution.

 

Fig. 9 The image intensity profiles comparison of raw image with that of realignment image and restoration image. (a) Image intensity profiles of dotted red line #1 and #1' in Figs. 7(a) and 7(b) and intensity profile of odd-rows image. (b) Image intensity profiles of dotted red line #2 and #2' in Figs. 7(a) and 7(b) and intensity profile of realignment image

Download Full Size | PDF

In frequency domain, a similar observation is presented in Fig. 10 . The black solid line depicts the normalized spatial-frequency of the raw TIR image, which is constantly lower than the red solid line of the restoration image. This shows that higher frequency components have been restored successfully with the proposed CI method; and there are more higher-frequency components in restoration image. The only exception is at about 0.5 position of normalized frequency due to the interlace noise; the higher frequency components circled with a black dotted line in Fig. 10, where the geometric distortion has been suppressed without sacrificing medium-high frequency components. Then, a higher spatial resolution and high-contrast TIR image was restored successfully with the proposed CI method.

 

Fig. 10 The normalized spatial frequency curves of raw image and restored image. The higher frequency components in the dotted circle resulting from non-uniform image distortion

Download Full Size | PDF

5. Conclusion

For non-uniform degradation of aerial TIR imagers, we have proposed a computational imaging method to deal with not only the geometric instantaneous distortion between time-division multiplexing channels, but also the blurring degradation due to optical diffraction, motion blurring and thermal diffusion. Using the 1D phase correlation iterative realignment algorithm, the non-uniform geometric distortion can be removed effectively. With a C² continuity prior and a sparse prior, the method is able to restore higher resolution TIR images without staircase and other artifacts introducted. Validation experiments proved that the proposed computational imaging method is a state-of-the-art approach to retrieve higher spatial resolution TIR images with enhanced contrast.