Fayaz Ali Dharejo
Skip Abstract Section
Abstract
Single Image Super-resolution (SISR) produces high-resolution images with fine spatial resolutions from a remotely sensed image with low spatial resolution. Recently, deep learning and generative adversarial networks (GANs) have made breakthroughs for the challenging task of single image super-resolution (SISR). However, the generated image still suffers from undesirable artifacts such as the absence of texture-feature representation and high-frequency information. We propose a frequency domain-based spatio-temporal remote sensing single image super-resolution technique to reconstruct the HR image combined with generative adversarial networks (GANs) on various frequency bands (TWIST-GAN). We have introduced a new method incorporating Wavelet Transform (WT) characteristics and transferred generative adversarial network. The LR image has been split into various frequency bands by using the WT, whereas the transfer generative adversarial network predicts high-frequency components via a proposed architecture. Finally, the inverse transfer of wavelets produces a reconstructed image with super-resolution. The model is first trained on an external DIV2 K dataset and validated with the UC Merced Landsat remote sensing dataset and Set14 with each image size of 256 × 256. Following that, transferred GANs are used to process spatio-temporal remote sensing images in order to minimize computation cost differences and improve texture information. The findings are compared qualitatively and qualitatively with the current state-of-art approaches. In addition, we saved about 43% of the GPU memory during training and accelerated the execution of our simplified version by eliminating batch normalization layers.
- [1] . 2020. A novel and efficient 3D multiple images encryption scheme based on chaotic systems and swapping operations. IEEE Access 8 (2020), 123536–123555.Google Scholar
Cross Ref
- [2] . 2020. Deep learning-based drivers emotion classification system in time series data for remote applications. Remote Sensing 12, 3 (2020), 587.Google ScholarCross Ref
- [3] . 2021. A plexus-convolutional neural network framework for fast remote sensing image super-resolution in wavelet domain. IET Image Processing 0, 0 (2021), 0–0.Google Scholar
- [4] . 2020. Sparse representation based computed tomography images reconstruction by coupled dictionary learning algorithm. IET Image Processing 14, 11 (2020), 2365–2375.Google ScholarCross Ref
- [5] . 1991. Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 53, 3 (1991), 231–239. Google ScholarDigital Library
- [6] . 2020. Wavelet-based enhanced medical image super resolution. IEEE Access 8 (2020), 37035–37044.Google ScholarCross Ref
- [7] . 2013. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In International Conference on Medical Image Computing and Computer-assisted Intervention. Springer, Nagoya, Japan, 9–16.Google ScholarCross Ref
- [8] . 2006. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. International Journal of Remote Sensing 27, 3 (2006), 473–491.Google ScholarCross Ref
- [9] . 2021. A remote-sensing image enhancement algorithm based on patch-wise dark channel prior and histogram equalisation with colour correction. IET Image Processing 15, 1 (2021), 47–56.Google ScholarCross Ref
- [10] . 2011. Very low resolution face recognition problem. IEEE Transactions on Image Processing 21, 1 (2011), 327–340. Google ScholarDigital Library
- [11] . 2020. Retinal image analysis for diabetes-based eye disease detection using deep learning. Applied Sciences 10, 18 (2020), 6185.Google ScholarCross Ref
- [12] . 2014. Single-image super-resolution: A benchmark. In European Conference on Computer Vision. Springer, Zurich, Switzerland, 372–386.Google ScholarCross Ref
- [13] . 2011. Image and video upscaling from local self-examples. ACM Transactions on Graphics (TOG) 30, 2 (2011), 1–11. Google ScholarDigital Library
- [14] . 2013. Fast image super-resolution based on in-place example regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Portland, OR, USA, 1059–1066. Google ScholarDigital Library
- [15] . 2010. Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 6 (2010), 1127–1133. Google ScholarDigital Library
- [16] . 2004. Super-resolution through neighbor embedding. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, Vol. 1. IEEE, Washington, DC, USA, I–I.Google ScholarCross Ref
- [17] . 2013. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Sydney, NSW, Australia, 1920–1927. Google ScholarDigital Library
- [18] . 2013. Image super-resolution via double sparsity regularized manifold learning. IEEE Transactions on Circuits and Systems for Video Technology 23, 12 (2013), 2022–2033. Google ScholarDigital Library
- [19] . 2013. Alternatively constrained dictionary learning for image superresolution. IEEE Transactions on Cybernetics 44, 3 (2013), 366–377.Google Scholar
- [20] . 2017. Remote sensing scene classification by unsupervised representation learning. IEEE Transactions on Geoscience and Remote Sensing 55, 9 (2017), 5148–5157.Google ScholarCross Ref
- [21] . 2017. A hybrid sparsity and distance-based discrimination detector for hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing 56, 3 (2017), 1704–1717.Google ScholarCross Ref
- [22] . 2017. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing 56, 4 (2017), 2183–2195.Google ScholarCross Ref
- [23] . 2018. A coarse-to-fine semi-supervised change detection for multispectral images. IEEE Transactions on Geoscience and Remote Sensing 56, 6 (2018), 3587–3599.Google ScholarCross Ref
- [24] . 2009. Super-resolution from a single image. In 2009 IEEE 12th International Conference on Computer Vision. IEEE, Kyoto, Japan, 349–356.Google ScholarCross Ref
- [25] . 2019. Achieving super-resolution remote sensing images via the wavelet transform combined with the recursive res-net. IEEE Transactions on Geoscience and Remote Sensing 57, 6 (2019), 3512–3527.Google ScholarCross Ref
- [26] . 2019. A color enhancement scene estimation approach for single image haze removal. IEEE Geoscience and Remote Sensing Letters 17, 9 (2019), 1613–1617.Google ScholarCross Ref
- [27] . 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian Conference on Computer Vision. Springer, Singapore, 111–126.Google Scholar
- [28] . 2014. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision. Springer, Zurich, Switzerland, 184–199.Google ScholarCross Ref
- [29] . 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Las Vegas, NV, USA, 1874–1883.Google ScholarCross Ref
- [30] . 2016. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision. Springer, Amsterdam, Netherlands, 391–407.Google ScholarCross Ref
- [31] . 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Las Vegas, NV, USA, 1646–1654.Google ScholarCross Ref
- [32] . 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Las Vegas, NV, USA, 770–778.Google ScholarCross Ref
- [33] . 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90. Google ScholarDigital Library
- [34] . 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEE, Las Vegas, NV, USA, 1637–1645.Google ScholarCross Ref
- [35] . 2017. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEE, Honolulu, HI, USA, 3147–3155.Google ScholarCross Ref
- [36] . 2018. A new deep generative network for unsupervised remote sensing single-image super-resolution. IEEE Transactions on Geoscience and Remote Sensing 56, 11 (2018), 6792–6810.Google ScholarCross Ref
- [37] . 2017. Super-resolution for remote sensing images via local–global combined network. IEEE Geoscience and Remote Sensing Letters 14, 8 (2017), 1243–1247.Google ScholarCross Ref
- [38] . 2018. A deep network architecture for super-resolution-aided hyperspectral image classification with classwise loss. IEEE Transactions on Geoscience and Remote Sensing 56, 8 (2018), 4650–4663.Google ScholarCross Ref
- [39] . 1999. Using wavelets transform in the analysis of electrochemical noise data. Electrochimica Acta 44, 26 (1999), 4805–4816.Google ScholarCross Ref
- [40] . 1992. Singularity detection and processing with wavelets. IEEE Transactions on Information Theory 38, 2 (1992), 617–643. Google ScholarDigital Library
- [41] . 1995. Wavelet transform signal processing for dispersion analysis of ultrasonic signals. In 1995 IEEE Ultrasonics Symposium. Proceedings. An International Symposium, Vol. 1. IEEE, Seattle, WA, USA, 751–755.Google ScholarCross Ref
- [42] . 1996. Wavelets for a vision. Proc. IEEE 84, 4 (1996), 604–614.Google ScholarCross Ref
- [43] . 2011. Ant colony optimization for wavelet-based image interpolation using a three-component exponential mixture model. Expert Systems with Applications 38, 10 (2011), 12514–12520. Google ScholarDigital Library
- [44] . 2010. Adaptive wavelet-Galerkin methods for limited angle tomography. Image and Vision Computing 28, 4 (2010), 696–703. Google ScholarDigital Library
- [45] . 2003. Wavelet algorithms for high-resolution image reconstruction. SIAM Journal on Scientific Computing 24, 4 (2003), 1408–1432. Google ScholarDigital Library
- [46] . 2001. Image interpolation using wavelet based hidden Markov trees. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221), Vol. 3. IEEE, Salt Lake City, UT, USA, 1957–1960. Google ScholarDigital Library
- [47] . 2007. Image interpolation based on statistical relationship between wavelet subbands. In 2007 IEEE International Conference on Multimedia and Expo. IEEE, Beijing, China, 1723–1726.Google ScholarCross Ref
- [48] . 2010. Remote-sensing image compression using two-dimensional oriented wavelet transform. IEEE Transactions on Geoscience and Remote Sensing 49, 1 (2010), 236–250.Google ScholarCross Ref
- [49] . 2003. Superresolution remote sensing image processing algorithm based on wavelet transform and interpolation. In Image Processing and Pattern Recognition in Remote Sensing, Vol. 4898. International Society for Optics and Photonics, Hangzhou, China, 259–263.Google Scholar
- [50] . 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Honolulu, HI, USA, 4681–4690.Google ScholarCross Ref
- [51] . 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE, Honolulu, HI, USA, 114–125.Google ScholarCross Ref
- [52] . 2006. Medical image fusion by multi-resolution analysis of wavelets transform. In Wavelet Analysis and Applications. Springer, 389–396.Google Scholar
- [53] . 2005. Advances in conversational case-based reasoning. The Knowledge Engineering Review 20, 3 (2005), 247. Google ScholarDigital Library
- [54] . 2017. Metric-based generative adversarial network. In Proceedings of the 25th ACM International Conference on Multimedia. ACM, Mountain View, California, USA, 672–680. Google ScholarDigital Library
- [55] . 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 0, o (2014), 0–0.Google Scholar
- [56] . 2016. Improved techniques for training GANs. arXiv preprint arXiv:1606.03498 0, 0 (2016), 0–0. Google ScholarDigital Library
- [57] . 2016. Task specific adversarial cost function. arXiv preprint arXiv:1609.08661 abs/1609.08661 (2016), 0–0.Google Scholar
- [58] . 2019. Sparse generative adversarial network. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. IEEE, Seoul, Korea (South), 0–0.Google ScholarCross Ref
- [59] . 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE, Honolulu, HI, USA, 136–144.Google ScholarCross Ref
- [60] . 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Honolulu, HI, USA, 3883–3891.Google ScholarCross Ref
- [61] . 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. Springer, Munich, Germany, 0–0.Google Scholar
- [62] . 2017. Improved training of Wasserstein GANs. arXiv preprint arXiv:1704.00028 0, 0 (2017), 0–0. Google ScholarDigital Library
- [63] . 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition. IEEE, Istanbul, Turkey, 2366–2369.Google ScholarDigital Library
Index Terms
- TWIST-GAN: Towards Wavelet Transform and Transferred GAN for Spatio-Temporal Single Image Super Resolution
Recommendations
Convolutional neural networks for wavelet domain super resolution
Proposed a super resolution method with higher reconstruction accuracy than before.Cast super resolution as a problem of estimating sparse wavelet detail coefficients.Estimated sparse wavelet coefficients using a convolutional neural network (CNN)...
Super-Resolution Image Reconstruction Using Wavelet Based Patch and Discrete Wavelet Transform
This paper proposes a novel method that combines the discrete wavelet transform (DWT) and example-based technique to reconstruct a high-resolution from a low-resolution image. Although previous interpolation- and example-based methods consider the ...
Microscopic image super resolution using deep convolutional neural networks
AbstractRecently, deep convolutional neural networks (CNNs) have achieved excellent results in single image super resolution (SISR). Owing to the strength of deep CNNs, it gives promising results compared to state-of-the-art learning based models on ...
Comments