Skip to main content
SpringerLink
Log in

NL-CS Net: Deep Learning with Non-local Prior for Image Compressive Sensing

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

Deep learning has been applied to compressive sensing (CS) of images successfully in recent years. However, existing network-based methods are often trained as the black box, in which the lack of prior knowledge is often the bottleneck for further performance improvement. To overcome this drawback, this paper proposes a novel CS method using non-local prior which combines the interpretability of the traditional optimization methods with the speed of network-based methods, called NL-CS Net. We unroll each phase from iteration of the augmented Lagrangian method solving non-local and sparse regularized optimization problem by a network. NL-CS Net is composed of the up-sampling module and the recovery module. In the up-sampling module, we use learnable up-sampling matrix instead of a predefined one. In the recovery module, patch-wise non-local network is employed to capture long-range feature correspondences. Important parameters involved (e.g. sampling matrix, nonlinear transforms, shrinkage thresholds, step size, etc.) are learned end-to-end, rather than hand-crafted. Furthermore, to facilitate practical implementation, orthogonal and binary constraints on the sampling matrix are simultaneously adopted. Extensive experiments on natural images and magnetic resonance imaging demonstrate that the proposed method outperforms the state-of-the-art methods while maintaining great interpretability and speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

We hereby declare that all data and materials used in this study are publicly available with no restrictions. The data used in this research has been made publicly available and can be accessed directly via https://github.com/bianshuai001/NL-CS-Net.

Code Availability

We declare that the code used in this study is open-source and publicly available for unrestricted use. The code used in this research can be accessed via the link https://github.com/bianshuai001/NL-CS-Net. Anyone can retrieve, download, and use the code for non-commercial purposes, subject to appropriate attribution of the source.

References

  1. A. Adler, D. Boublil, M. Elad, M. Zibulevsky, A deep learning approach to block-based compressed sensing of images. Available: arXiv:1606.01519 (2016)

  2. A. Adler, D. Boublil, M. Zibulevsky, Block-based compressed sensing of images via deep learning, in IEEE 19th International Workshop on Multimedia Signal Processing, pp. 1–6 (2017)

  3. A.M. Ali, B. Benjdira, A. Koubaa, W. El-Shafai, Z. Khan, W. Boulila, Vision transformers in image restoration: a survey. Sensors 23(5), 2385 (2023)

    Article  Google Scholar 

  4. A. Buades, B. Coll, J. Morel, A non-local algorithm for image denoising, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 60–65 (2005)

  5. E.J. Candes, T. Tao, Near-optimal signal recovery from random projections: universal encoding strategies. IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006)

    Article  MathSciNet  Google Scholar 

  6. K.Q. Dinh, H.J. Shim, B. Jeon, Measurement coding for compressive imaging using a structural measuremnet matrix, in IEEE International Conference on Image Processing, pp. 10–13 (2014)

  7. W. Dong, G. Shi, X. Li, Y. Ma, F. Huang, Compressive sensing via nonlocal low-rank regularization. IEEE Trans. Image Process. 23(8), 3618–32 (2014)

    Article  MathSciNet  Google Scholar 

  8. J. Du, X. Xie, C. Wang, G. Shi, X. Xu, Y. Wang, Fully convolutional measurement network for compressive sensing image reconstruction. Neurocomputing 328, 105–112 (2019)

    Article  Google Scholar 

  9. M.F. Duarte, M.A. Davenport, D. Takhar, J.N. Laska, Single-pixel imaging via compressive sampling. IEEE Sig. Process. Mag. 25(2), 83–91 (2008)

    Article  Google Scholar 

  10. J.M. Duarte-Carvajalino, G. Sapiro, Learning to sense sparse signals: simultaneous sensing matrix and sparsifying dictionary optimization. IEEE Trans. Image Process. 18(7), 1395–1408 (2009)

    Article  MathSciNet  Google Scholar 

  11. M. Elad, M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)

    Article  MathSciNet  Google Scholar 

  12. J.H. Ender, On compressive sensing applied to radar. Signal Process. 90(5), 1402–1414 (2010)

    Article  Google Scholar 

  13. A. Esmaeilzehi, M.O. Ahmad, M.N.S. Swamy, MISNet: multi-resolution level feature interpolating ultralight-weight residual image super resolution network, in IEEE International Symposium on Circuits and Systems, pp. 1–5 (2021)

  14. A. Esmaeilzehi, M.N.S. Swamy, M.O. Ahmad, HighBoostNet: a deep light-weight image super-resolution network using high-boost residual blocks. Vis. Comput. 40, 1111–1119 (2024)

    Article  Google Scholar 

  15. A. Esmaeilzehi, H. Zaredar, D. Hatzinakos, M.O. Ahmad, DPAN: a deep light-weight attention-based image super resolution network using multi-dimensional filter design technique. IEEE Signal Process. Lett. 30, 1637–1641 (2023)

    Article  Google Scholar 

  16. X. Gao, J. Zhang, W. Che, X. Fan, D. Zhao, Block-based compressive sensing coding of natural images by local structural measurement matrix, in Data Compression Conference, pp. 133–142 (2015)

  17. J. Haupt, W.U. Bajwa, G. Raz, R. Nowak, Toeplitz compressed sensing matrices with applications to sparse channel estimation. IEEE Trans. Inf. Theory 56(11), 5862–5875 (2010)

    Article  MathSciNet  Google Scholar 

  18. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in IEEE Conference on Computer Vision and Pattern Recognition, pp. 5862–5875 (2016)

  19. C. Hyun, H.P. Kim, S.M. Lee, S. Lee, J.K. Seo, Deep learning for undersampled MRI reconstruction. Phys. Med. Biol. 63(13), 135007 (2018)

    Article  Google Scholar 

  20. X. Jie, J. Ma, D. Zhang, Y. Zhang, S. Lin, Improved total variation minimization method for compressive sensing by intra-prediction. Signal Process. 92(11), 2614–2623 (2012)

    Article  Google Scholar 

  21. Y. Kim, M.S. Nadar, A. Bilgin, Compressed sensing using a Gaussian scale mixtures model in wavelet domain, in IEEE International Conference on Image Processing, pp. 3365–3368 (2010)

  22. K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, A. Ashok, ReconNet: non-iterative reconstruction of images from compressively sensed measurements, in IEEE Conference on Computer Vision and Pattern Recognition, pp. 449–458 (2016)

  23. W. Li, F. Liu, L. Jiao, F. Hu, Multi-scale residual reconstruction neural network with non-local constraint. IEEE Access 7, 70910–70918 (2019)

    Article  Google Scholar 

  24. S. Li, L.D. Xu, X. Wang, Compressed sensing signal and data acquisition in wireless sensor networks and internet of things. IEEE Trans. Ind. Inform. 9(4), 2177–2186 (2013)

    Article  Google Scholar 

  25. C. Li, W. Yin, H. Jiang, An efficient augmented Lagrangian method with applications to total variation minimization. Comput. Optim. Appl. 56(3), 507–530 (2013)

    Article  MathSciNet  Google Scholar 

  26. D. Liu, B. Wen, Y. Fu, C.C. Loy, T.S. Huang, Non-local recurrent network for image restoration, in Conference and Workshop on Neural Information Processing Systems (2018)

  27. C. Lu, H. Li, Z. Lin, Optimized projections for compressed sensing via direct mutual coherence minimization. Signal Process. 151, 45–55 (2018)

    Article  Google Scholar 

  28. M. Lustig, D.L. Donoho, J.M. Santos, J.M. Pauly, Compressed sensing MRI. IEEE Signal Process. Mag. 25(2), 72–82 (2008)

    Article  Google Scholar 

  29. D. Martin, C. Fowlkes, D. Tal, J. Malik, A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, in Proceedings Eighth IEEE International Conference on Computer Vision, pp. 416–423 (2001)

  30. C.A. Metzler, A. Maleki, R.G. Baraniuk, From denoising to compressed sensing. IEEE Trans. Inf. Theory 62(9), 5117–5144 (2016)

    Article  MathSciNet  Google Scholar 

  31. C. Mou, J. Zhang, X. Fan, H. Liu, R. Wang, COLA-Net: collaborative attention network for image restoration. IEEE Trans. Multimed. 24, 1366–1377 (2022)

    Article  Google Scholar 

  32. A. Mousavi, A.B. Patel, R.G. Baraniuk, A deep learning approach to structured signal recovery, in 53rd Annual Allerton Conference on Communication, Control, and Computing, pp. 1336–1343 (2015)

  33. X. Qu, Undersampled MRI reconstruction with patch-based directional wavelets. Magn. Reson. Imaging 30(7), 964–977 (2012)

    Article  Google Scholar 

  34. W. Shi, J. Caballero, F. Huszar, J. Totz, Z. Wang, Real-Time single image and video super-resolution using an efficient sub-pixel convolutional neural network, in IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  35. W. Shi, F. Jiang, S. Liu, D. Zhao, Scalable convolutional neural network for image compressed sensing, in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12282–12291 (2019)

  36. W. Shi, F. Jiang, S. Zhang, D. Zhao, Deep networks for compressed image sensing, in IEEE International Conference on Multimedia and Expo, pp. 877–882 (2017)

  37. J. Su, B. Xu, H. Yin, A survey of deep learning approaches to image restoration. Neurocomputing 487, 46–65 (2022)

    Article  Google Scholar 

  38. Y. Sun, J. Chen, Q. Liu, G. Liu, Learning image compressed sensing with sub-pixel convolutional generative adversarial network. Pattern Recognit. 98(12), 107051 (2019)

    Google Scholar 

  39. Z. Wang, J. Chen, S. Hoi, Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2020)

    Article  Google Scholar 

  40. X. Wang, R. Girshick, A. Gupta, K. He, Non-local neural networks, in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

  41. J. Xiang, Y. Dong, Y. Yang, FISTA-Net: learning a fast iterative shrinkage thresholding network for inverse problems in imaging. IEEE Trans. Med. Imag. 40(5), 1329–1339 (2021)

    Article  Google Scholar 

  42. Y. Yang, J. Sun, H. Li, Z. Xu, Deep ADMM-Net for compressive sensing MRI, in Advances in Neural Information Processing Systems (2016)

  43. J. Yang, Y. Zhang, W. Yin, A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data. IEEE J. Sel. Topics Sig. Process. 4(2), 288–297 (2010)

    Article  Google Scholar 

  44. H. Yao, F. Dai, S. Zhang, Y. Zhang, Q. Tian, C. Xu, DR2-Net: deep residual reconstruction network for image compressive sensing. Neurocomputing 359(24), 483–493 (2017)

    Google Scholar 

  45. D. You, J. Xie, J. Zhang, ISTA-Net++: flexible deep unfolding network for compressive sensing, in IEEE International Conference on Multimedia and Expo, pp. 1–6 (2021)

  46. J. Zhang, B. Ghanem, ISTA-Net: interpretable optimization-inspired deep network for image compressive sensing, in IEEE / CVF Computer Vision and Pattern Recognition Conference, pp. 1828–1837 (2018)

  47. Z. Zhang, Y. Liu, J. Liu, F. Wen, C. Zhu, AMP-Net: denoising-based deep unfolding for compressive image sensing. IEEE Trans. Image Process. 30(7), 1487–1500 (2021)

    Article  MathSciNet  Google Scholar 

  48. J. Zhang, S. Liu, R. Xiong, S. Ma, D. Zhao, Improved total variation based image compressive sensing recovery by nonlocal regularization, in IEEE International Symposium on Circuits and Systems, pp. 2836–2839 (2013)

  49. J. Zhang, C. Zhao, W. Gao, Optimization-inspired compact deep compressive sensing. IEEE J. Sel. Topics Sig. Process. 14(4), 765–774 (2020)

    Article  Google Scholar 

  50. J. Zhang, D. Zhao, C. Zhao, R. Xiong, S. Ma, W. Gao, Image compressive sensing recovery via collaborative sparsity. IEEE Emerg. Sel. Topics Circuits Syst. 2(3), 380–391 (2012)

    Article  Google Scholar 

  51. K. Zhang, W. Zuo, S. Gu, L. Zhang, Learning deep CNN denoiser prior for image restoration, in IEEE Conference on Computer Vision and Pattern Recognition, pp. 2808–2817 (2017)

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of Liaoning Province (2022-MS-114).

Funding

Natural Science Foundation of Liaoning Province (2022-MS-114).

Author information

Authors and Affiliations

  1. College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, 110169, China

    Shuai Bian, Shouliang Qi, Chen Li & Yueyang Teng

  2. Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ, 07030, USA

    Yudong Yao

  3. Key Laboratory of Intelligent Computing in Medicine, Ministry of Education, Shenyang, 110169, China

    Yueyang Teng

Authors
  1. Shuai Bian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shouliang Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yudong Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yueyang Teng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yueyang Teng.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bian, S., Qi, S., Li, C. et al. NL-CS Net: Deep Learning with Non-local Prior for Image Compressive Sensing. Circuits Syst Signal Process (2024). https://doi.org/10.1007/s00034-024-02699-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00034-024-02699-x

Keywords

Search

Navigation