Skip to main content
SpringerLink
Log in

Adaptive Deep PnP Algorithm for Video Snapshot Compressive Imaging

  • S.I. : Physics-Based Vision meets Deep Learning
  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Video Snapshot compressive imaging (SCI) is a promising technique to capture high-speed videos, which transforms the imaging speed from the detector to mask modulating and only needs a single measurement to capture multiple frames. The algorithm to reconstruct high-speed frames from the measurement plays a vital role in SCI. In this paper, we consider the promising reconstruction algorithm framework, namely plug-and-play (PnP), which is flexible to the encoding process comparing with other deep learning networks. One drawback of existing PnP algorithms is that they use a pre-trained denoising network as a plugged prior while the training data of the network might be different from the task in real applications. Towards this end, in this work, we propose the online PnP algorithm which can adaptively update the network’s parameters within the PnP iteration; this makes the denoising network more applicable to the desired data in the SCI reconstruction. Furthermore, for color video imaging, RGB frames need to be recovered from Bayer pattern or named demosaicing in the camera pipeline. To address this challenge, we design a two-stage reconstruction framework to optimize these two coupled ill-posed problems and introduce a deep demosaicing prior specifically for video demosaicing in SCI. Extensive results on both simulation and real datasets verify the superiority of our adaptive deep PnP algorithm. Code is available at https://github.com/xyvirtualgroup/AdaptivePnP_SCI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • A Sharif, S., Naqvi, R.A., & Biswas, M. (2021) Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 233–242

  • Bioucas-Dias, J. M., & Figueiredo, M. A. T. (2007). A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Transactions on Image Processing, 16(12), 2992–3004. https://doi.org/10.1109/TIP.2007.909319

    Article  MathSciNet  Google Scholar 

  • Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1), 1–122.

    Article  MATH  Google Scholar 

  • Brady, D. J., Fang, L., & Ma, Z. (2020). Deep learning for camera data acquisition, control, and image estimation. Advances in Optics and Photonics, 12(4), 787–846.

    Article  Google Scholar 

  • Chan, S. H., Wang, X., & Elgendy, O. A. (2017). Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3, 84–98.

    Article  MathSciNet  Google Scholar 

  • Chen, Z., Zheng, S., Tong, Z., & Yuan, X. (2022). Physics-driven deep learning enables temporal compressive coherent diffraction imaging. Optica, 9(6), 677–680.

    Article  Google Scholar 

  • Cheng, Z., Chen, B., Liu, G., Zhang, H., Lu, R., Wang, Z., & Yuan, X. (2021) Memory-efficient network for large-scale video compressive sensing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Cheng, Z., Lu, R., Wang, Z., Zhang, H., Chen, B., Meng, Z., & Yuan, X. (2020) Birnat: Bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. In: ECCV (24), vol. 12369, pp. 258–275

  • Chi, Z., Shu, X., & Wu, X. (2019) Joint demosaicking and blind deblurring using deep convolutional neural network. In: 2019 IEEE International conference on image processing (ICIP), pp. 2169–2173. IEEE

  • Gharbi, M., Chaurasia, G., Paris, S., & Durand, F. (2016). Deep joint demosaicking and denoising. ACM Transactions on Graphics, 35(6), 192.

    Article  Google Scholar 

  • Guo, S., Liang, Z., & Zhang, L. (2021) Joint denoising and demosaicking with green channel prior for real-world burst images. arXiv preprint arXiv:2101.09870

  • Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., & Nayar, S. K. (2011) Video from a single coded exposure photograph using a learned over-complete dictionary. In: 2011 International conference on computer vision (pp. 287–294). IEEE

  • Jalali, S., & Yuan, X. (2019). Snapshot compressed sensing: Performance bounds and algorithms. IEEE Transactions on Information Theory, 65(12), 8005–8024.

    Article  MathSciNet  MATH  Google Scholar 

  • Kamilov, U. S., Bouman, C. A., Buzzard, G. T., & Wohlberg, B. (2023). Plug-and-play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 40(1), 85–97.

    Article  Google Scholar 

  • Kokkinos, F., & Lefkimmiatis, S. (2018) Deep image demosaicking using a cascade of convolutional residual denoising networks. In: Proceedings of the European conference on computer vision (ECCV), pp. 303–319

  • Li, Y., Qi, M., Gulve, R., Wei, M., & Heidrich, W. (2020) End-to-end video compressive sensing using anderson-accelerated unrolled networks. In: 2020 IEEE international conference on computational photography (ICCP)

  • Liao, X., Li, H., & Carin, L. (2014). Generalized alternating projection for weighted-\(\ell _{2,1}\) minimization with applications to model-based compressive sensing. SIAM Journal on Imaging Sciences, 7(2), 797–823.

    Article  MathSciNet  MATH  Google Scholar 

  • Liu, Y., Yuan, X., Suo, J., Brady, D., & Dai, Q. (2019). Rank minimization for snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12), 2990–3006.

    Article  Google Scholar 

  • Llull, P., Liao, X., Yuan, X., Yang, J., Kittle, D., Carin, L., Sapiro, G., & Brady, D. J. (2013). Coded aperture compressive temporal imaging. Optics Express, 21(9), 10526–10545. https://doi.org/10.1364/OE.21.010526

    Article  Google Scholar 

  • Lu, S., Yuan, X., & Shi, W. (2020) An integrated framework for compressive imaging processing on CAVs. In: ACM/IEEE symposium on edge computing (SEC)

  • Ma, J., Liu, X., Shou, Z., & Yuan, X (2019) Deep tensor admm-net for snapshot compressive imaging. In: IEEE/CVF conference on computer vision (ICCV)

  • Malvar, R., He, L. W. & Cutler, R. (2004) High-quality linear interpolation for demosaicing of bayer-patterned color images. In: International conference of acoustic, speech and signal processing

  • Meng, Z., Jalali, S., & Yuan, X. (2020) Gap-net for snapshot compressive imaging. arXiv preprint arXiv:2012.08364

  • Meng, Z., Ma, J., & Yuan, X. (2020) End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. In: European conference on computer vision

  • Meng, Z., Yu, Z., Xu, K., & Yuan, X. (2021) Self-supervised neural networks for spectral snapshot compressive imaging. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 2622–2631

  • Menon, D., Andriani, S., & Calvagno, G. (2007). Demosaicing with directional filtering and a posteriori decision. IEEE Transactions on Image Processing, 16(1), 132–141. https://doi.org/10.1109/TIP.2006.884928

    Article  MathSciNet  Google Scholar 

  • Mittal, A., Soundararajan, R., & Bovik, A. C. (2012). Making a “ completely blind ’ ’ image quality analyzer. IEEE Signal Processing letters, 20(3), 209–212.

    Article  Google Scholar 

  • Mou, C., Zhang, J., & Wu, Z. (2021) Dynamic attentive graph learning for image restoration. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 4328–4337

  • Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., & Sorkine-Hornung, A. (2016) A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 724–732

  • Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., & Van Gool, L. (2017) The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675

  • Qiao, M., Meng, Z., Ma, J., & Yuan, X. (2020). Deep learning for video compressive sensing. APL Photonics, 5(3), 030801.

    Article  Google Scholar 

  • Reddy, D., Veeraraghavan, A., & Chellappa, R. (2011) P2c2: Programmable pixel compressive camera for high speed imaging. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 329–336

  • Ryu, E.K., Liu, J., Wang, S., Chen, X., Wang, Z., & Yin, W. (2019) Plug-and-play methods provably converge with properly trained denoisers. arXiv: 1905.05406

  • Song, J., Chen, B., & Zhang, J. (2021) Memory-augmented deep unfolding network for compressive sensing. In: Proceedings of the 29th ACM international conference on multimedia, pp. 4249–4258

  • Sreehari, S., Venkatakrishnan, S., Wohlberg, B., Drummy, L. F., Simmons, J. P., & Bouman, C. A. (2016). Plug-and-play priors for bright field electron tomography and sparse interpolation. IEEE Transactions on Computational Imaging, 2(4), 408–423.

    MathSciNet  Google Scholar 

  • Sun, Q., Liu, Y., Chen, Z., Chua, T.S., & Schiele, B. (2020) Meta-transfer learning through hard tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence

  • Tassano, M., Delon, J., & Veit, T. (2020) Fastdvdnet: Towards real-time deep video denoising without flow estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 1354–1363

  • Tirer, T., & Giryes, R. (2019). Super-resolution via image-adapted denoising cnns: Incorporating external and internal learning. IEEE Signal Processing Letters, 26(7), 1080–1084.

    Article  Google Scholar 

  • Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018) Deep image prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446–9454

  • Venkatakrishnan, S. V., Bouman, C. A., & Wohlberg, B. (2013) Plug-and-play priors for model based reconstruction. In: 2013 IEEE global conference on signal and information processing, pp. 945–948

  • Wang, L., Cao, M., Zhong, Y., & Yuan, X. (2022) Spatial-temporal transformer for video snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence

  • Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P., et al. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wang, Z., Zhang, H., Cheng, Z., Chen, B., & Yuan, X. (2021) Metasci: Scalable and adaptive reconstruction for video compressive sensing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  • Wei, K., Avilés-Rivero, A. I., Liang, J., Fu, Y., Huang, H., & Schönlieb, C. B. (2022). Tfpnp: Tuning-free plug-and-play proximal algorithms with applications to inverse imaging problems. Journal of Machine Learning Research, 23(16), 1–48.

    MathSciNet  MATH  Google Scholar 

  • Wu, Z., Zhang, J., & Mou, C. (2021) Dense deep unfolding network with 3d-cnn prior for snapshot compressive imaging. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp. 4892–4901

  • Xing, W., & Egiazarian, K. (2021) End-to-end learning for joint image demosaicing, denoising and super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3507–3516

  • Yang, C., Zhang, S., & Yuan, X. (2022) Ensemble learning priors driven deep unfolding for scalable video snapshot compressive imaging. In: Proceedings of 17th European conference computer vision–ECCV 2022, Tel Aviv, Israel, October 23–27, 2022, Part XXIII, pp. 600–618. Springer

  • Yang, J., Liao, X., Yuan, X., Llull, P., Brady, D. J., Sapiro, G., & Carin, L. (2015). Compressive sensing by learning a Gaussian mixture model from measurements. IEEE Transaction on Image Processing, 24(1), 106–119.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J., Yuan, X., Liao, X., Llull, P., Sapiro, G., Brady, D. J., & Carin, L. (2014). Video compressive sensing using Gaussian mixture models. IEEE Transaction on Image Processing, 23(11), 4863–4878.

    Article  MathSciNet  MATH  Google Scholar 

  • Yuan, X. (2016) Generalized alternating projection based total variation minimization for compressive sensing. In: 2016 IEEE international conference on image processing (ICIP), pp. 2539–2543 (2016)

  • Yuan, X., Brady, D. J., & Katsaggelos, A. K. (2021). Snapshot compressive imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 38(2), 65–88.

    Article  Google Scholar 

  • Yuan, X., Liu, Y., Suo, J., & Dai, Q. (2020) Plug-and-play algorithms for large-scale snapshot compressive imaging. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 1447–1457

  • Yuan, X., Liu, Y., Suo, J., Durand, F., & Dai, Q. (2022). Plug-and-play algorithms for video snapshot compressive imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 7093–7111. https://doi.org/10.1109/TPAMI.2021.3099035

    Article  Google Scholar 

  • Yuan, X., Llull, P., Liao, X., Yang, J., & Carin, L. (2014) Low-cost compressive sensing for color video and depth. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3318–3325

  • Yuan, X., Yang, J., Llull, P., Liao, X., Sapiro, G., Brady, D. J., & Carin, L. (2013) Adaptive temporal compressive sensing for video. IEEE international conference on image processing pp. 1–4

  • Zhang, B., Yuan, X., Deng, C., Zhang, Z., Suo, J., & Dai, Q. (2022). End-to-end snapshot compressed super-resolution imaging with deep optics. Optica, 9(4), 451–454.

    Article  Google Scholar 

  • Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., & Timofte, R. (2021) Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence

  • Zhang, K., Zuo, W., & Zhang, L. (2018). FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, 27(9), 4608–4622.

    Article  MathSciNet  Google Scholar 

  • Zhang, L., & Zuo, W. (2017). Image restoration: From sparse and low-rank priors to deep priors. IEEE Signal Processing Magazine, 34(5), 172–179. https://doi.org/10.1109/msp.2017.2717489

    Article  Google Scholar 

  • Zhang, Z., Zhang, B., Yuan, X., Zheng, S., Su, X., Suo, J., Brady, D. J., & Dai, Q. (2022). From compressive sampling to compressive tasking: Retrieving semantics in compressed domain with low bandwidth. PhotoniX, 3(1), 1–22.

    Article  Google Scholar 

  • Zheng, S., Wang, C., Yuan, X., & Xin, H. L. (2021) Super-compression of large electron microscopy time series by deep compressive sensing learning. Patterns pp. 100292

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 62271414, Zhejiang Provincial Natural Science Foundation of China under Grant No. LR23F010001. We would like to thank Research Center for Industries of the Future (RCIF) at Westlake University for supporting this work and the funding from Lochn Optics.

Author information

Author notes

  1. Z. Wu and C. Yang: contribute equally to this paper.

Authors and Affiliations

  1. Zhejiang University, Hangzhou, 310058, Zhejiang, China

    Zongliang Wu & Xiongfei Su

  2. Research Center for Industries of the Future (RCIF) and School of Engineering, Westlake University, Hangzhou, 310030, Zhejiang, China

    Zongliang Wu, Chengshuai Yang, Xiongfei Su & Xin Yuan

Authors
  1. Zongliang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengshuai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiongfei Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Yuan.

Additional information

Communicated by Ying Fu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Z., Yang, C., Su, X. et al. Adaptive Deep PnP Algorithm for Video Snapshot Compressive Imaging. Int J Comput Vis 131, 1662–1679 (2023). https://doi.org/10.1007/s11263-023-01777-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01777-y

Keywords

Search

Navigation