Skip to main content
Log in

Data-Driven Tight Frame for Multi-channel Images and Its Application to Joint Color-Depth Image Reconstruction

  • Published:
Journal of the Operations Research Society of China Aims and scope Submit manuscript

Abstract

In image restoration, we usually assume that the underlying image has a good sparse approximation under a certain system. Wavelet tight frame system has been proven to be such an efficient system to sparsely approximate piecewise smooth images. Thus, it has been widely used in many practical image restoration problems. However, images from different scenarios are so diverse that no static wavelet tight frame system can sparsely approximate all of them well. To overcome this, recently, Cai et. al. (Appl Comput Harmon Anal 37:89–105, 2014) proposed a method that derives a data-driven tight frame adapted to the specific input image, leading to a better sparse approximation. The data-driven tight frame has been applied successfully to image denoising and CT image reconstruction. In this paper, we extend this data-driven tight frame construction method to multi-channel images. We construct a discrete tight frame system for each channel and assume their sparse coefficients have a joint sparsity. The multi-channel data-driven tight frame construction scheme is applied to joint color and depth image reconstruction. Experimental results show that the proposed approach has a better performance than state-of-the-art joint color and depth image reconstruction approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. The dataset can be downloaded at http://web.cecs.pdx.edu/~fliu/project/depth-enhance/Middlebury.htm.

References

  1. Bao, C., Cai, J.-F., and Ji, H.: Fast sparsity-based orthogonal dictionary learning for image restoration, In: IEEE International Conference on Computer Vision, pp. 3384–3391, Sydney, (2013)

  2. Bao, C., Ji, H., Shen, Z.: Convergence analysis for iterative data-driven tight frame construction scheme, Appl. Comput. Harmon. Anal. (2014). doi:10.1016/j.acha.2014.06.007

  3. Cai, J.-F., Chan, R.H., Shen, L., Shen, Z.: Restoration of chopped and nodded images by framelets. SIAM J. Sci. Comput. 30, 1205–1227 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  4. Cai, J.-F., Chan, R.H., Shen, L., Shen, Z.: Convergence analysis of tight framelet approach for missing data recovery. Adv. Comput. Math. 31, 87–113 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  5. Cai, J.-F., Chan, R.H., Shen, Z.: A framelet-based image inpainting algorithm. Appl. Comput. Harmon. Anal. 24, 131–149 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  6. Cai, J.-F., Dong, B., Osher, S., Shen, Z.: Image restoration: total variation, wavelet frames, and beyond. J. Amer. Math. Soc. 25, 1033–1089 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  7. Cai, J.-F., Ji, H., Shen, Z., Ye, G.-B.: Data-driven tight frame construction and image denoising. Appl. Comput. Harmon. Anal. 37, 89–105 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  8. Cai, J.-F., Osher, S., Shen, Z.: Linearized Bregman iterations for compressed sensing. Math. Comp. 78, 1515–1536 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  9. Cai, J.-F., Osher, S., Shen, Z.: Linearized Bregman iterations for frame-based image deblurring. SIAM J. Imaging Sci. 2, 226–252 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  10. Candès, E.J., Donoho, D.L.: New tight frames of curvelets and optimal representations of objects with piecewise \(C^2\) singularities. Comm. Pure Appl. Math. 57, 219–266 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  11. Chan, R.H., Chan, T.F., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24, 1408–1432 (2003)

  12. Chan, T.F., Shen, J.: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods. Society for Industrial and Applied Mathematics, Philadelphia (2005)

  13. Daubechies, I.: Ten Lectures on Wavelets, vol. 61 of CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (1992)

  14. Daubechies, I., Han, B., Ron, A., Shen, Z.: Framelets: MRA-based constructions of wavelet frames. Appl. Comput. Harmon. Anal. 14, 1–46 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  15. Fornasier, M., Ramlau, R., Teschke, G.: The application of joint sparsity and total variation minimization algorithms to a real-life art restoration problem. Adv. Comput. Math. 31, 157–184 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  16. Fornasier, M., Rauhut, H.: Recovery algorithms for vector-valued data with joint sparsity constraints. SIAM J. Numer. Anal. 46, 577–613 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  17. Han, B., Kutyniok, G., Shen, Z.: Adaptive multiresolution analysis structures and shearlet systems. SIAM J. Numer. Anal. 49, 1921–1946 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  18. Hui, T.-W., and Ngi Ngan, K.: Motion-Depth: Rgb-d Depth Map Enhancement with Motion and Depth in Complement, In: IEEE conference on Computer Vision and Pattern Recognition, pp. 3962–3969, Columbus (2014)

  19. Jain, R., Kasturi, R., Schunck, B.G.: Machine Vision, vol. 5. McGraw-Hill, New York (1995)

    Google Scholar 

  20. Ji, H., Shen, Z., Xu, Y.: Wavelet frame based scene reconstruction from range data. J. Comput. Phys. 229, 2093–2108 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  21. Kuster, C., Popa, T., Zach, C., Gotsman, C., Gross, M. H., Eisert, P., Hornegger, J., and Polthier, K.: Freecam: A Hybrid Camera System for Interactive Free-viewpoint Video. In: Proceedings of International Workshop on Vision, Modeling and Visualization, pp. 17–24 (2011)

  22. Labate, D., Lim, W.-Q., Kutyniok, G., Weiss, G.: Sparse multidimensional representation using shearlets, In: Optics & Photonics 2005, International Society for Optics and Photonics, pp. 59140U–59140U, San Diego, (2005)

  23. Le Pennec, E., Mallat, S.: Sparse geometric image representations with bandelets. IEEE Trans. Image Process. 14, 423–438 (2005)

    Article  MathSciNet  Google Scholar 

  24. Liang, J., Ma, J., Zhang, X.: Seismic data restoration via data-driven tight frame. Geophysics 79, V65–V74 (2014)

    Article  Google Scholar 

  25. Lu, S., Ren, X., and Liu, F.: Depth enhancement via low-rank matrix completion, In: IEEE Conference on Computer Vision and Pattern Recognition, Columbus, (2014)

  26. Maimone, A., Fuchs, H.: Encumbrance-free telepresence system with real-time 3d capture and display using commodity depth cameras, In: 10th IEEE international symposium on Mixed and Augmented Reality, pp. 137–146, Basel, (2011)

  27. Mallat, S.: A wavelet tour of signal processing, Elsevier/Academic Press, Amsterdam, 3rd edn. The sparse way, With contributions from Gabriel Peyré (2009)

  28. Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C.: Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos, In: Proceedings of Eurographics, 31 (2012)

  29. Ron, A., Shen, Z.: Affine systems in \(L_2({{\bf R^d}})\): the analysis of the analysis operator. J. Funct. Anal. 148, 408–447 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  30. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)

    Article  MATH  Google Scholar 

  31. Seok Lee H., and Mu Lee, K.: Simultaneous super-resolution of depth and images using a single camera, In: The IEEE Conference on Computer Vision and Pattern Recognition, Sydney, (2013)

  32. Shekhar, S., Patel, V.M., Nasrabadi, N.M., Chellappa, R., Joint sparsity-based robust multimodal biometrics recognition, In: Computer Vision-ECCV 2012 Workshops and Demonstrations. Springer, pp. 365–374 (2012)

  33. Shen, Z.: Wavelet frames and image restorations, In: Proceedings of the International Congress of Mathematicians, Volume IV, pp. 2834–2863. Hindustan Book Agency, New Delhi (2010)

  34. Yang, X., Kim, S., and Xing, E. P.: Heterogeneous multitask learning with joint sparsity constraints, In: Advances in neural information processing systems, pp. 2151–2159, Vancouver, B.C., (2009)

  35. Zhou, W., Cai, J.-F., Gao, H.: Adaptive tight frame based medical image reconstruction: a proof-of-concept study for computed tomography. Inverse Prob. (2013). doi:10.1088/0266-5611/29/12/125006

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jian-Feng Cai.

Additional information

Jian-Feng Cai is partially supported by the National Natural Science Foundation of USA (No. DMS 1418737).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Cai, JF. Data-Driven Tight Frame for Multi-channel Images and Its Application to Joint Color-Depth Image Reconstruction. J. Oper. Res. Soc. China 3, 99–115 (2015). https://doi.org/10.1007/s40305-015-0074-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40305-015-0074-2

Keywords

Mathematics Subject Classification

Navigation