Skip to main content
SpringerLink
Log in

Adaptive weighting function for weighted nuclear norm based matrix/tensor completion

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Weighted nuclear norm provides a simple yet powerful tool to characterize the intrinsic low-rank structure of a matrix, and has been successfully applied to the matrix completion problem. However, in previous studies, the weighting functions to calculate the weights are fixed beforehand, and do not change during the whole iterative process. Such predefined weighting functions may not be able to precisely characterize the complicated structure underlying the observed data matrix, especially in the dynamic estimation process, and thus limits its performance. To address this issue, we propose a strategy of adaptive weighting function, for low-rank matrix/tensor completion. Specifically, we first parameterize the weighting function as a simple yet flexible neural network, that can approximate a wide range of monotonic decreasing functions. Then we propose an effective strategy, by virtue of the bi-level optimization technique, to adapt the weighting function, and incorporate this strategy to the alternating direction method of multipliers for solving low-rank matrix and tensor completion problems. Our empirical studies on a series of synthetic and real data have verified the effectiveness of the proposed approach, as compared with representative low-rank matrix and tensor completion methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

  1. More details are provided in Sect. 4.4.

  2. Interested readers can refer to [36] for more details about the unfolding operation.

  3. We set \(q=0.5\) for its representativeness for \(q\in (0,1)\) [69].

  4. http://www1.cs.columbia.edu/CAVE/databases/multispectral.

  5. http://trace.eas.asu.edu/yuv/.

References

  1. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Technical report

  2. Xiao T, Xia T, Yang Y, Huang C, Wang X (2015) Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2691–2699

  3. Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645

    Google Scholar 

  4. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  5. Candès EJ, Recht B (2009) Exact matrix completion via convex optimization. Found Comput Math 9:717–772

    MathSciNet  Google Scholar 

  6. Nie F, Huang H, Ding C (2012) Low-rank matrix recovery via efficient Schatten p-norm minimization. In: Proceedings of the 26th AAAI conference on artificial intelligence, pp 655–661

  7. Marjanovic G, Solo V (2012) On \(l_q\) optimization and matrix completion. IEEE Trans Signal Process 60(11):5714–5724

    MathSciNet  Google Scholar 

  8. Zhang D, Hu Y, Ye J, Li X, He X (2012) Matrix completion by truncated nuclear norm regularization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2192–2199

  9. Hu Y, Zhang D, Ye J, Li X, He X (2013) Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Trans Pattern Anal Mach Intell 35(9):2117–2130

    Google Scholar 

  10. Liu D, Zhou T, Qian H, Xu C, Zhang Z (2013) A nearly unbiased matrix completion approach. In: Proceedings of joint European conference on machine learning and knowledge discovery in databases, pp 210–225

  11. Gui H, Han J, Gu Q (2016) Towards faster rates and oracle property for low-rank matrix estimation. In: Proceedings of the 33rd international conference on machine learning, pp 2300–2309

  12. Fazel SM (2002) Matrix rank minimization with applications. PhD thesis, Stanford University

  13. Gu S, Xie Q, Meng D, Zuo W, Feng X, Zhang L (2017) Weighted nuclear norm minimization and its applications to low level vision. Int J Comput Vis 121:183–208

    Google Scholar 

  14. Liu J, Musialski P, Wonka P, Ye J (2009) Tensor completion for estimating missing values in visual data. In: Proceedings of the 12th IEEE international conference on computer vision, pp 2114–2121

  15. Liu J, Musialski P, Wonka P, Ye J (2013) Tensor completion for estimating missing values in visual data. IEEE Trans Pattern Anal Mach Intell 35(1):208–220

    Google Scholar 

  16. Nati NS, Jaakkola T (2003) Weighted low-rank approximations. In: Proceedings of the 20th international conference on machine learning, pp 720–727

  17. Mnih A, Salakhutdinov RR (2007) Probabilistic matrix factorization. Adv Neural Inf Process Syst 20:1257–1264

    Google Scholar 

  18. Buchanan AM, Fitzgibbon AW (2005) Damped newton algorithms for matrix factorization with missing data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 2, pp 316–322

  19. Okatani T, Deguchi K (2007) On the Wiberg algorithm for matrix factorization in the presence of missing components. Int J Comput Vis 72(3):329–337

    Google Scholar 

  20. Cai J-F, Candès EJ, Shen Z (2010) A singular value thresholding algorithm for matrix completion. SIAM J Optim 20(4):1956–1982

    MathSciNet  Google Scholar 

  21. Candes EJ, Plan Y (2010) Matrix completion with noise. Proc IEEE 98(6):925–936

    Google Scholar 

  22. Ma S, Goldfarb D, Chen L (2011) Fixed point and Bregman iterative methods for matrix rank minimization. Math Program 128:321–353

    MathSciNet  Google Scholar 

  23. Mazumder R, Hastie T, Tibshirani R (2010) Spectral regularization algorithms for learning large incomplete matrices. J Mach Learn Res 11(80):2287–2322

    MathSciNet  Google Scholar 

  24. Toh K-C, Yun S (2010) An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pac J Optim 6(3):615–640

    MathSciNet  Google Scholar 

  25. Lin Z, Chen M, Ma Y (2010) The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices. CoRR arXiv:1009.5055

  26. Lin Z, Liu R, Su Z (2011) Linearized alternating direction method with adaptive penalty for low-rank representation. Adv Neural Inf Process Syst 24:612–620

    Google Scholar 

  27. Liu L, Huang W, Chen D-R (2014) Exact minimum rank approximation via Schatten p-norm minimization. J Comput Appl Math 267:218–227

    MathSciNet  Google Scholar 

  28. Zhang R, Li S (2019) Optimal rip bounds for sparse signals recovery via \(\ell _p\) minimization. Appl Comput Harmon Anal 47(3):566–584

    MathSciNet  Google Scholar 

  29. Zhang C-H (2010) Nearly unbiased variable selection under minimax concave penalty. Ann Stat 38(2):894–942

    MathSciNet  Google Scholar 

  30. Fan J, Peng H (2004) Nonconcave penalized likelihood with a diverging number of parameters. Ann Stat 32(3):928–961

    MathSciNet  Google Scholar 

  31. Zhang H, Qian J, Zhang B, Yang J, Gong C, Wei Y (2020) Low-rank matrix recovery via modified Schatten-\(p\) norm minimization with convergence guarantees. IEEE Trans Image Process 29:3132–3142

    MathSciNet  Google Scholar 

  32. Lu C, Zhu C, Xu C, Yan S, Lin Z (2015) Generalized singular value thresholding. In: Proceedings of the twenty-ninth AAAI conference on artificial intelligence, pp 1805–1811

  33. Lai M-J, Xu Y, Yin W (2013) Improved iteratively reweighted least squares for unconstrained smoothed \(\ell _q\) minimization. SIAM J Numer Anal 51(2):927–957

    MathSciNet  Google Scholar 

  34. Li H, Lin Z (2015) Accelerated proximal gradient methods for nonconvex programming. Adv Neural Inf Process Syst 28:379–387

    Google Scholar 

  35. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends® Mach Learn 3(1):1–122

  36. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500

    MathSciNet  Google Scholar 

  37. Romera-Paredes B, Pontil M (2013) A new convex relaxation for tensor completion. Adv Neural Inf Process Syst 26:2967–2975

    Google Scholar 

  38. Cao W, Wang Y, Yang C, Chang X, Han Z, Xu Z (2015) Folded-concave penalization approaches to tensor completion. Neurocomputing 152:261–273

    Google Scholar 

  39. Zhao Q, Meng D, Kong X, Xie Q, Cao W, Wang Y, Xu Z (2015) A novel sparsity measure for tensor recovery. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 271–279

  40. Xie Q, Zhao Q, Meng D, Xu Z (2018) Kronecker-basis-representation based tensor sparsity and its applications to tensor recovery. IEEE Trans Pattern Anal Mach Intell 40(8):1888–1902

    Google Scholar 

  41. Xu Y, Hao R, Yin W, Su Z (2015) Parallel matrix factorization for low-rank tensor completion. Inverse Probl Imaging 9(2):601–624

    MathSciNet  Google Scholar 

  42. Bengua JA, Phien HN, Tuan HD, Do MN (2017) Efficient tensor completion for color image and video recovery: low-rank tensor train. IEEE Trans Image Process 26(5):2466–2479

    MathSciNet  Google Scholar 

  43. Oseledets IV (2011) Tensor-train decomposition. SIAM J Sci Comput 33(5):2295–2317

    MathSciNet  Google Scholar 

  44. Kilmer ME, Martin CD (2011) Factorization strategies for third-order tensors. Linear Algebra Appl 435(3):641–658

    MathSciNet  Google Scholar 

  45. Kilmer ME, Braman K, Hao N, Hoover RC (2013) Third-order tensors as operators on matrices: a theoretical and computational framework with applications in imaging. SIAM J Matrix Anal Appl 34(1):148–172

    MathSciNet  Google Scholar 

  46. Martin CD, Shafer R, LaRue B (2013) An order-\$p\$ tensor factorization with applications in imaging. SIAM J Sci Comput 35(1):474–490

    MathSciNet  Google Scholar 

  47. Zhang Z, Ely G, Aeron S, Hao N, Kilmer M (2014) Novel methods for multilinear data completion and de-noising based on tensor-SVD. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 3842–3849

  48. Zhang Z, Aeron S (2017) Exact tensor completion using t-SVD. IEEE Trans Signal Process 65(6):1511–1526

    MathSciNet  Google Scholar 

  49. Zheng Y-B, Huang T-Z, Zhao X-L, Jiang T-X, Ji T-Y, Ma T-H (2020) Tensor n-tubal rank and its convex relaxation for low-rank tensor recovery. Inf Sci 532:170–189

    MathSciNet  Google Scholar 

  50. Liu X, Aeron S, Aggarwal V, Wang X (2020) Low-tubal-rank tensor completion using alternating minimization. IEEE Trans Inf Theory 66(3):1714–1737

    MathSciNet  Google Scholar 

  51. Yuan L, Li C, Mandic DP, Cao J, Zhao Q (2019) Tensor ring decomposition with rank minimization on latent space: an efficient approach for tensor completion. In: Proceedings of the thirty-third AAAI conference on artificial intelligence, pp 9151–9158

  52. Yuan L, Li C, Cao J, Zhao Q (2020) Rank minimization on tensor ring: an efficient approach for tensor decomposition and completion. Mach Learn 109(3):603–622

    MathSciNet  Google Scholar 

  53. Wang W, Aggarwal V, Aeron S (2017) Efficient low rank tensor ring completion. In: Proceedings of the 2017 IEEE international conference on computer vision, pp 5698–5706

  54. Yu J, Zhou G, Li C, Zhao Q, Xie S (2021) Low tensor-ring rank completion by parallel matrix factorization. IEEE Trans Neural Netw Learn Syst 32(7):3020–3033

    MathSciNet  Google Scholar 

  55. Zhao Q, Zhou G, Xie S, Zhang L, Cichocki A (2016) Tensor ring decomposition. CoRR arXiv:1606.05535

  56. Gu S, Zhang L, Zuo W, Feng X (2014) Weighted nuclear norm minimization with application to image denoising. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2862–2869

  57. Lange K (2016) MM optimization algorithms. Society for Industrial and Applied Mathematics, Philadelphia

    Google Scholar 

  58. Candès EJ, Wakin MB, Boyd SP (2008) Enhancing sparsity by reweighted \(\ell _1\) minimization. J Fourier Anal Appl 14(5):877–905

    MathSciNet  Google Scholar 

  59. Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math Control Signals Syst 2(4):303–314

    MathSciNet  Google Scholar 

  60. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366

    Google Scholar 

  61. Barron AR (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inf Theory 39(3):930–945

    MathSciNet  Google Scholar 

  62. Daniels H, Velikova M (2010) Monotone and partially monotone neural networks. IEEE Trans Neural Netw 21(6):906–917

    Google Scholar 

  63. Dugas C, Bengio Y, Bélisle F, Nadeau C, Garcia R (2009) Incorporating functional knowledge in neural networks. J Mach Learn Res 10(42):1239–1262

    MathSciNet  Google Scholar 

  64. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th international conference on machine learning, pp 1126–1135

  65. Ren M, Zeng W, Yang B, Urtasun R (2018) Learning to reweight examples for robust deep learning. In: Proceedings of the 35th international conference on machine learning, pp 4334–4343

  66. Shu J, Xie Q, Yi L, Zhao Q, Zhou S, Xu Z, Meng D (2019) Meta-weight-net: learning an explicit mapping for sample weighting. Adv Neural Inf Process Syst 32:1919–1930

    Google Scholar 

  67. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 32:8024–8035

    Google Scholar 

  68. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray D.G, Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) Tensorflow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX conference on operating systems design and implementation, pp 265–283

  69. Xu Z, Chang X, Xu F, Zhang H (2012) \(l_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans Neural Netw Learn Syst 23(7):1013–1027

    Google Scholar 

  70. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Google Scholar 

  71. Roth S, Black MJ (2009) Fields of experts. Int J Comput Vis 82(2):205–229

    Google Scholar 

  72. Zhang L, Song L, Du B, Zhang Y (2021) Nonlocal low-rank tensor completion for visual data. IEEE Trans Cybern 51(2):673–685

    Google Scholar 

  73. Yasuma F, Mitsunaga T, Iso D, Nayar SK (2010) Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans Image Process 19(9):2241–2253. https://doi.org/10.1109/TIP.2010.2046811

    Article  MathSciNet  Google Scholar 

  74. Candès EJ, Li X, Ma Y, Wright J (2011) Robust principal component analysis? J ACM 58(3):11

    MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported National Key Research and Development Program of China (2020YFA0713900), China NSFC Projects (62076196) and the Macao Science and Technology Development Fund (061/2020/A2).

Author information

Author notes

  1. Qian Zhao and Yuji Lin contributed equally to this work.

Authors and Affiliations

  1. School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, People’s Republic of China

    Qian Zhao, Yuji Lin, Fengxingyu Wang & Deyu Meng

  2. Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, People’s Republic of China

    Deyu Meng

  3. Pazhou Laboratory (Huangpu), Guangzhou, 510555, Guangdong, People’s Republic of China

    Deyu Meng

  4. Macao Institute of Systems Engineering, Macau University of Science and Technology, Taipa, Macao, People’s Republic of China

    Deyu Meng

Authors
  1. Qian Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuji Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fengxingyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deyu Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qian Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Q., Lin, Y., Wang, F. et al. Adaptive weighting function for weighted nuclear norm based matrix/tensor completion. Int. J. Mach. Learn. & Cyber. 15, 697–718 (2024). https://doi.org/10.1007/s13042-023-01935-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01935-1

Keywords

Search

Navigation