Skip to main content
SpringerLink
Log in

Overcomplete-to-sparse representation learning for few-shot class-incremental learning

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Few-shot class-incremental learning (FSCIL) aims to continually learn new semantics given a few training samples of new classes. As training examples are too few to construct good representation upon, FSCIL is required to generalize learned semantics from old to new classes, as well as reduce the representation aliasing between them (old classes ‘forgetting’). This motivates us to develop overcomplete-to-sparse representation learning (O2SRL). It solves the ‘new class generalization’ and ‘old class forgetting’ problems systematically by regularizing both feature completeness and sparsity. Specifically, O2SRL consists of a spatial excitation module (SEM) and a channel purification module (CPM). SEM drives the model to learn overcomplete and generic features, which not only represent all classes well but also benefit generalization to new classes. CPM regularizes the sparsity and uniqueness of features, reducing semantic aliasing between classes and alleviating the forgetting of old classes. These two modules facilitate each other to configure unique and robust representation for both old and new classes. Experiments show that O2SRL improves the state-of-the-art of FSCIL by significant margins on public datasets including CUB200, CIFAR100, and mini-ImageNet. O2SRL’s effectiveness is also validated under the general few-shot learning setting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

CUB200 [52]: https://www.vision.caltech.edu/datasets/cub_200_2011/ mini-ImageNet [53]: https://www.kaggle.com/datasets/arjunashok33/miniimagenet CIFAR100: https://www.cs.toronto.edu/~kriz/cifar.html

References

  1. Reboud, A., Harrando, I., Lisena, P., Troncy, R.: Stories of love and violence: zero-shot interesting events’ classification for unsupervised tv series summarization. Multimed. Syst. (2023). https://doi.org/10.1007/S00530-022-01040-3

    Article  Google Scholar 

  2. Greenwald, A.G.: Cognitive learning, cognitive response to persuasion, and attitude change. Psychol. Found. Attitud. (1968)

  3. Zhang, C., Song, N., Lin, G., Zheng, Y., Pan, P., Xu, Y.: Few-shot incremental learning with continually evolved classifiers. In: IEEE CVPR (2021). https://doi.org/10.1109/CVPR46437.2021.01227

  4. Zhou, D.-W., Wang, F.-Y., Ye, H.-J., Ma, L., Pu, S., Zhan, D.-C.: Forward compatible few-shot class-incremental learning. In: IEEE CVPR, pp. 9046–9056 (2022). https://doi.org/10.1109/CVPR52688.2022.00884

  5. Liu, H., Gu, L., Chi, Z., Wang, Y., Yu, Y., Chen, J., Tang, J.: Few-shot class-incremental learning viaentropy-regularized data-free replay. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20053-3_9

  6. Chatterjee, R., Chatterjee, A., Islam, S.H., Khan, M.K.: An object detection-based few-shot learning approach for multimedia quality assessment. Multimedia Systems, 1–14 (2022). https://doi.org/10.1007/S00530-021-00881-8

  7. Li, Z., He, J., Ni, T., Huo, J.: Numerical computation based few-shot learning for intelligent sea surface temperature prediction. Multimed. Syst. 29(5), 3001–3013 (2022). https://doi.org/10.1007/S00530-022-00941-7

    Article  Google Scholar 

  8. Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. In: NeurIPS, pp. 3630–3638 (2016). https://proceedings.neurips.cc/paper/2016/hash/90e1357833654983612fb05e3ec9148c-Abstract.html

  9. Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning. In: NeurIPS, pp. 4077–4087 (2017)

  10. Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H.S., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning. In: IEEE CVPR, pp. 1199–1208 (2018). https://doi.org/10.1109/CVPR.2018.00131

  11. Zhang, C., Cai, Y., Lin, G., Shen, C.: Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers. In: IEEE CVPR, pp. 12200–12210 (2020). https://doi.org/10.1109/CVPR42600.2020.01222

  12. Liu, B., Jiao, J., Ye, Q.: Harmonic feature activation for few-shot semantic segmentation. IEEE Trans. Image Process. 30, 3142–3153 (2021). https://doi.org/10.1109/TIP.2021.3058512

    Article  Google Scholar 

  13. Yang, B., Wan, F., Liu, C., Li, B., Ji, X., Ye, Q.: Part-based semantic transform for few-shot semantic segmentation. IEEE Trans. Neural Netw. Learn. Syst. (2021). https://doi.org/10.1109/TNNLS.2021.3084252

    Article  Google Scholar 

  14. Zhang, C., Cai, Y., Lin, G., Shen, C.: Deepemd: differentiable earth mover’s distance for few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)

  15. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: ICML, pp. 1126–1135 (2017). http://proceedings.mlr.press/v70/finn17a.html

  16. Elsken, T., Staffler, B., Metzen, J.H., Hutter, F.: Meta-learning of neural architectures for few-shot learning. In: IEEE CVPR, pp. 12362–12372 (2020). https://doi.org/10.1109/CVPR42600.2020.01238

  17. Sun, Q., Liu, Y., Chua, T., Schiele, B.: Meta-transfer learning for few-shot learning. In: IEEE CVPR, pp. 403–412 (2019). https://doi.org/10.1109/CVPR.2019.00049

  18. Li, B., Yang, B., Liu, C., Liu, F., Ji, R., Ye, Q.: Beyond max-margin: class margin equilibrium for few-shot object detection. In: IEEE CVPR, pp. 7363–7372 (2021). https://doi.org/10.1109/CVPR46437.2021.00728

  19. Zhang, H., Zhang, J., Koniusz, P.: Few-shot learning via saliency-guided hallucination of samples. In: IEEE ICCV, pp. 2770–2779 (2019). https://doi.org/10.1109/CVPR.2019.00288

  20. Li, K., Zhang, Y., Li, K., Fu, Y.: Adversarial feature hallucination networks for few-shot learning. In: IEEE ICCV, pp. 13467–13476 (2020). https://doi.org/10.1109/CVPR42600.2020.01348

  21. Kim, J., Kim, H., Kim, G.: Model-agnostic boundary-adversarial sampling for test-time generalization in few-shot learning. In: ECCV, pp. 599–617 (2020). https://doi.org/10.1007/978-3-030-58452-8_35

  22. Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A.D., Weijer, J.: Class-incremental learning: survey and performance evaluation. CoRR (2020). arxiv:2010.15277

  23. Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation learning. In: IEEE CVPR, pp. 5533–5542 (2017). https://doi.org/10.1109/CVPR.2017.587

  24. Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.S.: Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: ECCV, pp. 556–572 (2018). https://doi.org/10.1007/978-3-030-01252-6_33

  25. Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large scale incremental learning. In: IEEE CVPR, pp. 374–382 (2019). https://doi.org/10.1109/CVPR.2019.00046

  26. Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NeurIPS, pp. 2990–2999 (2017). https://proceedings.neurips.cc/paper/2017/hash/0efbe98067c6c73dba1250d2beaa81f9-Abstract.html

  27. Xiang, Y., Fu, Y., Ji, P., Huang, H.: Incremental learning using conditional adversarial networks. In: IEEE ICCV, pp. 6618–6627 (2019). https://doi.org/10.1109/ICCV.2019.00672

  28. Kim, C.D., Jeong, J., Moon, S., Kim, G.: Continual learning on noisy data streams via self-purified replay. In: IEEE ICCV, pp. 537–547 (2021). https://doi.org/10.1109/ICCV48922.2021.00058

  29. Smith, J., Hsu, Y.-C., Balloch, J., Shen, Y., Jin, H., Kira, Z.: Always be dreaming: a new approach for data-free class-incremental learning. In: IEEE ICCV, pp. 9374–9384 (2021). https://doi.org/10.1109/ICCV48922.2021.00924

  30. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2018). https://doi.org/10.1109/TPAMI.2017.2773081

    Article  Google Scholar 

  31. Dhar, P., Singh, R.V., Peng, K., Wu, Z., Chellappa, R.: Learning without memorizing. In: IEEE CVPR, pp. 5138–5146 (2019). https://doi.org/10.1109/CVPR.2019.00528

  32. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: ICML, pp. 3987–3995 (2017). http://proceedings.mlr.press/v70/zenke17a.html

  33. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: IEEE ICCV, pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74

  34. Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: IEEE CVPR, pp. 831–839 (2019). https://doi.org/10.1109/CVPR.2019.00092

  35. Hu, X., Tang, K., Miao, C., Hua, X.-S., Zhang, H.: Distilling causal effect of data in class-incremental learning. In: IEEE CVPR, pp. 3957–3966 (2021). https://doi.org/10.1109/CVPR46437.2021.00395

  36. Cha, H., Lee, J., Shin, J.: Co2l: contrastive continual learning. In: IEEE ICCV, pp. 9516–9525 (2021). https://doi.org/10.1109/ICCV48922.2021.00938

  37. Mallya, A., Lazebnik, S.: Packnet: adding multiple tasks to a single network by iterative pruning. In: IEEE CVPR, pp. 7765–7773 (2018). https://doi.org/10.1109/CVPR.2018.00810

  38. Serrà, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. In: ICML. In: Proceedings of Machine Learning Research, pp. 4555–4564 (2018). http://proceedings.mlr.press/v80/serra18a.html

  39. Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In: ICLR (2018). https://openreview.net/forum?id=Sk7KsfW0-

  40. Mallya, A., Davis, D., Lazebnik, S.: Piggyback: adapting a single network to multiple tasks by learning to mask weights. In: ECCV, pp. 72–88 (2018). https://doi.org/10.1007/978-3-030-01225-0_5

  41. Liu, Y., Su, Y., Liu, A., Schiele, B., Sun, Q.: Mnemonics training: Multi-class incremental learning without forgetting. In: IEEE CVPR, pp. 12242–12251 (2020). https://doi.org/10.1109/CVPR42600.2020.01226

  42. Belouadah, E., Popescu, A.: IL2M: class incremental learning with dual memory. In: IEEE ICCV, pp. 583–592 (2019). https://doi.org/10.1109/ICCV.2019.00067

  43. Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: IEEE ICCV, pp. 3420–3429 (2017). https://doi.org/10.1109/ICCV.2017.368

  44. Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I., Tu, Y., Tesauro, G.: Learning to learn without forgetting by maximizing transfer and minimizing interference. In: ICLR (2019). https://openreview.net/forum?id=B1gTShAct7

  45. Tian, S., Li, L., Li, W., Ran, H., Ning, X., Tiwari, P.: A survey on few-shot class-incremental learning (2023). https://doi.org/10.48550/arXiv.2304.08130

  46. Kim, D.-Y., Han, D.-J., Seo, J., Moon, J.: Warping the space: Weight space rotation for class-incremental few-shot learning. In: International Conference on Learning Representations (2023). https://api.semanticscholar.org/CorpusID:259298246

  47. Yang, Y., Yuan, H., Li, X., Lin, Z., Torr, P.H.S., Tao, D.: Neural collapse inspired feature-classifier alignment for few-shot class incremental learning (2023). https://api.semanticscholar.org/CorpusID:256615229. ArXiv:2302.03004

  48. Tao, X., Hong, X., Chang, X., Dong, S., Wei, X., Gong, Y.: Few-shot class-incremental learning. In: IEEE CVPR, pp. 12180–12189 (2020)

  49. Zhu, K., Cao, Y., Zhai, W., Cheng, J., Zha, Z.-J.: Self-promoted prototype refinement for few-shot class-incremental learning. In: IEEE CVPR, pp. 6801–6810 (2021). https://doi.org/10.1109/CVPR46437.2021.00673

  50. Papyan, V., Romano, Y., Elad, M.: Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res. 2887–2938 (2017). http://jmlr.org/papers/v18/16-505.html

  51. Murdock, C., Lucey, S.: Reframing neural networks: deep structure in overcomplete representations. CoRR (2021). arxiv:2103.05804

  52. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S. Technical Report CNS-TR-2011-001, California Institute of Technology (2011)

  53. Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning. In: International Conference on Learning Representations (2016). https://api.semanticscholar.org/CorpusID:67413369

  54. Chi, Z., Gu, L., Liu, H., Wang, Y., Yu, Y., Tang, J.: Metafscil: a meta-learning approach for few-shot class incremental learning. In: IEEE CVPR, pp. 14166–14175 (2022). https://doi.org/10.1109/CVPR52688.2022.01377

  55. Zhuang, H., Weng, Z., He, R., Lin, Z., Zeng, Z.: Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task. In: IEEE CVPR (2023). https://doi.org/10.1109/CVPR52729.2023.00748

  56. Afrasiyabi, A., Larochelle, H., Lalonde, J.-F., Gagné, C.: Matching feature sets for few-shot image classification. In: IEEE CVPR, pp. 9014–9024 (2022). https://doi.org/10.1109/CVPR52688.2022.00881

Download references

Funding

This work was supported by National Natural Science Foundation of China (NSFC) under Grants 62225208 and 62171431.

Author information

Authors and Affiliations

  1. School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, 100049, China

    Fu Mengying, Liu Binghao, Ma Tianren & Ye Qixiang

Authors
  1. Fu Mengying
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liu Binghao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ma Tianren
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ye Qixiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Fu Mengying proposed the idea and was responsible for the validation of the main experiment. Liu Binghao wrote the main manuscript text and Ma Tianren made relevant pictures and tables. Ye Qixiang has revised the article and gave guidance. All authors reviewed the manuscript.

Corresponding author

Correspondence to Ye Qixiang.

Ethics declarations

Conflict of interest

The authors have no Conflict of interest to declare that are relevant to the content of this article.

Ethics approval and consent to participate

Not applicable.

Additional information

Communicated by J. Gao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mengying, F., Binghao, L., Tianren, M. et al. Overcomplete-to-sparse representation learning for few-shot class-incremental learning. Multimedia Systems 30, 102 (2024). https://doi.org/10.1007/s00530-024-01294-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01294-z

Keywords

Search

Navigation