Skip to main content

Advertisement

SpringerLink
Log in

A Multi-objective Optimization Model for Redundancy Reduction in Convolutional Neural Networks

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Until now, convolutional neural networks (CNNs) still among the powerful and robust deep neural networks that proved its efficiency through several real applications. However, their functioning requires a large number of parameters which in turn lead to some undesired effects such as the overparametrization, overfitting and the high consumption of computational resources. To deal effectively with these issues, we propose in this paper a new multi-objective optimization model for redundancy reduction in CNNs. The suggested model named MoRR-CNN allows to eliminate the unwanted parameters (kernels and weights) as well as to speeding up the CNN evaluation process. It consists of two objectives, the first one is related to the training task where the solution is the optimal parameters. These parameters are combined with a set of decision variables that controlling their contribution in the training process, making at the end a redundancy-related objective function. Both of the objectives are optimized using the non dominated sorting genetic algorithm NSGA-II. The robustness of MoRR-CNN has been demonstrated through different experimentation applied on three benchmark datasets including MNIST, Fashion-MNIST and CIFAR and using three of the most known CNNs such as VGG-19, Net-in-Net and VGG-16.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  2. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

    Article  Google Scholar 

  3. Baldeon-Calisto M, Lai-Yuen SK (2020) Adaresu-net: multiobjective adaptive convolutional neural network for medical image segmentation. Neurocomputing 392:325–340

    Article  Google Scholar 

  4. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: international conference on medical image computing and computer-assisted intervention, pp. 234–241. Springer

  5. Huang W, Zhang L, Wu H, Min F, Song A (2022) Channel-equalization-har: a light-weight convolutional neural network for wearable sensor based human activity recognition. IEEE Trans Mobile Comput. https://doi.org/10.1109/TMC.2022.3174816

    Article  Google Scholar 

  6. Huang W, Zhang L, Wang S, Wu H, Song A (2022) Deep ensemble learning for human activity recognition using wearable sensors via filter activation. ACM Trans Embedded Comput Syst 22(1):1–23

    Article  Google Scholar 

  7. Hssayni Eh, Joudar N-E, Ettaouil M (2022) A deep learning framework for time series classification using normal cloud representation and convolutional neural network optimization. Comput Intell 38(6):2056–2074

    Article  Google Scholar 

  8. Huang W, Zhang L, Teng Q, Song C, He J (2021) The convolutional neural networks training with channel-selectivity for human activity recognition based on sensors. IEEE J Biomed Health Inform 25(10):3834–3843

    Article  Google Scholar 

  9. Zhang Y, Sohn K, Villegas R, Pan G, Lee H (2015) Improving object detection with deep convolutional networks via bayesian optimization and structured prediction. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp. 249–258

  10. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 60(6):84–90

    Google Scholar 

  11. Xiong W, Droppo J, Huang X, Seide F, Seltzer ML, Stolcke A, Yu D, Zweig G (2017) Toward human parity in conversational speech recognition. IEEE/ACM Trans Audio Speech Lang Process 25(12):2410–2423

    Article  Google Scholar 

  12. Hssayni EH, Joudar N-E, Ettaouil M (2022) An adaptive drop method for deep neural networks regularization: estimation of dropconnect hyperparameter using generalization gap. Knowl Based Syst 253:109567

    Article  Google Scholar 

  13. Denil M, Shakibi B, Dinh L, Ranzato M, De Freitas N (2013) Predicting parameters in deep learning. In: advances in neural information processing systems, vol 26

  14. Ma R, Miao J, Niu L, Zhang P (2019) Transformed l 1 regularization for learning sparse deep neural networks. Neural Netw 119:286–298

    Article  MATH  Google Scholar 

  15. Xu Q, Zhang M, Gu Z, Pan G (2019) Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs. Neurocomputing 328:69–74

    Article  Google Scholar 

  16. Hu H, Peng R, Tai Y-W, Tang C-K (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250

  17. Kim Y-D, Park E, Yoo S, Choi T, Yang L, Shin D (2015) Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530

  18. Ding H, Chen K, Yuan Y, Cai M, Sun L, Liang S, Huo Q (2017) A compact CNN-DBLSTM based character model for offline handwriting recognition with tucker decomposition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol. 1, pp. 507–512. IEEE

  19. Lebedev V, Ganin Y, Rakhuba M, Oseledets I, Lempitsky V (2014) Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553

  20. Denton EL, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In: advances in neural information processing systems, vol 27

  21. Hssayni EH, Joudar N-E, Ettaouil M (2022) KRR-CNN: kernels redundancy reduction in convolutional neural networks. Neural Comput Appl 34(3):2443–2454

    Article  Google Scholar 

  22. Hssayni EH, Joudar N-E, Ettaouil M (2022) Localization and reduction of redundancy in CNN using l1-sparsity induction. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-022-04025-2

    Article  Google Scholar 

  23. Reiners M, Klamroth K, Heldmann F, Stiglmayr M (2022) Efficient and sparse neural networks by pruning weights in a multiobjective learning approach. Comput Oper Res 141:105676

    Article  MathSciNet  MATH  Google Scholar 

  24. Huang J, Sun W, Huang L (2020) Deep neural networks compression learning based on multiobjective evolutionary algorithms. Neurocomputing 378:260–269

    Article  Google Scholar 

  25. Guo Y, Chen G, Jiang M, Gong D, Liang J (2022) A knowledge guided transfer strategy for evolutionary dynamic multiobjective optimization. IEEE Trans Evol Comput. https://doi.org/10.1109/TEVC.2022.322284

    Article  Google Scholar 

  26. Chen G, Guo Y, Huang M, Gong D, Yu Z (2022) A domain adaptation learning strategy for dynamic multiobjective optimization. Inf Sciences 606:328–349

    Article  Google Scholar 

  27. Deb K, Agrawal S, Pratap A, Meyarivan T (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In: International Conference on Parallel Problem Solving from Nature, pp. 849–858. Springer

  28. Calin O (2020) Deep learning architectures. Springer, New York

    Book  MATH  Google Scholar 

  29. LeCun Y, Boser B, Denker J, Henderson D, Howard R, Hubbard W, Jackel L (1989) Handwritten digit recognition with a back-propagation network. In: Advances in neural information processing systems, vol 2

  30. Ranzato M, Boureau Y-L, Cun Y et al (2007) Sparse feature learning for deep belief networks. In: Advances in neural information processing systems, vol 20

  31. Collette Y, Siarry P (2011) Optimisation multiobjectif: algorithmes. Editions Eyrolles, Paris

    Google Scholar 

  32. Miettinen K (2012) Nonlinear multiobjective optimization. Springer, New York

    MATH  Google Scholar 

  33. Holland JH (1992) Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press, Cambridge

    Book  Google Scholar 

  34. Srinivas N, Deb K (1994) Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary computation 2(3):221–248

    Article  Google Scholar 

  35. Zitzler E, Deb K, Thiele L (2000) Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation 8(2):173–195

    Article  Google Scholar 

  36. Zitzler E, Thiele L (1998) Multiobjective optimization using evolutionary algorithms-a comparative case study. In: international conference on parallel problem solving from nature, pp. 292–301. Springer

  37. Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: proceedings of COMPSTAT’2010, pp. 177–186. Springer

  38. Hoseini F, Shahbahrami A, Bayat P (2019) Adaptahead optimization algorithm for learning deep CNN applied to MRI segmentation. J Digital Imaging 32(1):105–115

    Article  Google Scholar 

  39. Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400

  40. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  41. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

  42. Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images

  43. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  44. Guo Y-N, Zhang X, Gong D-W, Zhang Z, Yang J-J (2019) Novel interactive preference-based multiobjective evolutionary optimization for bolt supporting networks. IEEE Trans Evol Comput 24(4):750–764

    Article  Google Scholar 

  45. Ji J-J, Guo Y-N, Gao X-Z, Gong D-W, Wang Y-P (2021) Q-learning-based hyperheuristic evolutionary algorithm for dynamic task allocation of crowdsensing. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2021.3112675

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, FST of Fez, Sidi Mohamed Ben Abdellah University, Fez, Morocco

    Ali Boufssasse & Mohamed Ettaouil

  2. ENSIAS, Mohammed V University in Rabat, Rabat, Morocco

    El houssaine Hssayni

  3. ENSAM, Mohammed V University in Rabat, Rabat, Morocco

    Nour-Eddine Joudar

Authors
  1. Ali Boufssasse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. El houssaine Hssayni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nour-Eddine Joudar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Ettaouil
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to El houssaine Hssayni.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Boufssasse, A., Hssayni, E.h., Joudar, NE. et al. A Multi-objective Optimization Model for Redundancy Reduction in Convolutional Neural Networks. Neural Process Lett 55, 9721–9741 (2023). https://doi.org/10.1007/s11063-023-11223-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11223-2

Keywords

Search

Navigation