Skip to main content
SpringerLink
Log in

Pruning filters with L1-norm and capped L1-norm for CNN compression

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The blistering progress of convolutional neural networks (CNNs) in numerous applications of the real-world usually obstruct by a surge in network volume and computational cost. Recently, researchers concentrate on eliminating these issues by compressing the CNN models, such as pruning filters and weights. In comparison with the technique of pruning weights, the technique of pruning filters doesn’t effect in sparse connectivity patterns. In this article, we have proposed a fresh new technique to estimate the significance of filters. More precisely, we combined L1-norm with capped L1-norm to represent the amount of information extracted by the filter and control regularization. In the process of pruning, the insignificant filters remove directly without any loss in the test accuracy, providing much slimmer and compact models with comparable accuracy and this process is iterated a few times. To validate the effectiveness of our algorithm. We experimentally determine the usefulness of our approach with several advanced CNN models on numerous standard data sets. Particularly, data sets CIFAR-10 is used on VGG-16 and prunes 92.7% parameters with float-point-operations (FLOPs) reduction of 75.8% without loss of accuracy and has achieved advancement in state-of-art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778

  2. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 580–587

  3. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3431–3440

  4. Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision. pp 1520–1528

  5. Wu J, Leng C, Wang Y, et al (2016) Quantized convolutional neural networks for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 4820–4828

  6. Kadav A, Durdanovic I, Graf HP (2018) Pruning filters for efficient convolutional neural networks for image recognition in surveillance applications. Google Patents

  7. LeCun Y, Denker JS, Solla SA (1990) Optimal brain damage. In: Advances in neural information processing systems. pp 598–605

  8. Hassibi B, Stork DG (1993) Second order derivatives for network pruning: optimal brain surgeon. In: Advances in neural information processing systems. pp 164–171

  9. Mariet Z, Sra S (2015) Diversity networks: neural network compression using determinantal point processes. ArXiv Prepr ArXiv151105077

  10. Han S, Mao H, Dally WJ (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. ArXiv Prepr ArXiv151000149

  11. Li H, Kadav A, Durdanovic I, et al (2016) Pruning filters for efficient convnets. ArXiv Prepr ArXiv160808710

  12. He Y, Lin J, Liu Z, et al (2018) Amc: Automl for model compression and acceleration on mobile devices. In: Proceedings of the European conference on computer vision (ECCV). p 784–800

  13. Luo J-H, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: Proceedings of the IEEE international conference on computer vision. pp 5058–5066

  14. Molchanov P, Tyree S, Karras T, et al (2016) Pruning convolutional neural networks for resource efficient inference. ArXiv Prepr ArXiv161106440

  15. Jin X, Yuan X, Feng J, Yan S (2016) Training skinny deep neural networks with iterative hard thresholding methods. ArXiv Prepr ArXiv160705423

  16. Li Y, Wang L, Peng S, Kumar A, Yin B (2019) Using feature entropy to guide filter pruning for efficient convolutional networks. In: Tetko IV, Kůrková V, Karpov P, Theis F (eds) Artificial neural networks and machine learning – ICANN 2019: deep learning. Springer International Publishing, Cham, pp 263–274

    Chapter  Google Scholar 

  17. Hu H, Peng R, Tai Y-W, Tang C-K (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. CoRR abs/1607.03250

  18. Yu X, Liu T, Wang X, Tao D (2017) On compressing deep models by low rank and sparse decomposition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 7370–7379

  19. Jia K, Tao D, Gao S, Xu X (2017) Improving training of deep neural networks via singular value bounding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 4344–4352

  20. Chen W, Wilson J, Tyree S, et al (2015) Compressing neural networks with the hashing trick. In: International Conference on Machine Learning. pp 2285–2294

  21. Zhan Z, Gong Y, Li Z, et al (2020) A privacy-preserving DNN pruning and Mobile acceleration framework. ArXiv Prepr ArXiv200306513

  22. Zoph B, Le QV (2016) Neural architecture search with reinforcement learning. ArXiv Prepr ArXiv161101578

  23. Hüttenrauch M, Šošić A, Neumann G (2019) Deep reinforcement learning for swarm systems. J Mach Learn Res 20:1–31

    MathSciNet  MATH  Google Scholar 

  24. de Bruin T, Kober J, Tuyls K, Babuška R (2018) Experience selection in deep reinforcement learning for control. J Mach Learn Res 19:1–56

    MathSciNet  MATH  Google Scholar 

  25. Liu X, Zhu X, Li M et al (2019) Multiple kernel $ k $ k-means with incomplete kernels. IEEE Trans Pattern Anal Mach Intell 42:1191–1204

    Google Scholar 

  26. Yu X, Ye X, Gao Q (2020) Infrared handprint image restoration algorithm based on apoptotic mechanism. IEEE Access 8:47334–47343

    Article  Google Scholar 

  27. You Z, Yan K, Ye J, et al (2019) Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems. pp 2133–2144

  28. Garg I, Panda P, Roy K (2019) A low effort approach to structured cnn design using pca. IEEE Access 8:1347–1360

    Article  Google Scholar 

  29. Lee N, Ajanthan T, Torr PH (2018) Snip: single-shot network pruning based on connection sensitivity. ArXiv Prepr ArXiv181002340

  30. He Y, Kang G, Dong X, et al (2018) Soft filter pruning for accelerating deep convolutional neural networks. CoRR abs/1808.06866

  31. Roy S, Panda P, Srinivasan G, Raghunathan A (2020) Pruning filters while training for efficiently optimizing deep learning networks. ArXiv Prepr ArXiv200302800

  32. Yue L, Weibin Z, Lin S (2019) Really should we pruning after model be totally trained? Pruning based on a small amount of training. ArXiv Prepr ArXiv190108455

  33. Scardapane S, Comminiello D, Hussain A, Uncini A (2017) Group sparse regularization for deep neural networks. Neurocomputing 241:81–89

    Article  Google Scholar 

  34. Zhong J, Ding G, Guo Y, et al (2018) Where to prune: using LSTM to guide end-to-end pruning. In: IJCAI. pp 3205–3211

  35. Sun X, Zhou D, Pan X, et al (2019) Pruning filters with L1-norm and standard deviation for CNN compression. In: Eleventh international conference on machine vision (ICMV 2018). International Society for Optics and Photonics, p 110412J

  36. Xu Z, Huang G, Weinberger KQ, Zheng AX (2014) Gradient boosted feature selection. In: proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 522–531

  37. Tuv E, Borisov A, Runger G, Torkkola K (2009) Feature selection with ensembles, artificial variables, and redundancy elimination. J Mach Learn Res 10:1341–1366

    MathSciNet  MATH  Google Scholar 

  38. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. ArXiv Prepr ArXiv14091556

  39. Gonzalez TF (2007) Approximation algorithms for multilevel graph partitioning. In: Handbook of Approximation Algorithms and Metaheuristics. Chapman and Hall/CRC, pp 943–958

  40. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 4700–4708

  41. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp 1026–1034

  42. Liu Z, Li J, Shen Z, et al (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision. pp 2736–2744

  43. Aketi SA, Roy S, Raghunathan A, Roy K (2020) Gradual Channel pruning while training using feature relevance scores for convolutional neural networks. ArXiv Prepr ArXiv200209958

  44. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1389–1397

Download references

Author information

Authors and Affiliations

  1. Department of Automation, University of Science and Technology of China, Hefei, 230026, People’s Republic of China

    Aakash Kumar, Ali Muhammad Shaikh, Yun Li, Hazrat Bilal & Baoqun Yin

Authors
  1. Aakash Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Muhammad Shaikh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hazrat Bilal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Baoqun Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aakash Kumar.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, A., Shaikh, A.M., Li, Y. et al. Pruning filters with L1-norm and capped L1-norm for CNN compression. Appl Intell 51, 1152–1160 (2021). https://doi.org/10.1007/s10489-020-01894-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01894-y

Keywords

Search

Navigation