Skip to main content
SpringerLink
Log in

Fast and Robust Compression of Deep Convolutional Neural Networks

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2020 (ICANN 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12397))

Included in the following conference series:

Abstract

Deep convolutional neural networks (CNNs) currently demonstrate the state-of-the-art performance in several domains. However, a large amount of memory and computing resources are required in the commonly used CNN models, posing challenges in training as well as deploying, especially on those devices with limited computational resources. Inspired by the recent advancement of random tensor decomposition, we introduce a Hierarchical Framework for Fast and Robust Compression (HFFRC), which significantly reduces the number of parameters needed to represent a convolution layer via a fast low-rank Tucker decomposition algorithm, while preserving its expressive power. In the merit of randomized algorithm, the proposed compression framework is robust to noises in parameters. In addition, it is a general framework that any tensor decomposition method can be easily adopted. The efficiency and effectiveness of the proposed approach have been demonstrated via comprehensive experiments conducted on the benchmarks CIFAR-10 and CIFAR-100 image classification datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Astrid, M., Lee, S.I.: CP-decomposition with tensor power method for convolutional neural networks compression. In: 2017 IEEE International Conference on Big Data and Smart Computing, pp. 115–118. IEEE (2017). https://doi.org/10.1109/BIGCOMP.2017.7881725

  2. Ba, J., Caruana, R.: Do deep nets really need to be deep? In: Advances in Neural Information Processing Systems, pp. 2654–2662 (2014)

    Google Scholar 

  3. Balevi, E., Andrews, J.G.: Autoencoder-based error correction coding for one-bit quantization. IEEE Trans. Commun. (2020). https://doi.org/10.1109/tcomm.2020.2977280

    Article  Google Scholar 

  4. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009). https://doi.org/10.1109/CVPR.2009.5206848

  5. Denil, M., Shakibi, B., Dinh, L., Ranzato, M., De Freitas, N.: Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, pp. 2148–2156 (2013). https://doi.org/10.14288/1.0165555

  6. Erichson, N.B., Manohar, K., Brunton, S.L., Kutz, J.N.: Randomized CP tensor decomposition. arXiv preprint arXiv:1703.09074 (2017)

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  8. He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1389–1397 (2017). https://doi.org/10.1109/iccv.2017.155

  9. Kim, Y.D., Park, E., Yoo, S., Choi, T., Yang, L., Shin, D.: Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 (2015)

  10. Krizhevsky, A., et al.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)

    Google Scholar 

  11. Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., Lempitsky, V.: Speeding-up convolutional neural networks using fine-tuned CP-decomposition. arXiv preprint arXiv:1412.6553 (2014)

  12. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)

  13. Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: Sphereface: deep hypersphere embedding for face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 212–220 (2017). https://doi.org/10.1109/cvpr.2017.713

  14. Luo, J.H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017). https://doi.org/10.1109/ICCV.2017.541

  15. Ma, Y., et al.: A unified approximation framework for compressing and accelerating deep neural networks. In: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 376–383. IEEE (2019). https://doi.org/10.1109/ICTAI.2019.00060

  16. Minster, R., Saibaba, A.K., Kilmer, M.E.: Randomized algorithms for low-rank tensor decompositions in the tucker format. SIAM J. Math. Data Sci. 2(1), 189–215 (2020). https://doi.org/10.1137/19m1261043

    Article  MathSciNet  Google Scholar 

  17. Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11264–11272 (2019). https://doi.org/10.1109/CVPR.2019.01152

  18. Nakajima, S., Sugiyama, M., Babacan, S.D., Tomioka, R.: Global analytic solution of fully-observed variational bayesian matrix factorization. J. Mach. Learn. Res. 14, 1–37 (2013). https://doi.org/10.1016/j.imavis.2012.11.001

  19. Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  20. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018). https://doi.org/10.1109/CVPR.2018.00474

  21. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  22. Tan, M., Le, Q.V.: Efficientnet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019)

  23. Tjandra, A., Sakti, S., Nakamura, S.: Compressing recurrent neural network with tensor train. In: 2017 International Joint Conference on Neural Networks, pp. 4451–4458. IEEE (2017). https://doi.org/10.1109/ijcnn.2017.7966420

  24. Wang, F., et al.: Residual attention network for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2017). https://doi.org/10.1109/CVPR.2017.683

  25. Wang, K., Liu, Z., Lin, Y., Lin, J., Han, S.: Haq: hardware-aware automated quantization with mixed precision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612–8620 (2019). https://doi.org/10.1109/CVPR.2019.00881

  26. Wang, W., Sun, Y., Eriksson, B., Wang, W., Aggarwal, V.: Wide compression: tensor ring nets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9329–9338 (2018). https://doi.org/10.1109/cvpr.2018.00972

  27. Wang, Y., Tung, H.Y., Smola, A.J., Anandkumar, A.: Fast and guaranteed tensor decomposition via sketching. In: Advances in Neural Information Processing Systems, pp. 991–999 (2015)

    Google Scholar 

  28. Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9378–9387 (2018). https://doi.org/10.1109/cvpr.2018.00977

  29. Yuan, L., Li, C., Cao, J., Zhao, Q.: Randomized tensor ring decomposition and its application to large-scale data reconstruction. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2127–2131 (2019). https://doi.org/10.1109/ICASSP.2019.8682197

  30. Zhou, G., Cichocki, A., Xie, S.: Decomposition of big tensors with low multilinear rank. arXiv preprint arXiv:1412.1885 (2014)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61732011 and Grant 61702358, in part by the Beijing Natural Science Foundation under Grant Z180006, in part by the Key Scientific and Technological Support Project of Tianjin Key Research and Development Program under Grant 18YFZCGX00390, and in part by the Tianjin Science and Technology Plan Project under Grant 19ZXZNGX00050.

Author information

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, 300350, China

    Jia Wen & Liu Yang

  2. Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, 75390, USA

    Chenyang Shen

Authors
  1. Jia Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenyang Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liu Yang .

Editor information

Editors and Affiliations

  1. Department of Applied Informatics, Comenius University in Bratislava, Bratislava, Slovakia

    Igor Farkaš

  2. Department of Applied Mathematics and Computer Science, Technical University of Denmark, Kgs. Lyngby, Denmark

    Paolo Masulli

  3. Department of Informatics, University of Hamburg, Hamburg, Germany

    Stefan Wermter

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wen, J., Yang, L., Shen, C. (2020). Fast and Robust Compression of Deep Convolutional Neural Networks. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12397. Springer, Cham. https://doi.org/10.1007/978-3-030-61616-8_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61616-8_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61615-1

  • Online ISBN: 978-3-030-61616-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation