Skip to main content
SpringerLink
Log in

FP-DCNN: a parallel optimization algorithm for deep convolutional neural network

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Deep convolutional neural networks (DCNNs) have been successfully used in many computer visions task. However, with the increasing complexity of the network and continuous growth of data scale, training a DCNN model suffers from the following three problems: excessive network parameters, insufficient capability of the parameter optimization, and inefficient parallelism. To overcome these obstacles, this paper develops an optimization algorithm for deep convolutional neural network (FP-DCNN) in the MapReduce framework. First, a pruning method based on Taylor’s loss (FMPTL) is designed to trim redundant parameters, which not only compresses the structure of DCNN, but also reduces the computational cost of training. Next, a glowworm swarm optimization algorithm based on information sharing strategy (IFAS) is presented, which improves the ability of parameter optimization by adjusting the initialization of weights. Finally, a dynamic load balancing strategy based on parallel computing entropy (DLBPCE) is proposed to achieve an even distribution of data and thus improve the parallel performance of the cluster. Our experiments show that compared with other parallelized algorithms, this algorithm not only reduces the computational cost of network training, but also obtains a higher processing speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Hariri RH, Fredericks EM, Bowers KM (2019) Uncertainty in big data analytics: survey, opportunities, and challenges. Journal of Big Data 6:1–16

    Article  Google Scholar 

  2. Sewak M, Sahay SK, Rathore H (2020) An Overview of Deep Learning Architecture of Deep Neural Networks and Autoencoders. J Comput Theor Nanosci 17:182–188

    Article  Google Scholar 

  3. Dong, S., Wang, P., & Abbas, K. (2021). A survey on deep learning and its applications. Computer Science Review, 40, 100379

  4. Maheshwaram S (2019) A Review on Deep Convolutional Neural Network and its Applications. International Journal of Advanced Research in Computer and Communication Engineering 8:174–179

    Article  Google Scholar 

  5. Fredj, H.B., Bouguezzi, S., & Souani, C. (2020). Face recognition in unconstrained environment with CNN. The Visual Computer, 1–10

  6. Tabernik D, Skočaj D (2020) Deep Learning for Large-Scale Traffic-Sign Detection and Recognition. IEEE Trans Intell Transp Syst 21:1427–1440

    Article  Google Scholar 

  7. Cheng G, Zhou P, Han J (2016) Learning Rotation-Invariant Convolutional Neural Networks for Object Detection in VHR Optical Remote Sensing Images. IEEE Trans Geosci Remote Sens 54:7405–7415

    Article  Google Scholar 

  8. Jha S, Dey A, Kumar R, Solanki VK (2019) A Novel Approach on Visual Question Answering by Parameter Prediction using Faster Region Based Convolutional Neural Network. Int J Interact Multim Artif Intell 5:30–37

    Google Scholar 

  9. Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both Weights and Connections for Efficient Neural Network. https://arxiv.org/abs/1506.02626

  10. Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. (2017). Pruning Filters for Efficient ConvNets. https://arxiv.org/abs/1608.08710

  11. He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018). Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. IJCAI

  12. Zhuo, H., Qian, X., Fu, Y., Yang, H., & Xue, X. (2018). SCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners. https://arxiv.org/abs/1806.05320

  13. Ye, J., Lu, X., Lin, Z.L., & Wang, J.Z. (2018). Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers. https://arxiv.org/abs/1802.00124

  14. Chang, J., Lu, Y., Xue, P., Xu, Y., & Wei, Z. (2021). ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN. https://arxiv.org/abs/2101.06407

  15. Hidri, A. (2018). Optimization for training CNN deep models based on swarm intelligence. 2018 International Conference on Advanced Systems and Electric Technologies (IC_ASET), 284–289

  16. Anaraki AK, Ayati M, Kazemi F (2019) Magnetic resonance imaging-based brain tumor grades classification and grading via convolutional neural networks and genetic algorithms. Biocybernetics and Biomedical Engineering 39:63–74

    Article  Google Scholar 

  17. Banharnsakun A (2019) Towards improving the convolutional neural networks for deep learning using the distributed artificial bee colony method. Int J Mach Learn Cybern 10:1301–1311

    Article  Google Scholar 

  18. Zeng K, Ding S, Jia W (2018) Single image super-resolution using a polymorphic parallel CNN. Appl Intell 49:292–300

    Article  Google Scholar 

  19. Lee S, Kang Q, Madireddy S, Balaprakash P, Agrawal A, Choudhary A, Archibald R, Liao W (2019) Improving Scalability of Parallel CNN Training by Adjusting Mini-Batch Size at Run-Time. IEEE International Conference on Big Data 2019:830–839

    Google Scholar 

  20. Basit N, Zhang Y, Wu H, Liu H, Bin J, He Y, Hendawi AM (2016) MapReduce-based deep learning with handwritten digit recognition case study. IEEE International Conference on Big Data 2016:1690–1699

    Google Scholar 

  21. Maleki, N., Rahmani, A., & Conti, M. (2019). MapReduce: an infrastructure review and research insights. The Journal of Supercomputing, 1–69

  22. El-Ajou A, Arqub OA, Al-Smadi M (2015) A general form of the generalized Taylor’s formula with some applications. Appl Math Comput 256:851–859

    MathSciNet  MATH  Google Scholar 

  23. Rashmita Gupta, R.K.Bayal. (2020). A REVIEW ON GLOWWORM SWARM OPTIMIZATION TECHNIQUES[J]. Journal of Critical Reviews. Vol.7(No.11):3686–3694

  24. Dong X, Chen C, Geng Q, Zhang W, Zhang X (2021) Fast Algorithm Based on Parallel Computing for Sample Entropy Calculation. IEEE Access 9:20223–20234

    Article  Google Scholar 

  25. Yan C, Zhou L, Wan Y (2019) A Multi-Task Learning Model for Better Representation of Clothing Images. IEEE Access 7:34499–34507

    Article  Google Scholar 

  26. Chen, C., Reiz, S., Yu, C.D., Bungartz, H., & Biros, G. (2019). Fast Evaluation and Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron. https://arxiv.org/abs/1910.12184

  27. Giuste, F., & Vizcarra, J.C. (2020). CIFAR-10 Image Classification Using Feature Ensembles. https://arxiv.org/abs/2002.03846

  28. Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. https://arxiv.org/abs/1708.07747

  29. Litjens G, Kooi T, Bejnordi BE, Setio A, Ciompi F, Ghafoorian M, Laak JV, Ginneken B, Sánchez C (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  30. Cohen, G., Afshar, S., Tapson, J., & Schaik, A.V. (2017). EMNIST: an extension of MNIST to handwritten letters. https://arxiv.org/abs/1702.05373

Download references

Funding

This study was supported by the National Natural Science Foundation of China (41562019), the National Key Research and Development Program of China (2018YFC1504705) and the Natural Science Foundation of Jiangxi Province (2018BAB202004).

Author information

Authors and Affiliations

  1. School of Information Engineering, Jiangxi University of Science and Technology, Ganzhou, Jiangxi, China

    Ye Le, Y. A. Nanehkaran, Deborah Simon Mwakapesa, Ruipeng Zhang, Jianbing Yi & Yimin Mao

Authors
  1. Ye Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Y. A. Nanehkaran
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Deborah Simon Mwakapesa
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruipeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianbing Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yimin Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yimin Mao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Le, Y., Nanehkaran, Y.A., Mwakapesa, D.S. et al. FP-DCNN: a parallel optimization algorithm for deep convolutional neural network. J Supercomput 78, 3791–3813 (2022). https://doi.org/10.1007/s11227-021-04012-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-04012-y

Keywords

Search

Navigation