Skip to main content
SpringerLink
Log in

Advanced metaheuristic optimization techniques in applications of deep neural networks: a review

  • Review Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Deep neural networks (DNNs) have evolved as a beneficial machine learning method that has been successfully used in various applications. Currently, DNN is a superior technique of extracting information from massive sets of data in a self-organized method. DNNs have different structures and parameters, which are usually produced for particular applications. Nevertheless, the training procedures of DNNs can be protracted depending on the given application and the size of the training set. Further, determining the most precise and practical structure of a deep learning method in a reasonable time is a possible problem related to this procedure. Meta-heuristics techniques, such as swarm intelligence (SI) and evolutionary computing (EC), represent optimization frames with specific theories and objective functions. These methods are adjustable and have been demonstrated their effectiveness in various applications; hence, they can optimize the DNNs models. This paper presents a comprehensive survey of the recent optimization methods (i.e., SI and EC) employed to enhance DNNs performance on various tasks. This paper also analyzes the importance of optimization methods in generating the optimal hyper-parameters and structures of DNNs in taking into consideration massive-scale data. Finally, several potential directions that still need improvements and open problems in evolutionary DNNs are identified.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. DynamicNet is implemented as one of the Python library models: https://pypi.org/project/convtt/.

References

  1. McCulloch Warren S, Walter P (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133

    Article  MathSciNet  MATH  Google Scholar 

  2. Frank R (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65(6):386

    Article  Google Scholar 

  3. Rumelhart David E, Hinton Geoffrey E, Williams Ronald J (1985) Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science

  4. Khotimah C, Purnami SW, Prastyo DD, Chosuvivatwong V, Spriplung H (1905) Additive survival least square support vector machines: a simulation study and its application to cervical cancer prediction. In: Proceedings of the 13th IMT-GT International Conference on Mathematics, Statistics and their Applications (ICMSA), AIP Conference Proceedings

  5. Yann LC, Yoshua B, Geoffrey H (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  6. Geoffrey H (2018) Deep learning-a technology with the potential to transform health care. Jama 320(11):1101–1102

    Article  Google Scholar 

  7. Krizhevsky A, Sutskever I, Hinton GE (2012). 2012 alexnet. Adv Neural Inf Process Syst, pp 1–9

  8. Zhong Z, Jin L, Xie Z (2015) High performance offline handwritten chinese character recognition using googlenet and directional feature maps. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp 846–850. IEEE

  9. Targ S, Almeida D, Lyman K (2016) Resnet in resnet: Generalizing residual architectures. arXiv preprintarXiv: 1603.08029,

  10. Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T, Keutzer K (2014) Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprintarXiv: 1404.1869

  11. Mingxing T (1905) Le Quoc V (2019) Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv, Efficientnet, p 11946

  12. Faruk E (2020) A novel clustering method built on random weight artificial neural networks and differential evolution. Soft Comput, pp 1–12

  13. Laith A, Gandomi Amir H, Abd EM, Al HH, Mahmoud O, Mohammad A, Khasawneh Ahmad M (2021) Advances in meta-heuristic optimization algorithms in big data text clustering. Electronics 10(2):101

    Article  Google Scholar 

  14. Laith A, Gandomi Amir H, Abd EM, Hussien Abdelazim G, Khasawneh Ahmad M, Mohammad A, Houssein Essam H (2020) Nature-inspired optimization algorithms for text document clustering–a comprehensive analysis. Algorithms 13(12):345

    Article  MathSciNet  Google Scholar 

  15. Nigam A, Friederich P, Krenn M, Aspuru-Guzik A (2019) Augmenting genetic algorithms with deep neural networks for exploring the chemical space. arXiv preprint. arXiv: 1909.11655

  16. Shukla P, Kumar H, Nandi GC (2020) Robotic grasp manipulation using evolutionary computing and deep reinforcement learning. arXiv preprint. arXiv: 2001.05443

  17. Yuan Y, Fuchun S, Huaping L, Hongjiu Y (2014) Low-frequency robust control for singularly perturbed system. IET Control Theory Appl 9(2):203–210

    Article  MathSciNet  Google Scholar 

  18. Thomas W, Helmut B (2017) A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Trans Inf Theory 64(3):1845–1866

    MathSciNet  MATH  Google Scholar 

  19. van den Oord A, Dieleman S, Zen H, Simonyan K, Vinyals O, Graves A, Kalchbrenner N, Senior A, Koray K (2016). Wavenet A generative model for raw audio. arXiv preprint. arXiv: 1609.03499,

  20. Kumar K, Kumar R, de Boissiere T, Gestin L, Zhen TW, Sotelo J, de Brébisson A, Bengio Y, Courville AC (2019) Melgan: Generative adversarial networks for conditional waveform synthesis. In: Advances in Neural Information Processing Systems, pp 14910–14921

  21. Andrew H, Mark S, Grace C, Liang-Chieh C, Bo C, Mingxing T, Weijun W, Yukun Z, Ruoming P, Vijay V et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1314–1324

  22. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint. arXiv: 1409.1556

  23. Kaiming H, Xiangyu Z, Shaoqing R, Jian S (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  24. Xie S, Girshick Ross, Dollár Piotr, Tu Zhuowen , He Kaiming (2017) Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1492–1500

  25. Foret P, Kleiner A, Mobahi H, Neyshabur B (2020) Sharpness-aware minimization for efficiently improving generalization. arXiv preprint. arXiv: 2010.01412

  26. Szegedy C, Liu W , Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  27. Soria PX, Riba E, Sappa A (2020) Dense extreme inception network: Towards a robust cnn model for edge detection. In: The IEEE Winter Conference on Applications of Computer Vision, pp 1923–1932

  28. Huang G, Liu Z, Van Der Maaten L, Weinberger Kilian Q (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  29. Iandola Forrest N, Han Song, Moskewicz Matthew W, Ashraf Khalid, Dally William J , Keutzer Kurt (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and> 0.5 mb model size. arXiv preprint. arXiv: 1602.07360

  30. Howard Andrew G, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint. arXiv: 1704.04861

  31. Ma N, Zhang X, Zheng H-T, Sun J (2018) Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131

  32. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

  33. Bochkovskiy A, Wang C-Y, Mark Liao H-Y (2020) Yolov4: Optimal speed and accuracy of object detection. arXiv preprint. arXiv: 2004.10934

  34. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  35. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg Alexander C (2016) Ssd: Single shot multibox detector. In: European conference on computer vision, pp 21–37. Springer

  36. Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

  37. Merity S (2019) Single headed attention rnn: Stop thinking with your head. arXiv preprint.arXiv: 1911.11423

  38. Kalchbrenner N, Elsen E, Simonyan K, Noury S, Casagrande N, Lockhart E, Stimberg F, van den Oord A, Dieleman S, Kavukcuoglu K (2018) Efficient neural audio synthesis. arXiv preprint. arXiv: 1802.08435

  39. Melis G, Kočiskỳ T, Blunsom P (2019) Mogrifier lstm. arXiv preprint.arXiv: 1909.01792

  40. Ning J, Jiaxian W, Xiang M, Ke Y, Yuchang M (2020) Multi-task learning model based on multi-scale cnn and lstm for sentiment classification. IEEE Access 8:77060–77072

    Article  Google Scholar 

  41. Ganapathy K (2020) A study of genetic algorithms for hyperparameter optimization of neural networks in machine translation. arXiv preprint. arXiv: 2009.08928

  42. Jingzhao Z, Tianxing H, Suvrit S (1905) Jadbabaie A (2019) A theoretical justification for adaptivity. arXiv preprint arXiv, Why gradient clipping accelerates training, p 11881

  43. Young T, Hazarika D, Poria S, Cambria E (2018) Recent trends in deep learning based natural language processing. IEEE Comput Intell Magazine 13(3):55–75

    Article  Google Scholar 

  44. Kumar SA, Sandeep C, Kumar SD (2020) Sentimental short sentences classification by using cnn deep learning model with fine tuned word2vec. Proc Comput Sci 167:1139–1147

    Article  Google Scholar 

  45. Ashish V, Noam S, Niki P, Jakob U, Llion J, Aidan NG, Łukasz K, Illia P (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  46. Shoeybi M, Patwary M, Puri R , LeGresley P, Casper J, Catanzaro B (2019) Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint. arXiv: 1909.08053

  47. Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint. arXiv: 1810.04805

  48. Liu Y, Ott M , Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint. arXiv: 1907.11692

  49. Sanh V, Debut L, Chaumond J, Wolf T (2019) Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint. arXiv: 1910.01108

  50. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2019) Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint. arXiv: 1910.13461

  51. Raffel C., Shazeer N., Roberts A., Lee K., Narang S., Matena M., Zhou Y., Li W., Liu Peter J (2019) Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint. arXiv: 1910.10683

  52. Nikita K, Łukasz K (2001) Levskaya Anselm (2020). The efficient transformer. arXiv preprint arXiv, Reformer, p 04451

  53. Brown Tom B, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A et al (2020) Language models are few-shot learners. arXiv preprint. arXiv: 2005.14165

  54. Shin H-C, Zhang Y, Bakhturina E, Puri R, Patwary M, Shoeybi M, Mani R (2020) Biomegatron: Larger biomedical domain language model. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 4700–4706

  55. Jimmy L, Rodrigo N (2010) Yates Andrew (2020). Bert and beyond. arXiv preprint arXiv, Pretrained transformers for text ranking, p 06467

  56. Liu X, Duh K, Liu L, Gao J (2020) Very deep transformers for neural machine translation. arXiv preprint. arXiv: 2008.07772

  57. Siddhant G, Thuy V, Alessandro M (2020) Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection. Proc AAAI Conf Artif Intell 34:7780–7788

    Google Scholar 

  58. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  MATH  Google Scholar 

  59. Goodfellow I , Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  60. Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T (2020) Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8110–8119

  61. Vahdat A, Kautz J (2020) Nvae. A deep hierarchical variational autoencoder. arXiv preprint. arXiv:2007.03898

  62. Guo J, Lu S, Cai H, Zhang W, Yu Y, Wang J (2018) Long text generation via adversarial training with leaked information. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 32

  63. Karras T, Laine S, Aila T (2019) A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4401–4410

  64. Schonfeld E, Schiele B, Khoreva A (2020) A u-net based discriminator for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8207–8216

  65. Ronneberger OF, Philipp BT (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, pp 234–241. Springer

  66. Abualigah L, Diabat A (2021) Advances in sine cosine algorithm: a comprehensive survey. Artificial Intelligence Review, pp 1–42

  67. Abualigah L, Diabat A (2020) A comprehensive survey of the grasshopper optimization algorithm: results, variants, and applications. Neural Computing and Applications, pp 1–24

  68. Laith A, Ali D, Zong WG (2020) A comprehensive survey of the harmony search algorithm in clustering applications. Appl Sci 10(11):3827

    Article  Google Scholar 

  69. Abualigah L (2020) Group search optimizer: a nature-inspired meta-heuristic optimization algorithm with its results, variants, and applications. Neural Computing and Applications, pp 1–24

  70. Abualigah L (2020) Multi-verse optimizer algorithm: a comprehensive survey of its results, variants, and applications. Neural Computing Applications, pp 1–21

  71. Abualigah L, Diabat A, Mirjalili S, Abd Elaziz M, Gandomi AH (2021) The arithmetic optimization algorithm. Comput Methods Appl Mech Eng

  72. Xin Y, Yong L, Guangming L (1999) Evolutionary programming made faster. IEEE Trans Evolut Comput 3(2):82–102

    Article  Google Scholar 

  73. John H (1975) Adaptation in artificial and natural systems. The University of Michigan Press, Ann Arbor

    Google Scholar 

  74. Goldberg David E, Holland John H (1988) Genetic algorithms and machine learning. Mach Learn 3(2):95–99

    Article  Google Scholar 

  75. Michalewicz Z (1996) Evolution strategies and other methods . In: Genetic Algorithms+ Data Structures= Evolution Programs, pp 159–177. Springer

  76. Rainer S, Kenneth P (1997) Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J Global Optim 11(4):341–359

    Article  MathSciNet  MATH  Google Scholar 

  77. Koza John R (1994) Genetic programming as a means for programming computers by natural selection. Stat Comput 4(2):87–112

    Google Scholar 

  78. Eberhart R, Kennedy J (1995) A new optimizer using particle swarm theory. In: Proceedings of the Sixth International Symposium on Micro Machine and Human Science, 1995. MHS’95., pp 39–43. IEEE,

  79. Seyedali M, Gandomi Amir H, Zahra MS, Shahrzad S, Hossam F, Mohammad MS (2017) Salp swarm algorithm: a bio-inspired optimizer for engineering design problems. Adv Eng Softw 114:163–191

    Article  Google Scholar 

  80. Faramarzi A, Heidarinejad M, Mirjalili S, Gandomi AH (2020) Marine predators algorithm: A nature-inspired metaheuristic. Exp Syst Appl, pp 113377,

  81. Seyedali M, Andrew L (2016) The whale optimization algorithm. Adv Eng Softw 95:51–67

    Article  Google Scholar 

  82. Hadi E, Ali S, Ardeshir B, Mohd H (2012) Water cycle algorithm-a novel metaheuristic optimization method for solving constrained engineering optimization problems. Comput Struct 110:151–166

    Google Scholar 

  83. Kenichi T, Keiichiro Y (2011) Primary study of spiral dynamics inspired optimization. IEEJ Trans Electrical Electron Eng 6(S1):S98–S100

    Article  Google Scholar 

  84. Bayraktar Z, Komurcu M, Werner DH (2010) Wind driven optimization (wdo): a novel nature-inspired optimization algorithm and its application to electromagnetics. In: Antennas and Propagation Society International Symposium (APSURSI), 2010 IEEE, pp 1–4. IEEE

  85. Ali K (2014) Advances in metaheuristic algorithms for optimal design of structures. Springer, Berlin

    MATH  Google Scholar 

  86. İlker BŞ, Shu-Chering F (2003) An electromagnetism-like mechanism for global optimization. J Global Optim 25(3):263–282

    Article  MathSciNet  MATH  Google Scholar 

  87. Kaveh A, Talatahari S (2010) A novel heuristic optimization method: charged system search. Acta Mech 213(3–4):267–289

    Article  MATH  Google Scholar 

  88. Scott K, Daniel Gelatt C, Mario PV (1983) Optimization by simulated annealing. Science 220(4598):671–680

    Article  MathSciNet  Google Scholar 

  89. Esmat R, Hossein N-P, Saeid S (2009) Gsa: a gravitational search algorithm. Inf Sci 179(13):2232–2248

    Article  MATH  Google Scholar 

  90. Abualigah L, Yousri D, Abd Elaziz M, Ewees AA, Al-qanes, MA, Gandomi AH (2021) Aquila Optimizer: A novel meta-heuristic optimization Algorithm. Comput Indus Eng. https://doi.org/10.1016/j.cie.2021.107250

  91. Tahani M, Babayan N (2018) Flow regime algorithm (fra): a physics-based meta-heuristics algorithm. Knowledge and Information Systems, pp 1–38

  92. Ali Husseinzadeh Kashan (2015) A new metaheuristic for optimization: optics inspired optimization (oio). Comput Operat Res 55:99–125

    Article  MathSciNet  MATH  Google Scholar 

  93. Lam Albert YS, Li Victor OK (2010) Chemical-reaction-inspired metaheuristic for optimization. IEEE Trans Evolut Comput 14(3):381–399

    Article  Google Scholar 

  94. Laith A, Mohammad S, Mohammad A, Seyedali M, Abd EM (2020) Ant lion optimizer: A comprehensive survey of its variants and applications. Arch. Comput, Methods Eng

  95. Abualigah L, Abd EM, Hussien AG, Alsalibi B, Jafar J, Seyed M, Gandomi AH ( 2020) Lightning search algorithm: a comprehensive survey. Appl Intell, pp 1–24

  96. Ravipudi VR, Vimal JS, Vakharia DP (2011) Teaching-learning-based optimization: a novel method for constrained mechanical design optimization problems. Comput Aided Des 43(3):303–315

    Article  Google Scholar 

  97. Reza M, Khodakaram S (2018) Volleyball premier league algorithm. Appl Soft Comput 64:161–185

    Article  Google Scholar 

  98. Dai C, Zhu Y, Chen W (2006) Seeker optimization algorithm. In: International Conference on Computational and Information Science, pp 167–176. Springer

  99. Naser Moosavian and Babak Kasaee Roodsari (2014) Soccer league competition algorithm: a novel meta-heuristic algorithm for optimal design of water distribution networks. Swarm Evolut Comput 17:14–24

    Article  Google Scholar 

  100. Kashan AH (2009) League championship algorithm: a new algorithm for numerical function optimization. In: 2009 International Conference of Soft Computing and Pattern Recognition, pp 43–48. IEEE

  101. Fogel DB (1995) Phenotypes, genotypes, and operators in evolutionary computation. In Proceedings 1995 IEEE Int. Conf. Evolutionary Computation (ICEC’95), pp 193–198

  102. Kriegman S, Cheney N, Corucci F, Bongard JC (2017) A minimal developmental model can increase evolvability in soft robots. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp 131–138

  103. Parker A, Nitschke G (2017) Autonomous intersection driving with neuro-evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp 133–134

  104. Radu-Emil P, Radu-Codrut D (2019) Nature-inspired optimization algorithms for fuzzy controlled servo systems. Butterworth-Heinemann, Oxford

    Google Scholar 

  105. Constantin P, Radu-Emil P, Daniel I, Lucian-Ovidiu F, Radu-Codrut D, Florin D (2013) Optimal robot path planning using gravitational search algorithm. Int J Artif Intell 10(S13):1–20

    Google Scholar 

  106. Kaya Y, Faruk Ertugru O (2017) Determining the optimal number of body-worn sensors for human activity recognition. Soft Comput 21(17):5053–5060

    Article  Google Scholar 

  107. Justesen N, Risi S (2017) Continual online evolutionary planning for in-game build order adaptation in starcraft. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp 187–194

  108. Fan Z , Wei J, Zhu G, Mo J, Li W (2020) Evolutionary neural architecture search for retinal vessel segmentation. arXiv, pages arXiv–2001

  109. Abeer A-H, Shawki A (2017) Design space exploration of convolutional neural networks based on evolutionary algorithms. J Comput Vis Imag Syst, vol 3, no 1

  110. Xie L, Yuille A (2017) Genetic cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1379–1388

  111. Felbinger GC (2017) A genetic approach to design convolutional neural networks for the purpose of a ball detection on the nao robotic system. Project Work

  112. Yanan S, Bing X, Mengjie Z, Gary GY, Jiancheng L (2020) Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Trans Cybern

  113. Sun Y, Xue B, Zhang M, Yen GG (2018) Automatically evolving cnn architectures based on blocks. arXiv preprint. arXiv: 1810.11875

  114. Alejandro B, Yago S, Pedro I (2018) Evolutionary convolutional neural networks: an application to handwriting recognition. Neurocomputing 283:38–52

    Article  Google Scholar 

  115. Sun Y, Xue B, Zhang M, Yen GG (2019) Evolving deep convolutional neural networks for image classification. IEEE Trans Evolut Comput

  116. Stanley Kenneth O, D’Ambrosio David B, Jason G (2009) A hypercube-based encoding for evolving large-scale neural networks. Artificial Life 15(2):185–212

    Article  Google Scholar 

  117. Yang C, An Z, Li C, Diao B, Xu Y (2019) Multi-objective pruning for cnns using genetic algorithm. In: International Conference on Artificial Neural Networks, pp 299–305. Springer

  118. Jones D, Schroeder A, Nitschke G (2019) Evolutionary deep learning to identify galaxies in the zone of avoidance. arXiv preprint. arXiv: 1903.07461

  119. da Silveira BJ, Iochins GB, Dorn M (2020) Neuroevolution of neural network architectures using codeepneat and keras. arXiv preprint. arXiv: 2002.04634

  120. Miikkulainen R, Liang J , Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan Arshak,Duffy Nigel , et al (2019). Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp 293–312. Elsevier

  121. Agapitos A, O’Neill M, Nicolau M, Fagan D, Kattan A, Brabazon A, Curran K (2015) Deep evolution of image representations for handwritten digit recognition. In: 2015 IEEE Congress on Evolutionary Computation (CEC), pp 2452–2459. IEEE

  122. Masanori S, Shinichi S, Tomoharu N (2017) A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the genetic and evolutionary computation conference, pp 497–504

  123. Pak-Kan W, Kwong-Sak L, Man-Leung W (2019) Probabilistic grammar-based neuroevolution for physiological signal classification of ventricular tachycardia. Expert Syst Appl 135:237–248

    Article  Google Scholar 

  124. Chong Z, Pin L, Kai Qin A, Kay Chen T (2016) Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE Trans Neural Networks Learn Syst 28(10):2306–2318

    Google Scholar 

  125. Anupam T, Dipti S, Krishnendu S, Abhiroop G (2016) A survey of multiobjective evolutionary algorithms based on decomposition. IEEE Trans Evolut Comput 21(3):440–462

    Google Scholar 

  126. Swagatam D, Sankha SM, Ponnuthurai NS (2016) Recent advances in differential evolution-an updated survey. Swarm Evolut Comput 27:1–30

    Article  Google Scholar 

  127. Tae JC, Chang WA (2017) An improved differential evolution algorithm and its application to large-scale artificial neural networks. In: Journal of Physics: Conference Series, vol 806, p 012010. IOP Publishing

  128. Bin W, Yanan S, Bing X, Mengjie Z (2018) A hybrid differential evolution approach to designing deep convolutional neural networks for image classification. In: Australasian Joint Conference on Artificial Intelligence, pp 237–250. Springer

  129. Peng L, Shan L, Rui L, Lin W (2018) Effective long short-term memory with differential evolution algorithm for electricity price prediction. Energy 162:1301–1314

    Article  Google Scholar 

  130. Alejandro M, Raúl L-C, Félix F-H, Valery N, David C (2018) Evodeep: a new evolutionary approach for automatic deep neural networks parametrisation. J Parallel Distribut Comput 117:180–191

    Article  Google Scholar 

  131. Tirumala S S, Ali S, Ramesh C P (2016) Evolving deep neural networks: A new prospect. In: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp 69–74. IEEE

  132. Alexander O, AbdElRahman E, Travis D (2019) Investigating recurrent neural network memory structures using neuro-evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp 446–455

  133. ElSaid A,Benson S, Patwardhan S, Stadem D, Desell T (2019) Evolving recurrent neural networks for time series data prediction of coal plant parameters. In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar), pp 488–503. Springer

  134. Badan F, Sekanina L (2019) Optimizing convolutional neural networks for embedded systems by means of neuroevolution. In: International Conference on Theory and Practice of Natural Computing, pp 109–121. Springer

  135. Irwin-Harris W, Sun Y, Xue B, Zhang M (2019) A graph-based encoding for evolutionary convolutional neural network architecture design. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp 546–553. IEEE

  136. Bakhshi A, Noman N, Chen Z, Zamani M, Chalup S (2019)Fast automatic optimisation of cnn architectures for image classification using genetic algorithm. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp 1283–1290. IEEE

  137. Alejandro B, Yago S, Isasi P (2019) Hybridizing evolutionary computation and deep neural networks: an approach to handwriting recognition using committees and transfer learning. Complexity

  138. Benteng M, Xiang L, Yong X, Yanning Z (2020) Autonomous deep learning: a genetic dcnn designer for image classification. Neurocomputing 379:152–161

    Article  Google Scholar 

  139. Corne DW , Reynolds AP, Bonabeau E (2012) Swarm intelligence

  140. Yang X-S (2010) Nature-inspired metaheuristic algorithms. Luniver press

  141. Hossam F, Ibrahim A, Al-Betar MA, Mirjalili S (2018) Grey wolf optimizer: a review of recent variants and applications. Neural Comput Appl 30(2):413–435

    Article  Google Scholar 

  142. Bonyadi M, Reza MZ (2017) Particle swarm optimization for single objective continuous space problems: a review

  143. Dorigo M, Stützle T (2019) Ant colony optimization: overview and recent advances. In: Handbook of metaheuristics, pp 311–351. Springer

  144. Farhad Soleimanian Gharehchopogh and Hojjat Gholizadeh (2019) A comprehensive survey: whale optimization algorithm and its applications. Swarm Evolut Comput 48:1–24

    Article  Google Scholar 

  145. Mohit J, Vijander S, Asha R (2019) A novel nature-inspired algorithm for optimization: squirrel search algorithm. Swarm Evolut Comput 44:148–175

    Article  Google Scholar 

  146. Khalifa MH, Ammar M, Ouarda W, Alimi AM (2017) Particle swarm optimization for deep learning of convolution neural network. In: 2017 Sudan Conference on Computer Science and Information Technology (SCCSIT), pp 1–5. IEEE

  147. Fei Y (2017) Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data. PloS ONE 12(12):e0188746

    Article  Google Scholar 

  148. Qolomany B, Maabreh M, Al-Fuqaha A, Gupta A, Benhaddou D (2017) Parameters optimization of deep learning models using particle swarm optimization. In: 2017 13th International Wireless Communications and Mobile Computing Conference (IWCMC), pp 1285–1290. IEEE

  149. Wang B, Sun Y, Xue B, Zhang M (2018) Evolving deep convolutional neural networks by variable-length particle swarm optimization for image classification. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp 1–8. IEEE

  150. De Rosa GH, Papa João P, Xin-S Y (2018) Handling dropout probability estimation in convolution neural networks using meta-heuristics. Soft Comput 22(18):6147–6156

    Article  Google Scholar 

  151. Enrique A, Marco T (2002) Parallelism and evolutionary algorithms. IEEE Trans Evolut Comput 6(5):443–462

    Article  Google Scholar 

  152. Bin W, Bing X , Mengjie Z (2019) Particle swarm optimisation for evolving deep neural networks for image classification by evolving and stacking transferable blocks. arXiv preprint. arXiv: 1907.12659

  153. Junior FEF, Yen GG (2019) Particle swarm optimization of deep neural networks architectures for image classification. Swarm Evolut Comput 49:62–74

    Article  Google Scholar 

  154. de Pinho P, Cesar A, Nedjah N, de Macedo ML (2020) Detection and classification of pulmonary nodules using deep learning and swarm intelligence. Multimedia Tools Appl 79(21):15437–15465

    Article  Google Scholar 

  155. Desell T, Clachar S, Higgins J, Wild B (2015) Evolving deep recurrent neural networks using ant colony optimization. In: European Conference on Evolutionary Computation in Combinatorial Optimization, pp 86–98. Springer

  156. ElSaid A, Wild B, Jamiy FE, Higgins J, Desell T (2017) Optimizing lstm rnns using aco to predict turbine engine vibration. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp 21–22

  157. ElSaid A, Jamiy FE, Higgins J, Wild B, Desell T (2018) Using ant colony optimization to optimize long short-term memory recurrent neural networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp 13–20

  158. Byla E, Pang W (2019). Deepswarm: Optimising convolutional neural networks using swarm intelligence. In: UK Workshop on Computational Intelligence, pp 119–130. Springer

  159. ElSaid AA , Ororbia Alexander G, Desell Travis J (2019) The ant swarm neuro-evolution procedure for optimizing recurrent networks. arXiv preprint. arXiv:1909.11849

  160. Sharaf AI , Radwan E-SF (2020) An automated approach for developing a convolutional neural network using a modified firefly algorithm for image classification. In: Applications of Firefly Algorithm and its Variants, pp 99–118. Springer

  161. Bingham G, Macke W, Miikkulainen R (2020) Evolutionary optimization of deep learning activation functions. arXiv preprint. arXiv: 2002.07224

  162. Wang B, Sun Y, Xue B, Zhang M (2019). A hybrid ga-pso method for evolving architecture and short connections of deep convolutional neural networks. In: Pacific Rim International Conference on Artificial Intelligence, pp 650–663. Springer

  163. Yanan S, Yen Gary G, Zhang Y (2018) Evolving unsupervised deep neural networks for learning meaningful representations. IEEE Trans Evolut Comput 23(1):89–103

    Google Scholar 

  164. Verbancsics P , Harguess J (2015) Image classification using generative neuro evolution for deep learning. In: 2015 IEEE winter conference on applications of computer vision, pp 488–493. IEEE

  165. Adarsh T, Siddhant S, Apoorva M, Anupam S, Ritu T (2018) Hybrid evolutionary approach for Devanagari handwritten numeral recognition using convolutional neural network. Procedia Comput Sci 125:525–532

    Article  Google Scholar 

  166. Chenxi L, Barret Z, Maxim N, Jonathon S, Wei H, Li-Jia L, Li F-F, Alan Y, Jonathan H, Kevin M (2018) Progressive neural architecture search. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 19–34

  167. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8697–8710

  168. Alejandro M, Manuel VV, Antonio GP, David C, César H-M (2020) Optimising convolutional neural networks using a hybrid statistically-driven coral reef optimisation algorithm. Appl Soft Comput 90:106144

    Article  Google Scholar 

  169. Aly A, Weikersdorfer D, Delaunay C (2019) Optimizing deep neural networks with multiple search neuroevolution. arXiv preprint. arXiv: 1901.05988

  170. Peyrard M, Eckle-Kohler J (2016) A general optimization framework for multi-document summarization using genetic algorithms and swarm intelligence. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp 247–257

  171. Chen S-L, Lin S-C, Huang Y, Jen C-W, Lin Z-L, Su S-F (2020) A vision-based dual-axis positioning system with yolov4 and improved genetic algorithms. In: 2020 Fourth IEEE International Conference on Robotic Computing (IRC), pp 127–134. IEEE

  172. Zhicheng Y, Xiaoliang D, Peizhao Z, Tian Yuandong W, Bichen, (2011) Feiszli Matt (2020). Fast probabilistic neural architecture search. arXiv preprint arXiv, Fp-nas, p 10949

  173. Liu C, Chen L-C, Schroff F, Adam H, Hua W, Yuille W, Li F-F (2019) Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 82–92

Download references

Acknowledgements

This work is supported by the Hubei Provincinal Science and Technology Major Project of China under Grant No. 2020AEA011 and the Key Research & Developement Plan of Hubei Province of China under Grant No. 2020BAB100.

Author information

Authors and Affiliations

  1. Department of Mathematics, Faculty of Science, Zagazig University, Zagazig, 44519, Egypt

    Mohamed Abd Elaziz

  2. School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China

    Mohamed Abd Elaziz & Liyang Yu

  3. Mathematics and Computer Science department, University of Ahmed DRAIA, 01000, Adrar, Algeria

    Abdelghani Dahou

  4. Faculty of Computer Sciences and Informatics, Amman Arab University, 11953, Amman, Jordan

    Laith Abualigah, Mohammad Alshinwan & Ahmad M. Khasawneh

  5. Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong university of Science and Technology, Wuhan, 430074, China

    Songfeng Lu

Authors
  1. Mohamed Abd Elaziz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdelghani Dahou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Laith Abualigah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liyang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohammad Alshinwan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ahmad M. Khasawneh
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Songfeng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laith Abualigah.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abd Elaziz, M., Dahou, A., Abualigah, L. et al. Advanced metaheuristic optimization techniques in applications of deep neural networks: a review. Neural Comput & Applic 33, 14079–14099 (2021). https://doi.org/10.1007/s00521-021-05960-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05960-5

Keywords

Search

Navigation