Abstract
Although deep neural networks (DNNs) play important roles in many fields, the architecture design of DNNs can be challenging due to the difficulty of input data representation, the huge number of parameters and the complex layer relationships. To overcome the obstacles of architecture design, we developed a new method to generate the optimal structure of DNNs, named Evolutionary Strategy-based Architecture Evolution (ESAE), consisting of a bi-level representation and a probability distribution learning approach. The bi-level representation encodes architectures in the gene and parameter levels. The probability distribution learning approach ensures the efficient convergence of the architecture searching process. By using Fashion-MNIST and CIFAR-10, the effectiveness of the proposed ESAS is verified. The evolved DNNs, starting from a trivial initial architecture with one single convolutional layer, achieved the accuracies of 94.48% and 93.49% on Fashion-MNIST and CIFAR-10, respectively, and require remarkably less hardware costs in terms of GPUs and running time, compared with the existing state-of-the-art manual screwed architectures.
Supported by organizations of the National Natural Science Foundation of China (61972174, 61876069 and 61876207), the Key Development Project of Jilin Province (20180201045GX and 20180201067GX), the Guangdong Key-Project for Applied Fundamental Research (2018KZDXM076), and the Guangdong Premier Key-Discipline Enhancement Scheme (2016GDYSZDXK036).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Krizhevsky, A., Sutskever, I., Hinton, G.-E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2012)
Hinton, G., Deng, L., Yu, D., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)
Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Boston, pp. 1–9 (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Las Vegas, pp. 770–778 (2016)
Huang, G., Liu, Z., Van, D.M.L., et al.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Puerto Rico, pp. 2261–2269 (2017)
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(1), 281–305 (2012)
Snoek, J., Rippel, O., Swersky, K., et al.: Scalable Bayesian optimization using deep neural networks. Statistics 37, 1861–1869 (2015)
Miller, G.F., Todd, P.M., Hegde, S.U.: Designing neural networks using genetic algorithms. In: Proceedings of the Third International Conference on Genetic Algorithms, ICGA, San Francisco, pp. 379–384 (1989)
Yao, X., Liu, Y.: A new evolutionary system for evolving artificial neural networks. IEEE Trans. Neural Netw. 8(3), 694–713 (1997)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.-A.: Hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)
Suganuma, M., Shirakawa, S., Nagao, T.A.: Genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 497–504. ACM, Berlin (2017)
Miikkulainen, R., et al.: Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312. Academic Press (2017)
Real, E., Aggarwal, A., Huang, Y., et al.: Aging evolution for image classifier architecture search. In: Thirty-Third AAAI Conference on Artificial Intelligence, pp. 5048–5056. AAAI, Hawaii (2019)
Real, E., Moore, S., Selle, A., et al.: Large-scale evolution of image classifiers. In: 34th International Conference on Machine Learning, ICML, Sydney, pp. 2902–2911 (2017)
Assunção, F., et al.: DENSER: deep evolutionary network structured representation. Genet. Program. Evolvable Mach. 20(1), 5–35 (2019)
Zoph, B., Vasudevan, V., Shlens, J., et al.: Learning transferable architectures for scalable image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Salt Lake, pp. 8697–8710 (2018)
Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: International Conference on Learning Representations, ICLR, New Orleans (2019)
Hundt, A., Jain, V., Hager, G.-D.: sharpDARTS: faster and more accurate differentiable architecture search. arXiv preprint arXiv:1903.09900 (2019)
Zhang, J.-R., Zhang, J., Lok, T.-M., et al.: A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training. Appl. Math. Comput. 185(2), 1026–1037 (2007)
Wang, G.G., Deb, S., et al.: Monarch butterfly optimization. Neural Comput. Appl. 31(7), 1–20 (2015)
Ge, W.G., Suash, D., Santos, C.L.D.: Earthworm optimisation algorithm: a bio-inspired metaheuristic algorithm for global optimisation problems. Int. J. Bio-Inspir. Comput. 12(1), 1–17 (2018)
Wang, G.G., et al.: Elephant herding optimization. In: 2015 3rd International Symposium on Computational and Business Intelligence. IEEE, Bali Indonesia (2015)
Wang, G.-G., et al.: Moth search algorithm: a bio-inspired metaheuristic algorithm for global optimization problems. Mem. Comput. 10(2), 151–164 (2016)
Kiranyaz, S., Ince, T., et al.: Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw. 22(10), 1448–1462 (2009)
Das, G., Pattnaik, P.-K., Padhy, S.-K.: Artificial neural network trained by particle swarm optimization for non-linear channel equalization. Expert Syst. Appl. 41(7), 3491–3496 (2014)
Salama, K.-M., Abdelbar, A.-M.: Learning neural network structures with ant colony algorithms. Swarm Intell. 9(4), 229–265 (2015)
Zhou, X.-H., Zhang, M.X., et al.: Shallow and deep neural network training by water wave optimization. Swarm Evol. Comput. 50, 100561 (2019)
LeCun, Y., Boser, B.E., et al.: Handwritten digit recognition with a back-propagation network. In: Advances in neural Information Processing Systems, pp. 396–404 (1990)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
Gaier, A., et al.: Weight agnostic neural networks. arXiv preprint arXiv:1906.04358 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gu, X. et al. (2020). ESAE: Evolutionary Strategy-Based Architecture Evolution. In: Pan, L., Liang, J., Qu, B. (eds) Bio-inspired Computing: Theories and Applications. BIC-TA 2019. Communications in Computer and Information Science, vol 1159. Springer, Singapore. https://doi.org/10.1007/978-981-15-3425-6_16
Download citation
DOI: https://doi.org/10.1007/978-981-15-3425-6_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-3424-9
Online ISBN: 978-981-15-3425-6
eBook Packages: Computer ScienceComputer Science (R0)