Skip to main content
SpringerLink
Log in

Neural Architecture Search: Practical Key Considerations

  • Conference paper
  • First Online:
Distributed Computing and Artificial Intelligence, 20th International Conference (DCAI 2023)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 740))

  • 350 Accesses

Abstract

The rapid development of deep neural networks has highlighted the importance of research in this domain. Neural Architecture Search (NAS) has emerged as a pivotal technique for automating and optimizing neural network designs. However, due to the complex and evolving nature of this field, staying up to date with the latest research, trends, and best practices is challenging. This article addresses the need for practical considerations, best practices, and open frameworks to guide practitioners in NAS endeavors. It discusses key considerations, challenges, opportunities, and open problems, along with a compilation of best practices and open frameworks. Readers will gain a practical guide for developing, testing, or applying NAS techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. About vertex AI neural architecture search. https://cloud.google.com/vertex-ai/docs/training/neural-architecture-search/overview. Accessed 21 May 2023

  2. Benmeziane, H., Maghraoui, K.E., Ouarnoughi, H., Niar, S., Wistuba, M., Wang, N.: Hardware-aware neural architecture search: survey and taxonomy. In: International Joint Conference on Artificial Intelligence (2021)

    Google Scholar 

  3. Brock, A., Lim, T., Ritchie, J.M., Weston, N.: SMASH: one-shot model architecture search through hypernetworks (2017)

    Google Scholar 

  4. Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware (2019)

    Google Scholar 

  5. Cai, S., Li, L., Deng, J., Zhang, B., Zha, Z.J., Su, L., Huang, Q.: Rethinking graph neural architecture search from message-passing (2021)

    Google Scholar 

  6. Cha, S., Kim, T., Lee, H., Yun, S.Y.: SuperNet in neural architecture search: a taxonomic survey. ArXiv abs/2204.03916 (2022)

    Google Scholar 

  7. Chen, D., Chen, L., Shang, Z., Zhang, Y., Wen, B., Yang, C.: Scale-aware neural architecture search for multivariate time series forecasting (2021)

    Google Scholar 

  8. Chitnis, S., Hosseini, R., Xie, P.: Brain tumor classification based on neural architecture search. Sci. Rep. 12(1), 19206 (2022). https://doi.org/10.1038/s41598-022-22172-6

    Article  Google Scholar 

  9. Chu, J., Yu, X., Yang, S., Qiu, J., Wang, Q.: Architecture entropy sampling-based evolutionary neural architecture search and its application in osteoporosis diagnosis. Complex Intell. Syst. 9(1), 213–231 (2023). https://doi.org/10.1007/s40747-022-00794-7

    Article  Google Scholar 

  10. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey (2019)

    Google Scholar 

  11. Elsken, T., Staffler, B., Zela, A., Metzen, J.H., Hutter, F.: Bag of tricks for neural architecture search (2021)

    Google Scholar 

  12. Gao, Y., Yang, H., Zhang, P., Zhou, C., Hu, Y.: GraphNAS: graph neural architecture search with reinforcement learning (2019)

    Google Scholar 

  13. Guan, C., Wang, X., Chen, H., Zhang, Z., Zhu, W.: Large-scale graph neural architecture search. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 7968–7981. PMLR (2022). https://proceedings.mlr.press/v162/guan22d.html

  14. Gupta, A., Sheth, P., Xie, P.: Neural architecture search for pneumonia diagnosis from chest X-rays. Sci. Rep. 12(1), 11309 (2022). https://doi.org/10.1038/s41598-022-15341-0

  15. He, C., Ye, H., Shen, L., Zhang, T.: MileNAS: efficient neural architecture search via mixed-level reformulation (2020)

    Google Scholar 

  16. Hu, S., Xie, X., Liu, S., Geng, M., Liu, X., Meng, H.: Neural architecture search for speech recognition (2020)

    Google Scholar 

  17. Jiang, Y., Hu, C., Xiao, T., Zhang, C., Zhu, J.: Improved differentiable architecture search for language modeling and named entity recognition. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3585–3590. Association for Computational Linguistics, Hong Kong (2019). https://doi.org/10.18653/v1/D19-1367, https://aclanthology.org/D19-1367

  18. Jin, H., Song, Q., Hu, X.: Auto-keras: an efficient neural architecture search system (2019)

    Google Scholar 

  19. Kim, Y., Yun, W.J., Lee, Y.K., Jung, S., Kim, J.: Trends in neural architecture search: Towards the acceleration of search (2021)

    Google Scholar 

  20. Klyuchnikov, N., Trofimov, I., Artemova, E., Salnikov, M., Fedorov, M., Burnaev, E.: NAS-bench-NLP: neural architecture search benchmark for natural language processing. IEEE Access 1 (2020)

    Google Scholar 

  21. Li, C., et al.: BossNAS: exploring hybrid CNN-transformers with block-wisely self-supervised neural architecture search (2021)

    Google Scholar 

  22. Li, L., Talwalkar, A.: Random search and reproducibility for neural architecture search (2019)

    Google Scholar 

  23. Li, Y., Hao, C., Li, P., Xiong, J., Chen, D.: Generic neural architecture search via regression (2021)

    Google Scholar 

  24. Lindauer, M., Hutter, F.: Best practices for scientific research on neural architecture search. J. Mach. Learn. Res. 21(1), 9820–9837 (2020)

    MATH  Google Scholar 

  25. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search (2018)

    Google Scholar 

  26. Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search (2019)

    Google Scholar 

  27. Luo, R., Tian, F., Qin, T., Chen, E., Liu, T.Y.: Neural architecture optimization. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018)

    Google Scholar 

  28. Mehrotra, A., et al.: NAS-bench-ASR: reproducible neural architecture search for speech recognition. In: International Conference on Learning Representations (2021)

    Google Scholar 

  29. Microsoft: Neural network intelligence (2021). https://github.com/microsoft/nni. If you use this software, please cite it as above

  30. Miikkulainen, R., et al.: Evolving deep neural networks (2017)

    Google Scholar 

  31. Moser, B.B., Raue, F., Hees, J., Dengel, A.: DartsReNet: exploring New RNN cells in ReNet architectures. In: Farkaš, I., Masulli, P., Wermter, S. (eds.) ICANN 2020. LNCS, vol. 12396, pp. 850–861. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61609-0_67

    Chapter  Google Scholar 

  32. Ning, X., et al.: Evaluating efficient performance estimators of neural architectures. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021). https://openreview.net/forum?id=Esd7tGH3Spl

  33. Odema, M., Rashid, N., Faruque, M.A.A.: EExNAS: early-exit neural architecture search solutions for low-power wearable devices. In: 2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1–6 (2021)

    Google Scholar 

  34. Oyelade, O.N., Ezugwu, A.E.: A bioinspired neural architecture search based convolutional neural network for breast cancer detection using histopathology images. Sci. Rep. 11(1), 19940 (2021). https://doi.org/10.1038/s41598-021-98978-7

  35. Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing (2018)

    Google Scholar 

  36. Qian, G., et al.: When NAS meets trees: an efficient algorithm for neural architecture search (2022)

    Google Scholar 

  37. Rakhshani, H., et al.: Neural architecture search for time series classification. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020). https://doi.org/10.1109/IJCNN48605.2020.9206721

  38. Ren, P., et al.: A comprehensive survey of neural architecture search: Challenges and solutions (2021)

    Google Scholar 

  39. Robles, J.G., Vanschoren, J.: Learning to reinforcement learn for neural architecture search (2019)

    Google Scholar 

  40. Ru, B., Wan, X., Dong, X., Osborne, M.: Interpretable neural architecture search via Bayesian optimisation with Weisfeiler-Lehman kernels (2021)

    Google Scholar 

  41. Ruan, D., Han, J., Yan, J., Gühmann, C.: Light convolutional neural network by neural architecture search and model pruning for bearing fault diagnosis and remaining useful life prediction. Sci. Rep. 13(1), 5484 (2023). https://doi.org/10.1038/s41598-023-31532-9

  42. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002). https://doi.org/10.1162/106365602320169811

    Article  Google Scholar 

  43. Tao, T.M., Kim, H., Youn, C.H.: A compact neural architecture search for accelerating image classification models. In: 2021 International Conference on Information and Communication Technology Convergence (ICTC), pp. 1713–1718 (2021). https://doi.org/10.1109/ICTC52510.2021.9620797

  44. Vo-Ho, V.K., Yamazaki, K., Hoang, H., Tran, M.T., Le, N.: Chapter 19 - neural architecture search for medical image applications. In: Nguyen, H.V., Summers, R., Chellappa, R. (eds.) Meta Learning With Medical Imaging and Health Informatics Applications, The MICCAI Society book Series, pp. 369–384. Academic Press (2023). https://doi.org/10.1016/B978-0-32-399851-2.00029-6, https://www.sciencedirect.com/science/article/pii/B9780323998512000296

  45. Wang, D., Gong, C., Li, M., Liu, Q., Chandra, V.: AlphaNet: improved training of supernets with alpha-divergence (2021)

    Google Scholar 

  46. Wang, W., Zhang, X., Cui, H., Yin, H., Zhang, Y.: FP-DARTS: fast parallel differentiable neural architecture search for image classification. Pattern Recognit. 136, 109193 (2023). https://doi.org/10.1016/j.patcog.2022.109193, https://www.sciencedirect.com/science/article/pii/S0031320322006720

  47. Wang, Y., et al.: TextNAS: a neural architecture search space tailored for text representation (2019)

    Google Scholar 

  48. White, C., Neiswanger, W., Savani, Y.: BANANAS: Bayesian optimization with neural architectures for neural architecture search (2020)

    Google Scholar 

  49. White, C., Nolen, S., Savani, Y.: Exploring the loss landscape in neural architecture search (2021)

    Google Scholar 

  50. White, C., Zela, A., Ru, R., Liu, Y., Hutter, F.: How powerful are performance predictors in neural architecture search? Adv. Neural Inf. Process. Syst. 34 (2021)

    Google Scholar 

  51. Wistuba, M., Rawat, A., Pedapati, T.: A survey on neural architecture search (2019)

    Google Scholar 

  52. Wu, X., Hu, S., Wu, Z., Liu, X., Meng, H.: Neural architecture search for speech emotion recognition (2022)

    Google Scholar 

  53. Xie, L., Yuille, A.: Genetic CNN (2017)

    Google Scholar 

  54. Xie, S., Zheng, H., Liu, C., Lin, L.: SNAS: stochastic neural architecture search (2020)

    Google Scholar 

  55. Xie, X., Song, X., Lv, Z., Yen, G.G., Ding, W., Sun, Y.: Efficient evaluation methods for neural architecture search: a survey (2023)

    Google Scholar 

  56. Yao, X., Liu, Y.: A new evolutionary system for evolving artificial neural networks. IEEE Trans. Neural Netw. 8(3), 694–713 (1997). https://doi.org/10.1109/72.572107

    Article  Google Scholar 

  57. Yu, Q., Yang, D., Roth, H., Bai, Y., Zhang, Y., Yuille, A.L., Xu, D.: C2FNAS: coarse-to-fine neural architecture search for 3D medical image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4126–4135 (2020)

    Google Scholar 

  58. Zhou, H., Yang, M., Wang, J., Pan, W.: BayesNAS: a Bayesian approach for neural architecture search (2019)

    Google Scholar 

  59. Zimmer, L., Lindauer, M., Hutter, F.: Auto-pytorch: multi-fidelity metalearning for efficient and robust AutoDL. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 3079–3090 (2021). https://doi.org/10.1109/TPAMI.2021.3067763

    Article  Google Scholar 

  60. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning (2017)

    Google Scholar 

  61. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition (2018)

    Google Scholar 

Download references

Acknowledgements

This research has been funded through the call of the public business entity red.es, for 2021 grants for research and development projects in artificial intelligence and other digital technologies and their integration into value chains with Code: C005/21-ED, Funded by the European Union NextGenerationEU.

Author information

Authors and Affiliations

  1. AIR Institute, PCUVA Building, Paseo de Belén, 9A, 47011, Valladolid, Spain

    María Alonso-García & Juan M. Corchado

Authors
  1. María Alonso-García
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan M. Corchado
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to María Alonso-García .

Editor information

Editors and Affiliations

  1. ETSII / CETINIA, Universidad Rey Juan Carlos, Madrid, Madrid, Spain

    Sascha Ossowski

  2. Politechnika Śląska Kielce, Kielce University of Technology, Santa Cruz, Poland

    Pawel Sitek

  3. Department of Informatics, Engineering School, University of Minho, Braga, Portugal

    Cesar Analide

  4. Departamento de Engenharia Informática, Polytechnic Institute of Porto, Porto, Portugal

    Goreti Marreiros

  5. BISITE, University of Salamanca, Salamanca, Salamanca, Spain

    Pablo Chamoso

  6. BISITE, University of Salamanca, Salamanca, Salamanca, Spain

    Sara Rodríguez

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alonso-García, M., Corchado, J.M. (2023). Neural Architecture Search: Practical Key Considerations. In: Ossowski, S., Sitek, P., Analide, C., Marreiros, G., Chamoso, P., Rodríguez, S. (eds) Distributed Computing and Artificial Intelligence, 20th International Conference. DCAI 2023. Lecture Notes in Networks and Systems, vol 740. Springer, Cham. https://doi.org/10.1007/978-3-031-38333-5_17

Download citation

Publish with us

Policies and ethics

Search

Navigation