Skip to main content
SpringerLink
Log in

Neural Architecture Search as Self-assessor in Semi-supervised Learning

  • Conference paper
  • First Online:
Big Data (BigData 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1496))

Included in the following conference series:

  • 881 Accesses

Abstract

Neural Architecture Search (NAS) forms powerful automatic learning, which has helped achieve remarkable performance in several applications in recent years. Previous research focused on NAS in standard supervised learning to explore its performance, requiring labeled data. In this paper, our goal is to examine the implementation of NAS with large amounts of unlabeled data. We propose the NAS as a self-assessor, called NAS-SA, by adding the consistency method and prior knowledge. We design an adaptive search strategy, a balanced search space, and a multi-object optimization to generate a robust and efficient small model in NAS-SA. The image and text classification tasks proved that our NAS-SA method had achieved the best performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 2016), pp. 265–283 (2016)

    Google Scholar 

  2. Bender, G., Kindermans, P.J., Zoph, B., Vasudevan, V., Le, Q.: Understanding and simplifying one-shot architecture search. In: International Conference on Machine Learning, pp. 550–559 (2018)

    Google Scholar 

  3. Brock, A., Lim, T., Ritchie, J.M., Weston, N.J.: Smash: one-shot model architecture search through hypernetworks. In: 6th International Conference on Learning Representations (2018)

    Google Scholar 

  4. Chen, X., Hsieh, C.J.: Stabilizing differentiable architecture search via perturbation-based regularization. In: International Conference on Machine Learning, pp. 1554–1565. PMLR (2020)

    Google Scholar 

  5. Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719  2(3) (2019)

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long and Short Papers), pp. 4171–4186 (2019)

    Google Scholar 

  7. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)

    Article  Google Scholar 

  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  9. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. Statistics 1050, 9 (2015)

    Google Scholar 

  10. Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  11. Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  12. Kuo, C.-W., Ma, C.-Y., Huang, J.-B., Kira, Z.: FeatMatch: feature-based augmentation for semi-supervised learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12363, pp. 479–495. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58523-5_28

    Chapter  Google Scholar 

  13. Liu, C., Dollár, P., He, K., Girshick, R., Yuille, A., Xie, S.: Are labels necessary for neural architecture search? In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 798–813. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_46

    Chapter  Google Scholar 

  14. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436 (2017)

  15. Liu, H., Simonyan, K., Yang, Y.: Darts: differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)

  16. Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International Conference on Machine Learning, pp. 4095–4104. PMLR (2018)

    Google Scholar 

  17. Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4780–4789 (2019)

    Google Scholar 

  18. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  19. Sennrich, R., Haddow, B., Birch, A.: Improving neural machine translation models with monolingual data. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 86–96 (2016)

    Google Scholar 

  20. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    Article  Google Scholar 

  21. Sohn, K., et al.: Fixmatch: simplifying semi-supervised learning with consistency and confidence. In: Advances in Neural Information Processing Systems 33 (2020)

    Google Scholar 

  22. Strubell, E., Verga, P., Belanger, D., McCallum, A.: Fast and accurate entity recognition with iterated dilated convolutions. In: EMNLP (2017)

    Google Scholar 

  23. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 4278–4284 (2017)

    Google Scholar 

  24. Tan, M., et al.: Mnasnet: platform-aware neural architecture search for mobile. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820–2828 (2019)

    Google Scholar 

  25. Tang, Y., et al.: A semi-supervised assessor of neural architectures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1810–1819 (2020)

    Google Scholar 

  26. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  27. Vieira, J.P.A., Moura, R.S.: An analysis of convolutional neural networks for sentence classification. In: 2017 XLIII Latin American Computer Conference (CLEI), pp. 1–5. IEEE (2017)

    Google Scholar 

  28. Wan, A., et al.: FBNetV2: differentiable neural architecture search for spatial and channel dimensions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12965–12974 (2020)

    Google Scholar 

  29. Wei, J., Zou, K.: EDA: easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6383–6389 (2019)

    Google Scholar 

  30. Xue, C., Yan, J., Yan, R., Chu, S.M., Hu, Y., Lin, Y.: Transferable automl by model sharing over grouped datasets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9002–9011 (2019)

    Google Scholar 

  31. Yan, S., Zheng, Y., Ao, W., Zeng, X., Zhang, M.: Does unsupervised architecture representation learning help neural architecture search? In: Advances in Neural Information Processing Systems 33 (2020)

    Google Scholar 

  32. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. Adv. Neural. Inf. Process. Syst. 32, 5753–5763 (2019)

    Google Scholar 

  33. Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: NAS-bench-101: towards reproducible neural architecture search. In: International Conference on Machine Learning, pp. 7105–7114. PMLR (2019)

    Google Scholar 

  34. Zhou, T., Wang, S., Bilmes, J.: Time-consistent self-supervision for semi-supervised learning. In: International Conference on Machine Learning, pp. 11523–11533. PMLR (2020)

    Google Scholar 

  35. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

    Google Scholar 

Download references

Acknowledgement

This work is supported by National Key Research and Development Program of China under grant No. 2018YFB0204403, No. 2017YFB1401202 and No. 2018YFB1003500.

Author information

Authors and Affiliations

  1. Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China

    Zhenhou Hong, Jianzong Wang, Xiaoyang Qu, Chendong Zhao, Jie Liu & Jing Xiao

Authors
  1. Zhenhou Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianzong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoyang Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chendong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jing Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianzong Wang .

Editor information

Editors and Affiliations

  1. National University of Defense Technology, Changsha, China

    Xiangke Liao

  2. Shenzhen University of Technology, Chinese Academy of Sciences, Shenzhen, China

    Wei Zhao

  3. University of Science and Technology of China, Hefei, China

    Enhong Chen

  4. Sun Yat-sen University, Guangzhou, China

    Nong Xiao

  5. Taiyuan University of Technology, Taiyuan, China

    Li Wang

  6. Nanjing University, Nanjing, China

    Yang Gao

  7. Nanjing University, Nanjing, China

    Yinghuan Shi

  8. Sun Yat-sen University, Guangzhou, China

    Changdong Wang

  9. Sun Yat-sen University, Guangzhou, China

    Dan Huang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hong, Z., Wang, J., Qu, X., Zhao, C., Liu, J., Xiao, J. (2022). Neural Architecture Search as Self-assessor in Semi-supervised Learning. In: Liao, X., et al. Big Data. BigData 2022. Communications in Computer and Information Science, vol 1496. Springer, Singapore. https://doi.org/10.1007/978-981-16-9709-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-9709-8_7

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-9708-1

  • Online ISBN: 978-981-16-9709-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation