Skip to main content

Advertisement

Log in

A Robust Automated Machine Learning System with Pseudoinverse Learning

Cognitive Computation Aims and scope Submit manuscript

Abstract

Developing a robust deep neural network (DNN) for a specific task is not only time-consuming but also requires lots of experienced human experts. In order to make deep neural networks easier to apply or even take the human experts out of the design of network architecture completely, a growing number of researches focus on robust automated machine learning (AutoML). In this paper, we investigated the robustness problem of AutoML systems based on contractive pseudoinverse learners. In our proposed method, deep neural networks were built with stacked contractive pseudoinverse learners (CPILer). Each CPILer has a Jacobian regularized reconstruction loss function and is trained with pseudoinverse learning algorithm. When sigmoid activation function is adopted in the hidden layer, the graph Laplace regularizer is derived from square Frobenius norm of the Jacobian matrix. This learning scheme not only speeds up the training process dramatically but also reduces the effort of hyperparameter tuning. In addition, the graph Laplace regularization can improve the robustness of the learning systems by reducing the sensibility to noise. An ensemble network architecture consisting of several sub-networks was designed to build the AutoML systems. The architecture hyperparameters of the system were determined in an automated way which could be considered as a data-driven way. The proposed method shown good performance in the experiments in terms of efficiency and accuracy, and outperformed the baseline methods on a series of benchmark data sets. The robustness improvement of our proposed method was also demonstrated in the experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Zhang S, Huang K, Zhang R, Hussain A. Learning from few samples with memory network. Cogn Comput. 2018;10(1):15–22. https://doi.org/10.1007/s12559-01-9507-z.

    Article  Google Scholar 

  2. Perconti P, Plebe A. Deep learning and cognitive science. Cognition. 2020;203:104365. https://doi.org/10.1016/j.cognition.2020.104365. http://www.sciencedirect.com/science/article/pii/S0010027720301840

  3. Yao Q, Wang M, Escalante HJ, Guyon I, Hu YQ, Li YF, Tu WW, Yang Q, Yu Y. Taking human out of learning applications: A survey on automated machine learning. arXiv preprint 2018. arXiv:1810.13306

  4. Liang JZ, Meyerson E, Hodjat B, Fink D, Mutch K, Miikkulainen R. Evolutionary neural AutoML for deep learning. In A. Auger and T. Stützle, editors, Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2019, Prague, Czech Republic, July 13-17, 2019. ACM, 2019. pp. 401–409. https://doi.org/10.1145/3321707.3321721.

  5. Liu H, Simonyan K, Vinyals O, Fernando C, Kavukcuoglu K. Hierarchical representations for efficient architecture search. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. https://openreview.net/forum?id=BJQRKzbA-

  6. Real E, Moore S, Selle A, Saxena S, Suematsu YL, Tan J, Le QV, Kurakin A. Large-scale evolution of image classifiers. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research. PMLR, 2017. pp. 2902–2911. http://proceedings.mlr.press/v70/real17a.html

  7. Baker B, Gupta O, Naik N, Raskar R. Designing neural network architectures using reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. https://openreview.net/forum?id=S1c2cvqee

  8. Li T, Jiao W, Wang L, Zhong G. Automatic densenet sparsification. IEEE. Access. 2020;8:62561–71. https://doi.org/10.1109/ACCESS.2020.2984130.

    Article  Google Scholar 

  9. Zhong G, Jiao W, Gao W, Huang K. Automatic design of deep networks with neural blocks. Cogn Comput. 2020;12(1):1–12. https://doi.org/10.1007/s12559-019-09677-5.

    Article  Google Scholar 

  10. Zoph B, Le QV. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. https://openreview.net/forum?id=r1Ue8Hcxg

  11. Zoph B, Vasudevan V, Shlens J, Le QV. Learning transferable architectures for scalable image recognition. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2018. pp. 8697–8710. https://doi.org/10.1109/CVPR.2018.00907. http://openaccess.thecvf.com/content_cvpr_2018/

  12. Li L, Talwalkar A. Random search and reproducibility for neural architecture search. In A. Globerson and R. Silva, editors, Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019. AUAI Press, 2019. pp. 129. http://auai.org/uai2019/proceedings/papers/129.pdf

  13. Liu H, Simonyan K, Yang Y. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. https://openreview.net/forum?id=S1eYHoC5FX

  14. Kandasamy K, Neiswanger W, Schneider J, Póczos B, Xing EP. Neural architecture search with bayesian optimisation and optimal transport. In: Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, editors. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018. Montréal: Canada; 2018. p. 2020–9 http://papers.nips.cc/paper/7472.

    Google Scholar 

  15. He X, Zhao K, Chu X. Automl: A survey of the state-of-the-art. Knowledge-based Systems. 2021;212:106622. https://doi.org/10.1016/j.knosys.2020.106622.

    Article  Google Scholar 

  16. Elsken T, Metzen JH, Hutter F. Efficient multi-objective neural architecture search via lamarckian evolution. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. https://openreview.net/forum?id=ByME42AqK7

  17. Lu Z, Whalen I, Boddeti V, Dhebar Y, Deb K, Goodman E, Banzhaf W. Nsga-net: a multi-objective genetic algorithm for neural architecture search. arXiv preprint 2018. arXiv:1810.03522

  18. Pham H, Guan MY, Zoph B, Le QV, Dean J. Efficient neural architecture search via parameter sharing. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018. pp. 4092–4101. http://proceedings.mlr.press/v80/pham18a.html

  19. Zhong G, Li T, Jiao W, Wang L, Dong J, Liu C. DNA computing inspired deep networks design. Neurocomputing. 2020;382:140–7. https://doi.org/10.1016/j.neucom.2019.11.098.

    Article  Google Scholar 

  20. Keogh EJ, Mueen A. Curse of dimensionality. In C. Sammut and G. I. Webb, editors, Encyclopedia of Machine Learning. Springer, 2010. pp. 257–258. https://doi.org/10.1007/978-0-387-30164-8_192

  21. Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res. 2012;13:281–305 http://dl.acm.org/citation.cfm?id=2188395.

    MathSciNet  MATH  Google Scholar 

  22. Golovin D, Solnik B, Moitra S, Kochanski G, Karro J, Sculley D. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017. ACM, 2017. pp. 1487–1495. https://doi.org/10.1145/3097983.3098043

  23. Mutny M, Krause A. Efficient high dimensional bayesian optimization with additivity and quadrature fourier features. In: Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, editors. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018. Montréal: Canada; 2018. p. 9019–30 http://papers.nips.cc/paper/8115.

    Google Scholar 

  24. Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary MMA, Prabhat M, Adams RP. Scalable bayesian optimization using deep neural networks. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings. JMLR.org, 2015. pp. 2171–2180. http://proceedings.mlr.press/v37/snoek15.html

  25. Hutter F, Hoos HH, Leyton-Brown K. Sequential model-based optimization for general algorithm configuration. In C. A. C. Coello, editor, Learning and Intelligent Optimization - 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers, volume 6683 of Lecture Notes in Computer Science. Springer, 2011. pp. 507–523. https://doi.org/10.1007/978-3-642-25566-3_40

  26. Bergstra J, Bardenet R, Bengio Y, Kégl B. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pp. 2546–2554, 2011. http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization

  27. Chen B, Wu H, Mo W, Chattopadhyay I, Lipson H. Autostacker: a compositional evolutionary learning system. In H. E. Aguirre and K. Takadama, editors, Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, Kyoto, Japan, July 15-19, 2018.  ACM, 2018. pp. 402–409. https://doi.org/10.1145/3205455.3205586

  28. Lorenzo PR, Nalepa J, Kawulok M, Ramos LS, Pastor JR. Particle swarm optimization for hyper-parameter selection in deep neural networks. In P. A. N. Bosman, editor, Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, Berlin, Germany, July 15-19, 2017. ACM, 2017. pp. 481–488. https://doi.org/10.1145/3071178.3071208

  29. Maher M, Sakr S. Smartml: A meta learning-based framework for automated selection and hyperparameter tuning for machine learning algorithms. In M. Herschel, H. Galhardas, B. Reinwald, I. Fundulaki, C. Binnig, and Z. Kaoudi, editors, Advances in Database Technology - 22nd International Conference on Extending Database Technology, EDBT 2019, Lisbon, Portugal, March 26-29, 2019. OpenProceedings.org, 2019. pp. 554–557. https://doi.org/10.5441/002/edbt.2019.54

  30. Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science. 2006;313(5786):504–7. https://doi.org/10.1126/science.1127647.

    Article  MathSciNet  MATH  Google Scholar 

  31. Guo P, Chen CP, Sun Y. An exact supervised learning for a three-layer supervised neural network. In Proceedings of 1995 International Conference on Neural Information Processing (ICONIP’95), volume 2. Beijing, China, 1995. pp. 1041–1044. https://www.researchgate.net/publication/318445413_An_Exact_Supervised_Learning_for_a_Three-Layer_Supervised_Neural_Network

  32. Guo P, Lyu MR. A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data. Neurocomputing. 2004;56:101–21. https://doi.org/10.1016/S0925-2312(03)00385-0http://www.sciencedirect.com/science/article/pii/S0925231203003850.

    Article  Google Scholar 

  33. Guo P, Zhao D, Han M, Feng S. Pseudoinverse learners: New trend and applications to big data. In Recent Advances in Big Data and Deep Learning, Proceedings of the INNS Big Data and Deep Learning Conference INNSBDDL 2019, held at Sestri Levante, Genova, Italy 16-18 April 2019. Springer, 2019. pp. 158–168. https://doi.org/10.1007/978-3-030-16841-4_17

  34. Wang J, Guo P, Xin X. Review of pseudoinverse learning algorithm for multilayer neural networks and applications. In T. Huang, J. Lv, C. Sun, and A. V. Tuzikov, editors, Advances in Neural Networks - ISNN 2018 - 15th International Symposium on Neural Networks, ISNN 2018, Minsk, Belarus, June 25-28, 2018, Proceedings, volume 10878 of Lecture Notes in Computer Science. Springer, 2018. pp. 99–106. https://doi.org/10.1007/978-3-319-92537-0_12

  35. Wang K, Guo P. An ensemble classification model with unsupervised representation learning for driving stress recognition using physiological signals. IEEE Trans Intell Transp Syst, 2020. (Early Access). https://doi.org/10.1109/TITS.2020.2980555, https://ieeexplore.ieee.org/document/9040898/.

  36. Wang K, Guo P, Luo A-L. A new automated spectral feature extraction method and its application in spectral classification and defective spectra recovery. Mon Not R Astron Soc. 2017;465(4):4311–24. https://doi.org/10.1093/mnras/stw2894.

    Article  Google Scholar 

  37. Guo P, Lyu MR. Pseudoinverse learning algorithm for feedforward neural networks. In: Mastorakis NE, editor. Advances in Neural Networks and Applications., ppAthens, Greece: World Scientific and Engineering Society Press; 2001. p. 321–326.

    Google Scholar 

  38. Wang K, Guo P, Xin X, Ye Z. Autoencoder, low rank approximation and pseudoinverse learning algorithm. In 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, Banff, AB, Canada, October 5-8, 2017. IEEE, 2017. pp. 948–953. https://doi.org/10.1109/SMC.2017.8122732

  39. Wang K, Guo P, Yin Q, Luo AL, Xin X. A pseudoinverse incremental algorithm for fast training deep neural networks with application to spectra pattern recognition. In 2016 International Joint Conference on Neural Networks, IJCNN 2016, Vancouver, BC, Canada, July 24-29, 2016. pp. 3453–3460. https://doi.org/10.1109/IJCNN.2016.7727642

  40. Guo P, Lyu MR, Chen CLP. Regularization parameter estimation for feedforward neural networks. IEEE Trans Syst Man Cybern B Cybern. 2003;33(1):35–44. https://doi.org/10.1109/TSMCB.2003.808176.

  41. Xu X, Liu C, Zhang X. Laplacian black box variational inference. In Proceedings of the International Conference on Big Data and Internet of Things, London, United Kingdom, December 20-22, 2017. ACM, 2017. pp. 91–95. https://doi.org/10.1145/3175684.3175700.

  42. Xu B, Guo P. Pseudoinverse learning algorithm for fast sparse autoencoder training. In 2018 IEEE Congress on Evolutionary Computation, CEC 2018, Rio de Janeiro, Brazil, July 8-13, 2018, pp. 1–6. https://doi.org/10.1109/CEC.2018.8477907

  43. Rifai S, Vincent P, Muller X, Glorot X, Bengio Y. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011. pp. 833–840. https://doi.org/10.1007/978-3-642-23783-6_41.

  44. Rifai S, Mesnil G, Vincent P, Muller X, Bengio Y, Dauphin YN, Glorot. Higher order contractive auto-encoder. In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part II. 2011. pp. 645–660. https://doi.org/10.1007/978-3-642-23783-6_41.

  45. Thornton C, Hutter F, Hoos HH, Leyton-Brown K. Auto-weka: combined selection and hyperparameter optimization of classification algorithms. In I. S. Dhillon, Y. Koren, R. Ghani, T. E. Senator, P. Bradley, R. Parekh, J. He, R. L. Grossman, and R. Uthurusamy, editors, The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013. ACM, 2013. pp. 847–855. https://doi.org/10.1145/2487575.2487629.

  46. Guo P, Yin Q. Synergetic learning systems: Concept, architecture, and algorithms. preprint, arXiv, 01 2020. Presented at the third China Systems Science Conference (CSSC2019), Changsha, May 18-19, 2019. (in Chinese). https://doi.org/10.13140/RG.2.2.13681.12644/1.

Download references

Funding

The study is funded by the National Key Research and Development Program of China (No. 2018AA A0100203), the China Postdoctoral Science Foundation (2020M682348), the Joint Research Fund in Astronomy (U1531242) under cooperative agreement between the National Natural Science Foundation of China and Chinese Academy of Sciences (CAS), and the Key Research Foundation of Henan Higher Education Institutions (21A520002).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ping Guo.

Ethics declarations

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of Interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, K., Guo, P. A Robust Automated Machine Learning System with Pseudoinverse Learning. Cogn Comput 13, 724–735 (2021). https://doi.org/10.1007/s12559-021-09853-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-021-09853-6

Keywords

Navigation