Skip to main content
SpringerLink
Log in

Model Order Determination: A Multi-Objective Evolutionary Neural Network Scheme

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Knowing the relevant amount of information needed to correctly predict patterns present in a data series is an important question to address. This is known as the problem of model order determination and is not adequately solved yet. This paper proposes to determine model order based on the scheme of a topological dynamic neural network that examines the dimensionality of the non-linear function that reconstructs the process. The novelty of the approach lies in the use of a neural network optimized by an evolutionary multi-objective selection mechanism that is capable of determining model order and performing robust estimations based on joint minimization of the length of the past and of the prediction error. Since the size of the input layer of the neural network is associated with the model order, the results show that the model order can be determined by the Pareto-optimal solutions that emerge from the optimization process. The practicality of the model is demonstrated on three univariate examples extracted from a time series database.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Anders U, Korn O. Model selection in neural networks. Neural Netw. 1999;12(2):309–23.

    Article  Google Scholar 

  2. De Jan G G, Hyndman RJ. 25 years of time series forecasting. Int J Forecast. 2006;22(3):443–73.

    Article  Google Scholar 

  3. Cyril V, et al. Machine learning methods for solar radiation forecasting: a review. Renew Energy. 2017;105:569–82.

    Article  Google Scholar 

  4. Taylor SJ, Letham B. Forecasting at scale. Am Stat. 2018;72(1):37–45.

    Article  MathSciNet  Google Scholar 

  5. Georgia P, Tyralis H, Koutsoyiannis D. Predictability of monthly temperature and precipitation using automatic time series forecasting methods. Acta Geophys. 2018;66(4):807–31.

    Article  Google Scholar 

  6. Box GEP, et al. Time series analysis: forecasting and control. Hoboken: John Wiley & Sons; 2015.

    Google Scholar 

  7. Kalman RE. A new approach to linear filtering and prediction problems. J Basic Eng. 1960;82(1):35–45 (Transaction of the ASME).

    Article  MathSciNet  Google Scholar 

  8. Andy P, West M, Harrison J. Applied Bayesian forecasting and time series analysis. Boston: Springer; 1994. (Chapman and Hall/CRC).

    MATH  Google Scholar 

  9. Zhang CY, Chen CLP, Gan M, Chen L. Predictive deep Boltzmann machine for multiperiod wind speed forecasting. IEEE Trans Sustain Energy. 2015;6:1416–25.

    Article  Google Scholar 

  10. Busseti E, Osband I, Wong S. “Deep learning for time series modelling.” Technical report. Stanford: Stanford University; 2012.

    Google Scholar 

  11. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.

    Article  Google Scholar 

  12. Peter ZG, Qi M. Neural network forecasting for seasonal and trend time series. Eur J Oper Res. 2005;160(2):501–14.

    Article  MATH  Google Scholar 

  13. Khashei M, Bijari M. A novel hybridization of artificial neural networks and ARIMA models for time series forecasting. Appl Soft Comput. 2011;11(2):2664–75.

    Article  Google Scholar 

  14. da Matteo F, Yao X. Short-term load forecasting with neural network ensembles: a comparative study [application notes]. IEEE Comput Intell Mag. 2011;6(3):47–56.

    Article  Google Scholar 

  15. Rumelhart DE, Hinton GE, Williams RJ. Learning internal representation by error propagation. In: Rumelhart DE, McClelland JL, editors. Parallel distributed processing. Cambridge: MIT Press; 1986. (Vol 1, Chap 8).

    Chapter  Google Scholar 

  16. Lippmann RP. Pattern classification using neural networks. IEEE Commun Mag. 1989;27(11):47–50.

    Article  Google Scholar 

  17. Kamruzzaman Joarder, Syed Mahfuzul Aziz. A note on activation function in multilayer feedforward learning. Neural Networks, 2002. IJCNN'02. Proceedings of the 2002 International Joint Conference on. Vol. 1. IEEE, 2002.

  18. Principe JC, Chen B. Universal approximation with convex optimization: gimmick or reality? [Discussion forum]. IEEE Comput Intell Mag. 2015;10(2):68–77.

    Article  Google Scholar 

  19. He Xiangdong, Haruhiko Asada 1993 A new method for identifying orders of input-output models for nonlinear dynamic systems. American Control Conference. IEEE, 1993.

  20. Bomberger JD, Seborg DE. Determination of model order for NARX models directly from input-output data. J Process Control. 1998;8(5–6):459–68.

    Article  Google Scholar 

  21. Sragner L, Horvath G 2003 "Improved model order estimation for nonlinear dynamic systems." Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, 2003. Proceedings of the Second IEEE International Workshop on. IEEE, 2003.

  22. Darío B, Carvalho JP, Morgado-Dias F. Comparing different solutions for forecasting the energy production of a wind farm. Neural Comput Appl. 2020;32(20):15825–33.

    Article  Google Scholar 

  23. Kwok T-Y, Yeung D-Y. Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Trans Neural Netw. 1997;8(3):630–45.

    Article  Google Scholar 

  24. Yao X. Evolutionary artificial neural networks. Int J Neural Syst. 1993;4(03):203–22.

    Article  Google Scholar 

  25. Holland JH. Adaptation in natural and artificial systems. Ann Arbor: Univ. of Michigan Press; 1975.

    Google Scholar 

  26. Zheng Q, et al. Improvement of generalization ability of deep CNN via implicit regularization in two-stage training process. IEEE Access. 2018;6:15844–69.

    Article  Google Scholar 

  27. Qinghe Z, et al. PAC-Bayesian framework based drop-path method for 2D discriminative convolutional network pruning. Multidimens Syst Signal Process. 2020;31(3):793–827.

    Article  MathSciNet  MATH  Google Scholar 

  28. Qinghe Z, et al. Spectrum interference-based two-level data augmentation method in deep learning for automatic modulation classification. Neural Comput Appl. 2021;33(13):7723–45.

    Article  Google Scholar 

  29. Qinghe Z, et al. Layer-wise learning based stochastic gradient descent method for the optimization of deep convolutional neural network. J Intell Fuzzy Syst. 2019;37(4):5641–54.

    Article  Google Scholar 

  30. Qinghe Z, et al. A full stage data augmentation method in deep convolutional neural network for natural image classification. Discret Dyn Nat Soc. 2020;2020:1–11.

    MATH  Google Scholar 

  31. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436.

    Article  Google Scholar 

  32. Fonseca Carlos M, Fleming PJ. Genetic algorithms for multiobjective optimization: formulation discussion and generalization. Icga. 1993;93:416–23.

    Google Scholar 

  33. Fonseca CM, Fleming PJ. An overview of evolutionary algorithms in multiobjective optimization. Evol Comput. 1995;3(1):1–16.

    Article  Google Scholar 

  34. CoelloCoello CA. Evolutionary multi-objective optimization: a historical view of the field. IEEE Comput Intell Mag. 2006;1(1):28–36.

    Article  Google Scholar 

  35. Srinivas N, Deb K. Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evol Comput. 1994;2(3):221–48.

    Article  Google Scholar 

  36. Kalyanmoy D. Multi-objective optimization using evolutionary algorithms, vol. 16. Hoboken: John Wiley & Sons; 2001.

    MATH  Google Scholar 

  37. Kalyanmoy D, et al. A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. International Conference on Parallel problem solving from nature. Heidelberg: Springer; 2000.

    Google Scholar 

  38. Deb K, Jain H. An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: solving problems with box constraints. IEEE Trans Evol Comput. 2014;18(4):577–601.

    Article  Google Scholar 

  39. Figueiredo Mário AT, Anil K Jain. “Bayesian learning of sparse classifiers.” Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. Vol. 1. IEEE, 2001.

  40. Ligeiro R, Vilela Mendes R. Detecting and quantifying ambiguity: a neural network approach. Soft Comput. 2018;22(8):2695–703.

    Article  Google Scholar 

  41. Gecynalda S, da Gomes S, Teresa Ludermir B, Leyla Lima MMR. Comparison of new activation functions in neural network for forecasting financial time series. Neural Comput Appl. 2011;20(3):417–39.

    Article  Google Scholar 

  42. Karlik B, Vehbi Olgac A. Performance analysis of various activation functions in generalized MLP architectures of neural networks. Int J Artif Intell Expert Syst. 2011;1(4):111–22.

    Google Scholar 

  43. Hyndman RJ, Akram M. Time series data library. 2010. Available from internet: http://robjhyndman.com/TSDL.

  44. Norgaard M, Ravn O, Poulsen NK. NNSYSID-toolbox for system identification with neural networks. Math Comput Model Dyn Syst. 2002;8(1):1–20.

    Article  Google Scholar 

  45. Xue Yu, Wang Y, Liang J. A self-adaptive gradient descent search algorithm for fully-connected neural networks. Neurocomputing. 2022;478:70–80.

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by Fundação para a Ciência e a Tecnologia (FCT), through Portuguese national funds Ref. UIDB/50021/2020.

Author information

Authors and Affiliations

  1. Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal

    Rui Ligeiro

  2. INESC-ID/Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal

    Joao Paulo Carvalho

Authors
  1. Rui Ligeiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joao Paulo Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rui Ligeiro.

Ethics declarations

Conflict of Interest

The authors declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ligeiro, R., Carvalho, J.P. Model Order Determination: A Multi-Objective Evolutionary Neural Network Scheme. SN COMPUT. SCI. 3, 252 (2022). https://doi.org/10.1007/s42979-022-01134-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01134-9

Keywords

Search

Navigation