Skip to main content
SpringerLink
Log in

Nonlinear model structure detection and parameter estimation using a novel bagging method based on distance correlation metric

  • Original Paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

System identification has been applied in diverse areas over past decades. In particular, parametric modelling approaches such as linear and nonlinear autoregressive with exogenous inputs models have been extensively used due to the transparency of the model structure. Model structure detection aims to identify parsimonious models by ranking a set of candidate model terms using some dependency metrics, which evaluate how the inclusion of an individual candidate model term affects the prediction of the desired output signal. The commonly used dependency metrics such as correlation function and mutual information may not work well in some cases, and therefore, there are always uncertainties in model parameter estimates. Thus, there is a need to introduce a new model structure detection scheme to deal with uncertainties in parameter estimation. In this work, a distance correlation metric is implemented and incorporated with a bagging method. The combination of these two implementations enhances the performance of existing forward selection approaches in that it provides the interpretability of nonlinear dependency and an insightful uncertainty analysis for model parameter estimates. The new scheme is referred as bagging forward orthogonal regression using distance correlation (BFOR-dCor) algorithm. A comparison of the performance of the new BFOR-dCor algorithm with benchmark algorithms using metrics like error reduction ratio, mutual information, or the Reversible Jump Markov Chain Monte Carlo method has been carried out in dealing with several numerical case studies. For ease of analysis, the discussion is restricted to polynomial models that can be expressed in a linear-in-the-parameters form.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Billings, S.A.: Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains. Wiley, London (2013)

    Book  Google Scholar 

  2. Pope, K.J., Rayner, P.J.W.: Non-linear system identification using Bayesian inference. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP-94, 1994 vol. IV, pp. 457–460 (1994)

  3. Haber, R., Unbehauen, H.: Structure identification of nonlinear dynamic systems–a survey on input/output approaches. Automatica 26(4), 651–677 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  4. Aguirre, L.A., Letellier, C.: Modeling nonlinear dynamics and chaos: a review. Mathe. Prob. Eng. 2009, 35 (2009). doi:10.1155/2009/238960

  5. Billings, S.A., Coca, D.: Identification of NARMAX and related models. tech. rep., Department of Automatic Control and Systems Engineering, The University of Sheffield, UK, (2001)

  6. Guo, L.Z., Billings, S.A., Zhu, D.Q.: An extended orthogonal forward regression algorithm for system identification using entropy. Int. J. Control 81(4), 690–699 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  7. Koller, D., Sahami, M.: Toward optimal feature selection. In: In 13th International Conference on Machine Learning (1995)

  8. Billings, S.A., Wei, H.-L.: Sparse model identification using a forward orthogonal regression algorithm aided by mutual information. IEEE Trans. Neural Netw. 18(1), 306–310 (2007)

    Article  Google Scholar 

  9. Wei, H.-L., Billings, S.A.: Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information. Int. J. Model. Identif. Control 3(4), 341–356 (2008)

    Article  Google Scholar 

  10. Wang, S., Wei, H.-L., Coca, D., Billings, S.A.: Model term selection for spatio-temporal system identification using mutual information. Int. J. Syst. Sci. 44(2), 223–231 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Han, M., Liu, X.: Forward Feature Selection Based on Approximate Markov Blanket. In: Advances in Neural Networks-ISNN 2012, pp. 64–72, Springer, Berlin (2012)

  12. Baldacchino, T., Anderson, S.R., Kadirkamanathan, V.: Computational system identification for Bayesian NARMAX modelling. Automatica 49, 2641–2651 (2013)

    Article  MathSciNet  Google Scholar 

  13. Ninness, B., Brinsmead, T.: A Bayesian Approach to System Identification using Markov Chain Methods. Tech. Rep. EE02009, University of Newcastle, Australia, NSW (2003)

  14. Székely, G.J., Rizzo, M.L., Bakirov, N.K.: Measuring and testing dependence by correlation of distances. Annals Stat. 35(6), 2769–2794 (2007)

    Article  MATH  Google Scholar 

  15. Székely, G.J., Rizzo, M.L.: Energy statistics: a class of statistics based on distances. J. Stat. Plan. Inf. 143(8), 1249–1272 (2013)

    Article  MATH  Google Scholar 

  16. Chen, S., Billings, S., Luo, W.: Orthogonal least squares methods and their application to non-linear system identification. Int. J. Control 50(5), 1873–1896 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  17. Söderström, T., Stoica, P.: System Identification. Prentice Hall, New Jersey (1989)

    MATH  Google Scholar 

  18. Wei, H.-L., Balikhin, M.A., Billings, S.A.: Nonlinear time-varying system identification using the NARMAX model and multiresolution wavelet expansions. Tech. Rep. 829, The University of Sheffield, United Kingdom (2003)

  19. Wei, H.-L., Billings, S.A., Liu, J.: Term and variable selection for non-linear system identification. Int. J. Control 77(1), 86–110 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  20. Wei, H.-L., Billings, S.A., Zhao, Y., Guo, L.: Lattice dynamical wavelet neural networks implemented using particle swarm optimization for spatio-temporal system identification. IEEE Trans. Neural Netw. 20(1), 181–185 (2009)

  21. Rashid, M.T., Frasca, M., Ali, A.A., Ali, R.S., Fortuna, L., Xibilia, M.G.: Nonlinear model identification for Artemia population motion. Nonlinear Dyn. 69(4), 2237–2243 (2012)

    Article  MathSciNet  Google Scholar 

  22. Haynes, B.R., Billings, S.A.: Global analysis and model validation in nonlinear system identification. Nonlinear Dyn. 5(1), 93–130 (1994)

    Google Scholar 

  23. Billings, S.A., Wei, H.-L.: An adaptive orthogonal search algorithm for model subset selection and non-linear system identification. Int. J. Control 81(5), 714–724 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  24. Aguirre, L.A., Jácôme, C.: Cluster analysis of NARMAX models for signal-dependent systems. In: IEE Proceedings Control Theory and Applications, vol. 145, pp. 409–414, IET, July (1998)

  25. Feil, B., Abonyi, J., Szeifert, F.: Model order selection of nonlinear input-output models–a clustering based approach. J. Process Control 14(6), 593–602 (2004)

    Article  Google Scholar 

  26. Kukreja, S.L., Lofberg, J., Brenner, M.J.: A least absolute shrinkage and selection operator (LASSO) for nonlinear system identification. Syst. Identif. 14, 814–819 (2006)

    Google Scholar 

  27. Qin, P., Nishii, R., Yang, Z.-J.: Selection of NARX models estimated using weighted least squares method via GIC-based method and l 1-norm regularization methods. Nonlinear Dyn. 70(3), 1831–1846 (2012)

    Article  MathSciNet  Google Scholar 

  28. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. B 67(2), 301–320 (2005). (Statistical Methodology)

    Article  MathSciNet  MATH  Google Scholar 

  29. Hong, X., Chen, S.: An elastic net orthogonal forward regression algorithm. In: 16th IFAC Symposium on System Identification, pp. 1814–1819, July (2012)

  30. Sette, S., Boullart, L.: Genetic programming: principles and applications. Eng. Appl. Artif. Intell. 14(6), 727–736 (2001)

  31. Madár, J., Abonyi, J., Szeifert, F.: Genetic programming for the identification of nonlinear input-output models. Ind. Eng. Chem. Res. 44(9), 3178–3186 (2005)

    Article  Google Scholar 

  32. Martins, S.A.M., Nepomuceno, E.G., Barroso, M.F.S.: Improved structure detection for polynomial NARX models using a multiobjective error reduction ratio. J. Control Autom. Electr. Syst. 24(6), 764–772 (2013)

    Article  Google Scholar 

  33. Baldacchino, T., Anderson, S.R., Kadirkamanathan, V.: Structure detection and parameter estimation for NARX models in a unified EM framework. Automatica 48(5), 857–865 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  34. Teixeira, B.O., Aguirre, L.A.: Using uncertain prior knowledge to improve identified nonlinear dynamic models. J. Process Control 21(1), 82–91 (2011)

    Article  Google Scholar 

  35. Billings, S.A., Voon, W.S.F.: Correlation based model validity tests for nonlinear models. Tech. Rep. 285, The University of Sheffield, United Kingdom, October (1985)

  36. Billings, S.A., Wei, H.-L.: The wavelet-NARMAX representation: a hybrid model structure combining polynomial models with multiresolution wavelet decompositions. Int. J. Syst. Sci. 36(3), 137–152 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  37. Guo, Y., Guo, L., Billings, S., Wei, H.-L.: An iterative orthogonal forward regression algorithm. Int. J. Syst. Sci. 46(5), 776–789 (2015)

    Article  MathSciNet  Google Scholar 

  38. Billings, S.A., Chen, S., Backhouse, R.J.: The identification of linear and non-linear models of a turbocharged automotive diesel engine. Mech. Syst. Signal Process. 3(2), 123–142 (1989)

    Article  Google Scholar 

  39. Dietterich, T.G.: Machine Learning for Sequential Data: A Review. In: Structural, Syntactic, and Statistical Pattern Recognition, pp. 15–30, Springer, Berlin (2002)

  40. Efron, B.: Computers and the theory of statistics: thinking the unthinkable. SIAM Rev. 21, 460–480 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  41. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap, vol. 57 of Monographs on Statistics and Applied Probability. Chapman & Hall, London (1993)

    Book  Google Scholar 

  42. Davison, A.C.: Bootstrap Methods and their Application. Cambridge University Press, Cambridge (1997)

    Book  MATH  Google Scholar 

  43. Kukreja, S.L., Galiana, H., Kearney, R.: Structure detection of NARMAX models using bootstrap methods. In: Proceedings of the 38th IEEE Conference on Decision and Control, 1999. vol. 1, pp. 1071–1076 (1999)

  44. Kukreja,: A suboptimal bootstrap method for structure detection of NARMAX models. Tech. Rep. LiTH-ISY-R-2452, Linköpings universitet, Linköping, Sweden (2002)

  45. Wei, H.-L., Billings, S.A.: Improved parameter estimates for non-linear dynamical models using a bootstrap method. Int. J. Control 82(11), 1991–2001 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  46. Breiman, L.: Bagging predictors. Tech. Rep. 421, University of California, Berkeley, California, USA, September (1994)

  47. Hyndman, R.J., Athanasopoulos, G.: Forecasting: Principles and Practice. OTexts (2014). https://www.otexts.org/book/fpp

  48. James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning with Application in R, vol. 103 of Springer Texts in Statistics. Springer, Berlin (2013)

    Google Scholar 

  49. Sunspot Data, (2003)

  50. Kampstra, P.: Beanplot: A boxplot alternative for visual comparison of distributions. J. Stati. Softw. 28, 1–9 (2008)

  51. Lin, H., Varsik, J., Zirin, H.: High-resolution observations of the polar magnetic fields of the Sun. Solar Phys. 155(2), 243–256 (1994)

    Article  Google Scholar 

  52. Billings, S.A., Tao, Q.H.: Model validity tests for non-linear signal processing applications. Int. J. Control 54(1), 157–194 (1991)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors acknowledge the support for J. R. Ayala Solares from a University of Sheffield Full Departmental Fee Scholarship and a scholarship from the Mexican National Council of Science and Technology (CONACYT). The authors also gratefully acknowledge that this work was partly supported by EPSRC.

Author information

Authors and Affiliations

  1. Faculty of Engineering, Department of Automatic Control and Systems Engineering, The University of Sheffield, Sheffield, UK

    J. R. Ayala Solares & Hua-Liang Wei

Authors
  1. J. R. Ayala Solares
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hua-Liang Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hua-Liang Wei.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ayala Solares, J.R., Wei, HL. Nonlinear model structure detection and parameter estimation using a novel bagging method based on distance correlation metric. Nonlinear Dyn 82, 201–215 (2015). https://doi.org/10.1007/s11071-015-2149-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-015-2149-3

Keywords

Search

Navigation