Skip to main content
SpringerLink
Log in

Neurodynamical Optimization

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

Dynamical (or ode) system and neural network approaches for optimization have been co-existed for two decades. The main feature of the two approaches is that a continuous path starting from the initial point can be generated and eventually the path will converge to the solution. This feature is quite different from conventional optimization methods where a sequence of points, or a discrete path, is generated. Even dynamical system and neural network approaches share many common features and structures, yet a complete comparison for the two approaches has not been available. In this paper, based on a detailed study on the two approaches, a new approach, termed neurodynamical approach, is introduced. The new neurodynamical approach combines the attractive features in both dynamical (or ode) system and neural network approaches. In addition, the new approach suggests a systematic procedure and framework on how to construct a neurodynamical system for both unconstrained and constrained problems. In analyzing the stability issues of the underlying dynamical (or ode) system, the neurodynamical approach adopts a new strategy, which avoids the Lyapunov function. Under the framework of this neurodynamical approach, strong theoretical results as well as promising numerical results are obtained.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aluffi-Pentini, F., Parisi, V. and Zirilli, F. (1984), A differential-equations algorithm for nonlinear equations, ACM Trans. on Math. Software 10(3), 299–316.

    Google Scholar 

  2. Aluffi-Pentini, F., Parisi, V. and Zirilli, F. (1984) Algorithm 617 DAFNE: A differential-equations algorithm for nonlinear equations, ACM Trans. on Math. Software, 10(3), 317–324.

    Google Scholar 

  3. Anstreicher, K. M. (1988), Linear programming and the Newton barrier flow, Math. Prog. 41, 367–373.

    Google Scholar 

  4. Arrow, K. J., Hurwicz, L. and Uzawa, H. (1958), Studies in Linear and Nonlinear Programming, Stanford University Press, Stanford, CA.

    Google Scholar 

  5. Barron, A. R. (1993), Universal approximation bounds for superpositions of a sigmoidal function, IEEE Trans. Inform. Theory 39(3), 930–945.

    Google Scholar 

  6. Boggs, P. T. (1971), The solution of nonlinear systems of equations by A-stable integration techniques, SIAM J. Numer. Anal. 8(4), 767–785.

    Google Scholar 

  7. Botsaris, C. A. and Jacobson, D. H. (1976), A Newton-type curvilinear search method for optimization, JMAA, 54, 217–229.

    Google Scholar 

  8. Botsaris, C. A. (1978), Differential gradient methods, JMAA 63, 177–198.

    Google Scholar 

  9. Botsaris, C. A. (1978), A curvilinear optimization method based upon iterative estimation of the eigensystem of the Hessian matrix, JMAA 63, 396–411.

    Google Scholar 

  10. Botsaris, C. A. (1978), A class of methods for unconstrained minimization based on stable numerical integration techniques, JMAA 63, 729–749.

    Google Scholar 

  11. Bouzerdoum, A. and Pattison, T. R. (1993), Neural network for quadratic optimization with bound constrains, IEEE Trans. Neural Networks 4, 293–304.

    Google Scholar 

  12. Branin, Jr. F. H., (1972), Widely convergent method for finding multiple solutions of simultaneous nonlinear equations, IBM Journal of Research and Development 16, 504–522.

    Google Scholar 

  13. Brown, A. A. and Bartholomew-Biggs, M. C. (1989), Some effective methods for unconstrained optimization based on the solution of systems of ordinary differential equations, JOTA 62(2), 211–224.

    Google Scholar 

  14. Brown, A. A. and Bartholomew-Biggs, M. C. (1989), ODE versus SQP methods for constrained optimization, JOTA 62(3), 371–386.

    Google Scholar 

  15. Chen, Y. H. and Fang, S. C. (1998), Solving convex programming problem with equality constraints by neural networks, Computers Math. Appl. 36, 41–68.

    Google Scholar 

  16. Chu, M. T. (1988), On the continuous realization of iterative processes, SIAM Review 30(3), 375–387.

    Google Scholar 

  17. Cichocki, A. and Unbehauen, R. (1993), Neural Networks for Optimization and Signal Processing. Wiley, Chichester.

    Google Scholar 

  18. Cichocki, A., Unbehauen, R., Weinzierl, K. and Holzel, R., (1996), A new neural network for solving linear programming problems, European J. Operational Res., 93, 244–256.

    Google Scholar 

  19. Chua, L. O. and Lin, G. N. (1984), Nonlinear programming without computation, IEEE Trans. Circuits Syst., 31, 182–188.

    Google Scholar 

  20. Cybenko, G. (1989), Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems, 2, 303–314.

    Google Scholar 

  21. Evtushenko, Yu. G. and Zhadan, V. G., (1978), A relaxation method for solving problems of non-linear programming, U.S.S.R. Comput. Math. Math. Phys. 17(4), 73–87.

    Google Scholar 

  22. Diener, I. and Schaback, R., (1990), An extended continuous Newton method, JOTA 67(1), 57–77.

    Google Scholar 

  23. Glazos, M. P., Hui, S. and Żak, S., (1998), Sliding modes in solving convex programming problems, SIAM J. Control Optim. 36, 680–697.

    Google Scholar 

  24. Goldstein, A. A. (1964), Convex programming in Hilbert space, Bulletin of American Mathematical Society 70, 709–710.

    Google Scholar 

  25. Han, Q., Liao, L.-Z., Qi, H. and Qi, L., (2001), Stability analysis of gradient-based neural networks for optimization problems, J. Global Optim. 19(4), 363–381.

    Google Scholar 

  26. Hassan, N. and Rzymowski, W., (1990), An ordinary differential equation in nonlinear programming, Nonlinear Analysis, Theory, Method & Applications 15(7), 597–599.

    Google Scholar 

  27. Haykin, S. S., (1994), Neural Networks: A Comprehensive Foundation, Prentice-Hall, Englewood Cliffs, NJ.

    Google Scholar 

  28. He, B. S., (1994), Solving a class of linear projection equations, Numerische Mathematik 68, 71–80.

    Google Scholar 

  29. He, B. S., (1997), A class of projection and contraction methods for monotone variational inequalities, Applied Mathematics and Optimization 35, 69–76.

    Google Scholar 

  30. He, B. S., (1999), Inexact implicit methods for monotone general variational inequalities, Mathematical Programming, 86(1), 199–217.

    Google Scholar 

  31. He, B. S. and Yang H., (2000) A neural network model for monotone linear asymmetric variational inequalities, IEEE Trans. Neural Networks, 11, 3–16.

    Google Scholar 

  32. Hopfield, J. J., (1982), Neural networks and physical systems with emergent collective computational ability, Proc. Natl. Acad. Sci. USA, 79, 2554–2558.

    Google Scholar 

  33. Hopfield, J. J., (1984), Neurons with graded response have collective computational properties like those of two-state neurons, Proc. Natl. Acad. Sci., 81, 3088–3092.

    Google Scholar 

  34. Hopfield, J. J. and Tank, D. W., (1985), Neural computation of decisions in optimization problems, Biolog. Cybernetics, 52, 141–152.

    Google Scholar 

  35. Hornik, K., (1991), Approximation capabilities of multilayer feedforward networks, Neural Networks, 4, 251–257.

    Google Scholar 

  36. Hornik, K., Stinchcombe, M. and White, H., (1989), Multilayer feedforward networks are universal approximators, Neural Networks, 2, 359–366.

    Google Scholar 

  37. Hou, Z.-G., Wu, C.-P. and Bao, P., (1998), A neural network for hierarchical optimization of nonlinear large-scale systems, International Journal of Systems Science 29(2), 159–166.

    Google Scholar 

  38. Incerti, S., Parisi, V. and Zirilli, F., (1979), A new method for solving nonlinear simultaneous equations, SIAM J. Numer. Anal. 16, 779–789.

    Google Scholar 

  39. Kennedy, M. P. and Chua, L. O., (1988), Neural networks for nonlinear programming, IEEE Trans. Circuits Syst. 35, 554–562.

    Google Scholar 

  40. Liao, L.-Z. and Qi, H., (1999), A neural network for the linear complementarity problem, Math. Comput. Modelling 29(3), 9–18.

    Google Scholar 

  41. Liao, L.-Z., Qi, H. and Qi, L., (2001), Solving nonlinear complementarity problems with neural networks: a reformulation method approach, JCAM 131(12), 343–359.

    Google Scholar 

  42. Lillo, W. E., Loh, M. H., Hui S. and Zak, S. H., (1993), On solving constrained optimization problems with neural networks: a penalty method approach, IEEE Trans. Neural Networks, 4, 931–940.

    Google Scholar 

  43. Maa, C. Y. and Shanblatt, M. A., (1992), Linear and quadratic programming neural network analysis, IEEE Trans. Neural Networks 3, 580–594.

    Google Scholar 

  44. Maa, C. Y. and Shanblatt, M. A., (1992), A two-phase optimization neural network, IEEE Trans. Neural Networks 3, 1003–1009.

    Google Scholar 

  45. Mangasarian, O. L., (1993), Mathematical programming in neural networks, ORSA J. Comput. 5(4), 349–360.

    Google Scholar 

  46. Moré, J. J., Garbow, B. S. and Hillstrom, K. E., (1981), Testing unconstrained optimization software, ACM Trans. Math. Software 7(1), 17–41.

    Google Scholar 

  47. Novaković, Z. R., (1990), Solving systems of non-linear equations using the Lyapunov direct method, Computers Math. Applic. 20(12), 19–23.

    Google Scholar 

  48. Pan, P.-Q., (1992), New ODE methods for equality constrained optimization (1) — equations, JCM 10(1), 77–92.

    Google Scholar 

  49. Pan, P.-Q., (1992), New ODE methods for equality constrained optimization (2) — algorithms, JCM 10(2), 129–146.

    Google Scholar 

  50. Polyak, B. T., (1966), Constrained minimization problems, USSR Computational Mathematics and Mathematical Physics 6, 1–50.

    Google Scholar 

  51. Rodríguez-Vázquez, A., Domínguez-Castro, R., Rueda, A., Huertas J. L. and Sánchez-Sinencio, E., (1990), Nonlinear switch-capacitor ‘neural’ networks for optimization problems, IEEE Trans. Circuits Syst. 37, 384–398.

    Google Scholar 

  52. Schäffler, S. and Warsitz, H., (1990), A trajectory-following method for unconstrained optimization, JOTA 67(1), 133–140.

    Google Scholar 

  53. Schnabel, R. B. and Eskow, E., (1990), A new modified Cholesky factorization, SIAM J. Sci. Stat. Comput. 11, 1136–1158.

    Google Scholar 

  54. Slotine, J.-J. E. and Li, W., (1991), Applied Nonlinear Control, Prentice-Hall, Englewood Cliffs, NJ.

    Google Scholar 

  55. Snyman, J. A., (1982), A new and dynamic method for unconstrained minimization, Appl. Math. Modelling 6, 449–462.

    Google Scholar 

  56. Solodov, M. V. and Tseng, P., (1996), Modified projection-type methods for monotone variational inequalities, SIAM J. Control and Optimization 34, 1814–1830.

    Google Scholar 

  57. Sudharsanan, S. and Sundareshan, M., (1991), Exponential stability and a systematic synthesis of a neural network for quadratic minimization, Neural Networks 4, 599–613.

    Google Scholar 

  58. Tanabe, K., (1980), A geometric method in nonlinear programming, JOTA 30(2), 181–210.

    Google Scholar 

  59. Tank, D. W. and Hopfield, J. J., (1986), Simple neural optimization networks: An A/D convert, signal decision circuit, and a linear programming circuit, IEEE Trans. Circuits Syst. 33, 533–541.

    Google Scholar 

  60. Teo, K. L., Wong, K. H. and Yan, W. Y., (1995), Gradient-flow approach for computing a nonlinear-quadratic optimal-output feedback gain matrix, JOTA 85, 75–96.

    Google Scholar 

  61. Vincent, T. L., Goh, B. S. and Teo, K. L., (1992), Trajectory-following algorithms for min-max optimization problems, JOTA 75, 501–519.

    Google Scholar 

  62. Wilde, N. G., (1969), A note on a differential equation approach to nonlinear programming, Management Science 15(11), 739–739.

    Google Scholar 

  63. Williems, J. L., (1970), Stability Theory of Dynamical Systems, Nelson.

  64. Wu, X., Xia, Y., Li, J. and Chen, W. K., (1996), A high performance neural network for solving linear and quadratic programming problems, IEEE Trans. Neural Networks, 7, 643–651.

    Google Scholar 

  65. Xia, Y., (1996), A new neural network for solving linear programming problems and its applications, IEEE Trans. Neural Networks 7, 525–529.

    Google Scholar 

  66. Xia, Y., (1996) A new neural network for solving linear and quadratic programming problems, IEEE Trans. Neural Networks 7, 1544–1547.

    Google Scholar 

  67. Xia, Y. and Wang, J., (1998), A general methodology for designing globally convergent optimization neural networks, IEEE Trans. Neural Networks 9, 1331–1343.

    Google Scholar 

  68. Yamashita, H., (1980), A differential equation approach to nonlinear programming, Math. Prog. 18, 155–168.

    Google Scholar 

  69. Zabczyk, J., (1992), Mathematical Control Theory: An Introduction, Birkhauser, Boston.

    Google Scholar 

  70. Żak, S. H., Upatising, V. and Hui, S., (1995) Solving linear programming problems with neural networks: a comparative study, IEEE Trans. Neural Networks 6, 94–104.

    Google Scholar 

  71. Żak, S. H., Upatising, V., Lillo, W. E. and Hui, S., (1994), A dynamical systems approach to solving linear programming problems. In: K. D. Elworthy, W. N. Everitt and E. B. Lee (eds.), Differential Equations, Dynamical Systems, and Control Science, Marcel Dekker, New York.

    Google Scholar 

  72. Zhang, S. and Constantinides, A. G., (1992), Lagrange programming neural network, IEEE Trans. Circuits Syst. 39, 441–452.

    Google Scholar 

  73. Zhang, X.-S., (2000), Neural Network in Optimization, Kluwer Academic Publishers, Dordrecht.

    Google Scholar 

  74. Zhou, Z. and Shi, Y., (1997), An ODE method of solving nonlinear programming, Computers Math. Applic. 34(1), 97–102.

    Google Scholar 

  75. Zhou, Z. and Shi, Y., (1998), A convergence of ODE method in constrained optimization, JMAA 218(1), 297–307.

    Google Scholar 

  76. Zirilli, F., (1982), The use of ordinary differential equations in the solution of nonlinear systems of equations, Powell, M. J. D., (ed.) Nonlinear Optimization 1981, Academic Press, London.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Kowloon, Hong Kong; E-mail

    Li-Zhi Liao

  2. Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong; E-mail

    Houduo Qi & Liqun Qi

Authors
  1. Li-Zhi Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Houduo Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liqun Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liao, LZ., Qi, H. & Qi, L. Neurodynamical Optimization. Journal of Global Optimization 28, 175–195 (2004). https://doi.org/10.1023/B:JOGO.0000015310.27011.02

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/B:JOGO.0000015310.27011.02

Search

Navigation