Skip to main content
SpringerLink
Log in

Optimal synchronized control of nonlinear coupled harmonic oscillators based on actor–critic reinforcement learning

  • Original Paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

A distributed optimal control algorithm based on adaptive neural network is proposed for the synchronized control problem of a class of second-order nonlinear coupled harmonic oscillators. Firstly, the graph theory is used to establish the coupling relationship between the harmonic oscillator models; secondly, the neural network is used to fit the unknown nonlinearity in the harmonic oscillator model, and the virtual controller and the actual controller are designed based on the backstepping method; then, according to the state error and the controller, the cost function and the HJB function are designed. Since the HJB function cannot be solved directly, the critic neural network approximates its solution. The above two neural networks constitute a simplified reinforcement learning to achieve optimal consistent control of nonlinear coupled harmonic oscillators. Finally, the stability and effectiveness of the scheme are verified by the Lyapunov stability theorem and numerical simulation, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Yu, D., Yang, M., Liu, Y.-J., Wang, Z., Chen, C.L.P.: Adaptive fuzzy tracking control for uncertain nonlinear systems with multiple actuators and sensors faults. IEEE Trans. Fuzzy Syst. 31, 104–116 (2022)

    Article  Google Scholar 

  2. Yu, D., Long, J., Chen, C.P., Wang, Z.: Bionic tracking-containment control based on smooth transition in communication. Inf. Sci. 587, 393–407 (2022)

    Article  Google Scholar 

  3. Feng, J., Yu, F., Zhao, Y.: Exponential synchronization of nonlinearly coupled complex networks with hybrid time-varying delays via impulsive control. Nonlinear Dyn. 85(1), 621–632 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  4. Song, Q., Wen, G., Meng, D., Liu, Z.-W., Liu, F.: Distributed impulsive control for signed networks of coupled harmonic oscillators with sampled positions. IEEE Trans. Control Netw. Syst. 8(1), 111–122 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  5. Yang, Z., Li, S., Yu, D., Chen, C.L.P.: BLS-based formation control for nonlinear multi-agent systems with actuator fault and input saturation. Nonlinear Dyn. 109(4), 2657–2673 (2022)

    Article  Google Scholar 

  6. Jin, X., Wang, Z., Zhao, J., Yu, D.: Swarm control for large-scale omnidirectional mobile robots within incremental behavior. Inf. Sci. 614, 35–50 (2022)

    Article  Google Scholar 

  7. Tuna, S.E.: Synchronization of harmonic oscillators under restorative coupling with applications in electrical networks. Automatica 75, 236–243 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  8. Ballard, L., Cao, Y., Ren, W.: Distributed discrete-time coupled harmonic oscillators with application to synchronised motion coordination. Control Theory Appl. IET 4, 806–816 (2010)

    Article  MathSciNet  Google Scholar 

  9. Ma, G., Ren, J., Liu, Y., Lu, G.: Distributed event-triggered impulsive control for synchronization of coupled harmonic oscillators. IEEE Access 9, 126-231–126-240 (2021)

    Article  Google Scholar 

  10. Zhang, H., Wu, Q., Ji, J.: Synchronization of discretely coupled harmonic oscillators using sampled position states only. IEEE Trans. Autom. Control 63(11), 3994–3999 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  11. Wang, J., Xia, J., Shen, H., Xing, M., Park, J.H.: \(h_\infty \) synchronization for fuzzy markov jump chaotic systems with piecewise-constant transition probabilities subject to PDT switching rule. IEEE Trans. Fuzzy Syst. 29(10), 3082–3092 (2021)

    Article  Google Scholar 

  12. Shen, H., Hu, X., Wang, J., Cao, J., Qian, W.: Non-fragile \(h_\infty \) synchronization for Markov jump singularly perturbed coupled neural networks subject to double-layer switching regulation. IEEE Trans. Neural Netw. Learn. Syst. 34(5), 2682–2692 (2023)

    Article  MathSciNet  Google Scholar 

  13. Marcheggiani, L., Chacón, R., Lenci, S.: On the synchronization of chains of nonlinear pendula connected by linear springs. Eur. Phys. J. Spec. Top. 223(4), 729–756 (2014)

    Article  Google Scholar 

  14. Ren, W.: Synchronization of coupled harmonic oscillators with local interaction. Automatica 44(12), 3195–3200 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  15. Zhou, J., Zhang, H., Xiang, L., Wu, Q.: Synchronization of coupled harmonic oscillators with local instantaneous interaction. Automatica 48(8), 1715–1721 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  16. Su, H., Wang, X., Lin, Z.: Synchronization of coupled harmonic oscillators in a dynamic proximity network. Automatica 45(10), 2286–2291 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  17. Qin, J., Li, M., Shi, Y., Ma, Q., Zheng, W.X.: Optimal synchronization control of multiagent systems with input saturation via off-policy reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 30(1), 85–96 (2019)

    Article  MathSciNet  Google Scholar 

  18. Liu, Y., Wang, Z.: Optimal output synchronization of heterogeneous multi-agent systems using measured input-output data. Inf. Sci. 582, 462–479 (2022)

    Article  MathSciNet  Google Scholar 

  19. Li, H., Wei, Q.: Optimal synchronization control for multi-agent systems with input saturation: a nonzero-sum game. Front. Inf. Technol. Electron. Eng. 23(7), 1010–1019 (2022)

    Article  Google Scholar 

  20. Zhang, D., Yao, Y., Wu, Z.: Reinforcement learning based optimal synchronization control for multi-agent systems with input constraints using vanishing viscosity method. Inf. Sci. 637, 118949 (2023)

    Article  Google Scholar 

  21. Tong, S., Sun, K., Sui, S.: Observer-based adaptive fuzzy decentralized optimal control design for strict-feedback nonlinear large-scale systems. IEEE Trans. Fuzzy Syst. 26(2), 569–584 (2018)

    Article  Google Scholar 

  22. Cui, R., Yang, C., Li, Y., Sharma, S.: Adaptive neural network control of AUVs with control input nonlinearities using reinforcement learning. IEEE Trans. Syst. Man Cybern. Syst. 47(6), 1019–1029 (2017)

    Article  Google Scholar 

  23. Xu, H., Yu, D., Sui, S., Zhao, Y.-P., Chen, C.L.P., Wang, Z.: Nonsingular practical fixed-time adaptive output feedback control of MIMO nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 34(10), 7222–7234 (2023). https://doi.org/10.1109/TNNLS.2021.3139230

  24. Guo, X., Yan, W., Cui, R.: Neural network-based nonlinear sliding-mode control for an AUV without velocity measurements. Int. J. Control 92(3), 677–692 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  25. Yu, D., Chen, C.L.P., Xu, H.: Fuzzy swarm control based on sliding-mode strategy with self-organized omnidirectional mobile robots system. IEEE Trans. Syst. Man Cybern. Syst. 52(4), 2262–2274 (2022)

    Article  Google Scholar 

  26. Xu, H., Yu, D., Sui, S., Chen, C.L.P.: An event-triggered predefined time decentralized output feedback fuzzy adaptive control method for interconnected systems. IEEE Trans. Fuzzy Syst. 31(2), 631–644 (2023). https://doi.org/10.1109/TFUZZ.2022.3184834

  27. Wen, G., Chen, C.L.P., Feng, J., Zhou, N.: Optimized multi-agent formation control based on an identifier–actor–critic reinforcement learning algorithm. IEEE Trans. Fuzzy Syst. 26(5), 2719–2731 (2018)

    Article  Google Scholar 

  28. Yang, Y., Ding, Z., Wang, R., Modares, H., Wunsch, D.C.: Data-driven human–robot interaction without velocity measurement using off-policy reinforcement learning. IEEE/CAA J. Autom. Sin. 9(1), 47–63 (2022)

    Article  MathSciNet  Google Scholar 

  29. Qin, J., Li, M., Wang, J., Shi, L., Kang, Y., Zheng, W.X.: Optimal denial-of-service attack energy management against state estimation over an SINR-based network. Automatica 119, 109090 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  30. Li, Y., Tong, S.: Observer-based neuro-adaptive optimized control of strict-feedback nonlinear systems with state constraints. IEEE Trans. Neural Netw. Learning Syst 33(7), 3131–3145. (2021). https://doi.org/10.1109/TNNLS.2021.3051030

  31. Kennedy, J.O.S.: ‘Introduction to dynamic programming,’’ in Dynamic Programming, pp. 27–49. Springer, Berlin (1986)

    Google Scholar 

  32. Pontryagin, L.S.: Mathematical Theory of Optimal Processes. CRC Press, Boca Raton (1987)

    Google Scholar 

  33. Luo, B., Liu, D., Wu, H.-N., Wang, D., Lewis, F.L.: Policy gradient adaptive dynamic programming for data-based optimal control. IEEE Trans. Cybern. 47(10), 3341–3354 (2017)

    Article  Google Scholar 

  34. Luo, B., Liu, D., Huang, T., Wang, D.: Model-free optimal tracking control via critic-only q-learning. IEEE Trans. Neural Netw. Learn. Syst 27(10), 2134–2144 (2016). https://doi.org/10.1109/TNNLS.2016.2585520

  35. Lan, J., Liu, Y.-J., Yu, D., Wen, G., Tong, S., Liu, L.: Time-varying optimal formation control for second-order multiagent systems based on neural network observer and reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. pp. 1–12 (2022)

  36. Wen, G., Xu, L., Li, B.: Optimized backstepping for tracking control of strict-feedback systems. IEEE Trans. Neural Netw. Learn. Syst. 29(8), 3850–3862 (2018)

  37. Yang, Y., Gao, W., Modares, H., Xu, C.-Z.: Robust actor-critic learning for continuous-time nonlinear systems with unmodeled dynamics. IEEE Trans. Fuzzy Syst. 30(6), 2101–2112 (2022)

    Article  Google Scholar 

  38. Wu, C., Yao, W., Luo, W., Pan, W., Sun, G., Xie, H., Wu, L.: A secure robot learning framework for cyber attack scheduling and countermeasure. IEEE Trans. Robot., 39(5), 3722–3738 (2023). https://doi.org/10.1109/TRO.2023.3275875

  39. Wu, C., Pan, W., Staa, R., Liu, J., Sun, G., Wu, L.: Deep reinforcement learning control approach to mitigating actuator attacks. Automatica 152, 110999 (2023)

    Article  MathSciNet  MATH  Google Scholar 

  40. Wen, G., Chen, C.L.P., Liu, Y.-J., Liu, Z.: Neural network-based adaptive leader-following consensus control for a class of nonlinear multiagent state-delay systems. IEEE Trans. Cybern. 47(8), 2151–2160 (2017)

    Article  Google Scholar 

Download references

Funding

This work was funded in part by the National Natural Science Foundation of China Grant under numbers 62006192, 62373302, 62333009, 72001214, and 62106283 and in part by the China Postdoctoral Science Foundation under number 2021TQ0269.

Author information

Authors and Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, 710072, Shaanxi, China

    Zhiyang Gu

  2. Air and Missile Defense College, Air Force Engineering University, Xi’an, Shaanxi, China

    Chengli Fan

  3. Unmanned System Research Institute, Northwestern Polytechnical University, Xi’an, 710072, Shaanxi, China

    Dengxiu Yu

  4. Center for Optical Imagery Analysis and Learning, Northwestern Polytechnical University, Xi’an, 710072, Shaanxi, China

    Zhen Wang

Authors
  1. Zhiyang Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengli Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dengxiu Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dengxiu Yu.

Ethics declarations

Conflict of interest

No potential conflict of interest was reported by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, Z., Fan, C., Yu, D. et al. Optimal synchronized control of nonlinear coupled harmonic oscillators based on actor–critic reinforcement learning. Nonlinear Dyn 111, 21051–21064 (2023). https://doi.org/10.1007/s11071-023-08957-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-023-08957-y

Keywords

Search

Navigation