Homotopic policy iteration-based learning design for unknown linear continuous-time systems

Abstract

Recent results have emerged that policy iteration is a powerful reinforcement learning tool in designing a stabilizing control policy for continuous-time systems with unknown system dynamics. Policy iteration involves a model-based initialization stage, i.e., seeking an initial stabilizing control policy, which is, however, dependent on the full system dynamics including the drift dynamics and system input matrix. To remove such model requirements, this paper utilizes a homotopy-based initialization strategy for policy iteration, wherein a stabilizing control policy for continuous-time systems is obtained by gradually moving a stable system to the original system. We propose two homotopic policy iteration-based stabilizing control schemes, namely, a model-based design and a model-free design using system data, which are proved to place unstable poles into a stable region. The effectiveness of the proposed designs is validated through an illustrative example.

Introduction

Finding a stabilizing control without knowing the accurate system dynamics has been a long-standing goal in the field of system and control, which generates fruitful results on adaptive control (see e.g., Åström and Wittenmark, 2013, Goodwin and Sin, 2014, Ioannou and Sun, 2012, Krstić et al., 1995, Sastry and Bodson, 2011 and the references therein). It is often important for practical scenarios to design controllers with user-defined performance. It thus requires that adaptive controllers should guarantee system stability and performance. To this end, indirect adaptive controllers identify the system parameters, based on which an optimal control is derived within a model-based framework (Ioannou & Fidan, 2006). Extremum-seeking schemes are documented in Ariyur and Krstić (2003) and Scheinker and Krstić (2017) for system performance improvement with an analytical footing. Inverse optimality conditions are considered for adaptive controllers such as Freeman and Kokotović (2008) and Krstić and Tsiotras (1999).

As a method leveraging interactions between the environment and an agent, reinforcement learning (RL) e.g., (Abbasi-Yadkori and Szepesvári, 2011, Bradtke et al., 1994, Chen et al., 2021, Ibrahimi et al., 2012, Lewis, Vrabie, and Syrmos, 2012, Sutton and Barto, 1998, Vrabie et al., 2013), or approximate/adaptive dynamic programming (ADP) e.g., (Bertsekas, 1995, Jiang, Fan, et al., 2020, Jiang, Kiumarsi, et al., 2020, Lewis, Vrabie, and Vamvoudakis, 2012, Liu et al., 2017, Powell, 2007, Zhang et al., 2012), has recently been utilized to design control policies for both continuous-time (CT) and discrete-time (DT) systems. One structure for implementing RL algorithms is policy iteration, wherein the performance of a current control policy is evaluated and based on which an improved control policy is resulted (Kleinman, 1968, Lewis, Vrabie, and Syrmos, 2012, Sutton and Barto, 1998). RL algorithms via policy iteration can adaptively learn a stabilizing controller that minimizes certain performance indexes and guarantees the system stability.

This paper focuses on the stabilizing control design for CT systems without prior knowledge of the system dynamics. To obviate the system dynamics requirement in RL-based results (Doya, 2000, Murray et al., 2001), the Integral Reinforcement Learning (IRL) algorithm is proposed for CT systems without knowing prior knowledge of the drift dynamics (Lewis and Vrabie, 2009, Vrabie and Lewis, 2009, Vrabie et al., 2008, Vrabie et al., 2009). In Lewis and Vrabie (2009) and Vrabie et al. (2009), IRL algorithms are implemented via policy iteration for solving Linear Quadratic Regulator (LQR) problems and also for nonlinear systems. The IRL-based method directly learns the optimal control solution from real-time data along with the system trajectories (Lewis, Vrabie, & Syrmos, 2012). Further details of the IRL algorithms and their implementation for CT systems can be found in Chen et al., 2020, Chen, Modares, et al., 2019, Chen, Xie, et al., 2019, Jiang and Jiang, 2012, Lewis, Vrabie, and Syrmos, 2012 and Lewis, Vrabie, and Vamvoudakis (2012) and the references therein. Of note, value iteration-based results for the LQR design of CT systems are reported in Bian and Jiang, 2016, Bian and Jiang, 2021 and Vrabie et al. (2013).

Note that the policy iteration-based designs for CT systems require an initial stabilizing control policy, which, however, is largely dependent on prior knowledge of the full system dynamics including the drift dynamics and input matrix (see Kleinman, 1968, Kleinman, 1970, Vrabie et al., 2009 and the references therein). This results in a model-based initialization stage when implementing policy iteration. One way to approach a stabilizing control is a homotopy-based method (Feng and Lavaei, 2020a, Lamperski, 2020, Mobahi and Fisher, 2015). Homotopy can be used as an initialization strategy, the idea of which was studied in Broussard and Halyo (1983). Recently, Feng and Lavaei (2020b) considered improving the locally optimal solutions for CT systems via optimal decentralized control. Based on (Feng & Lavaei, 2020b), (Lamperski, 2020) studied the control policy design for DT systems. However, it is not clear how to solve the optimal CT solution without knowing the system dynamics. In contrast, a model-based projected gradient descent method was used in Feng and Lavaei (2020b) as the local search algorithm.

This paper aims to design a stabilizing control for linear CT systems without knowing the accurate system dynamics. We propose a homotopy-based policy iteration for linear CT systems, wherein a stabilizing control policy is now obtained by gradually moving an artificial stable system to the original control system. Note that the existing policy iteration-based results for CT systems usually required a model-based initialization stage, in which the full system dynamics including the drift dynamics and system input matrix may be used for design. In this paper, we utilize a homotopy-based strategy to remove such model requirements and propose homotopic policy iteration by integrating the homotopy with the policy iteration technique of CT systems. Using the homotopic policy iteration, we establish model-based and model-free designs for CT systems that place unstable closed-loop poles into a stable region. It is thus seen that we provide a method to reinforce the convergence rate for unknown linear CT systems.

The remainder of the paper is organized as follows. In Section 2, we formulate a stable closed-loop poles-oriented learning design problem for CT systems and review standard policy iteration-based techniques for LQR design. In Section 3, we give a homotopy-based policy iteration design for CT systems to obtain stable closed-loop poles within a model-based framework. In Section 4, we extend the model-based result of Section 3 to propose a model-free version using system data collected along with the trajectories of the system states. In Section 5, we present an illustrative example. In Section 6, the main results of this paper are summarized.

Notations: The notation ${ℂ}^{-}$ denotes the open left-half complex plane. Given a square matrix $X$, $\lambda \left(X\right)$ denotes its spectrum, and ${\sigma }_{min}\left(X\right)$ and ${\sigma }_{max}\left(X\right)$ are, respectively, the minimum and maximum singular values. The notation $\otimes$ indicates the Kronecker product. Given a matrix $X\in {\mathbb{R}}^{m\times n}$, $vec\left(X\right)={[x_1^T,x_2^T,\dots ,x_i^T,\dots ,x_{n-1}^T,x_n^T]}^T\in {\mathbb{R}}^{mn}$ denotes a vector with $x_i\in {\mathbb{R}}^m$ representing the $i$th column of the matrix $X$. Given a symmetric matrix $X\in {\mathbb{R}}^{m\times m}$, $vecs\left(X\right)={[x_{1,1},2x_{1,2},\dots ,2x_{1,m},x_{2,2},2x_{2,3},\dots ,2x_{m-1,m},x_{m,m}]}^T\in {\mathbb{R}}^{\frac{1}{2}m(m+1)}$ with $x_{i,j}$ being an entry. Given a vector $x_i\in {\mathbb{R}}^m$, $vecv\left(x_i\right)={[x_{i,1}^2,x_{i,1}{x}_{i,2},\dots ,x_{i,1}{x}_{i,m},x_{i,2}^2,x_{i,2}{x}_{i,3},\dots ,x_{i,m-1}{x}_{i,m},x_{i,m}^2]}^T\in {\mathbb{R}}^{\frac{1}{2}m(m+1)}$. Let $I_m$ be an identity matrix of size $m\times m$.