GEOMETRIC JACOBIAN LINEARIZATION AND LQR THEORY

. The procedure of linearizing a control-aﬃne system along a non-trivial reference trajectory is studied from a diﬀerential geometric perspective. A coordinate-invariant setting for linearization is presented. With the linearization in hand, the controllability of the geometric linearization is characterized using an alternative version of the usual controllability test for time-varying linear systems. The various types of stability are deﬁned using a metric on the ﬁbers along the reference trajectory and Lyapunov’s second method is recast for linear vector ﬁelds on tangent bundles. With the necessary background stated in a geometric framework, linear quadratic regulator theory is understood from the perspective of the Maximum Principle. Finally, the resulting feedback from solving the inﬁnite time optimal control problem is shown to uniformly asymptotically stabilize the linearization using Lyapunov’s second method.

1. Introduction and background. Jacobian linearization is a standard concept in control theory and is used to study controllability, stability, and stabilization of non-linear systems. Indeed, Jacobian linearization provides the setting for a significant number of the control algorithms implemented in practice for non-linear systems.
In this paper, the abstract setting of "affine systems" in [6] is used to develop a geometric theory of linearization for control-affine systems evolving on a differentiable manifold. The objective is not so much to broaden the applicability of linearization techniques, but to better understand the structure of linearization and to make explicit some of the choices that are made without mention in the standard practice of linearization. The motivation, in part, comes from examples in mechanics. Given an affine connection, what it means to linearize about a reference trajectory has a natural geometric interpretation provided by the Jacobi equation of geodesic variations. In the general setup of control-affine systems, a geometric setup is thus far not found in the literature. However, certain ideas presented here are implicit in the paper [13], although the presented geometric framework is less abstract and so has more structure.
1.1. Linear systems and quadratic optimal control. In order to provide a point of reference for our geometric formulation of control systems and their linearizations, this section outlines the standard manner in which linearization and stabilization is normally carried out for control-affine systems on R n . This standard strategy is, of course, correct but it "sweeps under the rug" various issues, listed in Section 1.2, that must be addressed to develop a geometric theory.
Let Ω ⊂ R n be an open subset and let f 0 , f 1 , . . . , f m be smooth vector fields, possibly depending measurably on t, on Ω. Consider a control system with governing equations where γ : I → Ω is locally absolutely continuous and u : I → R m is bounded and measurable for some interval I ⊂ R. For the purposes of linearization, fix a reference trajectory γ ref corresponding to a reference control u ref , both defined on I ⊂ R. To define the linearization, for each t ∈ I define m + 1 smooth vector fields f a,t , a ∈ {0, 1, . . . , m}, by f a,t (x) = f a (t, x). The linearization of (1) along γ ref is then defined by where , Here Df a,t denotes the Jacobian of the vector field f a,t , a ∈ {0, 1, . . . , m}. Once the linearization (2) is obtained, its controllability properties can be investigated using the standard controllability Gramian (see Section 4.2 for restatements of the standard Gramian results). If the linearization (2) is ascertained to be controllable on I = [0, ∞), then (1) can be locally stabilized along the reference trajectory by stabilizing the linearization (2) using linear feedback [8,7]. That is, if L(V ; W ) denotes the set of linear maps from a vector space V to a vector space W , a map F : I → L(R n ; R m ) is chosen with the property that the closed-loop system is uniformly asymptotically stable. If F a (t) ∈ (R n ) * is the ath row of F (t), then the non-linear closed-loop system is locally uniformly asymptotically stable along the trajectory γ ref [15]. In practice one might design F through optimal control methods using a quadratic cost, the so-called linear quadratic regulator (LQR). We review this next. Let A : R → L(R n ; R n ) and B : R → L(R m ; R n ) be continuous maps and define a time-varying linear system on R n to be a pair (A, B) satisfying The solution to (4) satisfying x(t 0 ) = x 0 for t 0 ∈ I is given by the variations of constants formula, where Φ(t, t 0 ) is the state transition matrix. That is, t → Φ(t, t 0 ) is the solution to the homogeneous system Φ (t, t 0 ) = A(t)Φ(t, t 0 ) with initial condition Φ(t 0 , t 0 ) = id R n .
Problem 1 (Finite time LQR problem). For a time-varying linear system (A, B), find a pair (x(t), u(t)), satisfying the equation (4), defined on I = [t 0 , t 1 ] which minimizes the quadratic cost function where F (t 1 ) ∈ L(R n ; R n ) is symmetric and positive-semidefinite, Q : I → L(R n ; R n ) is symmetric and positive-semidefinite for each t ∈ I, and R : I → L(R m ; R m ) is symmetric and positive-definite for each t ∈ I.
Solutions to Problem 1 for a finite time t 1 can be obtained by variational methods or by applying the Maximum Principle [3,9]. The original presentation of the Maximum Principle is provided in [10]. Using either method, the existence of a solution to Problem 1 is equivalent to the existence of a solution K(t, t 1 ) to the differential Riccati equation where the time dependence has been dropped for brevity. A solution to the Riccati equation then provides an optimal, in the sense of Problem 1, linear state feedback u(t) = −R −1 (t)B T (t)K(t, t 1 )x(t) [8].
In the study of the infinite-time problem, the terminal cost F (t 1 ) is considered to be zero.
Problem 2 (Infinite time LQR problem). For a time-varying linear system (A, B), find a pair (x(t), u(t)), satisfying the equation (4), defined on I = [t 0 , ∞) which minimizes the quadratic cost function where Q : I → L(R n ; R n ) is symmetric and positive-semidefinite for each t ∈ I, and R : I → L(R m ; R m ) is symmetric and positive-definite for each t ∈ I.
If the time-varying linear system (A, B) is controllable, then solutions to the Riccati equation exist as t 1 → ∞ and, similar to the finite time problem, the solution to the Riccati equation provides a linear feedback which is optimal in the sense of Problem 2 [8]. Furthermore, under suitable uniformity bounds on the timevarying linear system (A, B) and cost data, the uniform asymptotic stability of the closed-loop linear system follows from a Lyapunov argument. The mathematical details and proofs the standard time-varying case as the final time tends to infinity are found in [8] and short survey of the standard case is provided in [14].
1.2. Contribution and organization. This paper is a systematic investigation of Jacobian linearization and LQR theory from a differential geometric perspective. The above procedures by which (2), the linear system, is obtained and stabilized pose some problems when the Euclidean state space of the non-linear system (1) is replaced with a differentiable manifold. The main issues are outlined below and form the organizational basis of this paper. To answer Question 1, time-dependent affine systems on M are introduced in Section 3 and serve as the base object on which our geometric theory is built. In Section 3.1, what it means to linearize an affine system along a reference trajectory is understood in terms of the tangent lift. In that sense, the tangent lift plays the role of the Jacobian in Section 1.1.
Question 2. Since the control system given by (1) has a state space that is an open subset of R n , there are several natural identifications that can be (unknowingly) made. The fact that the state space is naturally identified with each tangent space implies that (2) lives in a vector space. Where does the geometric version of (2) live?
The geometry dictates that the linearization is an affine system on T M . Thus, in contrast to the standard case, an affine system and its linearization live on different manifolds. In Section 3.2, the complexity of the above development is seen to reduce significantly when the reference trajectory is chosen to be an equilibrium point. Question 3. By virtue of (2) living in a vector space, its controllability can be checked using the controllability Gramian which makes use of the standard inner product and the coordinate-dependent family of maps {B(t)}. What does it mean for the geometric version of (2) to be controllable, and how can it be checked whether such a system has this property?
In Section 4, Question 3 is addressed when the controllability of the linearization is considered. To begin, the standard controllability results for (2) are recharacterized in Section 4.2. These re-characterizations have the feature that they may be applied directly to the geometric setting of the linearization and this is done in Section 4.3. In Section 5.1 the geometric versions of both the finite and infinite time LQR problems are formulated for the linearization of an affine system. The bulk of this work involves characterizing solutions to the finite time LQR problem using the Maximum Principle. Given the geometric setup of the linearization along a reference trajectory as an affine system on T M , the regular Maximum Principle statements do not directly apply without reverting to working in a set of coordinates. Thus, a new Maximum Principle statement is provided by Theorem 5.1 and proved in Section 5.5. In Sections 5.2, 5.3, and 5.4, the key ingredients to prove the Maximum Principle are introduced. These include the variational and adjoint equations, needle variations, tangent cones, and, of course, the Hamiltonian. For readers familiar with the Hamiltonian in the standard setup, the Hamiltonian presented in Section 5.4 will look "different." However, it maintains the required maximization properties-see Lemmata 5.13 and 5.14-required to prove the Maximum Principle. In Section 5.6, the Maximum Principle is used, answering Question 4, to characterize solutions to the finite time LQR problem. In this characterization the geometric version of the Riccati equation is given.
In Section 5.7, the infinite time LQR problem is addressed. In particular, solutions to the Riccati equation are shown to exist as the final time in the LQR problem tends to infinity. To prove their existence, the geometric analogue of the classical minimum energy controller is developed. Finally, the trajectory corresponding to the solution of the Riccati equation, as the final time tends to infinity, is shown to be optimal in the sense of the infinite time LQR problem.
Question 5. Again, since the state space is naturally identified with each tangent space, the stability of both the non-linear system and its linearization are measured with respect to the standard Euclidean norm. What are the appropriate norms in a geometric setting?
In Section 6, stability and stabilization by LQR methods of the linearization are formulated to complete the geometric picture of LQR theory. In Section 6.1, the stability definitions are provided for a fixed reference vector field X ref and for linear vector fields over X ref . (For example, the linearisation X T ref is a linear vector field over X ref .) These definitions are made using both a metric on M and a metric on the fibres of T M over image(γ ref ). Such metrics are naturally induced by choosing a Riemannian metric, G, on M . This answers Question 5 and contrasts with the standard setup of Section 1.1 where standard Euclidean metric on R n is used for both the non-linear system and its linearization. It is noted that a metric G on the fibres over image (γ ref ), unlike the Euclidean norm, will in general be timedependent. As a consequence, any stability definitions made in terms of G, will be dependent on the choice of metric unless the state manifold is compact.
In Section 6.2, Lyapunov's direct method for linear vector fields on tangent bundles is introduced. As in the standard setup, the stability of the linear vector field is inferred from the properties of a Lyapunov candidate and its derivative along integral curves of the linear vector field. The derivative of the Lyapunov candidate along an integral curve is defined using the Lie derivative operator given by (5). Question 6. What is a linear state feedback for the geometric version of (2)?
In Section 6.3, Question 6 is addressed. After making geometric sense of the terms "linear state-feedback" and "closed-loop system," it is proved that the linearization of an affine system is uniformly asymptotically stabilized using the linear statefeedback provided by the infinite time LQR problem. The proof follows by showing that the solution to the Riccati equation, as the final time tends to infinity, is a suitable Lyapunov function.
Question 7. After stabilizing the linearisation, how can the stabilizing linear state feedback be implemented for the non-linear system?
Finally, in Section 7, a rough answer to Question 7 is posed as future work.
2. Geometric constructions. The basic geometric notation follows [1]. Let M be an n-dimensional Hausdorff manifold with a C ∞ differentiable structure. The letter I will always denote an interval in R. The set of class C r functions on M is denoted by C r (M ). The tangent bundle of M is denoted by τ M : T M → M and the cotangent bundle by π M : T * M → M . If φ : M → N is a differentiable map between manifolds, its derivative is denoted T φ : T M → T N . For a vector bundle π : E → M , Γ r (E) denotes the sections of E that are of class C r . The subbundle V E ker(T π) ⊂ T E is the vertical bundle of E.
Let V and W be R-vector spaces. The notation L(V ; W ) denotes the set of linear maps from V to W . The dual space to V is defined by V * = L(V ; R). For any nonempty set U ⊂ V , the annihilator of U is a subspace of V * defined by Similarly, for any nonempty set S ⊂ V * , the coannihilator of S is a subspace of V defined by For a bilinear map T : If T is invertible then its inverse, the sharp map, is denoted by T : V * → V .
2.1. Time-dependent objects on a manifold. To define time-dependent vector fields on manifolds in a general way, following [13, §3] it is convenient to first introduce time-dependent functions (see also [2]). A Carathéodory function on M is a map φ : I × M → R with the property that φ t φ(t, ·) is continuous for each t ∈ I, and φ x φ(·, x) is Lebesgue measurable for each x ∈ M . A Carathéodory function φ is locally integrally bounded (LIB) if, for each compact subset K ⊂ M , there exists a positive locally integrable function ψ K : I → R such that |φ(t, x)| ≤ ψ K (t) for each x ∈ K. A Carathéodory function φ : I × M → R is of class C r if φ t is of class C r for each t ∈ I and is locally integrally of class C r (LIC r ) it is of class C r and if X 1 · · · X r φ t is LIB for all t ∈ I and X 1 , . . . , X r ∈ Γ ∞ (T M ).
A Carathéodory vector field on M is a map X : I × M → T M with the property that X(t, x) ∈ T x M and with the property that the function α · X : (t, x) → α(x)·X(t, x) is a Carathéodory function for each α ∈ Γ ∞ (T * M ). For a Carathéodory vector field X on M , denote by X t : M → T M the map X t (x) = X(t, x). A Carathéodory vector field X on M is locally integrally of class C r (LIC r ) if α · X is LIC r for every α ∈ Γ ∞ (T * M ). The set of LIC r vector fields on M is denoted by LIC r (T M ).
The classical theory of time-dependent vector fields with measurable time dependence gives the existence of locally absolutely continuous integral curves for LIC ∞ vector fields [12,Appendix C]. An integral curve γ : I → M is locally absolutely continuous (LAC) if, for any φ ∈ C ∞ (M ), the map t → φ • γ(t) is locally absolutely continuous. Let γ (t) denote the tangent vector to γ at t ∈ I, noting that this is defined for almost every t ∈ I. The flow of X ∈ LIC ∞ (T M ) is denoted by Φ X t0,t and the curve γ : t → Φ X t0,t (x 0 ) is the integral curve for X with initial condition γ(t 0 ) = x 0 .
Let γ : I → M be an LAC curve. A vector field along γ is a map ξ : I → T M with the property that ξ(t) ∈ T γ(t) M . A vector field ξ along γ is locally absolutely continuous (LAC) if it is LAC as a curve in T M . A weaker notion than that of an LAC vector field along γ is that of a locally integrable (LI) vector field along γ, which is a vector field ξ along γ having the property that the function t → α(γ(t)) · ξ(t) is locally integrable for every α ∈ Γ ∞ (T * M ).
Let X ∈ LIC ∞ (T M ) and let γ : I → M be an integral curve for X. There is a naturally defined Lie derivative operator along γ that maps LAC sections of T M along γ to LI sections of T M along γ. This operator, denoted by L X,γ , is defined by where V ∈ Γ 1 (T M ) and V γ is the LAC section of T M along γ defined by V γ (t) = V (γ(t)). One easily verifies in coordinates that, for an LAC vector field ξ along γ, L X,γ (ξ) is given in coordinates (x 1 , . . . , x n ) by where a summation over i ∈ {1, . . . , n} is implied. The Lie differentiation of LAC vector fields along a curve will play an important role in future developments, particularly in Sections 2.2 and 4.3. The geometric details of Lie differentiation for vector fields that depend measurably on time are provided in [13, §4].

2.2.
Tangent bundle geometry. The various ways to lift a vector field is a prominent geometric idea that arises frequently in future sections. These constructions are contained in [16]. Let π : E → M be a vector bundle. An LIC ∞ vector field X on E is linear if, for each t ∈ I, 1. X t is π-projectable (denote the resulting vector field on M by πX t ) and 2. X t is a linear morphism of vector bundles relative to the following diagram: That is, the induced mapping from π −1 (x) to T π −1 (πX t (x)) is a linear mapping of R-vector spaces. The flow of a linear vector field has the property that Φ X t0,t |E x : E x → E Φ πX t 0 ,t is a linear transformation.
A linear vector field on a vector bundle generalizes the notion of a time-varying differential equation in the following manner. Let V be a finite-dimensional R-vector space and consider on V a linear differential equation where A : R → L(V ; V ) is locally integrable. Now define an LIC ∞ linear vector field on the trivial bundle pr 1 : R × V → R, where pr 1 is the projection onto the first factor, by X A (τ, v) = ((τ, v), (1, A(τ )(v))). Here the projected vector field on the base space is simply πX A = ∂ ∂τ . This special case of a linear vector field has the feature that the vector bundle admits a natural global trivialization. The lack of this feature in general accounts for some of the additional complexity in our development.
Now consider the case when E is the tangent bundle of M . Let X ∈ Γ ∞ (T M ) and define the tangent lift of X as the vector field X T ∈ Γ ∞ (T T M ) obtained by

ANDREW D. LEWIS AND DAVID R. TYNER
The definition of the tangent lift can be extended to time-varying vector fields as follows. For X ∈ LIC ∞ (T M ), the tangent lift of X is the vector field . In natural coordinates (x 1 , . . . , x n , v 1 , . . . , v n ) for T M , the vector field X T (t, v x ) is given by the coordinate expression To provide an interpretation of the tangent lift, let γ : I → M be an integral curve of X ∈ LIC ∞ (T M ). A variation of X along γ is a map σ : I × J → M satisfying 1. J ⊂ R is an interval for which 0 ∈ int(J), 2. σ is continuous, 3. the map I t → σ s (t) σ(t, s) ∈ M is an integral curve for X for each s ∈ J, 4. the map J s → σ t (s) σ(t, s) ∈ M is LAC for each t ∈ I, 5. the map I t → d ds | s=0 σ t (s) ∈ T M is LAC, and 6. σ 0 = γ. Corresponding to a variation σ of X along γ is an LAC vector field V σ along γ defined by With this notation, the following result records some useful properties of the tangent lift.
Proposition 1. Let X : I × M → T M be an LIC ∞ vector field, let v x0 ∈ T x0 M , let t 0 ∈ I, and let γ : I → M be the integral curve of X satisfying γ(t 0 ) = x 0 . For a vector field Υ along γ satisfying Υ(t 0 ) = v x0 , the following statements are equivalent: 1. Υ is an integral curve for X T ; 2. there exists a variation σ of X along γ such that V σ = Υ; 3. L X,γ (Υ) = 0.
Proof. The equivalence of (1) and (2) will follow from the more general Proposition 2 below. Thus only the equivalence of (1) and (3) needs to be proved. This, however, follows directly from the coordinate expressions (5) and (6).
The cotangent version of X T , used in Section 5, is defined in a similar manner. For X ∈ LIC ∞ (T M ), the cotangent lift of X is the vector field X T * ∈ LIC ∞ (T T * M ) defined by In natural coordinates (x 1 , . . . , x n , p 1 , . . . , p n ) for T * M , the vector field X T * (t, α x ) is given by the coordinate expression The LIC ∞ vector fields X T and X T * define an LIC ∞ vector field In natural coordinates (x 1 , . . . , x n , v 1 , . . . , v n ) for T M , the coordinate expression for the vertical lift is 3. Affine systems and their linearization. In this section time-dependent affine systems on M are introduced. In Section 3.1 the linearization of an affine system on M along a non-trivial reference trajectory is obtained using the tangent lift. The resulting linearization has the structure of an affine system on T M . In Section 3.2, it is seen that the complexity of the above development reduces significantly at an equilibrium point.
The vector fields X are called local generators for D. A time-dependent affine subbundle on M is a subset A ⊂ R×T M with the property that, for each x 0 ∈ M , there exists a neighbourhood N and LIC ∞ vector fields X = {X 0 , X 1 , . . . , X k } on N such that The vector fields X are called local generators for A. The linear part of a time-dependent affine subbundle is the time-dependent distribution L(A) defined by L(A) (t,x) being the subspace of T x M upon which the affine subspace A (t,x) is modelled. If X are local generators for A as above, then the vector fields {X 1 , . . . , X k } are local linear generators for L(A). In the setting of [6], the next step is to define an "affine system" in A to be an assignment to each (t, . This amounts to specifying the control set for the system. However, in order to focus on the geometry associated with an affine system and its linearization, it is assumed that A (t,x) = A (t,x) . This essentially means that the controls are unrestricted. Accepting a slight abuse of notation, a time-dependent affine subbundle A will be called a time-dependent affine system. A trajectory for A is then an LAC curve γ : I → M with the property that γ (t) ∈ A (t,γ(t)) .
Note that the specification of an affine system does not provide a natural notion of a drift vector field and control vector fields. As seen in [6], the basic properties like controllability can depend on the choice of a drift vector field. For the geometric development of the linearization, this is a non-issue since it is natural to assume the presence of a reference vector field, cf. the discussion of Section 1.1. To be formal about this, a reference vector field for an affine system A is an 1 The vector field V σ should be thought of as being the result of linearizing in the direction of the A-variation σ. Using the geometric constructions of Section 2.2, these vector fields along γ ref arise as trajectories for a time-dependent affine system on T M . Such a time-dependent affine subbundle A T ref on T M is defined as follows.

This is a time-dependent affine subbundle since it possesses local generators
Proposition 2. Let A be a time-dependent affine system, and let X ref be a reference vector field with LAC reference trajectory γ ref , as above. For a vector field Υ along γ ref , the following statements are equivalent: Proof. (2)⇒(1) Let σ be an A-variation giving rise to the vector field V σ along γ ref .

Using a set of local generators
since σ s is a trajectory for A. Differentiating with respect to s at s = 0 gives noting that the corresponding infinitesimal variation is The convexity of the set of variations of a given order (see [4]) now ensures the existence of a variation for any trajectory Υ that covers γ ref .

3.2.
Linearization about an equilibrium point. The above developments concerning linearization about a reference trajectory simplify significantly when dealing with an equilibrium point. Here the development looks a lot more like the standard non-geometric setup. Let A be a time-dependent affine subbundle on M and let X ref : Proof. This follows directly from the coordinate representation (6) for the tangent lift.
Thus the tangent lift is vertical-valued on T x0 M . Since V vx 0 T M T x0 M this means that the linearization is a time-dependent linear affine system on T x0 M for which for some measurable curve b : To make this look more like the usual notion of a time-varying linear system, for each t ∈ I let U be a finite-dimensional R-vector space and let B(t) ∈ L(U ; T x0 M ) have the property that image(B(t)) = L(A) (t,x0) . Then the equation governing trajectories become for a measurable curve u : I → U . This then recovers the usual notion of a timedependent linear system.
Similarly, a trajectory for the linearized time- For v x0 ∈ T x0 M and t ≥ t 0 , the reachable set from v x0 is defined by With these notions of the reachable sets, the controllability of each system is defined as follows.
Recasting the standard results. In a step toward a geometric theory of Jacobian linearization, the standard setup of, for example, [5] is recast on general R-vector spaces. In doing so, the extra structure available with Euclidean spaces, in particular the standard inner product, is removed. Let U and V be R-vector spaces with dim(U ) = m and dim(V ) = n. Let A : R → L(V ; V ) and B : R → L(U ; V ) be continuous and define a time-varying affine subbundle A (A,B) on V by The solution to (10) satisfying ξ(t 0 ) = ξ 0 for t 0 ∈ I is given by, where The transition matrix then has the following properties: In the standard case the controllability of a time-varying linear system is equivalent to the controllability Gramian, having full rank for t > t 0 . This definition makes use of the standard inner product Inducing an inner product on U by a symmetric map R : I → L(U ; U * ) which is positive-definite for each t ∈ I yields a Gramian of the form The derivation of (13) follows directly from the standard case in [5] and the timevarying affine subbundle A (A,B) is controllable at t 0 if and only if W (t 0 , t) is surjective for t > t 0 . Later, in Section 5.7, the quadratic cost in the LQR problem provides a natural choice of an inner product. The notion of a controllability Gramian does not make sense in the geometric framework of Section 3. There is no natural way to construct the analogue of W (t 0 , t) for the linearization of a reference vector field X ref along a reference trajectory γ ref since (13) is an integral of maps {t → B(t)} that depend on a choice of coordinates. Therefore, an alternative characterization of controllability that can be applied in the geometric setting is needed. The following result gives one such characterization.
Proof. For notational convenience define Let v ∈ image(W (t 0 , t)). Then there exists a continuous control u : [5]. Since A, B, and u are continuous, there exists a sequence of partitions Then v can be written as A useful characterization of points in image(W (t 0 , t)) is provided by the next lemma.
Using the composition property of the transition matrix, apply Φ(t 0 , t) to any point in this set: The lemma now follows by comparison with (12).
If the system can be steered from 0 to Φ(t, t 0 )v, this part of the theorem will follow from the lemma. Let µ j ∈ U have the property that B(t j )µ j = b tj j ∈ {1, . . . , k}. Now consider the distributional control u = k j=1 c j δ tj µ j , where δ tj is the deltadistribution with support {t j }. Applying this control, by (11) Thus the distributional control u steers from 0 to Φ(t, t 0 )v, as desired. To show the distributional control u can be replaced with a sequence of piecewise continuous controls, consider the following lemma.
Lemma There exists a sequence of controls {u i } i∈N such that Proof. For j ∈ {1, . . . , k} and i ∈ N define otherwise. Now note that, using the Peano-Baker series, Because A is continuous, all terms in the Peano-Baker series go to zero at least as fast as ( 1 i ) 2 . Thus only the first term remains in the limit, giving The result now follows by taking u i = k j=1 u j,i .
Let {u i } i∈N be a sequence of controls defined by the preceding lemma. For each i ∈ N, by the first of the above lemmata. Therefore, the limit as i → ∞ is also in image(W (t 0 , t)). But, by (14), giving the result.  1.
where pr 1 the projection onto the first factor.
The following lemma records some useful properties of the representation of trajectories of A T ref . Lemma The following statements hold: 1. there exists a vector bundle endomorphism A : Proof. The first assertion follows since X T ref is a vector bundle mapping over X ref . The second part of the lemma is merely the definition of ξ X ref .
From the definition of the set in (2), the equivalence of (2) and (3)  2. if A is linearly controllable at t 0 along γ ref then it is controllable at t 0 along γ ref .
5. LQR and the maximum principle. In this section the main geometric structure for LQR theory is presented by characterizing solutions to the finite time LQR problem using the Maximum Principle as stated in Theorem 5.1. Although Theorem 5.1 is a new formulation of the Maximum Principle, the ideas required to prove it come for the existing formulations of the Maximum Principle. Thus, many of the technicalities follow from the standard versions of the Maximum Principle found in [9]. After providing the geometric versions of both the finite and infinite time LQR problems in Section 5.1, the bulk of this section builds the tools to prove Theorem 5.1 in Section 5.5. In Sections 5.2, 5.3, and 5.4, the variational and adjoint equations, needle variations, tangent cones, and, of course, the Hamiltonian are defined. Again, it is noted that the Hamiltonian presented in Section 5.4 will look "different" from the standard case but maintains the required maximization properties-see Lemmata 5.13 and 5.14-required to prove the Maximum Principle. In Section 5.6, the Maximum Principle is used to characterize solutions to the finite time LQR problem and the geometric version of the Riccati equation is given. In Section 5.7, solutions to the Riccati equation are shown to exist as the final time in the LQR problem tends to infinity. In arriving at this result, the geometric analogue of the minimum energy controller is defined. Finally, the trajectory corresponding to the solution of the Riccati equation, as the final time tends to infinity, is shown to be optimal in the sense of the infinite time LQR problem.
LetL : T M → R be a smooth map and define the fiber derivative as the map FL : T M → T * M given by In the natural coordinates for T M and T * M , the local representative of the fibre derivative is given by

The variational and adjoint equations.
In the standard theory of optimal control for non-linear controls systems on a manifold M , the variational equations are given by the linearization of the dynamics. A trajectory of the variational equations is interpreted as an infinitesimal variation arising from varying the initial conditions of a fixed trajectory on M . In the present geometric framework, the varying of initial conditions for a trajectory for A T ref corresponds to a variation in the fiber over γ ref at the initial time. In other words, the trajectories of the variational equation will be vertical. For an A T ref -variation Σ of Υ, a vector field V Σ along Υ is defined by As a consequence of property 5, the vector field V Σ (t) along Υ is vertical. In the natural coordinates (x, v, u, w) for T T M , V Σ (t) is given by For a vertical vector field Ξ along Υ, the following statements are equivalent: using a set of local generators . . , X k } are local generators for A. Differentiating (17) in coordinates with respect to s at s = 0 yields , where Ξ is a vector field along γ ref .
Note that Definition 5.3 agrees with the statement "the linearization of a linear system is the original linear system." The upshot is that the adjoint equation will evolve on T M ⊕ T * M , which allows for the effect of the cost to be incorporated into the adjoint equations for the extended system; see Definition 5.6.
, where Λ is a covector field along γ ref .
The adjoint equations will play an important role in the statement and proof of the Maximum Principle. The relationship between the adjoint equations and variational equations is provided by Proposition 5. Proof. This follows from a direct computation of d dt Λ(t); Ξ(t) using the coordinate versions of the adjoint and variational equations.
A geometric interpretation of the adjoint equations is that they describe the evolution of a hyperplane in T M along γ ref .
is a solution of the adjoint equations with initial condition

Needle variations and tangent cones.
Roughly speaking the tangent cone is constructed by pushing forward needle variations. Its properties are instrumental in proving the Maximum Principle. The key property of the tangent cone is convexity. The main role of the tangent cone is to approximate the reachable set and it is interpreted as the set of "directions" from which a trajectory can start. In the case of a linear system, both the reachable set R(v γ ref (t0) , t, t 0 ) and the tangent cone at time t are contained in the tangent space T γ ref (t) M . In fact, they are equal [9]. However, since the proof of the Maximum Principle makes use of the extended system in Definition 5.5, which is not linear because of the cost being quadratic, this means that the general setup to construct the tangent cone is still required.
To prove the Maximum Principle, it is advantageous to include the cost as a state variable by defining the extended system. Definition 5.5. The extended system, denoted byÂ T ref ⊂ T M × R, is defined by asking that a trajectoryΥ = (Υ, Υ 0 ) satisfies the differential equationṡ The adjoint and variational equations can be obtained as before from the linearization of the extended system along a trajectory that projects to the reference trajectory. The effect of the cost enters the adjoint and variational equations using the fiber derivative of the Lagrangian.
1. The extended variational equation is defined bẏ where Ξ is a vector field along γ ref .

The extended adjoint equation is defined bẏ
where Λ(t) is a covector field along γ ref .
The first step toward constructing the tangent cone involves defining needle variations for the extended system, Definition 5.5. The motivation for using needle variations versus some other variety of variations is that the constructions involving needle variations are enough prove to the Maximum Principle.
Definition 5.7. Let t 0 , t 1 ∈ R satisfy t 0 < t 1 . LetÂ T ref be an extended system with initial conditionsΥ(t 0 ) and X a section of L(A) along γ ref .

The variation of X associated to the fixed interval needle variation data
where J = [0, s 0 ] is an interval sufficiently small such that X θ (s, ·) : t → X θ (s, t) is a section of L(A) along γ ref for each s ∈ J. 3. Let t → Σ(X θ (s, t),Υ(t 0 ), t 0 , t) be the trajectory ofÂ T ref corresponding to X θ (s, ·) with the fixed interval needle variation data θ = (τ θ , θ , Z θ ). The fixed interval needle variation associated with X is defined by and is a vertical curve in V T M × R which projects to γ ref .
The existence of the derivative in (19) It is noted that almost every τ θ ∈ (t 0 , t 1 ] is a Lebesgue point. For the fixed interval needle variation data θ = (τ θ , θ , Z θ ), where τ θ is a Lebesgue point, the fixed interval needle variation v θ has the form, [9, §4.1]. In light of (20), fixed interval needle variations will be considered as elements of L(A) × R. The set of fixed interval needle variations at Lebesgue points form a cone in L(A)×R. More precisely, if v θ is a fixed interval needle variation with data θ = (τ θ , θ , Z θ ) and λ ∈ R ≥0 , then λv θ is a fixed interval needle variation with data (τ θ , λ θ , Z θ ). Assigning the notation λθ = (τ θ , λ θ , Z θ ) implies the relation v λθ = λv θ . The above constructions are now extended to allow for multiple variations of X to contribute to corresponding fixed interval needle variations. Definition 5.9. Let t 0 , t 1 ∈ R satisfy t 0 < t 1 . LetÂ T ref be an extended system with initial conditionsΥ(t 0 ) and X a section of L(A).
Finally we define fixed interval tangent cones for the extended system.
Definition 5.10. Let t 0 , t 1 ∈ R satisfy t 0 < t 1 . LetÂ T ref be an extended system with initial conditionsΥ(t 0 ) and X a section of L(A). For t ∈ [t 0 , t 1 ] define the fixed interval tangent cone at t, denoted by K(X,Υ(t 0 ), t 0 , t), as the closure of the coned convex hull of the set {Φ τ,t v| v is a fixed interval needle variation at a Lebesgue point τ }.
The next lemma tells us that points in the interior of the fixed interval tangent cone are in the reachable set.

The extended maximum Hamiltonian is the function
The following lemmata provide a relationship between the Hamiltonian and the tangent cones of Section 5.3. It is interesting to note that these maximization statements only involve properties of tangent cones and do not rely on the optimal control problem data and, although they are stated for the extended system, they hold for general non-linear systems.
Lemma 5.14 (Hamiltonian maximization and the fixed interval tangent cone). Let A be an affine system with linearization A T ref along γ ref and extended systemÂ T ref . Let t 0 , t 1 ∈ R satisfy t 0 < t 1 and let X(t) ∈ L(A) t,γ ref (t) . For each t ∈ [t 0 , t 1 ] let κ t be a convex cone in L(A) t,γ ref (t) containing K(X(t),Υ(t 0 ), t 0 , t) and suppose, for some time τ ∈ [t 0 , t 1 ], that there exists a covector (Λ(τ ), Let t →Υ(t) ⊕Λ(t) be a solution to the extended adjoint equation forÂ T ref along γ ref with the above property at time τ . Then, for almost every t ∈ [t 0 , τ ], Proof. Let χ t ∈ L(A) (t,γ ref (t)) . Then, by definition of the fixed interval tangent cone, By hypothesis, Now use the definition of the adjoint equations to obtain Since this holds for every χ t ∈ L(A) t,γ ref (t) , the lemma follows from Lemma 5.13.
Proof. Let (Υ * , X * ) be an optimal trajectory for Problem 3(1). The proof relies on the construction of a hyperplane which is used to define the final condition for the extended adjoint equations.
Proof. At each time t ∈ [t 0 , t 1 ] the reachable set at time t for the extended system is contained in T γ ref (t) M × R, as is the fixed interval tangent cone. To prove the lemma, the important non-trivial fact that the fixed interval tangent cone is contained in the reachable set is utilized, Lemma 5.11. Suppose that (0 γ ref (t1) , −1) ∈ int(K(X * ,Υ(t 0 ), t 0 , t 1 )). Then , t 0 , t 1 ) whose final cost are lower than Υ 0 * (t 1 ) which contradicts the hypothesis that (Υ * , X * ) is optimal.
(1)⇒(2) Let Υ ∈ Traj(A T ref , t 1 , t 0 ) and let K be a symmetric (0, 2)-tensor field along γ ref such that K(t) is positive-definite for each t ∈ [t 0 , t 1 ] and that K satisfies the Riccati equation with final condition K(t 1 ) = F (t 1 ). Note that M is a linear map. Now integrate both sides of over the interval [t 0 , t 1 ] and add the result to the cost to obtain Now in local coordinates (for brevity the time dependence is no longer indicated) the right hand side is Using the hypothesis, the cost becomes and the cost is then minimized by choosing a trajectory Υ such that (2)⇒(3) By the Maximum Principle, Theorem 5.1, this follows. where are linear maps for each t ∈ [t 0 , t 1 ] and satisfy Proof. Note that F L(Υ(t)) = Q (t)Υ(t). Assuming (1), let (Υ, Λ) be the pair satisfying ). That is, η is the integral curve of X T ref such that η(t 1 ) = Υ(t 1 ). Next define the pair (Υ(t),Λ(t)) byΥ(t) = K 1 (t)η(t) andΛ(t) = K 2 (t)η(t), respectively. The coordinate calculations and similarly show that (Υ(t),Λ(t)) and (Υ(t), Λ(t)) satisfy the same differential equation. Since K 1 (t 1 ) = id T M , and K 2 (t 1 ) = F (t 1 ), it follows that (Υ(t 1 ),Λ(t 1 )) = (Υ(t 1 ), F (t 1 )Υ(t 1 )), and the lemma follows by the uniqueness of solutions to differential equations.
It is now shown that, given any and Λ * (t 1 ) = F (t 1 )Υ(t 1 ). By the lemma it follows that If (Υ, Λ) satisfy (3) and K 1 (t) is invertible for all t ∈ [t 0 , t 1 ], then by the lemma the following linear relationship holds: To show that K(t) = K 2 (t)K −1 1 (t) satisfies the Riccati equation in statement (1), it is first observed that K(t 1 ) = F (t 1 ) by construction. Using the linear relationship Λ(t) = K(t)Υ(t), the costate equation is In coordinates the above equation becomeṡ Again, using the linear relationship of state and costate implies thaṫ Thus K(t) satisfies Remark 1. The coordinate expressioṅ recovers the standard Riccati equatioṅ

5.7.
Infinite time LQR problems. In this section the solution to Problem 3 (2) is constructed by extending the ideas of Theorem 5.15 to the infinite time case. This will require various uniformity bounds on the problem data and ensuring the existence of solutions to the Riccati equation as the final time tends to infinity. The idea is to construct the analogue of the minimum energy controller in the present geometric framework. We quickly recall from [5] the development of this controller. We suppose that we have a controllable time-varying linear system (A, B). Given an initial state x 0 ∈ R n and t 0 , t 1 ∈ R satisfying t 0 < t 1 , we seek a control u : I → R m that steers x 0 to the origin at time t 1 while minimizing the energy t1 t0 u(t) R m dt.
One can show that the control is given by where η satisfies W (t 0 , t 1 )η = x 0 . The trajectory corresponding to this control is To develop our analogue to this minimum energy control law, define W (t, t 1 ), a (2, 0)-tensor on T γ ref (t) M , as the solution to The differential equation (26) should be thought of as the geometric analogue of the formula d dt for the derivative of the standard controllability Gramian (12). The rule for differentiating with respect to second parameter is provided by the following lemma.
To complete the proof it is shown that Φ also satisfies the differential equation (27). Applying the "backward differentiation lemma" [1], The next lemma plays a central role in the rest of the proof.
, then there exists a section X 1 of L(A), linear in v 0 , such that, for the resulting trajectory Υ 1 ∈ Traj(A T ref , t 1 , t 0 ) and time t 2 (v 0 , t 0 ) ≤ t 1 , the following hold: Proof. In line with the standard minimum energy controller [5], define a vector field along γ ref by where Υ(t 0 ) = v 0 . The lemma will follow if (28) is a trajectory of A T ref . We claim, this is the trajectory prescribed by the following section of L(A): . If Υ(τ ), as defined in (28), is a trajectory, then Υ(τ ) must satisfy t0,τ η. Consider the following coordinate computations: Combining the above coordinate calculations gives ,τ W (t 0 , τ )η)) = − ι(τ )R(τ ) ι * (τ )Φ X T * t0,τ η, as desired. Now Lemmata 5.16 and 5.17 are used to prove that solutions to the Riccati equation exist as the final time in the LQR problem tends to infinity. Proposition 6. For fixed t 1 , let K t1 (t) be a solution to for each t ∈ [t 0 , t 1 ]. If A T ref is controllable, then 1. lim t1→∞ K t1 (t) =K(t) and 2.K(t) satisfies (29).
The calculations in the proof of Theorem 5.15 show that the value of the minimal cost and the solution to the Riccati equation are related by K t1 (t 0 )(Υ * (t 0 ), Υ * (t 0 )) = J(Υ * (t 0 ), t 0 , t 1 ).
Using the above facts, the limitK(t 0 ) is constructed as follows. Define a quadratic form K(t 0 ) by The limit in (30) exists because it is a bounded non-deceasing function of t 1 . The polarization identity, then defines a symmetric (0, 2)-tensorK(t 0 ) for all v, w ∈ T γ ref (t0) M .
Theorem 5.18. The trajectory corresponding to the section of L(A) defined by is optimal in the sense of Problem 3(2).
6. Stability and stabilization. In this section, stability and stabilization by LQR methods of the linearization are addressed to complete the geometric picture of LQR theory. In Section 6.1, the stability definitions for a fixed reference vector field X ref and for linear vector fields over X ref are defined by using a metric on M and a metric on the fibres of T M over image(γ ref ), respectively. Such metrics are naturally induced by choosing a Riemannian metric G on M . Note that, unless the state manifold is compact, these stability definitions depend on the choice of metric. In Section 6.2, Lyapunov's direct method for linear vector fields on tangent bundles is introduced. In Section 6.3, after making geometric sense of the terms "linear state-feedback" and "closed-loop system," it is proved that the linearization of an affine system is uniformly asymptotically stabilized using the linear state-feedback provided by the infinite time LQR problem.