Asymptotic problems in optimal control with a vanishing Lagrangian and unbounded data

In this paper we give a representation formula for the limit of the fnite horizon problem as the horizon becomes infinite, with a nonnegative Lagrangian and unbounded data. It is related to the limit of the discounted infinite horizon problem, as the discount factor goes to zero. We give sufficient conditions to characterize the limit function as unique nonnegative solution of the associated HJB equation. We also briefly discuss the ergodic problem.


Introduction
The main goal of this paper is to discuss, in the case of a vanishing Lagrangian l ≥ 0 and truly unbounded data and controls, the limit as t tends to +∞ of the finite horizon value function V(t, x) . = inf α t 0 l(y(τ ), α(τ )) dτ, and the limit as δ tends to 0 + of the discounted infinite horizon value function where f , l are given functions, α(τ ) ∈ A ⊂ IR m is the control and the trajectory is given bẏ y(τ ) = f (y(τ ), α(τ )), y(0) = x.
These limits have been extensively studied in the literature. On the one hand, the approximability of the infinite horizon value function by the finite horizon value functions is classically required in most applications (see [CHL]) and it also represents the key point of several comparison results by viscosity solution methods. On the other hand, recently a lot of work has been devoted to the study of the two ergodic limits lim t→+∞ V(t, x)/t and lim δ→0 + δ V δ (x). We refer to [BCD] for a presentation of the basic results in the deterministic case, and to [AL] for the stochastic case. The same questions have been addressed in L ∞ control problems (see [AB] and the references therein).
The main novelty of this paper is the generality of the hypotheses under which the results are obtained, suitable to a wide range of applications in the framework of optimal control theory. Precise assumptions will be stated in Section 2, here we just point out that we can consider coercive and non coercive nonnegative Lagrangians, with arbitrary growth in the state variable and without restrictions on the set Z .
For instance, the dynamics can be control-affine, f (x, a) = f 0 (x) + F (x), a , where f 0 , F are locally Lipschitz functions with linear growth in x. In particular we cover (nonlinear generalizations of) LQR problems with l(x, a) = x T Qx + a T Ra, where Q and R are symmetric matrices, R is positive definite and Q is positive semidefinite. We can also allow for control-affine Lagrangians, l(x, a) = l 0 (x) + l 1 (x)|a| with l 0 ≥ 0, l 1 > 0 continuous and with arbitrary growth in x, used in some economics models, mostly in singular stochastic control (see [FS] and the references therein). We show that the function Σ(x) . = lim t→+∞ V(t, x) is l.s.c. and we characterize it as the minimal nonnegative supersolution to the limit HJB equation at every x where it is finite. The representation formula, when A is compact, is given, as expected, by the value function of the so-called relaxed infinite horizon problem. Adding some mild assumptions on the data, it is also equal to the l.s.c. envelope of the infinite horizon value function, V * (x). When A is unbounded, the relaxed problem is not defined. In this case, we can still give a representation formula for Σ by introducing an extended infinite horizon problem, which has a compact control set. Denoting by V the value function of the extended problem, we prove that Σ coincides with the relaxed version of V and also with its l.s.c. envelope, V * , under the same assumptions as for A compact. In particular, in classical impulsive control problems, the extended setting is equivalent to the replacement of controls with measures. In Theorem 3.1 we give sufficient conditions to have V equal to V. We obtain the same characterizations for lim δ→0 + V δ (x), assuming V δ bounded.
In general, Σ is not u.s.c. and the limit HJB equation does not have a unique solution. We give explicit sufficient conditions under which Σ turns out to be continuous and the unique nonnegative solution to the HJB equation.
We spend a few words on the ergodic problem. Starting from the papers [AL] and [A], a huge amount of literature has been devoted to the subject, initially in the case of bounded domains or periodic data and under some global controllability assumptions. The first results have been developed and generalized in several directions (see e.g. [BR], [GLM], [QR], and the references therein). Here we focus our attention mainly on the case where the set Z = ∅ and the infinite horizon value function is finite, case in which the ergodic limits turn out to be zero. We limit ourselves to showing how it is possible, under periodicity of the data and a complete controllability condition, to obtain the results of [A] in our framework.
Some final bibliographical remarks. In this paper, we extend to the dynamics and Lagrangians described above, many results already proved when some of the data of the problem are bounded. In so doing we get some results new also for the compact control case. When the control set is unbounded, our approach is based on a compactification method introduced in [BrRa] (see also [MiRu]); for a more complete survey we refer to [BP] and the references therein. In particular, the finite horizon problem with both coercive and weakly coercive Lagrangians was treated in [RS], while exit-time problems with a nonnegative Lagrangian were investigated in [MS]. Moreover some optimality principles were extended in [M] to the HJB equations involved in several optimal control problems of this kind. This approach has also been applied to some stochastic control problems (see e.g. [MS2] and the references therein).
In Section 2 we state the problem precisely. In Section 3 we introduce the extended setting for A unbounded and give sufficient conditions in order to have the extended infinite horizon value function coinciding with V(x); then we define the relaxed and the relaxed extended problems. Section 4 is devoted to characterize the limit as t tends to +∞ of the finite horizon value functions, while the limit as δ tends to 0 + of the discounted value functions is studied in Section 6. In Section 5 we state a uniqueness result for the solution of the limit HJB equation. The ergodic problem is investigated in Section 7. The discounted and the ergodic problems have been treated in the last two sections, since they are studied under assumptions not required for the previous results. Notations. For any function u : R n → R ∪ {+∞}, we will denote the set {x ∈ R n : u(x) < +∞} by Dom(u). IR + . = [0, +∞[. A function ω : IR + × IR + → IR + is called a modulus if: ω(·, R) is increasing in a neighborhood of 0, continuous at 0, and ω(0, R) = 0 for every R > 0; ω(r, ·) is increasing for every r. Let D ⊂ R N for some N ∈ IN. ∀r > 0 we will denote by D r the closed set B(D, r), while D c r = IR N \ D r . Moreover, χ D will denote the characteristic function of D, namely for any x ∈ IR N we set χ D (x) = 1 if x ∈ D and χ D (x) = 0 if x / ∈ D.

Assumptions and statement of the problem
We consider a nonlinear control system having the forṁ and an undiscounted payoff where α(τ ) ∈ A ⊂ IR m , and l is nonnegative. For any x ∈ IR n , we define the infinite horizon value function where the admissible controls set A is given by (7) below.
The following hypotheses (H0), (H1) will be assumed throughout the whole paper.
(H0) The control set A ⊂ R m is either compact or a convex, closed, nontrivial cone containing the origin.
The functions f : IR n × A → IR n , l : IR n × A → IR are continuous; there exist p, q ∈ IN, q ≥ p ≥ 1, M > 0, and for any R > 0 there are L R , M R > 0 and a modulus ω(·, R), such that ∀x, If A is compact, the above assumptions reduce to the continuity of l and to the usual hypotheses of sublinear growth and local Lipschitz continuity in x, uniformly w.r.t. a, for f . With a small abuse of notation, in this case we will denote again by L R the quantity max{L R (1 + |a| p ) : a ∈ A} and similarly for the other constants appearing in (H0).
When A is unbounded, we will always assume at least weak coercivity together with a regularity hypothesis in the control variable at infinity: (H1) There exist some constants C 1 ≥ 0, C 2 > 0 such that and q ≥ p, where q and p are the same as in (H0).
Let Φ ∈ {f, l}. There exists a continuous function Φ ∞ , called the recession function of Φ, verifying lim uniformly on compact sets of R n × A.
Condition (5), for q > p is known as coercivity and it is used to yield suitable compactness properties for the set of the admissible controls. It is satisfied, for instance, in the LQR problems anticipated in the Introduction. If q = p, instead, (5) is sometimes called weak coercivity. In this case the natural framework of all our optimization problems is that of generalized or impulsive controls, since minimizing sequences of trajectories may converge to a discontinuous function. In Section 3 the generalized setting will be introduced in terms of some extended problems. This approach is suitable to study, for instance, problems in which both the dynamics and the Lagrangian are control-affine.
Example 2.1 Functions f and l which are polynomials in the control variable a, admit the recession function introduced in (6). If, for instance, there are some continuous functions Let B denote the set of the Borel-measurable functions. The controls α are assumed to belong to the set coinciding with B when A is compact. For any x ∈ IR n and for any control α ∈ A, (1) admits just one solution, defined on the whole interval IR + . We use y x (·, α) to denote such a solution. When A is unbounded the control set A is the largest set where both payoff and trajectory are surely defined for all t ≥ 0. In fact, in view of the coercivity condition (5) (weak, if q = p), such a choice is not a restriction, since for any measurable control α, so that for controls α / ∈ A we will never obtain a finite cost. In particular, if C 1 = 0 we can consider merely controls in L q (IR + , A).
Let us write two estimates, useful in the sequel, that can be obtained by standard tools. For every x, z ∈ IR n , ∀α ∈ A, and ∀t ≥ 0 one has For some results we will use the following hypothesis (H2).
(H2) There is some nonempty closed set T ⊂ IR n with compact boundary such that V(x) = 0 for any x ∈ T and lim Remark 2.1 Assume that V(x) ≤ +∞ 0 l(y(τ ), α(τ )) dτ < +∞ for some x and a control α ∈ A with |y x (t)| ≤R for all t ≥ 0 and someR > 0. Then it is not difficult to show that there existsx where V * (x) = 0. Therefore, if V is continuous atx thenx ∈ T and hypothesis (H2) holds atx.
The hypothesis V ≡ 0 in T is satisfied, e.g., if T × {0} is a viability set for the vector field (f, l). 1 Sufficient viability conditions can be found e.g. in [AF].
As shown in [MR], a sufficient condition for (10) is the existence of a local MRF U , defined, for the case A compact, as follows.
Definition 2.1 [MR] Given an open set Ω ⊂ IR n , Ω ⊃ T we say that U : Ω \ and, moreover, ∃k > 0 such that, for every x ∈ Ω \ T , where D * U (x) is the set of limiting gradients of U at x.
For the case A unbounded, as proved in Remark 2.5 of [MS], we can consider the following hypothesis: There exists a local MRF U for l such that ∀x ∈ Ω \ T : where R : ]0, σ] →]0, +∞[ is a decreasing continuous function (in particular, we may have lim δ→0 + R(δ) = +∞). Let us observe that any MRF is a Control Lyapunov function for the system w.r.t. T , which yields local asymptotic controllability to T . For the notions borrowed from nonsmooth analysis, we refer to [CS]. .

Generalized and relaxed control problems
Following the so called graph-completion approach proposed in [BrRa], as developed in [RS], when A is unbounded we represent generalized controls and trajectories as reparametrizations (through a time-change, possibly discontinuous in case q = p) of controls and trajectories of the extended minimization problems below, involving bounded-valued controls. Then we investigate the well-posedness of the generalized setting, that is, when the infima over ordinary and generalized controls are the same. We do this for both the finite and for the infinite horizon problem. Let us remark that dealing with a compact set of controls as the generalized control set is, has two main advantages. On the one hand, it allows to introduce the relaxed problem for which an optimal control exists. On the other hand, the relative Hamiltonian, differently from the original, is continuous and satisfies some crucial growth and regularity properties. The exploitation of both these aspects yields many results.

Generalized problems and well posedness
Throughout this subsection we assume A unbounded. Let us define on IR n × (IR + × A) the extended dynamics and Lagrangian f , l as follows: where Φ ∞ is defined in (H1). f , l are continuous, q-positively homogeneous in the control variable (w 0 , w) and inherit properties analogous to those of f and l, respectively (see e.g. and ∀(w 0 , w) ∈ Γ denote by ξ(·) ≡ ξ x (·, w 0 , w) the extended trajectory solving the extended control system For any S > 0, the extended payoff is given by As recalled in Proposition 3.1 below, the solutions to (15) are simply time-reparametrizations of trajectories of (1) if the controls belong to Proposition 3.1 [MS] For any α ∈ A let us define s(t) = (1 − |w(·)| q ) 1/q , belongs to Γ + and y x (t(·), α) is the solution of (15) associated to (w 0 , w).
Remark 3.1 Considering extended controls where w 0 (s) = 0 for s in some intervals, is a way to introduce a notion of generalized control, where the (discontinuous) generalized solution to (1) corresponding to (w 0 , w), say y gen x is defined as y gen x (·) It is clear that, for q > p, one has f ∞ ≡ 0 and y gen x (·) ≡ y x (·) (for more details, see [RS]).
For any t ≥ 0, x ∈ IR n , we define the extended finite horizon value function and the extended infinite horizon value function Remark 3.2 In Proposition 3.1, we establish a correspondence between α ∈ A and (w 0 , w) ∈ Γ + , assuming (18). This is not a restriction, however, since (18) is satisfied by all (w 0 , w) ∈ Γ such that J(+∞, x, w 0 , w) < +∞, owing to the coercivity hypothesis (5) which, in the extended problem, reads as In fact, if we had +∞ 0 w q 0 (s) ds = T < +∞, (19) together with the constraint w q 0 + |w| q = 1 would yield a cost which is a contradiction.
For this reason in the definition of V (x) we can disregard the constraint (18), which should be naturally assumed, as in the definition of V (t, x). This is a key point: due to the coercivity hypothesis, the extended infinite horizon problem reduces to an unconstrained problem with a compact control set.
In view of Proposition 3.1 and Remark 3.2, in the extended setting we can recover V(x) and V(t, x) by restricting the minimization to Γ + in the definition of V (x) and V (t, x), respectively. In general, V(x) is neither l.s.c. nor u.s.c.. Moreover, as shown in the following example, if q = p it may happen that V (x) < V(x) at some x.
Let us now consider the associated extended system, given by Implementing the control w . = −1χ [0,1] the trajectory issuing from (1, 0), in time S = 1 reaches the origin, which is an equilibrium point for the extended system, and the corresponding extended cost is This yields V (1, 0) ≤ 1, obviously smaller than V(1, 0) = +∞.
When q = p, we can prove that V(x) ≡ V (x) using (H2) and the following condition.
(H3) Assume that there is some closed set T ⊂ IR n with compact boundary such that for When V ≡ 0 in T , both (SC1) and (SC2) below imply (20).
(SC1) means that (15) is UGAS (uniformly globally asymptotically stable) w.r.t. ∂T , so that all extended trajectories approach T , at least asymptotically, for any x ∈ T c (see e.g. [BaRo]). We point out that (SC1) allows the Lagrangian to be zero outside T . (SC2) instead, involving just the Lagrangian, implies that l is strictly positive outside T . For T ≡ {0}, it is satisfied in LQR problems, where l(x, a) = x T Qx + a T Ra and the matrices Q and R are symmetric and positive definite. (SC2) easily implies that J(+∞, x, w 0 , w) = +∞ for any control (w 0 , w) not satisfying the lim inf-condition in (20), in view of Remark 3.2.
We have the following well posedness results.
Theorem 3.1 For any t ≥ 0 and x ∈ IR n , one has Proof. Theorem 3.3 in [RS] yields (i) while Proposition 3.4 in [M] implies (ii) for q > p. It remains to prove thesis (ii) in case q = p. Being V ≤ V, for any x ∈ T the equality V(x) = V (x) = 0 follows trivially from (H2). Let x ∈ T c and V (x) < +∞ (if V (x) = +∞, V(x) = +∞ too). Assume by contradiction that there is some η > 0 such that By hypothesis (10), V is continuous on the compact set ∂T , therefore for some δ > 0. Owing to (H3), there is some Hence, for some S > 0, we have d(ξ x (S,w 0 ,w)) < δ and, using the Gronwall's Lemma, by standard calculations we get that the control (w n 0 ,w n ) ∈ Γ + wherew n . = n n+1w , for n large enough satisfies both d(ξ x (S,w n 0 ,w n )) < 2δ and Thanks to Proposition 3.1, setting T By (23) it follows that, ifx .
Thus the control α(t) .
At this point the first inequality implies that V(x) < +∞, which together with the last inequality yields the required contradiction. Statement (ii) for q = p is therefore proved.
V(x) is in general neither u.s.c. nor l.s.c., even if A is compact. Sufficient conditions for the upper semicontinuity are given in the following proposition. Proof . If A is unbounded condition (20) is assumed on the extended trajectories. However, (H2) implies that also in this case (and even if q = p), for any x with V(x) < +∞, there is some ε > 0 such that lim inf t→+∞ d(y x (t, α)) = 0 for any ε-optimal control α ∈ A. (24) Indeed, if (24) were not satisfied for some x and α, Proposition 3.1 and the equality V(x) = V (x) proved in Theorem 3.1, would imply a contradiction: (20) would not hold for the extended control (w 0 , w) corresponding to such an α. From now on, the proof is the same for a compact or non compact set A. Fix η > 0 and let δ > 0 be as in (23).
= y x (·, α). Estimates (8), (9) imply that one can choose δ ′ > 0 small enough to have, for all x ∈ B(x 0 , δ ′ ), for someC > 0 and for any δ ′′ > 0. Now by the Dynamic Programming Principle, in short DPP, choosing δ ′′ ≤ δ, we get for some C ′ > 0, where the second inequality holds since d(y x (T )) < 2δ. Therefore Dom (V) is an open set and a simple compactness argument yields that V is bounded on any compact subset of Dom(V).
The fact that V is u.s.c. in x 0 can now be easily deduced. Adding and subtracting , and with this the upper semicontinuity of V is proved.
Let us observe that the continuity on ∂T prescribed in (H2) plus (H3) does not yield the lower semicontinuity of V(x). The continuity of V in its whole domain will be discussed in Remark 5.1.

Relaxed problems
In this section we introduce the relaxed finite and infinite horizon problems, for the original problems when A is compact, and for the extended problems otherwise. In order to simplify the notation, the corresponding relaxed value functions, V r (if A is compact) and V r (in which A is replaced by S(A) and the extended data are considered), will be always denoted by V r .
A compact. As usual we define the relaxed controls is the set of Radon probability measures on the compact set A endowed with the weak * -topology, and we consider ψ ∈ {f, l} extended to IR n × A r by setting For any x ∈ IR n and µ ∈ A r , y r x (τ, µ) denotes the relaxed trajectory, solution oḟ Finally, we introduce where J r (t, x, µ) . = t 0 l r (y r x (τ, µ), µ(τ )) dτ for any t ∈]0, +∞].
Since for A compact, standard arguments yield that the relaxed finite and infinite horizon problems coincide with the original ones under the following convexity hypothesis.
(CV) Let A be compact. For each x ∈ IR n , the following set is convex: A unbounded. We define relaxed extended controls, denotes now the set of Radon probability measures on the compact set B(0, 1) ∩ A endowed with the weak * -topology and we consider ψ ∈ {f , l} extended to IR n × A r by setting For any x ∈ T c and µ ∈ Γ r , ξ r x (s, µ) is the relaxed trajectory, solution oḟ ξ r = f r (ξ r , µ) for s > 0, ξ r (0) = x.
In this case, V r (t, x) and V r (x) are given respectively by , µ(s)) ds for any S ∈]0, +∞].
If A is unbounded, in order to have V r ≡ V we could again invoke a convexity condition analogous to (CV), for the extended problem. However, in view of the definitions of f and l this condition would be very difficult to be satisfied, since the control set S(A) is not convex. Hence we introduce the weaker convexity condition (CV) ′ below, where S(A) is replaced by [0, 1] × B(0, 1) ∩ A and the space-time extended dynamics (w q 0 , f ) is considered. (CV) ′ is verified, for instance, by a control-affine dynamics and a convex Lagrangian.
(CV) ′ Let A be a unbounded. For any x ∈ T c , the following set is convex: Both for bounded and unbounded controls, the relaxed and the original finite horizon problems coincide.
Theorem 3.2 Finite horizon. For any (t, x) ∈]0, +∞[×IR n we have that V r (t, x) is continuous, there exists an optimal relaxed control, and Moreover, assuming either (CV) or (CV) ′ , there exists an optimal control α for the original problem in case either A is compact or q > p, and there exists an optimal extended control (w 0 , w) for p = q.
Proof. The equality, which could be proved directly, is a straightforward consequence of the uniqueness result in Theorem 5.1, since it is easy to show that V r (t, x) satisfies (52) in the viscosity sense. Moreover, it is continuous as V(t, x), since the relaxed data have the same properties of the original ones. The existence of an optimal control for the relaxed problem (which does not imply in general the existence of an optimal ordinary control) is well known.
As it is well known, this relaxation property is no more true for the infinite horizon problem and V(x) does not coincide in general with V r (x), even in the simplest case of compact valued controls, as shown by Example 4.1 below. The following weaker results hold.

Theorem 3.3 Infinite Horizon.
(i) Assume either (CV) or (CV) ′ and q > p. Then for any x ∈ IR n we have and there exists an optimal control α ∈ A for the original problem.
(ii) Assume (CV) ′ and q = p. Then for any x ∈ IR n , and there exists an optimal extended control, (w 0 , w) ∈ Γ. If moreover (H2) and (H3) hold for the same T , then we have (34).
From the same arguments in Remark 3.2 applied to the relaxed problem, we have that +∞ 0 w q 0 (s) ds = +∞. Now, (w 0 , w) / ∈ Γ in general, but by using the arc-lenght reparametrization and arguing as in the proof of Theorem 3.2, we can obtain an extended control in Γ with the same cost, and this proves (35). The last statement of (ii) follows from Theorem 3.1 (ii). If A is compact, statement (i) can be proved by standard arguments. When A is unbounded, the equality V(x) = V r (x) follows from the previous point together with Theorem 3.1 (ii). The existence of an optimal control α in the case q > p can be recovered as in the last part of the proof of Theorem 3.2.
Remark 3.3 In case A unbounded and q = p, even if V ≡ V r , both the original finite and infinite horizon problems may not have an optimal control.

Finite-horizon approximation
In this section we give a representation formula for the limit, as t tends to +∞ of the finite horizon value functions The following simple example describes what is expected to happen, for the compact control case.
The result suggested by the previous example can be extended to the case of unbounded controls as follows.
Theorem 4.1 For any x ∈ IR n , we have Moreover, V r is l.s.c. and there exists an optimal relaxed control.
In case A unbounded, we use the following preliminary result, true thanks to the coercivity hypothesis (5) and interesting in itself.
Proposition 4.1 For any x ∈ IR n , Proof. Let x ∈ IR n . We recall that for any t > 0, V(t, x) coincides with the relaxed finite horizon value function V r (t, x) in view of Theorem 3.2. Hence Σ(x) = sup t>0 V r (t, x). In order to conclude, it remains essentially to prove that the time constraint

Let us first show the simpler inequality
true even in non coercive problems. By Theorem 3.2, for any n ∈ IN, there exists an optimal relaxed trajectory-control pair (ξ r n , µ n ) and some s n > 0 such that V r (n, x) = sn 0 l r (ξ r n (s, µ n ), µ n (s)) ds, sn 0 (1 − |µ n (s)| q ) ds = n.
where s n ≥ n by definition, so that (37) follows easily by passing to the limit as n tends to +∞ in (38) (the lim s→+∞ W (s, x) exists and coincides with sup s>0 W (s, x) by monotonicity). Now, by (37) the converse inequality is trivially satisfied if sup s>0 W (s, x) = +∞. Let us assume by contradiction that there is some η > 0 such that Then for any n ∈ IN there is some (ξ r n , µ n ) such that n 0 l r (ξ r n (s, µ n ), µ n (s)) ds < Σ(x) − η.
Thus letting n tend to +∞ one obtains that Σ(x) = lim n V r (t n , x) ≤ Σ(x) − η, which yields the desired contradiction. If instead the sequence {t n } n is bounded, so that t n ≤ T for all n for some T > 0 by the coercivity assumption (5) we get When n tends to +∞, the l.h.s. tends to +∞ and we get a contradiction also in this case.
Proof of Theorem 4.1. We consider only the case A unbounded, the proof for A compact being similar and actually simpler. By the previous proposition, Therefore by Ascoli-Arzelà Theorem there exists a subsequence {ξ r n ′ } n ′ , uniformly converging to some functionξ r in [0, S], such that, owing to (H0), for some ρ S (n) with lim n ρ S (n) = 0. Moreover, since L ∞ ([0, S], P(B(0, 1) ∩ A) is sequentially weakly * -compact (see [W], p. 272), there exists a subsequence {µ n ′′ } n ′′ of {µ n ′ } n ′ which converges weakly to someμ in [0, S]. Therefore by a diagonal procedure we obtain a trajectory-control pair (ξ r ,μ) defined on the whole interval IR + and such that for any S > 0 there is some subsequence {(ξ r n , µ n )} n , where ξ r n converges uniformly toξ r and µ n weakly toμ in [0, S]. For any S > 0, by the weak convergence, passing to the limit in (41) one has S 0 l r (ξ r (s),μ(s)) ds ≤ Σ(x).
We are going now to discuss the relation of the previous approximation result with the original value function V. A straightforward consequence of Theorems 3.3 and 4.1 is the following Corollary 4.1 Assume either (CV) or (CV) ′ . If A is unbounded and q = p let (H2) and (H3) hold for the same T . Then for any x ∈ IR n we have where Σ is defined in (36).
If no convexity is assumed, we prove that Σ(x) = V * (x), the l.s.c. envelope of V, under some mild additional hypotheses (H0) 1 and (H0) 2 . Let us remark that, since the boundary value problem associated to the infinite horizon value function considered here has not a unique solution, we have to prove this relaxation result directly.
(i) If either A is compact or q > p, then for any x ∈ IR n , (ii) if A is unbounded and q = p, then for any x ∈ IR n , Moreover, if (H2) and (H3) hold for the same T , we have (43).
Proof. We prove the theorem only for A unbounded, the proof for A compact being analogous and actually simpler. We show that (44) holds for any x ∈ IR n . Both statement (i) for q > p and the last part of (ii) for q = p follow then from Theorem 3.1 (ii). Since V r (x) ≤ V (x) and V r is l.s.c., then V r (x) ≤ V * (x) for any x ∈ IR n . It remains to prove the converse inequality, where it is not restrictive to consider only x ∈ IR n with V r (x) < +∞. Let us first assume (H0) 1 . In this case it is easy to prove that f and l verify Assumption 3.1 of [AB], so that (44) holds in view of Theorem 3.2 of the same paper. Actually, in [AB] infinite horizon problems in L ∞ are considered, but for a nonnegative running cost l, one has ess sup s∈[0,+∞[ s 0 l(ξ(s), w 0 (s), w(s)) ds = +∞ 0 l(ξ(s), w 0 (s), w(s)) ds.
A sufficient condition to have (H0) 2 (i), is given in the next proposition. Let us remark that (47), even in the case A unbounded, involves only the original Lagrangian l and not the extended l.
Proof. Let A be unbounded. Then condition (47) together with the coercivity assumption (5) easily implies l(x, w 0 , w) ≥C ∀x with |x| ≥M and (w 0 , w) ∈ S(A) for some positive constantsM ,C, so that the same holds true for l r . Assume by contradiction that for some x with V r (x) < +∞, there exists some optimal relaxed control µ such that the corresponding trajectory ξ r (·) satisfies |ξ r (s n )| ≥ n for some increasing, positive sequence s n tending to +∞. Then ∃N > 0 such that |ξ r (s n )| >M for all n ≥ N . If |ξ r (s)| >M for all s ≥ s n for some n, then by (48) we should have an infinite cost, while J r (+∞, x, µ) = V r (x) < +∞. Otherwise, we can suppose that for any n > N there exists s n+1 ≥ c n > s n such that |ξ r (s)| >M for s ∈ [s n , c n [ and |ξ r (c n )| =M . Then by the estimate proved in Lemma 1, pag. 778 of [B], where M is the constant in (4), we get that is, the same contradiction as above.
Wo omit the proof in the case A compact, since it is completely similar.
In many applications (47) holds since for some r > 0, l satisfies the following stronger version of (5): l(x, a) ≥ C 2 |a| q + C 1 |x| r ∀(x, a) ∈ IR n × A where C 1 , C 2 > 0 and q ≥ p is the same as in (H0). Condition (49) holds, for instance, for in LQR problems, where l(x, a) = x T Qx + a T Ra and the matrices Q and R are symmetric and positive definite.

Maximal and minimal solutions and uniqueness
In this section we give sufficient conditions in order to characterize V(x) as unique solution of the associated HJB equation introduced below. As a byproduct we also obtain the characterization of the limit function Σ(x) = V r (x). We start by recalling a uniqueness theorem for the finite horizon problem obtained in [RS] (see also [MS2], where more general results, including second order PDEs, are obtained). We point out that these results cannot be derived by classical theorems within the viscosity theory, in view of the hypothesis l ≥ 0 and of the growth of the data considered here. Then we derive from the results in [M] and [MS] a uniqueness theorem for the infinite horizon case, generalizing that obtained for A compact in [MS1].
Let us define the Hamiltonian Notice that in case A unbounded and p = q, H can be discontinuous and equal to +∞ at some points. When A is unbounded and q ≥ p, H can be replaced, as shown in [RS] and [M], by the extended Hamiltonian which turns out to be continuous. Actually, considering H is useful even if q > p, since it allows to consider dynamics verifying |f (x, a)| ≤ M (1 + |a| p )(1 + |x|) instead of the more restrictive hypothesis |f (x, a)| ≤ M (1 + |a| p + |x|), assumed in most of the literature (see e.g. [BDL], [DL], and more recently, [GSor] and the references therein). An analogous remark holds for l. Therefore in the sequel we will use H and, in order to unify the exposition, we will set H . = H when A is compact.
Example 5.1 In control-affine problems, or, more precisely, when A is unbounded, q = p = 1, and ∀(x, a) ∈ IR n × A we have we showed in Section 5 of [MS2], that the evolutive PDE is equivalent to the following quasi-variational inequality: An analogous equivalence holds for the stationary equation. This is the more usual formulation of the PDE associated to impulsive control problems.
For the finite horizon problem we recall what follows.
Theorem 5.1 [Corollary 2.1, RS] We have V(t, x) = V (t, x) and it is continuous for any (t, x) ∈ IR + × IR n . Moreover, for every T > 0, it is the unique viscosity solution of the Cauchy problem   among the functions bounded from below and continuous on ( The above uniqueness result, for the case A compact, can be found in [BCD]. For A unbounded, some comparison theorems due in [BDL] (for the finite horizon problem) and in [DL] (for the infinite horizon case), address just the coercive case q > p, as observed above, require stronger hypotheses on f and l, and imply uniqueness in the class of the locally Lipschitz functions. We refer to [G] for a uniqueness result among convex functions.
Leu us now consider the infinite horizon problem with HJB equation In order to apply the results of [M], from now on we assume that where ω is the modulus of continuity of l in (H0). 4 We recall  (53) in Ω, and lim x→x u(x) = +∞ ∀x ∈ ∂Ω.} The proof of the following theorem follows from Theorem 5.4 below.
(i) Assume that either (H0) 1 or (H0) 2 holds. Moreover, let V be continuous in Dom(V) and satisfy the boundary condition (ii) assume that either (CV) or (CV) ′ holds.
Then V (≡ V r ≡ Σ) is the unique nonnegative viscosity solution to (53) in Dom(V), among the pairs (u, Ω) in S, where Ω ⊃ T , u ≡ 0 on T . Moreover V is continuous. If we drop (H0) 1 , (H0) 2 in (i), V (possibly = V r ) is the unique solution just among the continuous functions.
By the Kruzkov transform Ψ(v) . = 1 − e −v , the above free boundary problem, can be replaced by another boundary value problem in IR n \ T , whose solution, when unique, simultaneously gives both V and Dom(V). More precisely, let Theorem 5.4 Under the same hypotheses of Theorem 5.3, there is a unique nonnegative viscosity solution U to If we drop (H0) 1 , (H0) 2 in (i), U (possibly = Ψ(V r )) is the unique solution just among the continuous functions.
Proof. Let us prove the theorem in case (H0) 1 , (H0) 2 are not assumed. In order to apply the uniqueness result proved in Theorem 4.7 in [MS], let us observe that, under hypotheses (H2) and (H3), the asymptotic and the minimal exit-time value functions V and V m , as well as their extended versions V and V m there introduced, do all coincide. They also are equal to our infinite horizon value function V (≡ V by Theorem 3.1). Indeed, owing to (H2) and (H3), both original and extended nearly optimal trajectories have to approach at least asymptotically T . In fact, since V ≡ V, the conditions in hypothesis (H2) hold for V too, and as shown in the proof of Proposition 3.2, the liminf in (20) is zero also for the ε-optimal trajectories of the original system. Thanks to (5), the last statement follows now from (i) of Theorem 4.7 in [MS], while the first statement is a consequence of (ii) of Theorem 4.7 in [MS] together with either Theorem 3.3 when (ii) is assumed or Theorem 4.2, when (i) holds.
Remark 5.1 Since when (H2) and (H3) hold for the same T , the infinite horizon value function V coincides with the asymptotic exit-time value function considered in [MS], sufficient conditions for its continuity can be found there (see (TPK) ′ in [MS]). In particular, when (H2) holds for T , in view of Proposition 6.2 in [MS], (SC1) or (SC2) for the same T imply not only (H3), but also the continuity of V and the boundary condition (54). Moreover, as already observed, they also yield (H0) 2 (i). Since in this section we suppose l locally Lipschitz continuous in x, condition (H0) 2 (ii) is trivially verified.

Therefore we have
Corollary 5.1 Let T × {0} be a viability set for (f, l). Assume the existence of a local MRF and either (SC1) or (SC2) for T . Then (i) there is a unique nonnegative viscosity solution U to (56), which turns out to be continuous. Moreover, is the unique nonnegative viscosity solution to (53) in Dom(V) among the pairs (u, Ω) in S. Moreover, V is continuous.
When A is unbounded, the case q = p is the only one in which we could have V(x) > V (x) for some x. Since Σ(x) = V r (x), in order to characterize Σ, the well-posedness, that is the equality V ≡ V , is not required. Hence in this whole section assumption (H2) could be weakened, by replacing in it the function V with V . Accordingly, in Corollary 5.1 it would be enough to assume T × {0} viable for (f , l) and the existence of a MRF for the extended setting.

Discounted infinite horizon approximations
In this section we give a representation formula for the limit as δ tends to 0 + of the infinite horizon value function with discount rate δ > 0: To this aim, for any δ > 0, when A is unbounded, we also introduce the extended value function and, agreeing with the notation of Subsection 3.2, if A is compact [resp., unbounded], we consider the relaxed version of V δ , V r δ [resp., of V δ , V r δ ]. As a first step, by Proposition 3.2 in [M] all these value functions are supersolutions to δu + H(x, Du(x)) = 0 (57) in IR n . If they are locally bounded and with open domains, they also are subsolutions to (57) in their domains. Notice that, when A is unbounded, by Theorem 2.1 in [M], equation (57) can be replaced by where, for any (x, r, p) ∈ IR 2n+1 , H δ is the following continuous Hamiltonian H δ (x, r, p) . = max (w0,w)∈S (A) δr w q 0 − f (x, w 0 , w), p − l(x, w 0 , w) .
By Corollary 4 in [MS2], for any δ > 0 we have what follows.
Theorem 6.1 If V δ is bounded, then it is the unique bounded solution to (57) in IR n and it is continuous. Hence, if A is compact one has V δ ≡ V r δ , and V δ ≡ V δ ≡ V r δ otherwise.
Remark 6.1 It is easy to see that, when A is unbounded, sufficient conditions in order to have V δ bounded are, for instance, either |f (x, a)| ≤M + M (1 + |x|)|a| p and l(x, a) ≤M (1 + |x| r ) for someM > 0, r ≥ 1 (M R is the same as in (4)). Formally, the same conditions with a = 0 yield the boundedness of V δ for A bounded. We refer to Corollary 4 in [MS2], for a characterization of V δ as unique solution to (57) in IR n in some classes of unbounded functions with prescribed growth at infinity.
Theorem 6.2 Assume that each V δ is bounded. Then Proof. We give the proof in the case A unbounded, being the other case similar. Taking into account that the sequence δ → V δ is monotone non increasing, by Theorem 6.1, we have for every x ∈ IR n . In view of Theorem 5.2 (ii), V r is the minimal supersolution to (53) in IR n , hence it is now sufficient to show that Λ (= Λ * ) is a supersolution to (53) in IR n for any x such that Λ(x) < +∞. By the monotonicity of the sequence V δ and by the continuity of each V δ , it is known that [BCD]). The claim follows now from stability results of viscosity solutions, taking into account the continuity of the V δ and the fact that we can consider the regular Hamiltonian in (58).
In the above proof we used the upper optimality principle. Of course, it is also possible to obtain it by working directly on the control problem.

Ergodic problem
In this section we briefly investigate the so-called ergodic problem, that is the convergence of the limits lim t→+∞ V(t, x)/t, lim δ→0 + δ V δ (x). Our goal here is just to describe how known hypotheses and proofs can be adapted to the case of unbounded controls. Hence in the sequel we consider A unbounded and assume f and l periodic in the state variable and global controllability. Our precise assumptions, together with (H1), are the following.
(H4) (i) T i > 0 (i = 1, . . . , n) are real numbers and the functions f (x, a), l(x, a) are periodic in x i with the period T i (i = 1, . . . , n). Moreover there are L and M > 0 such that ∀x, where T T n denotes the n-dimensional torus (ii) There are C, γ > 0 such that for any pair x, z ∈ T T n there exist S > 0 and µ ∈ Γ r such that ξ r x (S, µ) = z and S ≤ C|x − z| γ . A sufficient condition to have (H4) (ii) (with γ = 1) is the usual hypothesis that, for some r > 0, B(0, r) ⊂ co f (x, S(A)) for any x ∈ IR n .
Remark 7.1 Owing to Theorems 4.1 and 6.2, at least when any V δ is bounded, lim t→+∞ V(t, x) = lim δ→0 + V δ (x) = V r (x) for every x ∈ IR n . As a consequence, the limits lim t→+∞ V(t, x)/t and lim δ→0 + δ V δ (x) converge obviously to zero when V r is finite in IR n . In fact, being l ≥ 0 such a convergence is locally uniform.
When l ≤ M (1 + |a| q ) and (H4) (ii) is in force, V r is finite as soon as (f, l)(x, a) = (0, 0) for some pair (x, a), or, more in general, if there exists a subset T ⊂ Z such that T × {0} is a viability set for (f, l). In this case indeed, for any x ∈ IR n it is possible to construct an admissible control α with finite cost, by concatenating a control steering x to T in time T , as in (H4) (ii), with a control keeping the trajectory inside T with null cost for all t > T . Such a control exists in view of the viability assumption.
Proposition 7.1 Assume (H3). Then, for any x, z ∈ T T n , Moreover, setting W δ (x) . = V δ (x) − V δ (0), one also has Proof. In view of Theorem 6.1, for any x ∈ IR n one has V δ (x) = V δ (x) ≡ V r δ (x). Therefore the first estimate in (60) follows immediately from the fact that l ≤ M , considering the relaxed control µ ≡ 0. Assuming V r δ (x) − V r δ (z) ≥ 0, as it is not restrictive, the second inequality in (60) can be obtained plugging in the DDP for V r δ (x) the control given by (H4) (see e.g. Theorem 2 in [A]). Both the estimates in (61) are easy consequence of (60).
Proof. By Proposition 7.1, the Ascoli-Arzelà Theorem and the periodicity of the solutions imply that there exists a sequence δ n → 0 + such that lim n→+∞ δ n V δn = λ ∈ C(IR n ) and lim n→+∞ W δn = W 0 ∈ C(IR n ). The second inequality in (60) implies that λ is a constant and consequently δ n W δn → 0 uniformly in IR n . It is now easy to check that W δ satisfies max (w0,w)∈S (A) δu w q 0 − f (x, w 0 , w), Du − l(x, w 0 , w) + δV δ (0)w q 0 = 0.
In order to prove that lim t→+∞ V(t, x)/t = λ uniformly, for the same λ as above, let us first introduce the function v(t, x) . = C + W 0 (x) + λt for all (t, x) ∈ IR + × IR n , where W 0 is a solution toH λ (x, Du) = 0 and C > 0 is chosen so that C + W 0 ≥ 0. Then v is a supersolution to (52) for any T > 0 and by the comparison principle underlying Theorem 5.1, V(t, x) ≤ v(t, x) = C + W 0 (x) + λt ∀(t, x) ∈ IR + × IR n .
Remark 7.2 Let us observe that the effective HamiltonianH λ really determines λ. This would not be the case, if there existed a function W 0 ∈ BU C(IR n ) such that the max in the definition ofH λ was reached for every x ∈ IR n in a vector (0, w) ∈ S(A). If fact, such a function would be a solution of max (0,w)∈S (A) − f (x, 0, w), Du − l(x, 0,w) = 0, and then it would also solveH λ (x, Du) = 0 for all λ. However, applying Theorem 5.2, such W 0 would be greater than the value function of an infinite horizon problem with compact controls (0, w) ∈ S(A) (where |w| q = 1) and lagrangianl(x, 0,w) ≥ C 2 , equal to +∞. Again, the coercivity hypothesis (5) plays a crucial role.