Stochastic Perron's method for optimal control problems with state constraints

We apply the stochastic Perron method of Bayraktar and S\^irbu to a general infinite horizon optimal control problem, where the state $X$ is a controlled diffusion process, and the state constraint is described by a closed set. We prove that the value function $v$ is bounded from below (resp., from above) by a viscosity supersolution (resp., subsolution) of the related state constrained problem for the Hamilton-Jacobi-Bellman equation. In the case of a smooth domain, under some additional assumptions, these estimates allow to identify $v$ with a unique continuous constrained viscosity solution of this equation.


Introduction and the main result
The aim of the paper is to extend the scope of applications of the stochastic Perron method, developed by Bayraktar and Sîrbu. This method allows to characterise the value function of a controlled diffusion problem as a viscosity solution of the corresponding Hamilton-Jacobi-Bellman (HJB) equation, bypassing the dynamic programming principle. Instead it requires a comparison result, implying the uniqueness of a viscosity solution of the HJB equation. Previously this method was applied to linear parabolic equations [5], stochastic differential games [7,25,26], regular [6,24] and singular control problems [8].
The method involves the construction of two families V − , V + of functions, bounding the value function from below and above u ≤ v ≤ w, u ∈ V − , w ∈ V + .
Elements of V − , V + are called stochastic sub-and supersolutions. By the superposition with the state process, u and w generate sub-and supermartingale-like processes. Similarly to the classical Perron method [14,Sections 2.8,6.3], the set V − (resp., V + ) is directed upward (resp., downward) with respect to the pointwise maximum (resp., minimum) operation. The essence of the method is to prove that the functions u − (x) = sup are respectively viscosity super-and subsolutions of the related HJB equation. If a comparison result, providing the inequality u − ≥ w + , holds true, it follows that u − = v = w + is a unique (continuous) viscosity solution. This construction differs from Perron's method of [17], which is not linked to the value function.
In the present paper we consider the stochastic control problem with state constraints in the form of [21]. In contrast to [23], where the drift is not assumed to be bounded, and the value function is singular near the boundary, in [21] the problem is "regular". To achieve the regularity it is assumed that the diffusion coefficient depends on the control and degenerates at the boundary. The same problem was considered in [18,11]. It was proved that under appropriate assumptions the value function v is a unique continuous constrained viscosity solution of the HJB equation. (The term "constrained" means, in particular, that v satisfies special boundary conditions, which in the deterministic situation were introduced in [27].) Roughly speaking, it is enough to assume that for each boundary point there exists a control, which kills the diffusion and directs the drift strictly inside the domain.
An application of the stochastic Perron method to state constrained problems seems rather interesting, since, as it is mentioned in [21], a direct proof of the dynamic programming principle is not available due to a complicated structure of admissible control processes, retaining a phase trajectory in a predetermined domain. Different penalization and approximation procedures were used instead in [21,18,11,10].
We turn to the precise statement of our main result (Theorem 1). Let Ω be the space C([0, ∞), R m ) of continuous R m -valued functions, endowed with the σ-algebra F • of cylindrical sets, and let P be the Wiener measure on F • . So, the canonical process W s (ω) = ω(s) is the standard m-dimensional Brownian motion under P. Denote by F • = (F • t ) t≥0 the natural filtration of W , and let F = (F t ) t≥0 be the correspondent minimal augmented filtration. The extension of the Wiener measure to the completion F of F • is still denoted by P.
Let α be an F-progressively measurable stochastic process with values in a compact set A ⊂ R k , 0 ∈ A. Consider the system of stochastic differential equations (1.1) We assume that the drift vector b : R d × A → R d and the diffusion matrix σ : with some constant K independent of x, y, a. Note, that the linear growth condition follows from the continuity of b, σ and compactness of A. Thus, there exist a unique F-adapted strong solution X x,α of (1.1) on [0, ∞): see [22,Chapter 2,Sect. 5].
Let G ⊂ R d be a closed set with the boundary ∂G and nonempty interior G • . It will be convenient to assume that 0 ∈ G • . Denote by A (x), x ∈ G the set of F-progressively measurable control processes α with values in A and such that X x,α t ∈ G, t ≥ 0 a.s. Elements of A (x) are called admissible controls for the initial condition x. The cost functional J and the value function v are defined as follows We assume that for any initial condition x ∈ G there exists an admissible control: A (x) = ∅. In this case the set G is called viable. A necessary condition for the validity of this property is given in [2] (Theorem 1). Let See [2, Section 3] for more concrete forms of this condition. We impose a slightly stronger requirement. For any function ψ : There exist a Borel measurable function ψ : R d → A such that b ψ , σ ψ are globally Lipschitz continuous and Under this assumption there exist a unique strong solution of the equation and X t ∈ G, t ≥ 0 a.s.: see [1, Theorem 3.1]. The correspondent control process Consider the Bellman operator Recall that a bounded upper semicontinuous (usc) function u is called a viscosity subsolution of the equation on a set E ⊂ R d if for any ϕ ∈ C 2 (R d ) and for any local maximum point ) ≤ 0 holds true. In the same way, a bounded lower semicontinuous (lsc) function w is called a viscosity supersolution of (1.6) on E if for any ϕ ∈ C 2 (R d ) and for any local minimum point x 0 of w − ϕ on E we have the inequality In these definitions one can assume that the maximum (resp., minimum) point x 0 is strict and ϕ(x 0 ) = u(x 0 ) (resp., ϕ(x 0 ) = w(x 0 )).
It is convenient to introduce the state constrained problem We say that a bounded usc (resp., lsc) function u, defined on G, is viscosity subsolution (resp., supersolution) of the state constrained problem (1.7) if F (x, u, Du, D 2 u) ≤ 0 on G • (resp., F (x, u, Du, D 2 u) ≥ 0 on G) in the viscosity sense. A bounded function u is called a viscosity solution of (1.7) (or a constrained viscosity solution), if its upper semicontinuous envelope u * is a viscosity subsolution, and its lower semicontinuous envelope u * is a viscosity supersolution of (1.7). Denote by Γ the set of points x ∈ ∂G such that for some α ∈ A (x) the solution X x,α of (1.1) immediately enters G • with probability 1: Theorem 1. There exist a viscosity subsolution w + and a viscosity supersolution u − of the state constrained problem (1.7) such that The nature of w + and u − is not explicitly indicated here. Their construction, which is presented in Sections 2 and 3 respectively, is based on the technique of stochastic semisolutions, developed in [5,6,7]. The details are quite similar to [6,24]. One only should take care of admissibility of controls.
Theorem 1 is useful if a sort of comparison result is available, and one can conclude that w + ≤ u − . In Section 4 we consider the case of a smooth domain and, under some additional assumptions, mention that such inequality follows from the known result, concerning the boundary behavior of viscosity subsolutions of linear equations [3], and the comparison result of [21]. In combination with Theorem 1 this allows to identify v with a unique continuous viscosity solution of (1.7). The related result (Theorem 2) is not new and is presented only to demonstrate the capabilities of the stochastic Perron method.

Proof. For an
, and there exists a regular conditional probability distribution Consider the SDE where ψ satisfies Assumption 1. To work with P τ ′ , related to the raw filtration F • , we pass from ξI {t≥τ } to an indistinguishable F • -adapted process of the same form.
Recall that any F-stopping time is predictable (see [4,Proposition 16.22]) and the filtration F is quasi-left continuous (see [15,Theorem 3.40]), that is, F τ − = F τ for any (predictable) F-stopping time τ . By Theorem IV.78 of [13] there exists an F • stopping time τ ′ such that P(τ ′ = τ ) = 0, and for any The process Z is a continuous martingale under P, and we can rewrite equation (2.3) in the form Recall the pathwise construction of a strong solution, presented in [20] (see also [9,19] is a strong solution of (2.4). Take Hence, under P τ ′ ,ω , the process H is indistinguishable from ξ ′ (ω)I {t≥τ ′ (ω)} , and X is a strong solution of the SDE with a non-random initial condition: By Assumption 1 the diffusion coefficients b ψ , σ ψ satisfy conditions of Theorem 3.1 of [1]. Since 0 ∈ G and ξ ′ (ω) ∈ G, we conclude that X t ∈ G, t ≥ 0 P τ ′ ,ω -a.s. It follows that G is invariant under P: The desired control process α ∈ A (τ, ξ) is given by the formula α = ψ(X).
Let w be a uniformly bounded continuous function: w ∈ C b (G). Consider the stochastic process Definition 1. We say that a control process for any stopping time ρ ≥ τ . A function w ∈ C b (G) is called a stochastic supersolution of (1.7) if for any randomized initial condition (τ, ξ) with ξ ∈ G • there exists a wsuitable control α.
The set of stochastic supersolutions is denoted by V + . Note that in the above definition the values X ∞ are irrelevant, since Z ∞ = ∞ 0 e −βs f (X τ,ξ,α s , α s ) ds. We emphasize also that the condition A (τ, ξ) = ∅ for all randomized initial conditions (τ, ξ), ξ ∈ G • is necessary for the existence of stochastic supersolutions.
A stochastic supersolution w is an upper bound for the value function (1.2) on G • . To see this put τ = 0, ξ = x ∈ G • , ρ = ∞ and take a w-suitable control α ∈ A (x). By Definition 1, with the convention Z x,α = Z 0,x,α , we get The set V + is non-empty and contains sufficiently large constants c: it is easy to see that Lemma 2. If w 1 , w 2 are stochastic supersolutions then w = w 1 ∧ w 2 is a stochastic supersolution.
belongs to A (τ, ξ) and that it is w-suitable.
The process Y = 2 i=1 X τ,ξ,α i t I A i satisfy the same equation as X τ,ξ,α . From the pathwise uniqueness property it follows that Y = X τ,ξ,α . We have X τ,ξ,α ∈ G, t ≥ τ P-a.s., and α is w-suitable for (τ, ξ): The following result was used in [5,6,24] (see, e.g., Lemmas 2 and 4 of [24]). Its proof use only the fact that V + is directed downward, that is, the statement of Lemma 2 holds true. Lemma 3. There exists a sequence w n ∈ V + , w n (x) ≥ w n+1 (x), x ∈ G such that lim n→∞ w n (x) = w + (x) := inf u∈V + w(x). The next assertion is the most important part of the stochastic Perron method.
Proof. If w + is not a viscosity subsolution then there exist x 0 ∈ G • , ϕ ∈ C 2 and ε > 0 such that w + (x 0 ) = ϕ(x 0 ), w + < ϕ on the set B ε (x 0 )\{0} ⊂ G • and Hence, there exists some a ∈ A such that βϕ( By the continuity of b, σ, f we may assume that for some ε > 0.
Since w + is upper semicontinuous, we have By Lemma 3 there exists a decreasing sequence w n ∈ V + , w n ց w + . The sets are compact, A n ⊃ A n+1 and ∩ ∞ n=1 A n = ∅. Thus, ∩ N n=1 A n = ∅ for some N. This means that there exists a function w = w N ∈ V + such that w − ϕ < −δ ′ on S ε .
To show that w + satisfies the last assertion of Theorem 1, we study its behavior near the points of Γ. Fix x ∈ Γ. By the definition of Γ there exists For ε > 0 consider the predictable set and its projection: D = {ω : (t, ω) ∈ E for some t ∈ [0, ∞)}. The equality (2.13) means that P(D) = 1. By the section theorem [4,Theorem 16.12] there exist an F-stopping time σ ε such that (2.14) Put D ε = {σ ε ≤ ε} = {σ ε < ∞}. Then (2.14) means that Let w be a stochastic supersolution, bounded from above by the constant f /β. Put ξ ε = I Dε X x,α 1 σ ε ∈ G • and take a w-suitable control α 2 ∈ A (σ ε , ξ ε ). Then Taking into account that σ ε = ∞ on D c ε , by the definitions of v and w we obtain: Moreover, by Lemma 3 and the monotone convergence theorem we can change w to w + in this inequality. Take ε n such that P(D c εn ) ≤ 1/2 n . By the Borel-Cantelli lemma for all ω in some set Ω ′ with P(Ω ′ ) = 1 we have ω ∈ D εn for sufficiently large n. Thus, and from (2.15) we obtain the estimate v(x) ≤ lim sup G • ∋y→x w + (y).
Any stochastic subsolution u is a lower bound for v: a). The set V − of stochastic subsolutions is non-empty and contains sufficiently large negative constants c. Indeed, it is easy to see that Lemma 5. Let u 1 , u 2 be stochastic subsolutions. Then u 1 ∨ u 2 is a stochastic subsolution.
The proof follows from the inequality This lemma is analogous to Lemma 3.
Proof. If u − is not a viscosity supersolution then there exist x 0 ∈ G, ϕ ∈ C 2 and ε > 0 such that By the continuity of F we can assume that Furthermore, by the lower-semicontinuity of u − we have for some δ > 0. In the same way as in the proof of Lemma 4, one can show that there exist u ∈ V − and δ ′ ∈ (0, δ) such that u ≥ ϕ + δ ′ on S ε . Take an η ∈ (0, δ ′ ) such that (3.2) holds true for ϕ η = ϕ + η instead of ϕ. We have To get a contradiction it is enough to prove that the function , contrary to the definition of u − . Clearly u η ∈ C b (G), and we only should to verify (3.1) for any randomized initial condition (τ, ξ), control process α ∈ A (τ, ξ) and stopping time ρ ≥ τ . Put We have Moreover, X τ 1 ,ξ 1 ,α = X τ,ξ,α on the stochastic interval τ 1 , ∞ .

The case of a smooth domain
Let G coincide with the closure of G • , and assume that ∂G is of class C 2 . Then the distance function ρ from ∂G: is of class C 2 in a neighbourhood of ∂G (see [14,Lemma 14.16]). Put −n(x) = Dρ(x), x ∈ G. If x ∈ ∂G, n(x) is the unit outer normal to ∂G at x. It is shown in [ To get a comparison result we need a stronger condition, presented in the next theorem.
Theorem 2. Assume that there exists a Borel measurable function ψ : G → A such that the functions (1.4) are globally Lipschitz continuous and (4.1) Then the value function v, defined by (1.2), is the unique continuous viscosity solution of the state constrained problem (1.7).
Proof. The viscosity subsolution w + , specified in Theorem 1, satisfies also the linear inequality in the viscosity sense. Consider the function Clearly, w + is a viscosity subsolution of (4.2), satisfying all conditions of Theorem 1. Now we use conditions (4.1). By Lemma 4.1 of [3] the function w + is a viscosity subsolution of (4.2) on G. Furthermore, by Theorem 4.1(ii) of [3], for any x ∈ ∂G there exists a sequence x k ∈ G • , x k → x such that w + (x) = lim k→∞ w + (x k ) and or, equivalently, for some β ∈ (0, 1). This is the nontangential upper semicontinuity property of w + , which, by the comparison result of [21] (Theorem 2.2), implies that Let us prove that ∂G = Γ. For x ∈ ∂G denote by X the solution of the equation Since conditions (4.1) imply the viability, we get an admissible control α t = ψ(X t ): X t = X x,α t ∈ G, t ≥ 0 a.s. Take ε > 0 such that ρ ∈ C 2 (B ε (x)) and inf y∈Bε(x)∩G −n(y) · b ψ (y) + 1 2 Tr σ ψ (y)σ T ψ (y)D 2 ρ(y) > 0.
Furthermore, put τ = inf{t ≥ 0 : X t ∈ B ε (x)}. By Ito's formula we have where M is a continuous martingale with M 0 = 0. From the representation of M as a time-changed Brownian motion on an extended filtered probability space (see [16,Theorem 7.2']) it follows that 0 is a limit point of the set {t > 0 : M t = 0} a.s. For a sequence t k (ω) → 0 with M t k = 0 we have Tr σ ψ (X s )σ T ψ (X s )D 2 ρ(X s ) ds > 0 a.s.
Hence, v = w + = u − is a continuous function, and it satisfies (1.7) in the viscosity sense. Note also that the uniqueness of a continuous constrained viscosity solution is a more classical result: see [12,Theorem 7.10].
Theorem 2 is similar to Theorem 4.1 of [21]. Although, the second condition (4.1) is presented there in the form which is formally not comparable to ours local condition −n(x) · b ψ (x) > 0, x ∈ ∂G, the result of [21] is more sophisticated. To get the comparison result in Theorem 2 we used only the fact that any subsolution, being suitably modified at the boundary points, possesses the nontangential upper semicontinuity property under conditions (4.1). In [21] it is shown that a subsolution u ≥ v with this property exists even some diffusion in the tangent direction to ∂G is allowed: see conditions A3 of [21]. Certainly, the stochastic Perron method can be applied in the case of finite horizon as well. However, some work is required to study the parabolic problem, corresponding to (1.7). In particular, a new boundary condition at the terminal time appears, and the viability notion should be modified. Such a problem was studied in [10] by another methods. We mention a comparison result, ensuring the continuity of the value function, proved under conditions similar to (4.1): see [10,Theorem A.1].