1 Introduction

The Markov property is a main tool used in the dynamic evaluation of risk, for example in the solution of risk-sensitive optimisation problems. In this paper we present a probabilistic formulation of the Markov property under a risk framework with minimal assumptions, which we call dynamic conditional risk mappings, and give applications to optimal prediction, a class of risk-sensitive stochastic optimisation problems.

To fix ideas, let \(X = (X_t)_{t \in {\mathbb {N}}_0}\) be a Markov chain taking values in a measurable space E and let \((\Omega ,{\mathcal {F}},({\mathcal {F}}_t)_{t \in {\mathbb {N}}_0},{\mathbb {P}}^x)\) be its canonical probability space, where \(X_0=x\), \({\mathbb {P}}^x-\)a.s. and \({\mathbb {N}}_0=\{0,1,\ldots \}\). Let \(\varrho =((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) be the family of conditional linear expectations given by

$$\begin{aligned} \rho ^{x}_{t}(Z) = {\left\{ \begin{array}{ll} {\mathbb {E}}^{x}[Z], &{} t=0, \\ {\mathbb {E}}^{x}[Z \vert {\mathcal {F}}_{t}], &{} t \ge 1, \end{array}\right. } \end{aligned}$$
(1)

where Z is an arbitrary bounded random variable depending on the whole sample path (that is, Z is measurable with respect to \((\Omega ,{\mathcal {F}})\)). Then \(\varrho \) is Markovian in the sense that

$$\begin{aligned} \rho ^{x}_{t}(Z \circ \theta _{t}) = \rho ^{X_{t}}_0(Z) \;\;{\mathbb {P}}^{x}\text {-a.s. for each }\,t \in {\mathbb {N}}_0, \end{aligned}$$
(2)

where \(\theta _t\) is the shift operator, and we would like to generalise this property to an appropriately large class of nonlinear (that is, risk-sensitive) families \(\varrho \).

A number of settings have been given for the Markov property under dynamic risk frameworks. Broadly they are formulated either on functions of the state of the Markov process (that is, analytically), or on the canonical probability space (that is, probabilistically). In the linear case the probabilistic and analytic formulations are equivalent, and below we obtain sufficient conditions on \(\varrho \) for their equivalence (Proposition 2.11).

Analytic formulations, which are often based on so-called transition risk mappings (cf. Definition 4.1), are useful in recursive solution techniques which evaluate risk only one step ahead, taking \(Z=f(X_{t+1})\) in (1). Since probabilistic formulations apply to the whole path of X, they are useful for recursions which directly evaluate risk multiple steps ahead, taking \(Z=f(X_{t+1}, X_{t+2},...)\). In optimal prediction problems, for example, the evaluation of risk depends on the evolution of the process after a user-selected stopping time: for instance, the problem of stopping as close as possible to the ultimate maximum of a time-homogeneous Markov chain X taking values in \(E = {\mathbb {R}}\) (cf. Allaart 2010; Yam et al. 2009 in the case of linear expectation):

$$\begin{aligned} V_\text {pred}^T(x) := \inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^x_0(X_T^* - X_\tau ), \end{aligned}$$
(3)

where \(T \in {\mathbb {N}}_0\), \(X_T^* := \max _{0 \le s \le T} X_s\), and \({\mathscr {T}}_{[0,T]}\) is the set of stopping times taking values in \(\{0,1,\ldots ,T\}\) (see also du Toit and Peskir 2007; Pedersen 2003 for work in continuous time). In the aforementioned studies, explicit solutions have been obtained for this problem by applying the probabilistic Markov property to represent the objective as a function of \(\tau \) and \(X_\tau \), obtaining a function F such that

$$\begin{aligned} {\mathbb {E}}^x[ f(X_T^* - X_\tau )] = {\mathbb {E}}^x[ F(\tau ,X_\tau )], \end{aligned}$$

see e.g. page 1077 in Allaart (2010). The probabilistic Markov property, which is satisfied by the commonly used entropic, mean semi-deviation, \({\text {VaR}}\), \({\text {AVaR}}\) and worst-case risk mappings (see Sect. 3), is applied in Sect. 5.1 to solve (3) recursively.

The evaluation of risk for random variables via so-called sublinear (and therefore convex) functionals goes back to Lebedev (1992, 1993), where results including dual representations are obtained in both the static and conditional settings. To enable dynamic programming for risk-sensitive Markov decision processes, dynamic risk-sensitive frameworks have also been proposed using analytic formulations of the Markov property under the assumption of time consistency. In Ruszczyński (2010) a dynamic setting is introduced in which risk-sensitive Markov decision processes are studied in both finite and infinite time horizon. Also with infinite time horizon, the average risk of controlled Markov processes is studied in Shen et al. (2013) while Çavuş and Ruszczyński (2014) address the undiscounted total risk of transient controlled Markov processes. In Fan and Ruszczyński (2018a, 2018b) a structure for dynamic risk measures is introduced based on a stronger concept of stochastic conditional time consistency. Utility-based (also known as certainty equivalent) frameworks are special cases using the Markov property under linear expectation, see for example Bäuerle and Rieder (2014), Bäuerle and Rieder (2017). Other frameworks are presented in Bartl (2020) using analytic sets and in Pichler and Schlotter (2020) using the Kusuoka representation.

While the dynamic risk-sensitive frameworks above involve a reference probability measure, analytic settings of the Markov property also exist in risk-sensitive frameworks without such a measure. When the state space E is finite, these frameworks include Markov chains under imprecise expectations, which are related to sensitivity analyses under a set of possible transition probabilities for the Markov process \((X_{t})_{t \in {\mathbb {N}}_0}\), see for example de Cooman et al. (2009), Hartfiel (1998), Krak et al. (2017). More generally they include nonlinear expectations which, in Peng (2005) and Nendel (2021), are related to finite-dimensional properties of so-called nonlinear Markov chains. As in the present paper, in Denk et al. (2018) the framework is related to the infinite dimensional path space of Markov processes, although convexity of the nonlinear expectation is then assumed. Also without a reference measure, the risk forms of Dentcheva and Ruszczyński (2020) have been applied to the optimisation of partially observable two-stage systems.

The general study of dynamic conditional risk mappings can also be approached via backward stochastic differential or difference equations, see Cohen and Elliott (2008), Cohen and Elliott (2010). In contrast to the latter setup our risk mappings do not assume time consistency. In the other direction, in Martyr et al. (2022) reflected backward stochastic difference equations are derived from dynamic conditional risk mappings, in the study of non-Markovian optimal switching problems.

In the present work we assume a reference measure and make minimal further assumptions. Time consistency is not assumed, making our formulation applicable to risk mappings including mean semi-deviation and average value at risk (cf. Sect. 3). In the time-consistent case we make the connection to analytic formulations, and provide a recursive solution to the optimal prediction problem.

For convex risk mappings we characterise the Markov property in terms of the dual representation (see for example Artzner et al. 1999; Delbaen 2002; Detlefsen and Scandolo 2005; Frittelli and Rosazza Gianin 2002; Lebedev 1992, 1993). More precisely, we show that a Markovian convex risk mapping can be characterised as a supremum over penalised linear expectations with respect to certain transition kernels, extending the dual representation of transition risk mappings beyond the coherent case studied in Ruszczyński (2010). We also obtain sufficient conditions under which the latter structure implies the probabilistic Markov property.

The paper is structured as follows. Section 2 provides the probabilistic framework, together with equivalences between versions of the Markov property, and a representation in terms of acceptance sets. Section 3 gives examples and Sect. 4 addresses the dual representation, while applications to optimisation problems are given in Sect. 5.

2 A probabilistic Markov property for risk mappings

After presenting the setup and briefly recalling necessary definitions (Sect. 2.1), in Sects. 2.2 and 2.3 we provide our novel probabilistic setting for the Markov property and establish equivalent forms. The Markov property in terms of acceptance sets is studied in Sect. 2.4.

2.1 Setup and notation

Suppose we have an E-valued time-homogeneous Markov process \((X_{t})_{t \in {\mathbb {N}}_0}\) with respect to the filtered probability space \((\Omega ,{\mathcal {F}},{\mathbb {F}},{\mathbb {P}})\), where:

  • E is a Polish space equipped with its Borel \(\sigma \)-algebra \({\mathcal {E}}\),

  • \({\mathbb {N}}_0= \{0,1,2,\ldots \}\) is the discrete time parameter set,

  • \(\Omega \) is the canonical space of trajectories \(\Omega = E^{{\mathbb {N}}_0}\),

  • X is the coordinate mapping, \(X_{t}(\omega ) = \omega (t)\) for \(\omega \in \Omega \) and \(t \in {\mathbb {N}}_0\),

  • \({\mathbb {F}} = ({\mathcal {F}}_{t})_{t \in {\mathbb {N}}_0}\) with \({\mathcal {F}}_{t} = \sigma (\{X_{s} :s \le t\})\) the natural filtration generated by X and \({\mathcal {F}} = \sigma (\bigcup _{t \in {\mathbb {N}}_0} {\mathcal {F}}_{t})\).

Let \({\mathscr {P}}({\mathcal {F}})\) denote the set of probability measures on \((\Omega ,{\mathcal {F}})\). Unless otherwise specified, all inequalities between random variables will be interpreted in the almost sure sense with respect to the appropriate probability measure. We write \({\mathscr {T}}\) for the set of finite-valued stopping times and \({\mathscr {T}}_{[t,T]}\) for the set of stopping times taking values in \(\{t,t+1,\ldots ,T\}\). We denote by \(b{\mathcal {F}}\) the space of bounded random variables on \((\Omega ,{\mathcal {F}})\) and similarly for other \(\sigma \)-algebras. It will also be convenient to define \({\mathcal {F}}_{t,\infty } = \sigma (X_{s} :s \ge t)\) and \({\mathcal {F}}_{t,t} = \sigma (X_t)\).

In the above setup the following objects exist:

  • The law \(\mu ^{X_{0}}\) of \(X_0\) under \({\mathbb {P}}\) and a family of probability measures defined by the measurable mapping \(x \mapsto {\mathbb {P}}^x\) from E to \({\mathscr {P}}({\mathcal {F}})\), which is a disintegration of \({\mathbb {P}}\) with respect to \(X_0\) (see Dellacherie and Meyer 1978, p. 78). To be precise, this family satisfies \({\mathbb {P}}^{x}(X_{0} = x) = 1\) and for every \(F \in {\mathcal {F}}\) we have

    $$\begin{aligned} {\mathbb {P}}(F) = \int _{E}{\mathbb {P}}^{x}(F)\,\mu ^{X_{0}}({\textrm{d}}x). \end{aligned}$$

  • A time-homogeneous Markov transition kernel \(q^{X} :{\mathcal {E}} \times E \rightarrow [0,1]\) such that for every \(x \in E\) and \(B \in {\mathcal {E}}\) we have \(q^{X}(B \vert x) = {\mathbb {P}}^{x}\big (X_{1} \in B \big )\),

  • Markov shift operators \(\theta _{t} :\Omega \rightarrow \Omega \), \(t \in {\mathbb {N}}_0\) such that \(\theta _{0}(\omega ) = \omega \), \(\theta _{t} \circ \theta _{s} = \theta _{t+s}\) and \((X_{t} \circ \theta _{s})(\omega ) = X_{t+s}(\omega )\) for each \(\omega \in \Omega \) and \(s,t \in {\mathbb {N}}_0\).

For \(\tau \in {\mathscr {T}}\) define the random shift operator \(\theta _{\tau }\) by

$$\begin{aligned} \begin{aligned} \theta _{\tau }(\omega )&= \theta _{\tau (\omega )}(\omega ), \\&= \theta _{t}(\omega )\;\; \text {on}\;\; \{\tau (\omega ) = t\}. \end{aligned} \end{aligned}$$

We recall the definitions of risk mapping and conditional risk mapping (which are interchangeable via the mapping \(Z \mapsto \rho (-Z)\) with the monetary conditional risk measures of Föllmer and Schied 2016, Def. 11.1):

Definition 2.1

(Risk mapping) A risk mapping on the probability space \((\Omega ,{\mathcal {F}},{\mathbb {P}}^x)\) is a function \(\rho ^x :b{\mathcal {F}} \rightarrow {\mathbb {R}}\) satisfying

  • Normalisation: \(\rho ^x(0) = 0\),

  • Translation invariance: \(\forall \; Z \in b{\mathcal {F}}\) and \(c \in {\mathbb {R}}\) we have \(\rho ^x(Z + c) = c + \rho ^x(Z)\),

  • Monotonicity: \(\forall \; Z,Z' \in b{\mathcal {F}}\), we have \(Z \le Z' \, {\mathbb {P}}^x\text {-a.s.} \implies \rho ^x(Z) \le \rho ^x(Z')\).

Definition 2.2

(Conditional risk mapping) A conditional risk mapping on the probability space \((\Omega ,{\mathcal {F}},{\mathbb {P}}^x)\) with respect to the \(\sigma \)-algebra \({\mathcal {F}}_t \subseteq {\mathcal {F}}\) is a function \(\rho _t^x :b{\mathcal {F}} \rightarrow b{\mathcal {F}}_t\) satisfying:

  • Normalisation: \(\rho _t^x(0) = 0\) \({\mathbb {P}}^x\)-a.s.,

  • Conditional translation invariance: \(\forall \; Z \in b{\mathcal {F}}\) and \(Z' \in b{\mathcal {F}}_t\),

    $$\begin{aligned} \rho _t^x(Z + Z') = Z' + \rho _t^x(Z), \qquad {\mathbb {P}}^x\text {-a.s.} \end{aligned}$$

  • Monotonicity: \(\forall \; Z,Z' \in b{\mathcal {F}}\),

    $$\begin{aligned} Z \le Z' \, {\mathbb {P}}^x\text {-a.s.} \implies \rho _t^x(Z) \le \rho _t^x(Z') \,{\mathbb {P}}^x\text {-a.s.} \end{aligned}$$

Conditional risk mappings also satisfy the following property (cf. Cheridito et al. 2006, Prop. 3.3 and Föllmer and Schied 2016, Ex. 11.1.2):

  • Conditional locality: for every Z and \(Z'\) in \(b{\mathcal {F}}\) and \(A \in {\mathcal {F}}_t\), we have \({\mathbb {P}}^x\)-a.s.

    $$\begin{aligned} \rho _t^x(\mathbbm {1}_{A}Z + \mathbbm {1}_{A^{c}}Z') = \mathbbm {1}_{A}\rho _t^x(Z) + \mathbbm {1}_{A^{c}}\rho _t^x(Z'). \end{aligned}$$

Definition 2.3

(Dynamic conditional risk mapping) For each \(x \in E\) a dynamic conditional risk mapping on the filtered probability space \((\Omega ,{\mathcal {F}},{\mathbb {F}},{\mathbb {P}}^x)\) is a sequence \((\rho _{t}^x)_{t \in {\mathbb {N}}_0}\) where

  • \(\rho _0^x\) is a risk mapping,

  • for each \(t \ge 1\), \(\rho _{t}^x\) is a conditional risk mapping on \((\Omega ,{\mathcal {F}},{\mathbb {P}}^x)\) with respect to \({\mathcal {F}}_{t}\).

We use the superscript x in \((\rho _{t}^x)_{t \in {\mathbb {N}}_0}\) to indicate a dynamic conditional risk mapping on \((\Omega ,{\mathcal {F}},{\mathbb {F}},{\mathbb {P}}^x)\).

Note that the codomain of \(\rho _0^x\) is \({\mathbb {R}}\) while, for each \(t \ge 1\), the codomain of \(\rho _t^x\) is \(b{{\mathcal {F}}_t}\). This setup is motivated by the fact that any \({\mathcal {F}}_0\)-measurable random variable is \({\mathbb {P}}^x\)-a.s. constant. For example, for each \(x \in E\), the sequence \((\rho _t^x)_{t \in {\mathbb {N}}_0}\) given by (1) is a dynamic conditional risk mapping.

For a finite stopping time \(\tau \) define

$$\begin{aligned} \rho _{\tau } = \sum _{t \in {\mathbb {N}}_0}\mathbbm {1}_{\{\tau = t\}}\rho _{t}, \end{aligned}$$

noting that \(\rho _{\tau } :b{\mathcal {F}} \rightarrow b{\mathcal {F}}_{\tau }\).

In some results below we will assume continuity.

Definition 2.4

Let \(t\in {\mathbb {N}}_0\), \(x \in E\). We say that \(\rho ^x_{t}\) is continuous from below (resp. from above) if \(\rho ^x_{t}(Y_n) \rightarrow \rho ^x_{t}(Y)\) \({\mathbb {P}}^x\)-a.s. for every increasing (resp. decreasing) sequence \((Y_n)_{n \in {\mathbb {N}}_0}\) in \(b{\mathcal {F}}\) converging \({\mathbb {P}}^x\)-a.s. to \(Y \in b{\mathcal {F}}\).

Note that results for decreasing risk maps (e.g. in Föllmer and Penner 2006) requiring continuity from above can be applied to increasing risk maps of Definitions 2.1–2.3 if continuity from below is assumed.

2.2 Markov property

We begin with measurability with respect to the initial state of the Markov process, referring to this as regularity.

Definition 2.5

(Regularity) A collection of risk mappings \((\rho ^x)_{x \in E}\) is said to be regular if for all \(Z \in b{\mathcal {F}}\) the map \(x \mapsto \rho ^x(Z)\) is bounded and measurable.

Definition 2.6

(Markov property) The family \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) of dynamic conditional risk mappings satisfies the Markov property (for the chain \((X_t)_{t \in {\mathbb {N}}_0}\)) if

  1. 1.

    \((\rho _0^x)_{x \in E}\) is regular,

  2. 2.

    for each \(x \in E\), \(Z \in b{\mathcal {F}}\) and \(t \in {\mathbb {N}}_0\) we have

    $$\begin{aligned} \rho ^{x}_{t}(Z \circ \theta _{t}) = \rho ^{X_{t}}(Z) \;\;{\mathbb {P}}^{x}\text {-a.s.}, \end{aligned}$$
    (4)

where \(\rho ^{X_{t}}(Z)\) is interpreted as the random variable \(\omega \mapsto \rho ^{X_{t}(\omega )}(Z)\).

By construction, if \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) is a family of dynamic conditional risk mappings then \((\rho _0^x)_{x \in E}\) is a collection of risk mappings. For convenience we often write \(\rho ^x\) for \(\rho ^x_0\).

In particular we have

$$\begin{aligned} \rho ^x(Z)=\rho ^x(\mathbbm {1}_{\{x\}}(X_0)Z), \qquad Z \in b{\mathcal {F}}, \; x \in E. \end{aligned}$$
(5)

Note that the linear conditional expectation (1) satisfies this Markov property and corresponds to the risk-neutral case. Examples of \(\rho \) which are risk sensitive are presented in Sect. 3.

Remark 2.7

Note that (4) could have been specified differently. For example, by relating all risk mappings \(\rho _t^x\) to the same regular collection \((\rho ^x)_{x \in E}\) in (4) we have imposed a time homogeneity on the measurement of risk. This is not essential, since taking a collection \(\{\rho ^{x,s}: x \in E, s \in {\mathbb {N}}_0\}\) indexed also by time and specifying

$$\begin{aligned} \rho ^{x}_{t}(Z \circ \theta _{t}) = \rho ^{X_{t},t}(Z) \;\;{\mathbb {P}}^{x}\text {-a.s.}, \end{aligned}$$
(3')

the family of dynamic conditional risk mappings may be time-heterogeneous.

A regular collection \((\rho ^x)_{x \in E}\) of risk mappings can also be used to construct a Markovian family \(\varrho =((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) satisfying Definition 2.6, as follows. We use the fact that any bounded \({\mathcal {F}}\)-measurable random variable Z can be represented as \(Z=f(X_0,X_1,\ldots )\) for some measurable and bounded function \(f :E^{{\mathbb {N}}_0} \rightarrow {\mathbb {R}}\), which follows by standard monotone class arguments, see Blumenthal and Getoor (1968), Prop. 0.2.7 or Çinlar (2011), Th. 2.4.4. As it is obtained without reference to any probability measure, the equality \(Z=f(X_0,X_1,\ldots )\) holds for all (rather than almost all) \(\omega \in \Omega \) and therefore the function f is unique.

Proposition 2.8

Let \((\rho ^x)_{x \in E}\) be regular. For each \(x \in E\), \(t \in {\mathbb {N}}_0\) and \(Z=f(X_0,X_1,\ldots )\) let

$$\begin{aligned} \rho ^x_t(Z)(\omega ) :=\rho ^{X_t(\omega )}(Z_t(X_0(\omega ),\ldots ,X_t(\omega ))), \qquad \omega \in \Omega , \end{aligned}$$

where

$$\begin{aligned} Z_t(x_0,\ldots ,x_t) :=f(x_0,\ldots , x_t, X_1,X_2,\ldots ). \end{aligned}$$

Then for each \(x \in E\), \((\rho ^x_t)_{t \in {\mathbb {N}}_0}\) is a dynamic conditional risk mapping and the family \(\varrho = ((\rho ^x_t)_{t \in {\mathbb {N}}_0})_{x \in E}\) satisfies the Markov property.

Proof

Let \(x \in E\), \(t \in {\mathbb {N}}_0\) and \(\omega \in \Omega \) be arbitrary. For compactness we will write \(X_{0:t}(\omega )\) for \((X_0(\omega ),\ldots , X_t(\omega )) \in E^{t+1}\). Clearly \(\rho _t^x\) is normalised, so we check conditional translation invariance and monotonicity. Taking \(Z = f(X_0,X_1,\ldots ) \in b{\mathcal {F}}\) and \(W = g(X_0,\ldots ,X_t) \in b{\mathcal {F}}_t\), by construction we have

$$\begin{aligned} \rho _t^x(Z+W)(\omega )&= \rho ^{X_t(\omega )}(Z_t(X_0(\omega ),\ldots , X_t(\omega )) + W_t(X_0(\omega ),\ldots , X_t(\omega ))) \\&= \rho ^{X_t(\omega )}(Z_t(X_0(\omega ),\ldots , X_t(\omega ))) + W_t(X_0(\omega ),\ldots , X_t(\omega )) \\&= \rho _t^x(Z)(\omega ) + W(\omega ). \end{aligned}$$

To check monotonicity let \(Z = f(X_0,X_1,\ldots )\) and \(Z' = f'(X_0,X_1,\ldots )\) be two bounded random variables such that \(Z \le Z'\) \({\mathbb {P}}^x\)-a.s. We first show that \(Z_t(X_{0:t}(\omega )) \le Z'_t(X_{0:t}(\omega )), {\mathbb {P}}^{X_t(\omega )}\)-a.s. Writing as usual \({\mathbb {P}}^x(A \vert {\mathcal {F}}_t)\) for \({\mathbb {E}}^x[1_A \vert {\mathcal {F}}_t]\) for each \(A \in {\mathcal {F}}\), and applying conditional locality and the Markov property, for almost all \(\omega \) we have, with a slight abuse of notation

$$\begin{aligned} 1&= {\mathbb {P}}^x(Z \le Z' \vert {\mathcal {F}}_t)(\omega ) \\&= {\mathbb {P}}^x(f(X_{0:t}(\omega ),X_{t+1}, \ldots ) \le f'(X_{0:t}(\omega ), X_{t+1}, \ldots )\vert {\mathcal {F}}_t)(\omega ) \\&= {\mathbb {P}}^{X_t(\omega )}(f(X_{0:t}(\omega ),X_1, \ldots ) \le f'(X_{0:t}(\omega ),X_1, \ldots )) \\&= {\mathbb {P}}^{X_t(\omega )}(Z_t(X_{0:t}(\omega )) \le Z'_t(X_{0:t}(\omega ))). \end{aligned}$$

By the monotonicity of \(\rho ^{X_t(\omega )}\) we then have that \({\mathbb {P}}^x\)-a.s.,

$$\begin{aligned} \rho _t^x(Z)(\omega ) = \rho ^{X_t(\omega )}\big (Z_t(X_{0:t}(\omega ))\big ) \le \rho ^{X_t(\omega )}\big (Z_t'(X_{0:t}(\omega ))\big ) = \rho _t^x(Z')(\omega ). \end{aligned}$$

Lastly we verify the Markov property for the family \(\varrho \). For \(Z=f(X_0,X_1,\ldots )\) we have by construction and (5) that \({\mathbb {P}}^x\)-a.s.

$$\begin{aligned} \rho _t^x(Z \circ \theta _t)(\omega ) = \rho ^{X_t(\omega )}(f(X_t(\omega ),X_1,X_2,\ldots )) = \rho ^{X_t(\omega )}(Z). \end{aligned}$$

\(\square \)

2.3 Equivalent forms of the Markov property

Just as for the linear conditional expectation, the Markov property for risk mappings can be stated in several equivalent forms. We begin with the strong Markov property.

Proposition 2.9

(Strong Markov Property) If \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) satisfies the Markov property then for any stopping time \(\tau \in {\mathscr {T}}\) and \(Z \in b{\mathcal {F}}\) we have

$$\begin{aligned} \rho ^{x}_{\tau }(Z \circ \theta _{\tau }) = \rho ^{X_{\tau }}(Z) \;\;{\mathbb {P}}^{x}\text {-a.s.}. \end{aligned}$$

Proof

Using \(\{\tau = t\} \in {\mathcal {F}}_{t}\), conditional locality and the Markov property we have \({\mathbb {P}}^{x}\)-a.s.:

$$\begin{aligned} \rho ^{x}_{\tau }(Z \circ \theta _{\tau }) = \sum _{t=0}^{\infty }\mathbbm {1}_{\{\tau = t\}}\rho ^{x}_{t}(Z \circ \theta _{t}) = \sum _{t=0}^{\infty }\mathbbm {1}_{\{\tau = t\}}\rho ^{X_{t}}(Z) = \rho ^{X_{\tau }}(Z). \end{aligned}$$

\(\square \)

To make a connection to one-step Markov properties we will require time consistency:

Definition 2.10

The family \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) is said to be time consistent if for all \(Y,Z \in b{\mathcal {F}}\), \(t \in {\mathbb {N}}_0\) and \(x \in E\) we have

$$\begin{aligned} \rho ^x_{t+1}(Y) \le \rho ^x_{t+1}(Z) \; {\mathbb {P}}^x\text {-a.s.} \implies \rho ^x_t(Y) \le \rho ^x_t(Z) \; {\mathbb {P}}^x\text {-a.s.} \end{aligned}$$

We say that a regular collection \((\rho ^x)_{x \in E}\) of risk mappings is time consistent if the associated Markovian dynamic conditional risk mapping (constructed in Proposition 2.8) is time consistent.

It is well known (see e.g. Acciaio and Penner 2011, Prop. 1.16) that we then have the following recursive relation: for every \(x \in E\) and \(0 \le s\le t\),

$$\begin{aligned} \rho ^x_s = \rho ^x_s \circ \rho ^x_t. \end{aligned}$$

As noted in Föllmer and Schied (2016), Exercise 11.2.2, this relation can be generalised to stopping times: for any bounded stopping times \(\tau _1 \le \tau _2\) one has

$$\begin{aligned} \rho _{\tau _1}^x = \rho _{\tau _1}^x \circ \rho _{\tau _2}^x. \end{aligned}$$
(6)

Since risk mappings are nonlinear in general, in the next proof we use a non-standard version of the Monotone Class Theorem (see Appendix 1) which, unlike Blumenthal and Getoor (1968), Th. 0.2.3, does not appeal to vector spaces.

Proposition 2.11

Let \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) be a family of dynamic conditional risk mappings such that \((\rho ^x)_{x \in E}\) is regular. Let each \(\rho _t^x\) be continuous from above and below: that is, \(\rho ^x_{t}(Y_n) \rightarrow \rho ^x_{t}(Y)\) a.s. for every \(t\in {\mathbb {N}}_0\), \(x \in E\) and monotone sequence \((Y_n)_{n \in {\mathbb {N}}_0}\) in \(b{\mathcal {F}}\) converging to \(Y \in b{\mathcal {F}}\). Then

  1. (i)

    \(\varrho \) is Markov iff for all \(k\ge 0\) the k-step Markov property holds:

    $$\begin{aligned} \rho _t^x(f(X_{t+1},\ldots ,X_{t+k})) = \rho ^{X_t}(f(X_1,\ldots ,X_k)), \end{aligned}$$
    (7)

    \({\mathbb {P}}^x\)-a.s. for all \(t \in {\mathbb {N}}_0\), \(x \in E\) and bounded measurable functions \(f:E^{k} \rightarrow {\mathbb {R}}\).

  2. (ii)

    If the family \(\varrho \) is time consistent, then \(\varrho \) is Markov iff the one-step Markov property holds: for every \(t\in {\mathbb {N}}_0\), \(x \in E\) and bounded measurable function \(f :E \rightarrow {\mathbb {R}}\) we have

    $$\begin{aligned} \rho _t^x(f(X_{t+1})) = \rho ^{X_t}(f(X_1)), \qquad {\mathbb {P}}^x\text {-a.s.}. \end{aligned}$$
    (8)

Remark 2.12

The assumptions of this proposition simplify in the case of convex risk mappings, for which continuity from above implies continuity from below (see proof of Corollary 4.3).

Proof

For both claims (i) and (ii), the ‘only if’ part is trivial and so it remains to establish the ‘if’ part. We first prove this for claim (ii).

Therefore let the family \(\varrho \) be time consistent and suppose that the one-step Markov property (8) holds. We begin by showing, proceeding by induction on k, that the Markov property (4) holds for the class of simple functions – that is, functions of the form

$$\begin{aligned} f(x_{t+1},\ldots ,x_{t+k}) = \sum _{j=1}^{n}\alpha _{j} g_{j}(x_{t+1},\ldots ,x_{t+k}), \quad t \in {\mathbb {N}}_0, n \ge 1, \alpha _{i} \in {\mathbb {R}}. \end{aligned}$$
(9)

This is true for \(k = 1\) since this is a special case of the one-step Markov property (8). Suppose it is also true for some \(k \ge 1\). We have

$$\begin{aligned} f(x_{t+1},\ldots ,x_{t+k+1})&= \sum _{j=1}^{n}\alpha _{j} g_{j}(x_{t+1},\ldots ,x_{t+k+1}) \nonumber \\&= \sum _{j=1}^{n}\alpha _{j} \left( \prod _{i=1}^{k+1}\mathbbm {1}_{A_{ij}}(x_{t+i})\right) \nonumber \\&= \sum _{j=1}^{n}\mathbbm {1}_{A_{1j}}(x_{t+1})\left( \alpha _{j}\prod _{i=2}^{k+1}\mathbbm {1}_{A_{ij}}(x_{t+i})\right) . \end{aligned}$$
(10)

By taking all possible intersections of the sets \(A_{11},\ldots ,A_{1n}\) and their complements, we can define \(N \ge n\) mutually disjoint sets \({\tilde{A}}_{1},\ldots ,{\tilde{A}}_{N}\) belonging to \({\mathcal {E}}\) such that

$$\begin{aligned} \sum _{j=1}^{n}\mathbbm {1}_{A_{1j}}(x_{t+1})\left( \alpha _{j}\prod _{i=2}^{k+1}\mathbbm {1}_{A_{ij}}(x_{t+i})\right)&= \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(x_{t+1})\left( \sum _{j=1}^{n}{\tilde{\alpha }}_{\ell j}\prod _{i=2}^{k+1}\mathbbm {1}_{A_{ij}}(x_{t+i})\right) , \end{aligned}$$

where \({\tilde{\alpha }}_{\ell j} = \alpha _{j}\) if \(A_{1j} \cap {\tilde{A}}_{\ell } \ne \emptyset \) and \({\tilde{\alpha }}_{\ell j} = 0\) otherwise. Therefore we can rewrite f in (10) as

$$\begin{aligned} f(x_{t+1},\ldots ,x_{t+k+1}) = \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(x_{t+1})f_{\ell }(x_{t+2},\ldots ,x_{t+k+1}), \end{aligned}$$
(11)

where the \({\tilde{A}}_{\ell }\) are mutually disjoint and each \(f_{\ell }\) has the form (9). Using the local property and time consistency for \(\rho _t^x\), the induction hypothesis and the one-step Markov property we have

$$\begin{aligned} \rho _t^x(f(X_{t+1},\ldots , X_{t+k+1}))&= \rho _t^x\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{t+1})f_{\ell }(X_{t+2},\ldots ,X_{t+k+1})\right) \nonumber \\&= \rho _t^x\left( \rho _{t+1}^x\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{t+1})f_{\ell }(X_{t+2},\ldots ,X_{t+k+1})\right) \right) \nonumber \\&= \rho _t^x\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{t+1}) \rho _{t+1}^x\left( f_{\ell }(X_{t+2},\ldots ,X_{t+k+1})\right) \right) \nonumber \\&= \rho _t^x\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{t+1})\rho ^{X_{t+1}}(f_{\ell }(X_{1},\ldots ,X_{k}))\right) \nonumber \\&= \rho ^{X_{t}}\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{1}) \rho ^{X_{1}}(f_{\ell }(X_{1},\ldots ,X_{k}))\right) . \end{aligned}$$
(12)

Note that for every realisation \(x_{t}\) of \(X_{t}(\omega )\) we have that almost surely under \({\mathbb {P}}^{x_t}\),

$$\begin{aligned} \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{1})\rho ^{X_1}\left( f_{\ell }(X_{1},\ldots ,X_{k})\right)&= \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{1})\rho _1^{x_t}\left( f_{\ell }(X_{2},\ldots ,X_{k+1})\right) \nonumber \\&= \rho _1^{x_t}\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{1}) f_{\ell }(X_{2},\ldots ,X_{k+1})\right) \end{aligned}$$
(13)

Therefore, by (11)–(13) and time-consistency we have for almost every \(\omega \in \Omega \):

$$\begin{aligned} \rho _t^x(f(X_{t+1},\ldots , X_{t+k+1}))(\omega )&= \rho ^{X_t(\omega )} \left( \rho _1^{X_t(\omega )}\left( \sum _{\ell =1}^{N}\mathbbm {1}_{{\tilde{A}}_{\ell }}(X_{1}) f_{\ell }(X_{2},\ldots ,X_{k+1})\right) \right) \nonumber \\&= \rho ^{X_t(\omega )}(f(X_1,\ldots , X_{k+1})), \end{aligned}$$

and, by induction, the Markov property (4) holds for all functions f of the form (9).

Next we appeal to the monotone class theorem. Let \({\mathscr {H}}_0\) be the set of random variables having the form \(Z=f(X_0,\ldots ,X_{k})\) for some \(k \in {\mathbb {N}}_0\) and some f of the form (9). Clearly \({\mathscr {H}}_0\) is closed under the operation of taking the pointwise minimum. Let

$$\begin{aligned} {\mathscr {H}} :=\{Z \in b{\mathcal {F}} :\rho _t^x(Z \circ \theta _t) = \rho ^{X_t}(Z)\,\,\, {\mathbb {P}}^x\text {-a.s. for all } x \in E, t \in {\mathbb {N}}_0\}. \end{aligned}$$

We show that \({\mathscr {H}}_0 \subset {\mathscr {H}}\). Suppose that \(Z=f(X_0,\ldots , X_k) \in {\mathscr {H}}_0\). Then by conditional locality and the fact that \(Z\in {\mathscr {H}}_0\) we have that for each \(\omega \in \Omega \),

$$\begin{aligned} \rho _t^x(Z \circ \theta _t)(\omega )&= \rho _t^x(f(X_t(\omega ), X_{t+1}, \ldots , X_{t+k}))(\omega ) \\&= \rho ^{X_t(\omega )}(f(X_t(\omega ), X_1, \ldots , X_{k}))(\omega ) = \rho ^{X_t(\omega )}(f(X_0, X_1, \ldots , X_{k}))(\omega ), \end{aligned}$$

i.e. \(Z \in {\mathscr {H}}\).

The space \({\mathscr {H}}\) is closed under monotone limits and Theorem A.1 implies that \({\mathscr {H}}\) contains all bounded \(\sigma ({\mathscr {H}}_0)\)-measurable functions. Since \(\sigma ({\mathscr {H}}_0) = {\mathcal {F}}\) we conclude that the Markov property (4) holds on \({\mathscr {H}} = b{\mathcal {F}}\), completing the proof of claim (ii).

To prove claim (i), note that (7) applies directly to all functions f of the form (9), in which case time consistency does not need to be assumed. We then appeal to the monotone class theorem as we did for claim (ii). \(\square \)

The following result shows that Markovian families of conditional risk mappings which are continuous from above and below can be represented using the canonical form given in Proposition 2.8.

Proposition 2.13

Let \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) be a family of dynamic conditional risk mappings continuous from above and below. Then \(\varrho \) satisfies the Markov property if and only if for all \(x \in E\), \(t \in {\mathbb {N}}_0\), \(Z = f(X_{0},X_{1},\ldots ) \in b{\mathcal {F}}\) and \({\mathbb {P}}^{x}\)-almost every \(\omega \in \Omega \) we have

$$\begin{aligned} \rho ^{x}_{t}(Z)(\omega ) = \rho ^{X_{t}(\omega )}(Z_{t}(X_{0}(\omega ),\ldots ,X_{t}(\omega ))), \end{aligned}$$

where \(Z_{t}(x_{0},\ldots ,x_{t}) :=f(x_{0},\ldots ,x_{t},X_{1},X_{2},\ldots )\).

The proof is omitted as it follows the path analogous to that of Proposition 2.11, namely showing the claimed property for simple random variables and appealing to the monotone class theorem.

Remark 2.14

  1. (i)

    If in addition to the hypotheses of Proposition 2.13 the family \(\varrho \) is time consistent, then we recover a version of the Markov property which is similar to that of Nendel (2021):

    $$\begin{aligned} \rho ^{x}(Z) = \rho ^{x}\Big (\rho ^{X_{t}}(Z_{t}(X_{0},\ldots ,X_{t}))\Big ), \quad \forall x \in E, t \in {\mathbb {N}}_0, Z \in b{\mathcal {F}}, \end{aligned}$$

    where \(Z = f(X_{0},X_{1},\ldots )\) and \(Z_{t}(x_{0},\ldots ,x_{t}) :=f(x_{0},\ldots ,x_{t},X_{1},X_{2},\ldots )\). Note that in Definition 1.2 in Nendel (2021) random variables of the form \(Z = f(X_{0},X_{1},\ldots , X_t,X_{t+s})\) are taken into account.

  2. (ii)

    In Denk et al. (2018) a Kolmogorov-type theorem is established for conditional risk mappings which, like Proposition 2.13, leads to a risk mapping on path space, and Example 5.3 of the latter paper explores the case of discrete-time Markov chains.

2.4 Markov property in terms of acceptance sets

Particularly in the context of mathematical finance, conditional risk mappings can be characterised by their acceptance sets \({\mathcal {A}}_t^x\) (see Acciaio and Penner 2011, Sec. 1.4.1 or Föllmer and Schied 2016, Sec. 4.1), where

$$\begin{aligned} {\mathcal {A}}_t^x := \{ Y \in b{\mathcal {F}} : \rho _t^x(Y) \le 0 \,\, {\mathbb {P}}^x\text {-a.s.}\}, \end{aligned}$$

and so for completeness we also formulate the Markov property in these terms. First define another acceptance set, which will be useful in formulating the Markov property:

$$\begin{aligned} \tilde{{\mathcal {A}}}_t^x = \{ Y\in b{\mathcal {F}} : \rho ^{X_t}(Y) \le 0 \,\, {\mathbb {P}}^x\text {-a.s.}\}. \end{aligned}$$

Note that for any \({\mathcal {F}}_{t,\infty }\)-measurable random variable \(Y={\hat{Y}} \circ \theta _t\) with the representation \(Y = f(X_t,X_{t+1},\ldots )\) one can define \(Y\circ \theta _{-t} := {\hat{Y}} = f(X_0,X_1,\ldots )\).

Lemma 2.15

The family \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) is Markov if and only if \(\rho _t^x :b{\mathcal {F}}_{t,\infty } \rightarrow b{\mathcal {F}}_{t,t}\) and for each \(x \in E\), \(t \in {\mathbb {N}}_0\) and \(Z \in b{\mathcal {F}}\) we have

$$\begin{aligned} Z\circ \theta _t \in {\mathcal {A}}_t^x \iff Z \in \tilde{{\mathcal {A}}}_t^x. \end{aligned}$$
(14)

Proof

Necessity is obvious. Conversely, suppose that \(\rho _t^x :b{\mathcal {F}}_{t,\infty } \rightarrow b{\mathcal {F}}_{t,t}\) and that the equivalence (14) holds. Fix \(Z \in b{\mathcal {F}}\), \(t \in {\mathbb {N}}_0\) and \(x \in E\).

Step 1. We show that

$$\begin{aligned} \rho ^{X_t}(Z) = {{\,\mathrm{ess\,inf}\,}}\{ Y=g(X_t) \in b{\mathcal {F}}_{t,t} :Z - Y \circ \theta _{-t} \in \tilde{{\mathcal {A}}}_t^{x} \}, \qquad {\mathbb {P}}^{x}\text {-a.s..} \end{aligned}$$
(15)

Proof of ‘\(\ge \)’. Let \(g(y) := \rho ^y(Z)\) for \(y\in E\). Then \({\mathbb {P}}^y\)-a.s. we have \(g(X_t)\circ \theta _{-t} = \rho ^{X_0}(Z) = \rho ^y(Z)\). Since \(\rho ^y(Z-g(X_t)\circ \theta _{-t}) = \rho ^y(Z-\rho ^y(Z)) = 0\) we have \(\rho ^{X_t}(Z-g(X_t)\circ \theta _{-t}) =0\), implying \(Z - g(X_t)\circ \theta _{-t} \in \tilde{{\mathcal {A}}}_t^{x}\).

Proof of ‘\(\le \)’. Let \(Y=g(X_t)\) belong to the set on the right-hand side of (15). Then for \(\Omega _1 = \{\omega \in \Omega : \rho ^{X_t}(Z - Y \circ \theta _{-t}) \le 0\}\) we have \({\mathbb {P}}^{x}(\Omega _1) = 1\). Let \(\Omega _0 = \{\omega \in \Omega : \rho ^{X_t}(Z)>Y\}\). To show that \(\Omega _0 \subset \Omega _1^c\), let \(\omega \in \Omega _0\) and \(x_t := X_t(\omega ) = \omega (t)\). Since \(\omega \in \Omega _0\), we have that \(\rho ^{X_t(\omega )}(Z) > Y(\omega )\), which is equivalent to \(\rho ^{x_t}(Z)>g(x_t)\). Then

$$\begin{aligned}{} & {} \rho ^{X_t(\omega )}(Z-Y\circ \theta _{-t}) = \rho ^{x_t}(\mathbbm {1}_{x_t}(X_0)(Z-Y\circ \theta _{-t})) \\{} & {} \quad = \rho ^{x_t}(Z-g(x_t)) = \rho ^{x_t}(Z) - g(x_t) > 0, \end{aligned}$$

i.e. \(\omega \in \Omega _1^c\). Since \({\mathbb {P}}^x(\Omega _1^c) = 0\), it follows that \({\mathbb {P}}^x(\Omega _0) = 0\), which finishes the proof of the claim of Step 1.

Step 2. To finish the proof, note from Acciaio and Penner (2011), Prop. 1.2 (modulo a minus sign which appears because Acciaio and Penner (2011) considers decreasing risk mappings) that

$$\begin{aligned} \rho _t^x(Z \circ \theta _t)&= {{\,\mathrm{ess\,inf}\,}}\{ Y \in b{\mathcal {F}}_t :Z\circ \theta _t - Y \in {\mathcal {A}}_t^x \} \\&\le {{\,\mathrm{ess\,inf}\,}}\{ Y \in b{\mathcal {F}}_{t,t} :Z \circ \theta _t - Y \in {\mathcal {A}}_t^x \} \le \rho _t^x(Z \circ \theta _t), \end{aligned}$$

where the last inequality follows from the fact that \(Y = \rho _t^x(Z \circ \theta _t) \in b{\mathcal {F}}_{t,t}\) and \(Z \circ \theta _t - \rho _t^x(Z \circ \theta _t) \in {\mathcal {A}}_t^x\). Since for \(Y \in b{\mathcal {F}}_{t,t}\) we have from (14) that

$$\begin{aligned} Z \circ \theta _t - Y \in {\mathcal {A}}_t^x \iff Z - Y \circ \theta _{-t} \in \tilde{{\mathcal {A}}}_t^x, \end{aligned}$$

Step 1 completes the proof. \(\square \)

Remark 2.16

The above lemma implies in particular that for a Markovian risk map, if \(Z \in b{\mathcal {F}}_{t,\infty }\), then \(\rho _t^x(Z)\) is \(\sigma (X_t)\)-measurable.

3 Examples

In this section we provide examples of Markovian families of dynamic conditional risk mappings. Note that the entropic and worst case risk mappings are time consistent (see Detlefsen and Scandolo 2005, Prop. 6 and Barron et al. 2003, Th. 2.8(b)(ii) respectively), while the mean semi-deviation risk mapping and average value at risk are not (Föllmer and Schied 2016, Ex. 11.13, Artzner et al. 2007, pp. 20–21). Below we take \(Z \in b{\mathcal {F}}, t \in {\mathbb {N}}_0, x \in E\).

3.1 Composite risk mappings

Let \(K \in {\mathbb {N}}_0\) and for \(k = 0,\ldots ,K\) let \(g_{k} :{\mathbb {R}}^{m_{k}} \times E \rightarrow {\mathbb {R}}\) be measurable functions bounded on compact sets such that \(m_{0} = 1\) and \(m_k = 2\) for \(k\ge 1\), with the map \(x \mapsto g_{k}(r_{k},x)\) bounded on E for every \(r_{k} \in {\mathbb {R}}^{m_{k}}\). Assume also that for each \(x \in E\), the sequence \((\rho _t^x)_{t \in {\mathbb {N}}_0}\) is a dynamic conditional risk mapping, where \(\rho _0^{x}(Z) = R_{K}^{x}(Z)\) and \(\rho ^{x}_{t}(Z) = R_{K}^{x}(Z \vert {\mathcal {F}}_{t})\) for \(t \ge 1\), with

$$\begin{aligned} R_{k}^{x}(Z)= & {} {\left\{ \begin{array}{ll} {\mathbb {E}}^{x}\big [g_{0}(Z,X_{0})\big ], &{} \text {if}\;\;k =0,\\ {\mathbb {E}}^{x}\Big [g_{k}\big (Z,R_{k-1}^{X_{0}}(Z),X_{0}\big )\Big ], &{}\text {if}\;\; 1 \le k \le K, \end{array}\right. } \end{aligned}$$
(16)
$$\begin{aligned} R_{k}^{x}(Z \vert {\mathcal {F}}_{t})= & {} {\left\{ \begin{array}{ll} {\mathbb {E}}^{x}\big [g_{0}(Z,X_{t}) \big \vert {\mathcal {F}}_{t}\big ], &{} \text {if}\;\;k =0,\\ {\mathbb {E}}^{x}\Big [g_{k}\big (Z,R_{k-1}^{x}(Z \vert {\mathcal {F}}_{t}),X_{t}\big ) \big \vert {\mathcal {F}}_{t}\Big ], &{}\text {if}\;\; k \ge 1. \end{array}\right. } \end{aligned}$$
(17)

This family clearly includes the linear expectation (\(K = 0\), \(g_{0}(z,x) = z\)) and its statistical estimation properties are studied in Dentcheva et al. (2017).

Lemma 3.1

The family of dynamic conditional risk mappings \(\varrho =((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) defined through (16) and (17) is Markovian.

Proof

The Markov property holds at \(k = 0\) since \({\mathbb {P}}^x\)-a.s.

$$\begin{aligned} R_{0}^{x}(Z \circ \theta _{t} \vert {\mathcal {F}}_{t})&= {\mathbb {E}}^{x}\big [g_{0}(Z \circ \theta _{t},X_{t}) \big \vert {\mathcal {F}}_{t}\big ] \\&= {\mathbb {E}}^{x}\big [g_{0}(Z,X_{0}) \circ \theta _{t} \big \vert {\mathcal {F}}_{t}\big ] \\&= {\mathbb {E}}^{X_{t}}\big [g_{0}(Z,X_{0})\big ] = R_{0}^{X_{t}}(Z). \end{aligned}$$

Assuming that it holds at \(k-1\), the Markov property also holds at k:

$$\begin{aligned} R_{k}^{x}(Z \circ \theta _{t} \vert {\mathcal {F}}_{t})&= {\mathbb {E}}^{x}\Big [g_{k}\big (Z \circ \theta _{t},R_{k-1}^{x}(Z \circ \theta _{t} \vert {\mathcal {F}}_{t}),X_{t}\big ) \big \vert {\mathcal {F}}_{t}\Big ] \\&= {\mathbb {E}}^{x}\Big [g_{k}\big (Z,R_{k-1}^{X_{0}}(Z),X_{0}\big ) \circ \theta _{t} \big \vert {\mathcal {F}}_{t}\Big ] \\&= {\mathbb {E}}^{X_{t}}\Big [g_{k}\big (Z,R_{k-1}^{X_{0}}(Z),X_{0}\big ) \Big ] = R_{k}^{X_{t}}(Z) \qquad {\mathbb {P}}^x\text {-a.s.} \end{aligned}$$

\(\square \)

3.1.1 Entropic risk mapping

The entropic risk mapping (a special case of a certainty equivalent risk mapping, see Föllmer and Schied 2016, Def. 2.36. or Bäuerle and Rieder 2017) is Markovian since it is recovered from (17) by taking \(K = 1\), \(g_{1}(z,r,x) = \frac{1}{\gamma (x)}\ln (r)\) (restricting the domain of \(r \mapsto g_{1}(z,r,x)\) to \((0,\infty )\)) and \(g_{0}(z,x) = e^{\gamma (x)z}\) in (17), where \(\gamma :E \rightarrow (0,\infty )\) is measurable and bounded away from both 0 and \(\infty \), giving

$$\begin{aligned} \rho ^{x}_{t}(Z) = {\left\{ \begin{array}{ll} \frac{1}{\gamma (x)}\ln \left( {\mathbb {E}}^{x}\left[ e^{\gamma (x) Z} \right] \right) , &{} t=0, \\ \frac{1}{\gamma (X_{t})}\ln \left( {\mathbb {E}}^{x}\left[ e^{\gamma (X_{t}) Z} \big \vert {\mathcal {F}}_{t}\right] \right) , &{} t \ge 1. \end{array}\right. } \end{aligned}$$

3.1.2 Mean-semideviation risk mapping

Similarly, the mean–semideviation risk mapping satisfies the Markov property since it is recovered from (17) by taking \(K = 2\), \(g_{2}(z,r,x) = z + \kappa (x) \, r^{\frac{1}{p}}\), \(g_{1}(z,r,x) = ((z - r)^{+})^{p}\) and \(g_{0}(z,x) = z\) in (17), where \(\kappa :E \rightarrow [0,1]\) is measurable and \(p \ge 1\) is an integer, giving

$$\begin{aligned} \rho ^{x}_{t}(Z) = {\left\{ \begin{array}{ll} {\mathbb {E}}^{x}[Z] + \kappa (x)\left( {\mathbb {E}}^{x}\left[ \big (\left( Z - {\mathbb {E}}^{x}[Z]\right) ^{+}\big )^{p} \right] \right) ^{\frac{1}{p}}, &{} t=0, \\ {\mathbb {E}}^{x}[Z \vert {\mathcal {F}}_{t}] + \kappa (X_{t})\left( {\mathbb {E}}^{x}\left[ \big (\left( Z - {\mathbb {E}}^{x}[Z \vert {\mathcal {F}}_{t}]\right) ^{+}\big )^{p} \big \vert {\mathcal {F}}_{t}\right] \right) ^{\frac{1}{p}}, &{} t \ge 1. \end{array}\right. } \end{aligned}$$

3.2 Worst-case risk mapping

The worst-case risk mapping is given by the family

$$\begin{aligned} \rho ^{x}_{t}(Z) = {\left\{ \begin{array}{ll} {\mathbb {P}}^{x}-{{\,\mathrm{ess\,sup}\,}}(Z), &{} t = 0, \\ {\mathbb {P}}^{x}-{{\,\mathrm{ess\,sup}\,}}\left( Z\,\vert \,{\mathcal {F}}_{t}\right) , &{} t \ge 1. \end{array}\right. } \end{aligned}$$
(18)

For \(t \ge 1\) this is the \({\mathcal {F}}_{t}\)-conditional \({\mathbb {P}}^{x}\)-essential supremum of Z, that is, the smallest \({\mathcal {F}}_{t}\)-measurable random variable dominating Z almost surely with respect to \({\mathbb {P}}^{x}\)Artzner et al. (2007), pp. 20-21.

Lemma 3.2

The family of dynamic conditional risk mappings given by (18) is Markovian.

Proof

Supposing first that Z is non-negative, then using Barron et al. (2003), Prop. 2.12 and the Markov property of the conditional expectation, we have \({\mathbb {P}}^{x}\)-a.s.:

$$\begin{aligned} \rho ^{x}_{t}(Z \circ \theta _{t})&= \lim _{p \rightarrow \infty }\left( {\mathbb {E}}^{x}\big [(Z \circ \theta _{t})^{p}\,\vert \, {\mathcal {F}}_{t}\big ]\right) ^{\frac{1}{p}} \nonumber \\&= \lim _{p \rightarrow \infty }\left( {\mathbb {E}}^{x}\big [Z^{p} \circ \theta _{t} \,\vert \, {\mathcal {F}}_{t}\big ]\right) ^{\frac{1}{p}} \nonumber \\&= \lim _{p \rightarrow \infty }\left( {\mathbb {E}}^{X_{t}}[Z^{p}]\right) ^{\frac{1}{p}} = \rho ^{X_{t}}(Z), \end{aligned}$$

while the case \(t=0\) establishes measurability in x. For general \(Z \in b{\mathcal {F}}\) we first set \(Z_{c} :=Z + c\) with \(c = \sup _{\omega }\vert Z(\omega ) \vert \), then use translation invariance with respect to constants (see Barron et al. 2003, Prop. 2.1),

$$\begin{aligned} \rho ^{x}_{t}(Z \circ \theta _{t}) = \rho ^{x}_{t}(Z_{c} \circ \theta _{t}) - c = \rho ^{X_{t}}(Z_{c}) - c = \rho ^{X_{t}}(Z), \end{aligned}$$

completing the proof. \(\square \)

3.3 Value at risk

The value at risk may be defined by the family

$$\begin{aligned} \rho ^{x}_{t}(Z) = {\left\{ \begin{array}{ll} {\text {VaR}}^{x}_{\lambda }(-Z), &{} t=0,\\ {\text {VaR}}^{x}_{\lambda }(-Z \vert {\mathcal {F}}_{t}), &{} t \ge 1, \end{array}\right. } \end{aligned}$$
(19)

where \(\lambda \in (0,1)\),

$$\begin{aligned} {\text {VaR}}^{x}_{\lambda }(-Z) := \inf \{m \in {\mathbb {R}} :{\mathbb {P}}^{x}(m < Z) \le \lambda \} \end{aligned}$$

and

$$\begin{aligned} {\text {VaR}}^{x}_{\lambda }(-Z \vert {\mathcal {F}}_{t}) := {\mathbb {P}}^x- {{\,\mathrm{ess\,inf}\,}}\{m_{t} \in b{\mathcal {F}}_{t} :{\mathbb {P}}^{x}(m_{t} < Z \vert {\mathcal {F}}_{t}) \le \lambda \} \end{aligned}$$

for \(t \ge 1\), see e.g. Föllmer and Schied (2016), Sec. 4.4 & Ex. 11.4.

Lemma 3.3

The family of dynamic conditional risk mappings given by (19) is Markovian.

Proof

We first show that \(x \mapsto \rho ^{x}(Z)\) is measurable. For \(y \in {\mathbb {R}}\) we have

$$\begin{aligned} \{ x \in E : \rho ^{x}(Z)< y \}&= \{ x \in E : \inf \{m \in {\mathbb {R}} :{\mathbb {P}}^{x}(m< Z) \le \lambda \}< y \} \\&= \{ x \in E : \exists m<y : {\mathbb {P}}^x(m< Z) \le \lambda \} \\&= \{ x \in E : \exists m \in {\mathbb {Q}}, m<y :{\mathbb {P}}^x(m< Z) \le \lambda \} \\&= \bigcup _{m\in (-\infty ,y) \cap {\mathbb {Q}}} \{ x \in E : {\mathbb {P}}^x(m < Z) \le \lambda \} \\&= \bigcup _{m\in (-\infty ,y) \cap {\mathbb {Q}}} f_m^{-1}((-\infty ,\lambda ]), \end{aligned}$$

where for each \(m\in {\mathbb {R}}\) the function \(f_m :E \rightarrow {\mathbb {R}}\) is defined by \(f_m(x) = {\mathbb {P}}^x(m < Z)\). Note that each \(f_m\) is measurable because for every \(A \in {\mathcal {F}}\) the mapping \(E \ni x \mapsto {\mathbb {P}}^x(A)\in {\mathbb {R}}\) is measurable by the measurability of \(E \ni x \mapsto {\mathbb {P}}^x\in {\mathscr {P}}({\mathcal {F}})\). Thus \(\{ x \in E :\rho ^{x}(Z) < y \}\) is a measurable set.

To show that \(x \mapsto \rho ^x(Z)\) is bounded note that, since Z is bounded, there exists \(M\in {\mathbb {R}}\) such that \(\{\vert Z\vert <M\}=\Omega \). Thus \(-M \le \rho ^x(Z) \le M\) for all \(x \in E\).

Next we show that \(\rho ^{x}_{t}(Z \circ \theta _{t}) = \rho ^{X_{t}}(Z)\) almost surely. Since Z is bounded, let \(m_{t}(X_{0},\ldots ,X_{t}) \in b{\mathcal {F}}_{t}\) satisfy \({\mathbb {P}}^x\)-a.s.

$$\begin{aligned} {\mathbb {P}}^{x}(m_{t}(X_{0},\ldots ,X_{t}) < Z \circ \theta _{t} \vert {\mathcal {F}}_{t}) \le \lambda . \end{aligned}$$

Then for \({\mathbb {P}}^x\)-almost all \(\omega \in \Omega \), by conditional locality and the Markov property we have

$$\begin{aligned} \lambda&\ge {\mathbb {P}}^{x}(m_{t}(X_{0},\ldots ,X_{t})< Z \circ \theta _{t} \vert {\mathcal {F}}_{t})(\omega ) \\&= {\mathbb {P}}^{x}(m_{t}(X_{0}(\omega ),\ldots ,X_{t}(\omega ))< Z \circ \theta _{t} \vert {\mathcal {F}}_{t})(\omega ) \\&= {\mathbb {P}}^{X_t(\omega )}(m_{t}(X_{0}(\omega ),\ldots ,X_{t}(\omega )) < Z ), \end{aligned}$$

giving \(m_{t}(\omega ) \ge \rho ^{X_{t}(\omega )}(Z)\). We conclude that \(\rho ^{x}_{t}(Z \circ \theta _{t}) \ge \rho ^{X_{t}}(Z)\) almost surely under \({\mathbb {P}}^{x}\).

Conversely we have by the Markov property that \({\mathbb {P}}^{x}\text {-a.s.}\),

$$\begin{aligned} {\mathbb {P}}^{x}(\rho ^{X_{t}}(Z)< Z \circ \theta _{t} \vert {\mathcal {F}}_{t})(\omega )&= {\mathbb {P}}^{X_{t}(\omega )}(\rho ^{X_{0}}(Z)< Z) \\&= {\mathbb {P}}^{X_{t}(\omega )}(\rho ^{X_{t}(\omega )}(Z) < Z) \le \lambda , \end{aligned}$$

and, since \(\omega \mapsto \rho ^{X_{t}(\omega )}(Z)\) is bounded and \({\mathcal {F}}_{t}\)-measurable, we conclude that \(\rho ^{x}_{t}(Z \circ \theta _{t}) \le \rho ^{X_{t}}(Z)\). \(\square \)

3.4 Average value at risk

For \(\lambda \in (0,1)\) the average value at risk (see Acciaio and Penner 2011, Ex. 1.10) may be defined by the following family of dynamic conditional risk mappings:

$$\begin{aligned} \rho ^{x}_{t}(Z) = {\left\{ \begin{array}{ll} \text {AVaR}_{\lambda }^x(-Z), &{} t = 0, \\ \text {AVaR}_{\lambda ,t}^x(-Z), &{} t \ge 1, \end{array}\right. } \end{aligned}$$
(20)

where

$$\begin{aligned} \text {AVaR}_{\lambda }^x(-Z) = {\mathbb {E}}^{x}\left[ {\text {VaR}}^{x}_{\lambda }(-Z) + \frac{1}{\lambda }(Z - {\text {VaR}}^{x}_{\lambda }(-Z))^{+}\right] \end{aligned}$$

and

$$\begin{aligned} \text {AVaR}_{\lambda ,t}^x(-Z) ={\mathbb {E}}^{x}\left[ {\text {VaR}}^{x}_{\lambda }(-Z \vert {\mathcal {F}}_{t}) + \frac{1}{\lambda }(Z - {\text {VaR}}^{x}_{\lambda }(-Z \vert {\mathcal {F}}_{t}))^{+} \Big \vert {\mathcal {F}}_{t}\right] \end{aligned}$$

for \(t \ge 1\).

Lemma 3.4

The family of dynamic conditional risk mappings given by (20) is Markovian.

Proof

This follows from the Markov property for \({\text {VaR}}^{x}_{\lambda }(\cdot \vert {\mathcal {F}}_{t})\) (Lemma 3.3), since \({\mathbb {P}}^x\)-a.s.

$$\begin{aligned} \text {AVaR}_{\lambda ,t}^x(-Z \circ \theta _t)&= {\text {VaR}}_\lambda ^{X_t}(-Z) + {\mathbb {E}}^x \left[ \frac{1}{\lambda } (Z \circ \theta _t - {\text {VaR}}_\lambda ^{X_t}(-Z))^+ \big \vert {\mathcal {F}}_t \right] \\&= {\text {VaR}}_\lambda ^{X_t}(-Z) + {\mathbb {E}}^{X_t} \left[ \frac{1}{\lambda } (Z - {\text {VaR}}_\lambda ^{X_0}(-Z))^+ \right] \\&= {\text {AVaR}}_\lambda ^{X_t}(-Z). \end{aligned}$$

\(\square \)

4 Dual representation of convex Markovian risk mappings

In this section we characterise the dual representation of convex Markovian risk mappings. Recalling from Sect. 2.1 that \((q^X(B\vert x) :B \in {\mathcal {E}}, x \in E)\) is the kernel associated to the Markov process X under \({\mathbb {P}}\), we begin with the necessary definitions:

Definition 4.1

  1. (i)

    \({\mathcal {R}} :E \times b{\mathcal {E}} \rightarrow {\mathbb {R}}\) is a transition risk mapping (cf. Çavuş and Ruszczyński 2014; Fan and Ruszczyński 2018a; Ruszczyński 2010) if:

    • for all \(f \in b{\mathcal {E}}\), \(x \mapsto {\mathcal {R}}(x,f)\) is bounded and measurable,

    • for all \(x \in E\), \(f \mapsto {\mathcal {R}}(x,f)\) satisfies

      • normalisation: \({\mathcal {R}}(x,0) = 0\),

      • monotonicity: \({\mathcal {R}}(x,f) \le {\mathcal {R}}(x,g)\) for all \(f \le g\),

      • constant translation invariance: \({\mathcal {R}}(x,f+c) = {\mathcal {R}}(x,f) + c\) for all constants c.

  2. (ii)

    A transition risk mapping is convex if for all \(x \in E\), \(f,g \in b{\mathcal {E}}\) and \(\lambda \in [0,1]\) we have

    $$\begin{aligned} {\mathcal {R}}(x,\lambda f + (1-\lambda ) g) \le \lambda {\mathcal {R}}(x,f) + (1-\lambda ) {\mathcal {R}}(x,g). \end{aligned}$$

Note that by Definitions 2.3 and 2.5, a transition risk mapping can be derived from a regular collection of risk mappings \((\rho ^x)_{x \in E}\) by writing

$$\begin{aligned} {\mathcal {R}}(x,f) := \rho ^{x}(f(X_{1})) \qquad \text {for } f \in b{\mathcal {E}}. \end{aligned}$$
(21)

If the transition risk mapping \({\mathcal {R}}\) defined by (21) is convex and continuous from below it has the following dual representation (cf. Föllmer and Penner 2006, Th. 2.3):

$$\begin{aligned} {\mathcal {R}}(x,f) = \sup _{\begin{array}{c} Q \in {\mathscr {P}}({\mathcal {E}}), \\ Q \ll {\mathbb {P}}^x \circ X_1^{-1} \end{array}} \left( \int _E f(y) \, Q({\textrm{d}}y) - \alpha ^x(Q) \right) , \end{aligned}$$
(22)

where the penalty functions \(\alpha ^x : {\mathscr {P}}({\mathcal {E}}) \rightarrow {\mathbb {R}}\) are defined by

$$\begin{aligned} \alpha ^x(Q) = \sup _{g \in b{\mathcal {E}}} \left( {\mathbb {E}}_Q[g] - {\mathcal {R}}(x,g) \right) . \end{aligned}$$

Letting \({\mathcal {K}}\) denote the set of kernels q such that \(q(\cdot \vert x)\) is absolutely continuous with respect to \(q^X(\cdot \vert x)\) for every \(x \in E\), we have the following proposition:

Proposition 4.2

Let \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) be a family of dynamic conditional risk mappings such that \((\rho ^x)_{x \in E}\) is regular and each risk mapping \(\rho ^x\) is convex and continuous from below. Then \(\varrho \) satisfies the one-step Markov property (8) if and only if for all \(x \in E\) and \(f \in b{\mathcal {E}}\) we have

$$\begin{aligned} \rho ^x(f(X_{1}))&= \sup _{q \in {\mathcal {K}}} \left( \int _E f(y) \, q({\textrm{d}}y \vert x) - \alpha ^{x}(q(\cdot \vert x)) \right) , \end{aligned}$$
(23)
$$\begin{aligned} \rho _t^x(f(X_{t+1}))&= \sup _{q \in {\mathcal {K}}} \left( \int _E f(y) \, q({\textrm{d}}y \vert X_t) - \alpha ^{X_t}(q(\cdot \vert X_t)) \right) ,&t = 1,2,\ldots . \end{aligned}$$
(24)

\({\mathbb {P}}^x\)-almost surely.

Proof

Using kernels, for all \(x \in E\) and \(f \in b{\mathcal {E}}\) the representation (22) can be rewritten as (23). Indeed, it is clear that the right-hand side of (23) is less than or equal to the right-hand side of (22). For the reverse inequality, simply note that for every given \(x \in E\) and every \(Q \in {\mathscr {P}}({\mathcal {E}})\) such that \(Q \ll {\mathbb {P}}^x \circ X_1^{-1}\), we can associate a kernel \(q \in {\mathcal {K}}\) by setting \(q( \cdot \vert x') = Q\mathbbm {1}_{\{x\}}(x') + ({\mathbb {P}}^{x'} \circ X_1^{-1})(1-\mathbbm {1}_{\{x\}}(x'))\). Then Equation (23) implies

$$\begin{aligned} \sup _{q \in {\mathcal {K}}} \left( \int _E f(y) \, q({\textrm{d}}y \vert X_t) - \alpha ^{X_t}(q(\cdot \vert X_t)) \right) = \rho ^{X_t}(f(X_1)), \end{aligned}$$

which shows that (24) is equivalent to the one step Markov property (8). \(\square \)

Note that in (23) we take the supremum (rather than essential supremum) over a potentially uncountable family of kernels. Therefore the regularity of the collection \((\rho ^x)_{x \in E}\) follows from the assumptions of Proposition 4.2 rather than from (23).

Corollary 4.3

Let \(\varrho :=((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) be a time-consistent family of dynamic conditional risk mappings such that each risk mapping \(\rho ^x\) is convex and continuous from above. Then \(\varrho \) satisfies (23)–(24) (for all \(x \in E\) and \(f \in b{\mathcal {E}}\)) iff the Markov property of Definition 2.6 holds.

Proof

Combining Lemma 4.21 and Theorem 4.22 in Föllmer and Schied (2016), we see that a convex conditional risk mapping that is continuous from above is also continuous from below (recall the sign difference in our work). The corollary is then an application of Proposition 2.11 to Proposition 4.2. \(\square \)

The following example identifies a maximising kernel in (24) in the case of the entropic risk mapping of Sect. 3.1.1.

Example 4.4

(Entropic risk) For \(q \in {\mathcal {K}}\) let

$$\begin{aligned} \alpha ^x(q(\cdot \vert x)) = \frac{1}{\gamma (x)} \int _E \ln \frac{{\textrm{d}}q(\cdot \vert x)}{{\textrm{d}}q^X(\cdot \vert x)}(y) \, q({\textrm{d}}y \vert x). \end{aligned}$$
(25)

Fixing a bounded, measurable function \(f:E \rightarrow {\mathbb {R}}\), define the kernel \(q_{op}\) by

$$\begin{aligned} \frac{{\textrm{d}}q_{op}(\cdot \vert x)}{{\textrm{d}}q^X(\cdot \vert x)}(y) = \frac{e^{\gamma (x) f(y)}}{\int _E e^{\gamma (x) f(z)} \, q^X({\textrm{d}}z\vert x) }. \end{aligned}$$
(26)

It is well known (Detlefsen and Scandolo 2005, Rem. 9) from the non-Markovian setting that for each \(x \in E\) the function (25) is the minimal penalty corresponding to the entropic risk mapping on \(b{\mathcal {E}}\) and that (26) defines a measure attaining the maximum in (24). In order to show that \(q_{op}\) defined in this way is indeed a kernel we show that for each \(A \in {\mathcal {E}}\) the function \(x \mapsto q_{op}(A\vert x)\) is measurable. Indeed we have \(q_{op}(A\vert x) = \int _E \mathbbm {1}_A(y) \frac{e^{\gamma (x)f(y)}}{\int _E e^{\gamma (x)f(z)} \, q^X({\textrm{d}}z\vert x)} \, q^X({\textrm{d}}y\vert x)\) and measurability follows since more generally, for any jointly measurable function \(g:E \times E \rightarrow {\mathbb {R}}\) and any kernel p, the function \(x \mapsto \int _E g(x,y) \, p({\textrm{d}}y \vert x)\) is measurable.

5 Applications

The probabilistic Markov property can provide a convenient tool to address, for example, optimal stopping problems with costs which are measurable only after the chosen stopping time. A first example is the case of exercise lag, where we seek

$$\begin{aligned} L^T(x):=\inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^x \left( \sum _{i=0}^{\tau -1} c(X_i) + g(X_{\sigma \circ \theta _\tau + \tau }) \right) , \end{aligned}$$

where functions \(c, g:E \mapsto {\mathbb {R}}\) and a potentially unbounded stopping time \(\sigma \in {\mathscr {T}}\) represent respectively an observation cost, exercise cost and exercise lag, and \(\varrho =((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) is a time-consistent Markovian family of dynamic risk mappings. The strong Markov property of Proposition 2.9 then allows dynamic programming to be applied indirectly by first transforming the objective function. Indeed it then follows by the recursive property (6), conditional locality, conditional translation invariance, the identity \(X_{\sigma \circ \theta _\tau + \tau } = X_\sigma \circ \theta _\tau \) and the strong Markov property that

$$\begin{aligned} L^T(x)&=\inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^x \left( \rho _\tau ^x \left( \sum _{i=0}^{\tau -1} c(X_i) + g(X_{\sigma \circ \theta _\tau + \tau }) \right) \right) \nonumber \\&= \inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^x \left( \sum _{t=0}^T \rho _t^x \left( \mathbbm {1}_{\{\tau =t \}} \sum _{i=0}^{t-1} c(X_i) + \mathbbm {1}_{\{\tau =t \}} g(X_{\sigma \circ \theta _\tau + \tau }) \right) \right) \nonumber \\&= \inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^x \left( \sum _{i=0}^{\tau -1} c(X_i) + h(X_\tau ) \right) , \end{aligned}$$

where \(h(x) :=\rho ^x(g(X_\sigma ))\), and standard dynamic programming arguments can then be applied to obtain the Wald-Bellman equations

$$\begin{aligned} {\left\{ \begin{array}{ll} L^{0}(x) = h(x),&{} \\ L^{m}(x) = h(x) \wedge \left( c(x) + \rho ^{x}\big ( L^{m-1}(X_{1})\big )\right) ,&{} m = 1,\ldots ,T. \end{array}\right. } \end{aligned}$$

In the optimal prediction problem of the next section, use of the probabilistic Markov property enables dynamic programming to instead be applied directly.

5.1 Optimal prediction

Generalising (3), let

$$\begin{aligned} V_\text {pred}^T(x) := \inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^x(g(X_T^* - X_\tau )), \end{aligned}$$

where \(x \in E = {\mathbb {R}}\), \(\Omega = {\mathbb {R}}^{{\mathbb {N}}_0}\), \(T \in {\mathbb {N}}_0\), \(\varrho =((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) is a Markovian family of dynamic conditional risk mappings, \(X_T^* = \max _{0 \le s \le T} X_s\) and \(g:{\mathbb {R}} \rightarrow {\mathbb {R}}\) is bounded and measurable.

We extend this probability space to include the process’ running maximum by letting \({\tilde{\Omega }} = ({\mathbb {R}} \times {\mathbb {R}})^{{\mathbb {N}}_0}\). On this space, we have the canonical process \((X_t({\tilde{\omega }}),M_{t}({\tilde{\omega }})) = ({\tilde{\omega }}^{1}(t),{\tilde{\omega }}^{2}(t)) = {\tilde{\omega }}(t)\). Setting \(\tilde{{\mathbb {F}}} = (\tilde{{\mathcal {F}}}_{t})_{t \in {\mathbb {N}}_0}\) with \(\tilde{{\mathcal {F}}}_{t} = \sigma (\{(X_{s},M_{s}) :s \le t\})\) and \(\tilde{{\mathcal {F}}} = \sigma \left( \cup _{t} \tilde{{\mathcal {F}}}_{t}\right) \), there exists (see, for example, Çinlar 2011, Th. 4.4.18) a unique probability measure \(\tilde{{\mathbb {P}}}^{x,m}\) on \(({\tilde{\Omega }},\tilde{{\mathcal {F}}})\) such that (X, M) is a time-homogeneous Markov chain on \(({\tilde{\Omega }},\tilde{{\mathcal {F}}},\tilde{{\mathbb {F}}},\tilde{{\mathbb {P}}}^{x,m})\) with \(\tilde{{\mathbb {P}}}^{x,m}(X_{0} = x, M_{0} = m) = 1\) and transition kernel \(q^{X,M}\) satisfying \(q^{X,M}({\textrm{d}}x',{\textrm{d}}m' \vert x, m) = \delta _{m\vee x'}({\textrm{d}}m') \, q^{X}({\textrm{d}}x' \vert x)\) for all \((x, m) \in {\mathbb {R}}^2\). Note that \(\tilde{{\mathbb {P}}}^{x,m}(M_{n} = X_n^*\vee m) = 1\) and, in particular, for \(m=x\) we have \(\tilde{{\mathbb {P}}}^{x,x}(M_{n} = X_n^*) = 1\). Recalling Proposition 2.8, define a regular collection of risk mappings by

$$\begin{aligned} \rho ^{x,m}(f(X_0, M_0, X_1, M_1, \ldots )) := \rho ^x(f(X_0, m, X_1, X_1^* \vee m, \ldots )), \end{aligned}$$

and let \(((\rho _t^{x,m})_{t \in {\mathbb {N}}_0})_{x,m \in {\mathbb {R}}}\) be the associated Markovian family of dynamic conditional risk mappings.

Theorem 5.1

If \(((\rho _t^x)_{t \in {\mathbb {N}}_0})_{x \in E}\) is time consistent then, for each bounded measurable function \(g:{\mathbb {R}} \rightarrow {\mathbb {R}}\), the extended value function

$$\begin{aligned} {\tilde{V}}^T(x,m) := \inf _{\tau \in {\mathscr {T}}_{[0,T]}} \rho ^{x,m}(g(M_T-X_\tau )) \end{aligned}$$

satisfies the following modified Wald-Bellman equations:

$$\begin{aligned} {\tilde{V}}^0(x,m)&= g(m-x), \\ {\tilde{V}}^n(x,m)&= \rho ^{x,m}(g(M_n-x)) \wedge \rho ^{x,m}({\tilde{V}}^{n-1}(X_1,M_1)). \end{aligned}$$

The optimal prediction problem (3) satisfies \(V_{\text {pred}}^n(x) = {\tilde{V}}^n(x,x)\).

Proof

Set

$$\begin{aligned} S_T^T&= g(M_T-X_T), \\ S_n^T&= \rho _n^{x,m}(g(M_T - X_n)) \wedge \rho _n^{x,m}(S_{n+1}^T). \end{aligned}$$

Following the outline of Peskir and Shiryaev (2006), Sec. 1.2, we may now proceed in five steps:

Step 1. We show that for all \(n = 0,1,\ldots ,T\) and \(k = T-n, T-n-1, \ldots , 0\), we have

$$\begin{aligned} S_k^{T-n} \circ {\tilde{\theta }}_n = S_{k+n}^T. \end{aligned}$$

One can easily see that the claim is true for \(k=T-n\). Further, by the Markov property and backward induction, for all n the random variable \(S_n^T\) is \(\sigma (M_n,X_n)\)-measurable (cf. Remark 2.16). All subsequent equalities hold \({\tilde{{\mathbb {P}}}}^{x,m}\)-almost surely. For any \(Z = {{\hat{Z}}} \circ {\tilde{\theta }}_k \in b\tilde{{\mathcal {F}}}_{k,\infty }\) we have

$$\begin{aligned} \rho ^{x,m}_{k}(Z) \circ {\tilde{\theta }}_{n}&= \rho ^{x,m}_{k}({{\hat{Z}}} \circ {\tilde{\theta }}_k) \circ {\tilde{\theta }}_{n} = \rho ^{X_k,M_k}({{\hat{Z}}}) \circ {\tilde{\theta }}_{n} = \rho ^{X_{k+n},M_{k+n}}({{\hat{Z}}}) \\&= \rho ^{x,m}_{k+n}({{\hat{Z}}} \circ {\tilde{\theta }}_{k+n}) = \rho ^{x,m}_{k+n}(Z \circ {\tilde{\theta }}_{n}). \end{aligned}$$

This and the induction hypothesis imply

$$\begin{aligned} S_k^{T-n} \circ {\tilde{\theta }}_n&= \rho _k^{x,m}(g(M_{T-n} - X_k)) \circ {\tilde{\theta }}_n \wedge \rho _k^{x,m}(S_{k+1}^{T-n}) \circ {\tilde{\theta }}_n \\&= \rho _{k+n}^{x,m}(g(M_{T} - X_{k+n})) \wedge \rho _{k+n}^{x,m}(S_{k+1}^{T-n} \circ {\tilde{\theta }}_n) \\&= \rho _{k+n}^{x,m}(g(M_{T} - X_{k+n})) \wedge \rho _{k+n}^{x,m}(S_{k+n+1}^T) \\&= S_{k+n}^T. \end{aligned}$$

Step 2. Let \(\tau _n^T := \inf \{k=n,\ldots , T : S_k^T = \rho _k^{x,m}(g(M_T-X_k)) \}\). We show that \(\tau _n^T = n + \tau _0^{T-n} \circ {\tilde{\theta }}_n\).

Indeed,

$$\begin{aligned} \tau _n^T&= \inf \{k=n,\ldots , T : S_{k-n}^{T-n} \circ {\tilde{\theta }}_n = \rho _{k-n}^{x,m}(g(M_{T-n}-X_{k-n})) \circ {\tilde{\theta }}_n \} \\&= n + \inf \{k=0,\ldots , T-n : S_{k}^{T-n} \circ {\tilde{\theta }}_n = \rho _{k}^{x,m}(g(M_{T-n}-X_{k})) \circ {\tilde{\theta }}_n \} \\&= n + \tau _0^{T-n} \circ {\tilde{\theta }}_n. \end{aligned}$$

Step 3. We show that for \(n=T, \ldots , 0\) we have

$$\begin{aligned} S_n^T = \rho _n^{x,m}(g(M_T-X_{\tau _n^T})) \end{aligned}$$
(27)

Note that on \(\{\tau _{n-1}^T \ge n\}\) we have \(\tau _{n-1}^T = \tau _n^T\) (by definition of these stopping times). From this and time consistency we have

$$\begin{aligned}{} & {} \rho _{n-1}^{x,m}(g(M_T-X_{\tau _{n-1}^T})) \\{} & {} \quad = \mathbbm {1}_{\{\tau _{n-1}^T=n-1\}} \rho _{n-1}^{x,m}(g(M_T-X_{n-1})) + \mathbbm {1}_{\{\tau _{n-1}^T \ge n\}} \rho _{n-1}^{x,m}(\rho _n^{x,m}(g(M_T-X_{\tau _n^T})). \end{aligned}$$

By the induction hypothesis

$$\begin{aligned} \begin{aligned} \rho _{n-1}^{x,m}(g(M_T-X_{\tau _{n-1}^T})) = {}&\mathbbm {1}_{\{\tau _{n-1}^T=n-1\}} \rho _{n-1}^{x,m}(g(M_T-X_{n-1})) \\&+ \mathbbm {1}_{\{\tau _{n-1}^T \ge n\}} \rho _{n-1}^{x,m}(S_n^T). \end{aligned} \end{aligned}$$
(28)

Note that

$$\begin{aligned} S_{n-1}^T&= \rho _{n-1}^{x,m}(g(M_T-X_{n-1})) \qquad \text { on } \qquad \{ \tau _{n-1}^T = n-1 \}, \\ S_{n-1}^T&= \rho _{n-1}^{x,m}(S_n^T) \qquad \text { on } \qquad \{ \tau _{n-1}^T \ge n \}. \end{aligned}$$

Thus, (28) implies that \(\rho _{n-1}^{x,m}(g(M_T-X_{\tau _{k-1}^T})) = S_{n-1}^T\).

Step 4. We prove that

$$\begin{aligned} S_n^T = {\tilde{V}}^{T-n}(X_n,M_n). \end{aligned}$$
(29)

We have

$$\begin{aligned} S_n^T= & {} \rho _n^{x,m}(g(M_T-X_{\tau _n^T})) = \rho _n^{x,m}(g(M_T-X_{n + \tau _0^{T-n} \circ {\tilde{\theta }}_n})) \nonumber \\= & {} \rho _n^{x,m}(g(M_{T-n}-X_{\tau _0^{T-n}}) \circ {\tilde{\theta }}_n) = \rho ^{X_n,M_n}(g(M_{T-n}-X_{\tau _0^{T-n}})). \end{aligned}$$
(30)

On the other hand, one can show by induction that for each \(k=T, \ldots , 0\) and every \(\tau \in {\mathscr {T}}_{[k,T]}\) we have \(\rho _k^{x,m}(g(M_T-X_\tau )) \ge S_k^T\). The claim is true for \(k=T\), and we may write

$$\begin{aligned} \rho ^{x,m}_{k-1}(g(M_T-X_\tau ))&= \mathbbm {1}_{\{\tau =k-1\}} \rho _{k-1}^{x,m}(g(M_T-X_\tau )) \\&\quad + \mathbbm {1}_{\{\tau \ge k\}} \rho _{k-1}^{x,m}(\rho _k^{x,m}(g(M_T-X_{\tau \vee k}))). \end{aligned}$$

By the induction hypothesis, since \(\tau \vee k \in {\mathscr {T}}_{[k,T]}\) we have

$$\begin{aligned} \rho ^{x,m}_{k-1}(g(M_T-X_\tau ))&\ge \mathbbm {1}_{\{\tau =k-1\}} \rho _{k-1}^{x,m}(g(M_T-X_k)) + \mathbbm {1}_{\{\tau \ge k\}} \rho _{k-1}^{x,m}(S_k^T) \\&\ge \mathbbm {1}_{\{\tau =k-1\}} S_{k-1}^T + \mathbbm {1}_{\{\tau \ge k\}} S_{k-1}^T \\&=S_{k-1}^T. \end{aligned}$$

In particular for \(k=0\) we conclude by Step 3 that for every \(T \in {\mathbb {N}}_0\), the stopping time \(\tau _0^T\) is optimal and \({\tilde{V}}^T(x,m)=\rho ^{x,m}(g(M_T-X_{\tau _0^T}))\). Combining this with (30) gives (29).

Step 5. We have by the previous step and the Markov property that

$$\begin{aligned} {\tilde{V}}^{T-n}(X_n,M_n)&= S_n^T \\&= \rho _n^{x,m}(g(M_T-X_n)) \wedge \rho _n^{x,m}(S_{n+1}^T) \\&= \rho _n^{x,m}(g(M_N-X_n)) \wedge \rho _n^{x,m}({\tilde{V}}^{T-n-1}(X_{n+1},M_{n+1})) \\&= \rho _n^{x,m}(g(M_T-X_n)) \wedge \rho ^{X_n,M_n}({\tilde{V}}^{T-n-1}(X_1,M_1)). \end{aligned}$$

Taking \(n=0\) we get \({\tilde{V}}^{T}(x,m) = \rho ^{x,m}(g(M_T-x)) \wedge \rho ^{x,m}({\tilde{V}}^{T-1}(X_1,M_1))\), and the result follows by construction. \(\square \)