Simulation optimization of risk measures with adaptive risk levels

Abstract

Optimizing risk measures such as Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR) of a general loss distribution is usually difficult, because (1) the loss function might lack structural properties such as convexity or differentiability since it is often generated via black-box simulation of a stochastic system; (2) evaluation of risk measures often requires rare-event simulation, which is computationally expensive. In this paper, we study the extension of the recently proposed gradient-based adaptive stochastic search to the optimization of risk measures VaR and CVaR. Instead of optimizing VaR or CVaR at the target risk level directly, we incorporate an adaptive updating scheme on the risk level, by initializing the algorithm at a small risk level and adaptively increasing it until the target risk level is achieved while the algorithm converges at the same time. This enables us to adaptively reduce the number of samples required to estimate the risk measure at each iteration, and thus improving the overall efficiency of the algorithm.

A Proof of theorems

Proof

Proof of Lemma 4.1. Since $S_{\theta _k}(\cdot )$ is continuous in both ${C}_{\alpha ^*}$ and $\gamma _{\theta _k}$, it suffices to show that for all $x\in \mathcal {X}$

$$\begin{aligned} \lim _{k\rightarrow \infty }\widehat{C}_{\alpha ^*}(x)\rightarrow {C}_{\alpha ^*}(x),\; w.p.1., \quad \text{ and }\quad \lim _{k\rightarrow \infty }|\widehat{\gamma }_{\theta _k}-\gamma _{\theta _k}| = 0,\; w.p.1. \end{aligned}$$

(A.1)

Let us first show the left part of the above statement. Recall that by Assumption 1.(iii), we have $M_k\rightarrow \infty$ as $k\rightarrow \infty$. Then, we only need to show that the one-layer CVaR estimator $\widehat{C}_{\alpha ^*}(x)$ is strongly consistent. By Lemma A.1 in [22] under Assumption 2.(i) this holds, where note that Assumption 3.1 in [22] is satisfied by Assumption 2 here.

It remains to establish the right part of (A.1). In view of Assumptions 1.(ii) and 1.(iii), we have $N_k, M_k\rightarrow \infty$ as $k\rightarrow \infty$. That is, $N_k, M_k$ go to infinity simultaneously as $k\rightarrow \infty$. Therefore, it suffices to show

$$\begin{aligned} \lim _{N_k, M_k\rightarrow \infty }|\widehat{\gamma }_{\theta _k}-\gamma _{\theta _k}| = 0,\; w.p.1. \end{aligned}$$

(A.2)

Note that

$$\begin{aligned} \gamma _{\theta _k}=V_{1-\rho }(-C_{\alpha ^*}(x)), \end{aligned}$$

i.e., the $(1-\rho )$-level Value-at-Risk (VaR) of $(-C_{\alpha ^*}(x))$ w.r.t. $f(x;\theta _k)$. Furthermore,

$$\begin{aligned} \widehat{\gamma }_{\theta _k}=\widehat{V}_{1-\rho } (-\widehat{C}_{\alpha ^*}(x)), \end{aligned}$$

i.e., the sample $(1-\rho )$-quantile of $\{-\widehat{C}_{\alpha ^*}(x_k^i): i=1,\ldots ,N_k\}$. Therefore, $\widehat{\gamma }_{\theta _k}$ is a nested estimator of $\gamma _{\theta _k}$, where $N_k$ outer-layer samples are drawn, and for each outer-layer sample $M_k$ inner-layer samples are drawn.

Rewrite $\widehat{C}_{\alpha ^*}(x)$ as

$$\begin{aligned} \widehat{C}_{\alpha ^*}(x) =C_{\alpha ^*}(x)+\frac{1}{\sqrt{M_k}}\cdot \mathcal {E}_k(x),\quad \forall x\in \mathcal {X}, \end{aligned}$$

(A.3)

where $\mathcal {E}_k(x)$ is the standardized error. By Theorem 3.3 in [22], we have

$$\begin{aligned} \lim _{M_k\rightarrow \infty } \sqrt{M_k}\left( \widehat{C}_{\alpha ^*}(x) -c\right) \overset{\mathcal {D}}{\Rightarrow }\mathcal {N}\left( 0, \sigma ^2(x) \right) , \end{aligned}$$

where “$\overset{\mathcal {D}}{\Rightarrow }$” denotes the convergence in distribution, and $\mathcal {N}\left( 0, \sigma ^2(x)\right)$ denotes a normal distribution with mean zero and variance $\sigma ^2(x)$, where $\sigma ^2(x)$ is the variance parameter that only depends on x. Combined with (A.3), we can see that the standardized error $\mathcal {E}_k(x)$ converges to $\mathcal {N}\left( 0, \sigma ^2(x)\right)$ in distribution. Having established this, the remaining proof is identical to the proof of Theorem 3.2 in [22], where note that Assumption 2 here is parallel with Assumption 3.2 in [22]. For simplicity, here we list the main steps as follows.

• 1. Show that the p.d.f. of $\widehat{C}_{\alpha ^*}(x)$ and its first-order derivative, which are induced jointly by the sampling distribution $f(\cdot , \theta _k)$ and the noise $\xi _x$, converge to the p.d.f. of $C_{\alpha ^*}(x)$ and its first order derivative, respectively, as $k\rightarrow \infty$. Here note that the p.d.f. of $C_{\alpha ^*}(x)$ is induced by $f(\cdot , \theta _k)$. [Lemma B.1 in [22]].

• 2. Show that the VaR of $\widehat{C}_{\alpha ^*}(x)$ at risk level $\rho$, denoted by $VaR_{\rho }(\widehat{C}_{\alpha ^*}(x)$, converges to VaR of ${C}_{\alpha ^*}(x)$ at risk level $\rho$ in the order of $O(\frac{1}{M_k})$. This is done by Taylor expansion analysis on the p.d.f. of $\widehat{C}_{\alpha ^*}(x)$ and its derivative. [Lemma B.3 in [22]].

• 3. Show that the difference between nested risk estimator $\widehat{VaR}_{\rho }(\widehat{C}_{\alpha ^*}(x)$, i.e., $\widehat{\gamma }_{\theta _k}$, and $VaR_{\rho }(\widehat{C}_{\alpha ^*}(x)$ is in the order of $O(\frac{1}{N_k} \log N_k)$ uniformly for all $M_k$. [Lemma B.4 in [22]].

Proof

Proof of Lemma 4.2. With a slight abuse of notation, we also use $\left||A\right||_2$ to denote the spectral norm of a real square matrix A induced by the vector Euclidean norm. In particular, $\left||A\right||_2=\sqrt{\lambda _{max}(A^T A)}$, i.e., $\left||A\right||_2$ is the largest eigenvalue of the positive-semidefinite matrix $A^TA$. When the matrix A is positive-semidefinite, $\left||A\right||_2$ is just the largest eigenvalue of A.

To facilitate the proof, let us also introduce the following notations:

$$\begin{aligned} {\begin{matrix} &{}\widetilde{\mathbb {Y}}_k\overset{\triangle }{=}\frac{1}{N_k} \sum _{i=1}^{N_k}S_{\theta _k}\left( -C_{\alpha ^*}(x_k^i)\right) \Gamma (x_k^i),\;\; \widetilde{\mathbb {Z}}_k\overset{\triangle }{=}\frac{1}{N_k} \sum _{i=1}^{N_k}S_{\theta _k}\left( -C_{\alpha ^*}(x_k^i)\right) , \\ &{}\widehat{\mathbb {Y}}_k\overset{\triangle }{=}\frac{1}{N_k} \sum _{i=1}^{N_k}\widehat{S}_{\theta _k}\left( -\widehat{C}_{\alpha ^*}(x_k^i)\right) \Gamma (x_k^i),\;\; \widehat{\mathbb {Z}}_k\overset{\triangle }{=}\frac{1}{N_k} \sum _{i=1}^{N_k}\widehat{S}_{\theta _k}\left( -\widehat{C}_{\alpha ^*}(x_k^i)\right) . \end{matrix}} \end{aligned}$$

Here note that $\widetilde{\mathbb {Y}}_k$, $\widehat{\mathbb {Y}}_k$ are vectors because $\Gamma (\cdot )$ are vector-valued functions, and $\widetilde{\mathbb {Z}}_k$, $\widehat{\mathbb {Z}}_k$ are scalar-valued.

Since $C_{\alpha ^*}(x)$ and $\Gamma (x)$ are both bounded on $\mathcal {X}$, we immediately have $|\widetilde{\mathbb {Z}}_k|$ bounded below from zero and $\frac{\left||\widehat{\mathbb {Y}}_k\right||_2}{\left| \widehat{\mathbb {Z}}_k\right| }$ bounded for all k. Note that

$$\begin{aligned} b_k= & {} \widehat{V}_k^{-1}\left( \widehat{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}_{q_k}[\Gamma (x)]\right) \\= & {} \widehat{V}_k^{-1}\left( \frac{\widehat{\mathbb {Y}}_k}{\widehat{\mathbb {Z}}_k}- \frac{\widetilde{\mathbb {Y}}_k}{\widetilde{\mathbb {Z}}_k}\right) \\= & {} \widehat{V}_k^{-1}\left( \frac{\widehat{\mathbb {Y}}_k}{\widehat{\mathbb {Z}}_k}- \frac{\widehat{\mathbb {Y}}_k}{\widetilde{\mathbb {Z}}_k}+ \frac{\widehat{\mathbb {Y}}_k}{\widetilde{\mathbb {Z}}_k}- \frac{\widetilde{\mathbb {Y}}_k}{\widetilde{\mathbb {Z}}_k}\right) \\= & {} \widehat{V}_k^{-1}\widehat{\mathbb {Y}}_k\left( \frac{\widetilde{\mathbb {Z}}_k-\widehat{\mathbb {Z}}_k}{\widehat{\mathbb {Z}}_k\widetilde{\mathbb {Z}}_k}\right) + \widehat{V}_k^{-1} \frac{\widehat{\mathbb {Y}}_k-\widetilde{\mathbb {Y}}_k}{\widetilde{\mathbb {Z}}_k}. \end{aligned}$$

Therefore,

$$\begin{aligned} \left||b_k\right||_2\le & {} \frac{\left||\widehat{V}_k^{-1}\right||_2}{\left| \widetilde{\mathbb {Z}}_k\right| } \frac{\left||\widehat{\mathbb {Y}}_k\right||_2}{\left| \widehat{\mathbb {Z}}_k\right| } \left| \widetilde{\mathbb {Z}}_k-\widehat{\mathbb {Z}}_k\right| + \frac{\left||\widehat{V}_k^{-1}\right||_2}{\left| \widetilde{\mathbb {Z}}_k\right| } \left| \widehat{\mathbb {Y}}_k-\widetilde{\mathbb {Y}}_k\right| \nonumber \\\le & {} \frac{\left||\widehat{V}_k^{-1}\right||_2}{\left| \widetilde{\mathbb {Z}}_k\right| } \frac{\left||\widehat{\mathbb {Y}}_k\right||_2}{\left| \widehat{\mathbb {Z}}_k\right| } \frac{1}{N_k}\sum _{i=1}^{N_k} \left| S_{\theta _k}\left( -C_{\alpha ^*}(x_k^i)\right) - \widehat{S}_{\theta _k}\left( -\widehat{C}_{\alpha ^*}(x_k^i) \right) \right| \nonumber \\&+\frac{\left||\widehat{V}_k^{-1}\right||_2}{\left| \widetilde{\mathbb {Z}}_k\right| } \frac{1}{N_k}\sum _{i=1}^{N_k} \left| \widehat{S}_{\theta _k}\left( -\widehat{C}_{\alpha ^*}(x_k^i) \right) -S_{\theta _k}\left( -C_{\alpha ^*}(x_k^i)\right) \right| \left||\Gamma (x_k^i)\right||_2. \end{aligned}$$

Recall that $\widehat{V}_k=\left( \widehat{Var}_{\theta _k}[\Gamma (x)]+\epsilon I\right)$. Thus, it is a positive-definite matrix and its minimum eigenvalue is at least $\epsilon$. It follows that the maximum eigenvalue of $\widehat{V}_k^{-1}$ is no greater than $\epsilon ^{-1}$, i.e., $\left||\widehat{V}_k^{-1}\right||_2\le \epsilon ^{-1}$. Since $|\widetilde{\mathbb {Z}}_k|$ is bounded below from zero, $\frac{\left||\widehat{\mathbb {Y}}_k\right||_2}{\left| \widehat{\mathbb {Z}}_k\right| }$ is bounded, and $\Gamma (x)$ is bounded on $\mathcal {X}$, Lemma 4.1 implies that $\left||b_k\right||_2\rightarrow 0$ w.p.1 as $k\rightarrow \infty$.

Proof

Proof of Theorem 4.2. Let us first show the following lemma. $\square$

Lemma A.1

Suppose Assumptions 1 and 2 hold. Further suppose the risk level sequence $\{\alpha _k\}$ generated by (3.2) converges to the target risk level $\alpha ^*$ w.p.1. Then the sequence $\{\theta _k\}$ generated by (4.8) converges to a limit set of the ODE (4.7) w.p.1.

Proof of Lemma A.1. Similar to the proof of Theorem 4.1, we will reformulate the updating scheme (4.8) as a noisy discretization of the constrained ODE (4.7), and show both the bias and the noise are properly bounded. Specifically, rewrite (4.8) as

$$\begin{aligned} \theta _{k+1}=\theta _k+\beta _k\left[ G(\theta _k)+ \bar{b}_k+e_k+\bar{p}_k\right] , \end{aligned}$$

(a.4)

where $G(\theta _k)$ and $e_k$ are defined as previously, $\bar{b}_k\overset{\triangle }{=}\widehat{V}_k^{-1}\left( \bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}_{q_k}[\Gamma (x)]\right)$, and $\bar{p}_k$ is the projection error term that takes the current iterate back onto the constraint set $\widetilde{\Theta }$ with minimum Euclidean norm. In view of Theorem 2 in [9], it suffices to show

$$\begin{aligned} \lim _{k\rightarrow \infty }\left||\bar{b}_k\right||_2= 0,\quad w.p.1. \end{aligned}$$

To ease the presentation, let us denote

$$\begin{aligned} \widetilde{\mathbb {E}}^{\alpha }_{q_k}[\Gamma (x)]\overset{\triangle }{=} \sum _{i=1}^{N_k}\frac{S_{\theta _k} \left( -C_{\alpha }(x_k^i)\right) }{\sum _{j=1}^{N_k}S_{\theta _k} \left( -C_{\alpha }(x_k^j)\right) }\Gamma (x^i_k). \end{aligned}$$

It immediately implies that $\widetilde{\mathbb {E}}_{q_k}[\Gamma (x)]= \widetilde{\mathbb {E}}^{\alpha ^*}_{q_k}[\Gamma (x)]$. Furthermore,

$$\begin{aligned} \left||\bar{b}_k\right||_2= & {} \left||\widehat{V}_k^{-1}\left( \bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}_{q_k}[\Gamma (x)]\right) \right||_2\nonumber \\= & {} \left||\widehat{V}_k^{-1}\left( \bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\alpha _k}_{q_k}[\Gamma (x)]\right) + \widehat{V}_k^{-1}\left( \widetilde{\mathbb {E}}^{\alpha _k}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\alpha ^*}_{q_k}[\Gamma (x)]\right) \right||_2\nonumber \\\le & {} \left||\widehat{V}_k^{-1}\left( \bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\alpha _k}_{q_k}[\Gamma (x)]\right) \right||_2+ \left||\widehat{V}_k^{-1}\right||_2 \left||\widetilde{\mathbb {E}}^{\alpha _k}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\alpha ^*}_{q_k}[\Gamma (x)]\right||_2.\nonumber \\ \end{aligned}$$

(a.5)

Following an argument almost identical to the proof of Lemma 4.2, the first term in (a.5) converges to 0 w.p.1 as $k\rightarrow \infty$. Note that $S_{\theta _k}(\cdot )$ is a continuous function and $C_{\alpha _k}(x)$ is continuous in $\alpha _k$. Thus, $\widetilde{\mathbb {E}}^{\alpha _k}_{q_k}[\Gamma (x)]$ is a continuous function in $\alpha _k$. Therefore, the second term in (a.5) converges to 0 w.p.1 as $k\rightarrow \infty$ since $\left||\widehat{V}_k^{-1}\right||_2$ is bounded and $\alpha _k$ converges to $\alpha ^*$ as $k\rightarrow \infty$. Proof of Lemma A.1 is now complete.

In view of Lemma A.1, it remains to show that the risk level sequence $\{\alpha _k\}$ generated by (3.2) converges to the target risk level $\alpha ^*$ w.p.1. Proof by contradiction. Since the sequence $\{\alpha _k\}$ is non-decreasing and bounded above by $\alpha ^*$, let us assume $\lim _{k\rightarrow \infty }\alpha _k=\bar{\alpha }^*$ and $\bar{\alpha }^*< \alpha ^*$ w.p.1. Conditioning on this, Lemma A.1 still holds when the target risk level $\alpha ^*$ is replaced by $\bar{\alpha }^*$. That is, the algorithm GASS-CVaR-ARL converges, and the gradient sequence $\{g_k\}$ converges to 0 w.p.1. as $k\rightarrow \infty$. Note that $g_k$ is bounded (since $\Gamma (x)$ is bounded), by bounded convergence theorem we have

$$\begin{aligned} \lim _{k\rightarrow \infty }\mathbb {E}\left[ \left||g_k\right||_2\right] =0. \end{aligned}$$

(a.6)

Furthermore, note that

$$\begin{aligned} \mathbb {E}\left[ \left||\bar{g}_k-g_k\right||_2\right]= & {} \mathbb {E}\left[ \left||\bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2\right] \nonumber \\\le & {} \mathbb {E}\left[ \left||\bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2\right] +\mathbb {E}\left[ \left||\widetilde{\mathbb {E}}^ {\bar{\alpha }^*}_{q_k}[\Gamma (x)]- \mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2\right] ,\nonumber \\ \end{aligned}$$

(a.7)

where

$$\begin{aligned} \mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\overset{\triangle }{=} \frac{\int S_{\theta _k}\left( -C_{\bar{\alpha }^*}(x)\right) \Gamma (x)f(x;\theta _k)dx}{\int S_{\theta _k}\left( -C_{\bar{\alpha }^*}(x)\right) f(x;\theta _k)dx}. \end{aligned}$$

We have shown in the proof of Lemma A.1 that

$$\begin{aligned} \lim _{k\rightarrow \infty }\left||\bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2=0, \quad w.p.1. \end{aligned}$$

Since $\left||\bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2$ is bounded, again by bounded convergence theorem

$$\begin{aligned} \lim _{k\rightarrow \infty } \mathbb {E}\left[ \left||\bar{\mathbb {E}}_{q_k}[\Gamma (x)]- \widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k} [\Gamma (x)]\right||_2\right] =0. \end{aligned}$$

(a.8)

Moreover, notice that $\widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]$ is a self-normalized importance sampling estimator of $\mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]$. Applying Theorem 9.1.10 (pp. 294) in [4], we have

$$\begin{aligned} \mathbb {E}\left[ \left| \widetilde{\mathbb {E}}^ {\bar{\alpha }^*}_{q_k}[\Gamma _j(x)]- \mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma _j(x)]\right| ^2\right] \le \frac{c_j}{N_k}, \; j=1,\ldots ,d_\theta , \end{aligned}$$

where $\Gamma _j(x)$ is the $j^{th}$ element in the vector $\Gamma (x)$, and $c_j$’s are positive constants that depend on the bounds of $\Gamma _j(x)$’s on $\mathcal {X}$. Therefore, by Cauchy–Schwarz Inequality we have

$$\begin{aligned} \mathbb {E}\left[ \left||\widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]- \mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2\right] \le \sqrt{\frac{d\cdot \max _j c_j}{N_k}}. \end{aligned}$$

That is,

$$\begin{aligned} \lim _{k\rightarrow \infty } \mathbb {E}\left[ \left||\widetilde{\mathbb {E}}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]- \mathbb {E}^{\bar{\alpha }^*}_{q_k}[\Gamma (x)]\right||_2\right] =0. \end{aligned}$$

(a.9)

Combining (a.7), (a.8) with (a.9), we have

$$\begin{aligned} \lim _{k\rightarrow \infty } \mathbb {E}\left[ \left||\bar{g}_k-g_k\right||_2\right] =0. \end{aligned}$$

In view of (a.6), we have

$$\begin{aligned} \lim _{k\rightarrow \infty } \mathbb {E}\left[ \left||\bar{g}_k\right||_2\right] =0. \end{aligned}$$

(a.10)

Since $\bar{\alpha }^*<\alpha ^*$, the sequence $\{\left||\bar{g}_k\right||_2\}$ generated by (3.2) will always be above a certain positive value w.p.1 (otherwise $\alpha _k$ will converge to $\alpha ^*$), which contradicts with (a.10). Proof of Theorem 4.2 is complete. $\square$