Note on the (non-)smoothness of discrete time value functions

We consider the discrete time stopping problem \[ V(t,x) = \sup_{\tau}E_{(t,x)}[g(\tau, X_\tau)],\] where $X$ is a random walk. It is well known that the value function $V$ is in general not smooth on the boundary of the continuation set $\partial C$. We show that under some conditions $V$ is not smooth in the interior of $C$ either. More precisely we show that $V$ is not differentiable in the $x$ component on a dense subset of $C$. As an example we consider the Chow-Robbins game. We give evidence that as well $\partial C$ is not smooth and that $C$ is not convex, even if $g(t,\cdot)$ is for every $t$.

assumptions it turns out, that for every t there is a dense subset of C ∩ ({t} × R) on which V (t, ·) is not differentiable, see Theorem 3.2. As a guiding example we consider the Chow-Robbins game in Section 2. These results lead to the conjecture that the stopping boundary ∂C is not smooth on a dense set either. We will not prove this in general, but give numerical examples in Section 4. These furthermore illustrate, that in the Chow-Robbins game the continuation region C is not convex.
These results are interesting for different reasons. They show that we can not hope to find a closed form solution for these problems. On the other hand, they give an interesting qualitative characterization of V and C and show that discrete time problems behave quite differently from their time continuous counterparts.

Some properties of the Chow-Robbins game
As a main example we consider the Chow-Robbins game, also known as the Sn nproblem. Let ξ 1 , ξ 2 , . . . , be i.i.d. random variables with P (ξ i = −1) = P (ξ i = 1) = 1 2 , x + X τ t + τ . (2.1) This stopping problem was introduced by Chow and Robbins in 1965 [2]. It was originally defined on the lattice N 0 × Z where stopping in t = 0 is ruled out. We will use (t, x) ∈ R ≥0 × R instead and allow for N 0 -valued τ if t > 0. Only for t = 0 we require τ > 0, i.e. V (0, x) := EV (1, x + X 1 ). We denote the stopping set by D, the continuation set by C and its boundary by ∂C. In [3] the authors recently showed how to calculate a good approximation for (2.1).
We want to collect two lists of properties of (2.1), that are sufficient to show the following theorems about the non-smoothness of value functions. 1. There exist x 1 < 0 < x 2 with P (ξ 1 = x 1 ) > 0 and P (ξ 1 = x 2 ) > 0.
4. g(t, ·) is convex for every t, therefore is convex for every stopping time τ . It follows that V (t, ·) is convex as the supremum over convex functions.
To show 5. we calculate the left and right derivative for (t, x) ∈ ∂C, where we write g for ∂ ∂x g and b t for b(t): where we used the convexity of V in the last line.

Main results
We now show that for stopping problems, which have the Properties 1 resp. 1 and 2, the value function is not smooth. Let X be a random walk with X 0 = 0 and g : R ≥0 × R → R a measurable gain function such that the stopping problem is solvable by an a.s. finite stopping time τ * . Theorem 3.1. If the stopping problem (3.1) has Properties 1, then V is not differentiable with respect to x on C, i.e. there exists at least one (t, x) ∈ C such that V (t, ·) is not differentiable in x.
We call points (t, x) where V (t, ·) is not differentiable in x non-smoothness points.
Proof. Let (s, b(s)) be a point on ∂C where smooth fit does not hold. Due to convexity we therefore have We denote with τ * = τ t,x * the optimal stopping time starting in (t, x). Let be (t, and The convexity of V (t, ·) implies The lefthand side in (3.4) is decreasing in h and is integrable for every h since the arguments are in D and therefore The lefthand side in (3.5) is increasing and V is non-decreasing in the x component, since g(t, ·) is non-decreasing by assumption.
Therefore the lefthand side in (3.5) is non-negative and has the integrable lower bound 0. Taking limits in (3.2) and (3.3) and using the dominated convergence theorem we get With the more restrictive additional Properties 2, the non-smoothness points lie dense in C.

Theorem 3.2.
If the stopping problem (3.1) has Properties 1 and 2, the non-smoothness x ) is a non-smoothness point in the sense of Theorem 3.1, i.e. we show that there exists j ∈ N, such that P (x + X j = b(t + j), τ * = j) > 0. We distinguish two cases.
We choose N such that b(s + m * ) − b(s) < ε for all s ≥ N . We choose m and y, such that t + m ≥ N , y ≥ b(t + m) and P (x + X m = y) > 0. Again we find a series of increments An illustration of the construction in this part is shown in Figure 1.
. Now x := x + ε is our candidate starting point. We need to show that there is a path from (t, x ) to (t + m + k * m * , b(t + m + k * m * )), that has positive probability and lies in C. The path with the increments Λ m * . . . Λ m * k * times Λ y has positive probability and the first k * m * steps clearly lie in C. The last part however might not lie in C, since we can not assume C to be convex. If it does not we choose an integer l and start the same procedure again with (t + lm, l(y − x) + x) instead of (t + m, y). Since b(t) grows slower with increasing t, but the jump sizes in Λ y does not change, the claim will hold for some l.
Remark 3.4. By general theory we know that b(t) = inf{x | V (x) = g(x)}. If V is not smooth, there is no reason to assume that b is. In particular, if there exist t and m such that P (b(t) + X m = b(t + m)) > 0 we would expect that (if existent) Furthermore, if the non-smooth points of V lie dense in C we expect that the non-smooth points of b lie dense in R >0 . For the Chow-Robbins game this means in particular that the continuation set C is not convex. We will not prove these conjectures in general, but study numerical evidence for examples in the next section.

Remark 3.5.
Many discrete stopping problems possess Properties 1. The condition that ∂C is the graph of a function can be relaxed quite a bit. We only need that specific points on ∂C can be reached from the interior of C with positive probability, hence similar results could be obtained for two-sided problems as well. It might be difficult to prove results in a more general setting because different effect may cancel out. Nevertheless, we believe that the observed properties are quite common for discrete time stopping problems.
If X is a random walk and the stopping problem has a one-sided solution, there is a sufficient and easy to check condition for the convexity of V (t, ·). Let be ξ * = sup ω ξ + 1 (ω), if there exists ε > 0 such that g(t, ·) is convex on [b(t) − ε, b(t) + ξ * + ε] for all t, then V (t, ·) is convex on (−∞, b(t) + ξ * + ε] for all t. This can be proven analogously to Lemma 2.1. For more general processes X however, a convex gain function g(t, ·) does not always yield a convex value function. In [9] an example for a diffusion X with linear gain function, is given, that has a non-convex value function, see also [1].
Also the stronger Properties 2 are by no means restricted to the Chow-Robbins game. For example, if the increments are centered and have second moments, the slow growth condition for b, (b(n + 1) − b(n)) → 0, follows from the law of the iterated logarithm. Also the arguments in Lemma 2.1 can be extended.

Examples
We have seen that the value function of the Chow-Robbins game is non-smooth on a dense subset of C ∩ ({t} × R) for every t > 0. Figure 2 shows V for the fixed time t = 1.
Some non-smoothness points can be seen in the plot: • x 0 = 0.46 is the smallest value of x for which it is optimal to stop. We see that V does not follow the smooth fit principle.
In Figure 3 V is given for t = 5.  ·) (blue) and the gain function g(1, ·) (orange). Some non-smoothess points can be seen.
As mentioned in Remark 3.4 we expect b to be non-smooth either and C to be nonconvex. We found some numerical evidence for these conjectures. It is unlikely to find closed form solutions for V or b. Yet it may be helpful to study functions with these properties in order to get a better understanding of discrete stopping problems. Examples of functions that are continuous but not differentiable on a dense subset can be found in [4].
The ξ i are centered, have unit variance and the value function V has the upper bound given in [3]. We numerically calculate an estimate of V and the stopping boundary b(t), the absolute error of our calculation is approximately 10 −6 . In Figure 5 we see the stopping boundary  b(t). It looks smooth and concave, but if we zoom in and tilt it for better visibility, we see that this conception is misleading.
The point (3.697, 1.089) lies on the boundary ∂C and (3.697 + 1, 1.089 + 0.2) lies on ∂C again, hence we expect b(t) to be non-smooth in t = 3.697. In Figure 6 we see a plot of b(t) − 0.2085t. The effect is small, but we can see clearly that C is not convex. If we zoom in further, we can see more non-smooth points, see Figure 7.
Clearly V has some non-smoothness points, but they do not lie dense in C, see Figure 8.

Conclusion
We have shown that the value functions of the Chow-Robbins game is not differentiable on a dense subset of C and that this can be generalized. We have shown non-smoothness in the x-component, but in the Chow-Robbins game and most other cases the value function will not be differentiable in t in the non-smoothness points either. Furthermore, it seems likely that the stopping boundary b(t) is not smooth on a dense set as well. This shows that it is highly unlikely to find a closed form for V or b.     because our intuition might be misleading. We restricted our proof to specific cases, but the described phenomena seem to be typical for discrete time stopping problems.

A Numerical methods
The mathematical background of the methods used for our computations can be found in [3]. We will give a brief description here. To calculate the value function V (t, x) of the Chow-Robbins game and its variant in Example 4.1 at time t, we choose a time horizon T and use backward induction from T + t. At T + t we use the upper bound given in [3]. This can be interpreted as switching to the favorable Wt t -game for a standard Brownian motion W . With the lower bound given in [3] we estimate the absolute error of our calculations. To calculate the stopping boundary b(t) we use for every t a series of M nested intervals I 1 ⊃ . . . ⊃ I M that contain our approximation of b(t) and have the length λ(I k ) = C2 −k . For our calculations we used T = 5000 and M = 40. The resolution in t is 500 evaluation points per plot, that is ∆t = 10 −5 for Figure 7.