Why (some) abnormal problems are "normal"

In abnormal optimal control problems it is necessary to basically ignore the objective for certain state values in order to be able to determine the optimal control. In the past, abnormal problems were considered to be degenerated problems that did not ﬁt to any real application. In the present paper we discuss reasons for the occurrence of abnormality. We show that abnormality can be an integral part of a meaningful problem rather than to be a sign for degeneracy


Introduction
In his often-quoted paper Halkin (8) presents a relatively simple infinite time horizon problem with free end-state that is abnormal. In particular, this means that the Lagrange multiplier corresponding to the objective function in the Hamiltonian, which is often denoted by λ 0 , is equal to zero. In finite dimensional optimization it was John (9) who formulated Lagrange's rule in case that no conditions on the constraints are specified. Since then much effort has been undertaken to find so called constraint qualifications, cf. Kuhn and Tucker (10), Mangasarian and Fromovitz (11), Evans (4), Bazaraa et al.
(3), Peterson (12), that guarantee the existence of regular Lagrange-multipliers, i.e. λ 0 = 1. Results on this topic can also be found for the infinite dimensional optimization problems and abstract constraints, see e.g. Gould and Tolle (6), Arutyunov (1), Arutyunov and Izmailov (2). But in its generality these conditions are hard to verify.
In optimal control theory normality can be guaranteed for finite time and free end-state problems. However, normality does not need to hold for infinite time horizon problems with free end-state. Halkin (8) shows that even relatively simple problems can be abnormal, if the time horizon is increased to infinity.
Nonetheless, abnormality in real applications seems to occur only in degenerate cases and it was thought that it relates to an ill posed model. The fact that in the abnormal case the objective function does not play any role in the optimization process seems convincing for the latter viewpoint. In this note we design and analyze an optimal control model about the optimal accumulation of reputation. We show that abnormality is an integral part and not a sign for degeneracy.
The paper is structured as follows. Section 2 discusses the occurrence of abnormality in Halkin's example. In Section 3 we consider a model dealing with the accumulation of reputation. Section 4 concludes.

Halkin's example
We start with Halkin's example and carry out the details leading to abnormality. We also shortly discuss a possible modification of this example, which reveals more clearly the cause for abnormality. The details for this analysis and its conclusions for economic applications can be found in Grass (7).
In Halkin (8) the following stylized model is used to show that optimal control problems over an infinite time horizon with free end state are not necessarily normal contrary to finite time horizon problems. Note that in Halkin (8) the problem is only considered for x 0 = 0. To derive the necessary optimality conditions we consider the Hamiltonian function and the Lagrangian together with the derivatives For an optimal solution (x * (·), u * (·)) the maximizing condition For x 0 = 0 it is immediately clear that the optimal solution is (x * (.), u * (.)) = (0, 0). The reason is that for this solution the objective value is equal to zero, whereas for every other choice of the control u the objective value is strictly negative. Now it is important to realize that, when taking Eq. (2f) into account, we find that u = 0 can only be achieved for λ 0 = 0 or λ (.) = −∞. The latter choice is not an absolutely continuous function, as is required by the necessary optimality conditions. Then we are left with λ 0 = 0, implying that the problem is abnormal.
It follows that for problem (1) the objective value depends on x 0 , where it is discontinuous at x 0 = 0 : This model is degenerate in the sense that the state diverges and the objective value immediately jumps from zero to minus infinity.

A model with self-enforcing reputation
Let x(t) be the reputation of the decision maker at time t. The decision maker wants his or her reputation to be high and therefore the objective is to maximize the discounted stream of reputation values over time. The decision maker can improve reputation by networking efforts u(t), which are assumed to be more effective if reputation is already large. Reputation has a self-enforcing effect, see e.g. Feichtinger et al. (5), i.e. if it is sufficiently large reputation grows without any efforts by the decision maker (e.g. due to word-of-mouth propagation or the so-called Matthew effect). Parameter a is the depreciation rate of reputation and is a threshold for the self-enforcement effect. In particular, if reputation is larger than a, it can grow without any further efforts by the decision maker. The maximum possible networking efforts are denoted by u max , whereas the maximum obtainable reputation is given by A.
The optimal control problem can be written as and the parameter values have to satisfy the inequalities The inequalities (4e) guarantee that the maximum reputation A is sustainable.
Noting that the usage of control is costless 1 and that it is advantageous to stay at the highest possible state value, the optimal solution is

Necessary optimality conditions
We start out presenting the Hamiltonian from which it follows that Then the Maximum Principle yields At the state constraint (4d) we have according to (5) u(t) = 0. Since condition (6a) has to be fulfilled, this means that at the switching time T , where constraint (4d) becomes active, it has to hold The condition (6c) on the costate at T follows from the continuity of the costate at the switching time.
Next we analyze the geometric properties of the Stalling Equilibriumx > 0, which is a steady state at which effort is at its maximum (see Feichtinger et al. (5)).
Due to the assumption u max < a, it follows from equation (4b) that the Stalling Equilibrium with x = a − u max > 0 always exists.

Stalling Equilibrium
In this section we consider the problem (4) for x(0) =x and the according equilibrium solution. We see that the adjoint equation (6b) exhibits an equilibriumλ atx. The properties of the equilibrium (x,λ) in the state-costate space are: At the Stalling Equilibrium the maximizing condition (6a) yields for λ 0 > 0: The JacobianJ at the equilibrium (x,λ) is given as This matrix exhibits the eigenvalues and eigenvectors The eigenvalue ξ 1 > 0 is always strictly positive and hence corresponds to an unstable direction. The sign of the eigenvalue ξ 2 depends on the relation between the discount rate r and the size of the Stalling Equilibriumx.

Solution Structure
From (7c) we obtain that the relationship between the Stalling Equilibriumx and the discount rate r is crucial. Therefore, we have to distinguish between the cases where r is larger or smaller thañ x.
Case r −x < 0. The equilibrium (x,λ) is not admissible for λ 0 > 0. This is becauseλ < 0 and hence the maximizing condition (6a) for u max is violated since Eq. (7c) yields H u (x, u max ,λ) < 0. (8) In Section 3.2 we showed that for r −x < 0 the equilibrium is a saddle point (see Eq. (7e)) and the vertical line is the stable manifold (see Eq. (7f)). This situation is depicted in Figure 1a. The maximized objective value for different initial state values is shown in Figure 2a.
To show that the problem is abnormal we have to prove that no costate path λ(·) exists that satisfies the adjoint equation (6b) and the maximizing condition (6a) for u = u max . We already showed that λ(·) ≡λ is not admissible and hence does not satisfy the necessary optimality conditions. Therefore we choose some λ(0) > 0, for which the maximizing condition is satisfied, i.e.
Since (x, λ(0)) lies on the stable manifold of (x,λ) the costate path approachesλ. Consequently there exists a time τ (λ(0)) such that for t ≥ τ (λ(0)) This proves that we cannot find any finite value λ(0) such that (6a) for u max is satisfied for every t > 0. Thus, the maximizing condition only holds if λ 0 = 0, yielding the abnormal case.
The infinite time horizon is crucial. Otherwise, for some finite time T , we could chose λ(0) large enough, such that λ(T ) = 0 satisfies the transversality condition and hence yields H u (T ) = 0 and H u (t) > 0 for t ∈ [0, T ).
A more intuitive explanation is the following. Figure 2a shows that a solution ending at A gives a significantly higher value than a solution ending up at zero. Due to the control constraint u ≤ u max , however, reaching A is only possible for x(0) >x. Exactly atx, setting u ≤ u max is just sufficient to keep x equal tox forever. This implies that an infinitesimal increase of x atx would make a solution of reaching A possible, which would result in a significant value increase. This value increase translates into an infinite value of the costate, as is confirmed by the blue trajectories in Figure 1a. The maximum principle does not allow infinite costate values, which is the reason that the abnormal problem applies for x(0) =x.
Case r−x > 0. In that case the equilibrium (x,λ) is admissible and is an unstable node, see Figure 1b. Thus, it is a threshold point separating the solutions converging to the origin or moving to a and staying there. An important difference with Figure 2a is that in Figure 2b the value function is smooth, especially also at the Stalling Equilibriumx. Still it is the case that solutions ending at A have a higher value but differences with the alternative solutions, like staying atx or converging to zero are not that large. The reason is that future proceeds are to a large extent discounted away. Note that what distinguishes the scenario of Figure 2b from Figure 2a is the large discount rate. The smoothness of the value function implies that the costate value is finite atx, so that considering the abnormal solution is not needed here.

Conclusion
In the present paper we saw that the presence of constraints can lead to the occurrence of abnormal behavior. The essential feature of these problems is that at a certain point in the state space the decision maker would like to steer the system into a favorable direction, but is not able to as the control does not have the desired impact on the state dynamics. This property is something that can occur in economically meaningful problems e.g. in environmental or health economics, marketing, capital accumulation etc. Therefore, the possibility of abnormality must not be neglected in models exhibiting this feature as it is a central part of the underlying problem. . The black curves denote the unstable paths of the equilibria and green shows the stable path, that is non-admissible for λ0 > 0, the normal case.  Figure 1. For r = 0.03 the optimal objective value is not differentiable in the Stalling Equilibriumx, and since the derivative diverges, it is not Lipschitz continuous in this point. For r = 2 the optimal objective value is continuously differentiable for every initial x0 in the state space [0, A].