Optimal Spatial-Dynamic Management of Stochastic Species Invasions

Abstract

Recent analyses demonstrate that the spatial–temporal behavior of invasive species requires optimal management decisions over space and time. From a spatial perspective, this bioeconomic optimization model broadens away from invasive species spread at a frontier or to neighbors by examining short and long-distance dispersal, directionality in spread, and network geometry. In terms of uncertainty and dynamics, this framework incorporates several sources of stochasticity, decisions with multi-year implications, and temporal ecological processes. This paper employs a unique Markov decision process planning algorithm and a Monte Carlo simulation of the stochastic system to explore the spatial-dynamic optimal policy for a river network facing a bioinvasion, with Tamarisk as an example. In addition to exploring the spatial, stochastic, and dynamic aspects of management of invasions, the results demonstrate how the interaction of spatial and multi-period processes contributes to finding the optimal policy. Those interactions prove critical in determining the right management tool, in the right location, at the right time, which informs the management implications drawn from simpler frameworks. In particular, as compared to other modeling framework’s policy prescriptions, the framework here finds more use of the management tool restoration and more management in highly connected locations, which leads to a less invaded system over time.

Appendix

The code for the simulator can be obtained from

http://2013.rl-competition.org/domains/invasive-species

We assume we are given:

• 1. The state space S.

• 2. The action space A.

• 3. A simulator $:S\times A\rightarrow S\times {\mathbb {R}}$. Given a state s and an action a, the simulator produces the next state $s'$ and the immediate (deterministic) reward r according to a probability distribution ${\upvarphi } (s'| s,a)$.

• 4. A discount factor $\gamma$

• 5. A confidence level ${\updelta }$

• 6. An error tolerance $\upepsilon$

Data Structures:

• 1. $N(s,a,s'):$ Number of times that the transition $(s,a)\rightarrow s'$ has been observed.

• 2. $N(s,a)=\sum \nolimits _{s'} {N(s,a,s')}$. The number of times that action a has been simulated in state s.

• 3. $\hat{{\varvec{\varphi }}}(\mathrm {s}'|\hbox {s},\hbox {a})$: The maximum likelihood estimate of the transition probabilities:

  $$\begin{aligned} N(s,a,s')/N(s,a). \end{aligned}$$

• 4. A(s): The set of actions that can be executed in state $\mathrm {s}$.

• 5. R(s, a): The reward of taking action a in state s

• 6. V(s): The value function for state s

• 7. Q(s, a): The state-action value function for (s, a)

• 8. $\pi (s)$: The (deterministic) policy, which is a mapping from states to actions, $\pi {:}\,S\rightarrow A$

Algorithm 1: $\pi =\hbox {Planner}(S,A,\delta ,\epsilon )$

• 1. Initialization

  • a. $Q(s,a)=0$ for all (s, a)

  • b. $V(s)=0$ for all s

• 2. $(R,\hat{\varphi })=$ Call Sampler($S,A,\delta ,\epsilon$)

• 3. Repeat %value iteration

  • a. $\mathrm {\Delta }=0$

  • b. For each $s\in S$

    • i. $v=V(s)$

    • ii. For each $a\in A(s)$

      $$\begin{aligned} 1. \quad Q(s,a)=R(s,a)+\gamma \sum \nolimits _{s'\in S} {\hat{\varvec{\varphi }}}(s'|s,a) \max \nolimits _{a'} Q(s',a') \end{aligned}$$

    • iii. $V(s)=\max \nolimits _{a} Q(s,a)$

    • iv. ${\Delta }=\max ({\Delta },|v-V(s))$

    Until ${\Delta }<\epsilon ({\textit{small}}\; {\textit{positive}}\;{\textit{number}})$

• 4. $\pi (s)={\textit{argmax}}_{a}Q(s,a)$ for each $s\in S$

• 5. Return $\pi$

Algorithm 2: $(R,{\hat{\varvec{\varphi }}})=\hbox {Sampler}(S,A,\delta ,\epsilon )$

• 1. Initialization

  • a. $R(s,a)=0$ for all (s, a)

  • b. $N(s,a,s')=0$ for all $(s,a,s')$

• 2. For each $s\in S$

  • a. For each $a\in A(s)$

    • i. Repeat

      • 1. Invoke the simulator F on (s, a) and obtain $(R(s,a),s')$

      • 2. Update $N(s,a,s'), N(s,a)$, and R(s, a)

      • 3. $\Delta (\mathrm {s}')$=Clopper-Pearson confidence bound using $N(s,a,s'), N(s,a), \delta$

      Until $\max _{{\mathrm{s}}\prime }\Delta (\mathrm {s}')<\epsilon$

    • ii. $\hat{{\varvec{\varphi }}}(s'|s,a)=N(s,a,s')/N(s,a)$ for each $s'\in S$

• 3. Return $R,\varphi$