Beyond expected values: Making environmental decisions using value of information analysis when measurement outcome matters

In ecological and environmental contexts, management actions must sometimes be chosen urgently. Value of information (VoI) analysis provides a quantitative toolkit for projecting the improved management outcomes expected after making additional measurements. However, traditional VoI analysis reports metrics as expected values (i.e. risk-neutral). This can be problematic because expected values hide uncertainties in projections. The true value of a measurement will only be known after the measurement's outcome is known, leaving large uncertainty in the measurement's value before it is performed. As a result, the expected value metrics produced in traditional VoI analysis may not align with the priorities of a risk-averse decision-maker who wants to avoid low-value measurement outcomes. In the present work, we introduce four new VoI metrics that can address a decision-maker's risk-aversion to different measurement outcomes. We demonstrate the benefits of the new metrics with two ecological case studies for which traditional VoI analysis has been previously applied. Using the new metrics, we also demonstrate a clear mathematical link between the often-separated environmental decision-making disciplines of VoI and optimal design of experiments. This mathematical link has the potential to catalyse future collaborations between ecologists and statisticians to work together to quantitatively address environmental decision-making questions of fundamental importance. Overall, the introduced VoI metrics complement existing metrics to provide decision-makers with a comprehensive view of the value of, and risks associated with, a proposed monitoring or measurement activity. This is critical for improved environmental outcomes when decisions must be urgently made.


Introduction
In many fields, including ecology, environmental monitoring, economics and health, decisionmakers must identify whether there is sufficient information to confidently choose manage-ment actions, or if further information should be acquired before an action is chosen (Bennett et al., 2018;Yokota and Thompson, 2004). Ideally, the expected benefit of additional measurements outweighs the cost (Koerkamp et al., 2006), but in ecological and environmental contexts, management actions must sometimes be chosen urgently (McDonald-Madden et al., 2010). Management actions always carry a risk of unanticipated and detrimental outcomes (Bergstrom et al., 2009), but delays in management actions can be extraordinarily costly, resulting in ecosystem damage or potential species extinctions (Martin et al., 2012). Thus, decision-makers may need to quickly decide if more information should be obtained about the system of interest before choosing an action, or if decisions should be promptly made based on current knowledge (Bennett et al., 2018). If a timed sequence of decisions about management actions is to account for the system's change following each action, this sequential decisionmaking process can be informed using mathematical approaches that explicitly describe the iterative interaction between management actions and system changes (Chadès et al., 2008(Chadès et al., , 2021. However, even if an immediate decision about which one management action to choose at one specific time would benefit from additional information about the system, conservation monitoring is very costly and time consuming. Thus, it is crucial that such monitoring is justified by its usefulness for improving management decisions (McDonald-Madden et al., 2010).
Value of information (VoI) analysis provides a quantitative toolkit for assessing the expected improvement that additional information can make to management decisions . VoI represents the difference between expected outcomes when a management decision is made in the presence of new information versus only current information (Yokota and Thompson, 2004). VoI analysis therefore addresses the question of whether or not it is beneficial to collect additional information when improved decision-making is the primary objective. When the management action to be chosen must be performed at one specific time, then VoI analysis precedes both the (proposed) information collection activity and the choice of management action (Figure 1).

VoI analysis is performed
Collect or don't collect data Near future

Implement action
Far future Value of system, that we wish to maximise, is assessed Figure 1: Overview of the timing of value of information (VoI) analysis, for a decision problem where a management action needs to be chosen at some 'near future' time, for a system whose value we want to maximise in the 'far future'. In this system, there is a measurement ('collect data') that could be performed in the meantime to improve the action that is chosen. VoI analysis informs whether this measurement should be performed or not. In this paper we focus on decision problems which adhere to this timeline.
Traditional calculations in VoI analysis assume that the decision-maker is risk-neutral.
This is not realistic in many management contexts, especially in conservation where it is expected that many decision-makers responsible for environmental management are riskaverse (Tulloch et al., 2015). Risk aversion of the decision-maker can be incorporated into VoI analysis calculations (Xiao et al., 2019), and previous work has focused on reducing the risk of undesirable outcomes for the system as a whole (Nadiminti et al., 1996;Borgonovo and Cillo, 2017). For example, risk aversion to poor outcomes can be incorporated in VoI analysis by requiring that the value function used in this analysis is a strategically-chosen nonlinear function of the system's properties that the decision-maker aims to maximise (e.g. Hilton 1981;Willinger 1989;Bleichrodt et al. 2020).
A similarly important concern is that the information obtained from a measurement does not substantially improve the decision-maker's choice of management action. For example, if new information does not change the action the decision-maker was otherwise going to select, the data collection activity may be deemed retrospectively as "worthless". Whilst this potential outcome is not directly detrimental to the system being managed, the possibility of this outcome may be perceived by the decision-maker as a "risk" of wasted investment of time and/or money in monitoring.
More broadly, there may be a range of potential measurement outcomes which are viewed as "low-value measurement outcomes" -they may change the decision-maker's chosen man-agement action, but due to their expected low improvement values they are deemed by the decision-maker to be not worth the cost of measurement or delay in action (Grantham et al., 2009). Even if traditional VoI analysis suggests that a new measurement has a substantial expected value, an expected value says nothing about the uncertainty in this value, so there may still be an unacceptably high probability that the measurement yields low value to the decision-maker. Whilst it may be impossible to know if a low-value measurement outcome will definitely occur prior to the data being collected, the risk of low-value measurement outcomes, on the other hand, can be predicted, and to our best knowledge the calculation of this risk has not yet been explored. Such risks are particularly critical to quantify in environmental decision-making when the consequences of a delayed management action can be catastrophic (Martin et al., 2012).
Because of its focus on informing data collection activities, VoI is also related to optimal design (Williams and Brown, 2020), a class of experimental design methods that aims to recommend monitoring activities based on some statistical optimality criteria (Holling and Schwabe, 2017). VoI analysis has evolved primarily from the economics literature (Raiffa, 1968), with subsequent uptake of this analysis in medical decision-making (Felli and Hazen, 1998) and increasingly recent uptake by the ecological and environmental science communities (Runge et al., 2011;Nicol et al., 2019). The research area of optimal design has been expanding rapidly in parallel in the statistics literature (Ryan et al., 2016). However, the topics of VoI and optimal design have not substantially overlapped thus far in the literature due to these topics' different origins in economics and statistics respectively. The conceptual link between VoI analysis and optimal design is already clear, as both are decision-theoretic approaches that estimate the expected benefits of collecting further information (Jackson et al., 2022). Making the mathematical link between VoI analysis and optimal design clear has the potential to unlock the rapid technical advances made by statisticians in the field of optimal design to help address VoI questions in ecology and environmental science.
Here, we extend VoI analysis by introducing four new metrics that explicitly address risk tolerance to different measurement outcomes. In doing so we also show how the new metrics can be used to clearly establish the mathematical link between VoI and optimal design. Our new metrics quantify (Table 1): 1. The shift ∆EV x in the expected value of the system, dependent on measurement outcome x; 2. The value VSI x of sample information, dependent on measurement outcome (this value of the information can be different to the shift in expected value of the system resulting from the collection of this information); 3. The uncertainty (specifically, standard deviation σVSI) in the value of information provided by the measurement; and 4. The risk rVSI δ of the measurement providing low value (specifically, probability of value ≤ δ) for decision-making.
When reported alongside traditional VoI metrics, the new metrics provide the decision-maker with a broader perspective on the potential value of proposed measurements before choosing whether (and what) information should be collected. For example, the decision-maker would be able to choose monitoring activities that either maximise the expected value of the monitoring or minimise the risk of low-value monitoring outcomes. Finally, we demonstrate the usefulness of our new metrics via two ecological case studies that traditional VoI metrics have been previously applied to . Using these case studies, we show how our new metrics help decision-makers to form a comprehensive view of the value of, and risks associated with, a proposed monitoring or measurement activity; this is critical for improved environmental outcomes when decisions must be urgently made.

Materials and methods
To provide context for our new VoI analysis metrics (Table 1), we first introduce the traditional metrics used in VoI analysis (Section 2.1) and then explain each of the four new metrics in turn in Sections 2.2.1-2.2.4. We then mathematically describe the link between There is a 40% risk that the measurement's impact on decision-making is no change in the expected population of frogs.
VoI and optimal design (Section 2.3) as this link requires knowledge of our second new metric (VSI x , introduced in Section 2.2.2). Finally, we apply our new metrics to two case studies (Sections 3.1 and 3.2), to compare and contrast the insights obtained from the traditional and new VoI metrics for environmental decision-making.

Common calculations and terminology in VoI analysis
To perform VoI analysis, one must first describe how the value V (a, s) of the system being managed depends on the system's state s and the management action a being undertaken . Values V (a, s) are quantified in units of the management objective, for example, population size or the probability of species survival. It is assumed that the decision-maker wishes to maximise the system's value. The current uncertainty about the system can be represented as alternative potential states s of the system, with each state assigned a prior probability p(s) of being true. If the management action a is performed on the system without collecting any new data prior to this action, the action is said to be taken "under uncertainty". An example of such uncertainty in a system is the presence or absence of a species when no new information is obtained to confirm species presence or absence (Bennett et al., 2018).
The expected value EV(a) of the system if an action a is taken under uncertainty is calculated from the expectation of the system's value over all possible states s, given that the action a is definitely chosen: A risk-neutral decision-maker will choose the action that yields the largest expected value of the system. If this action is selected, the expected value of the system, denoted as EV uncertainty , Conversely, if perfect information is obtained for the system state, a risk-neutral decisionmaker can choose the action that yields the maximum value possible for that system state.
Such a decision is said to be made "under certainty". In this case, the expected value of the system, denoted as EV certainty , is Equation (3) indicates that under each fully-known system state s, the action a that yields the maximum value of V (a, s) is chosen by the decision-maker. However, the use of the expectation over all system states s in Equation (3) acknowledges that this calculation is carried out before the perfect information is obtained (see Figure 1).
When performing VoI analysis, it is assumed that every decision-maker seeks to choose the action that leads to the greatest net benefit (Yokota and Thompson, 2004). Thus, a first commonly calculated VoI metric -the expected value of perfect information (EVPI) -is the expected benefit of resolving or eliminating uncertainty entirely. Mathematically, EVPI is the difference between the expected values of the system when the best management actions are taken under certainty versus under uncertainty: Since access to perfect information is usually not possible in practice, Equation (4) represents a best case scenario of information gain from measurements prior to action. The EVPI therefore puts an upper bound on the expected value of imperfect information, and as such is a useful quantity to estimate even when the proposed measurements are expected to yield information that is nowhere near perfect. For example, if a risk-neutral decision-maker deems the EVPI to be low enough, then no measurement should be performed before the management action is chosen. This outcome is likely to be common (McDonald-Madden et al., 2010), and can be advantageous because actions are not delayed by potentially costly measurements.
Measurements that do not yield perfect information can, however, still empower the decision-maker to choose better management actions (Yokota and Thompson, 2004). Thus, a second commonly calculated VoI metric -the expected value of sample information (EVSI) -is the increase in the system's expected value when the decision-maker obtains a sample of imperfect measurements, observations, or information . Calculation of EVSI is slightly more complicated than EVPI; it requires an expression for the expected value of additional imperfect information, but this imperfect information, denoted as x, is not guaranteed to completely eliminate uncertainty regarding which state s the system is in.
For a risk-neutral decision-maker, the expected value of the system after obtaining imperfect information, denoted as EV less uncertainty , is where x is the specific outcome obtained by the proposed measurement of imperfect information. For example, a monitoring activity to determine the number of koalas in a forest area may locate 10 koalas (x = 10) when there are in fact 14 koalas present in the area (s = 14). However, we do not know prior to monitoring how many koalas will actually be found (x = ?), so calculation of EV less uncertainty requires an expectation over all possible measurement outcomes x.
The EVSI is thereafter calculated as follows: The EVSI can be used by the decision-maker to decide whether or not to perform a measurement of imperfect information prior to choosing the management action: if the EVSI is judged to be sufficiently large, this provides greater impetus to do the measurement first.
However, EVSI is only appropriate if the decision-maker is risk-neutral to the potentially wide variety of improvements (from zero to small to large) in the expected system state that may result from different potential measurement outcomes. The reason for this is that EVSI is, by definition, an expected value of information calculated before the measurement is performed ( Figure 1) and so does not provide any indication of the uncertainty in the actual value of information that will be obtained from the measurement. To inform a decision-maker that wishes to avoid low-value measurement outcomes, new metrics are needed.

New metrics to quantify risk and uncertainty in VoI analysis
If the analysis of a decision problem explicitly accounts for the decision-maker's risk tolerance (e.g. risk-neutral or risk-averse), it is hoped that the outcomes of the analysis will be of greater practicality to the decision-maker (Markowitz, 1959). With this in mind, we assume that a risk-neutral decision-maker will aim for the highest expected outcome, whilst a riskaverse decision-maker will aim to minimise the risks or regrets associated with an undesirable outcome (Tulloch et al., 2015;Xiao et al., 2019). This "risk" is equally applicable to a decision-maker's tolerance to low-value system outcomes (Nadiminti et al., 1996;Borgonovo and Cillo, 2017) and low-value measurement outcomes (addressed in the present work). We propose four new metrics that describe the risk and uncertainty associated with different potential measurement outcomes.
The four new metrics are summarised in Table 1, and mathematically detailed in Sections 2.2.1-2.2.4. All four metrics are designed for the realistic situation in which the proposed measurement yields imperfect information, and thus the new metrics complement EVSI.
However, although not discussed further here, the new metrics can also be applied to situations in which the proposed measurement yields perfect information (to complement EVPI), as this latter case is a mathematical simplification of imperfect information in which every measurement outcome x reveals the true state s of the system. Thus, any formulae introduced in this paper for the case of imperfect information can also be used for the case of perfect information, by mathematically substituting the system state s into all instances of the measurement outcome x (which subsequently eliminates x from these formulae).
2.2.1. Metric 1: Shift in expected value of the system, ∆EV x The first new metric we introduce is the shift in expected system value due to the measurement, and is denoted by ∆EV x . This metric focuses on the value of the system rather than the value of the information associated with the measurement, although it has a clear and strong link to the traditional VoI metric of EVSI that makes it a logical new metric to discuss first. ∆EV x also aids with understanding the three other, more decision-focused, metrics we introduce in this work.
The metric ∆EV x is defined for a particular measurement outcome x, so every potential measurement outcome x yields a different value of ∆EV x . This metric is obtained by removing the expectation over measurement outcomes x from the formula for EVSI (Equation (6)) so Since the second term in Equation (6) has no dependence on measurement outcome x, any expectation over x introduced to or removed from this term has no effect (see Supplementary Material Section S1.1 for further mathematical details).
Hence, we obtain: Whilst EVSI specifies the expected value of the measurement information and is calculated without knowing what the measurement outcome is, ∆EV x specifies the change in the expected value of the system if we know what the measurement outcome is (x). Hence, ∆EV x estimates the shift in expected system value for a particular measurement outcome x, but not necessarily the shift in decision. Note that new knowledge gained from a measurement can lead to a re-evaluation of the system state to possess either a higher or lower expected value, so unlike the strictly non-negative VoI metrics EVSI and EVPI, the metric ∆EV x can be positive, zero or negative. The system-focused metric ∆EV x is also a useful complement to the more decision-focused metric VSI x , which we introduce next.

Metric 2: Value of sample information, VSI x
The second new metric is the value of sample information (VSI). VSI is the value that imperfect information has for decision-making, specified after the information is collected.
Similarly to ∆EV x , the VSI is defined for a particular measurement outcome x, so is denoted mathematically as VSI x . However, unlike ∆EV x , VSI x is a measure of information value rather than system value. Specifically, VSI x is the expected value of the measurement information if we know what the measurement outcome is (x).
To explain how VSI x is obtained, we first consider the action that yields the maximum expected value of the system if no measurement is performed. If we denote this optimal action under uncertainty as a * , from Equations (1) and (2) it is clear that Action a * would be chosen by the decision-maker if no measurement is made. Now, suppose that the measurement has taken place and a particular measurement outcome x is observed. We calculate VSI x associated with this measurement outcome x as the difference between the expected value of the system if the best action is taken after knowing the measurement outcome is x and the expected value of the system if the best action is taken without the knowledge that the measurement outcome will indeed be x: Note that the dependence on x of the expectation in the second term of Equation (9) can be interpreted in the following way: the VSI is calculated as if the measurement has already taken place and the decision-maker is reflecting on the expected value gained from the measurement in hindsight.
From the definition of VSI x provided in Equation (9), we also note that the expectation of VSI x taken over all possible measurement outcomes x is equal to the EVSI defined in Equation (6) (see Section S1.2 for further mathematical details). Hence, the expectations over x of the two new metrics ∆EV x and VSI x both yield EVSI, even though ∆EV x and VSI x have different mathematical definitions. The metric ∆EV x represents the shift in expected value of the system after obtaining the measurement outcome x, so ∆EV x can be positive, zero or negative. On the other hand, VSI x represents the value for decision-making gained from the measurement outcome x, so VSI x is, like EVSI, strictly non-negative. We will see later in the case studies (Section 3) that sometimes ∆EV x = VSI x , but this is not true in general.
While the two new metrics ∆EV x and VSI x reveal far more information than the traditional EVSI metric alone, these two new metrics can also be unwieldy to report if there are many (or infinite) possible measurement outcomes x that can be obtained. The next two metrics we introduce are not dependent on measurement outcome, so advantageously yield only one number each, although the calculation of both of these metrics requires that VSI x is calculated first.

Metric 3: Standard deviation of the value of sample information, σVSI
The traditional metric of EVSI is the expected value of sample information, and thus provides no measure of the uncertainty in the value of information that might actually be obtained.
Hence, EVSI can be viewed as an "average" VoI, but it would be useful to be able to report VoI as "average ± uncertainty". Variance (the square of standard deviation) has long been suggested as a measure of risk when deciding which management actions to undertake (Markowitz, 1991), although standard deviation is easier to interpret because it is in the same units as the quantity being estimated. Hence, the third metric we introduce is the standard deviation in the value of sample information, denoted as σVSI: where Var x [VSI x ] is the variance in the distribution of VSI x across all different measurement outcomes x. If σVSI is reported alongside EVSI, then these two metrics together characterise the mean and standard deviation of VoI across all potential outcomes x of a proposed imperfect measurement, i.e. EVSI ± σVSI.

Metric 4: Risk of low value of sample information, rVSI δ
The fourth and final metric that we introduce quantifies the risk that the measurement will yield low value for decision-making. For this metric, a threshold value of VSI is chosen (ideally by the decision-maker) at or below which the value of the measurement is considered "too low". This threshold value of VSI is denoted as δ and reflects the decision-maker's risk aversion to low-value measurement outcomes. The risk of low value of sample information, denoted as rVSI δ , is defined as the probability across all measurement outcomes x that VSI x ≤ δ: Here, δ can be interpreted as a measure of the decision-maker's level of risk tolerance, because it defines the minimum value of the measurement above which they are willing to accept.
The threshold value δ must be non-negative: for example, if it is set to zero, then rVSI 0 is the probability (i.e. risk) that the measurement has no value because its outcome did not change the management action chosen by the decision-maker. It is important to note here that, if an obtained measurement outcome suggests that (1) the originally proposed management action is still the best and (2) this best action is expected to have an even greater positive effect on the system than previously thought, the value of the measurement for decision-making is still zero because it did not change the decision-maker's action.
Positive values of δ may also be considered; for example, rVSI 10 = 0.15 indicates that the decision-maker desires that the measurement's expected value should exceed ten, and the risk that this desire is not achieved is 15%. To obtain rVSI 10 , we would first identify all measurement outcomes x that yield VSI x ≤ 10, and calculate the probabilities p(x) of each of these outcomes. Adding together these probabilities p(x) would yield rVSI 10 . If the decision-maker considers this risk to be acceptable, they may wish to proceed with the proposed measurement prior to choosing a management action.

The mathematical link between VoI and optimal design
Our new metrics allow us to establish the mathematical connection between the research fields of optimal design and VoI theory. It is our hope that the establishment of this theoretical link will encourage advances made in either of these two fields to advance the other field as well. Briefly, optimal design is a class of experimental design based on optimisation of certain statistical criteria in relation to a prescribed task such as parameter estimation (Holling and Schwabe, 2017). Experimental design provides the theoretical framework for allocating resources when the design of an experiment or gathering more information about an uncertain system state would enable the decision-maker to better understand an underlying process (Raiffa, 1968;Ryan et al., 2016).
Optimal design minimises the experimental resources required to make precise inferences on the relevant features of an experimental design (Ryan et al., 2016;Holling and Schwabe, 2017). Thus, optimal design has broad application in many fields, including the medical, social, natural, and educational sciences, and in business, finance and engineering (Drovandi and Pettitt, 2013;Gill et al., 2022). With the advent of increasing computational power, optimal design utilises Bayesian approaches to incorporate prior information and/or the relevant system uncertainties within a utility function that describes the aim of the experiment (Ryan et al., 2016).
In statistical experimental design, there may be some uncertainty about the parameters controlling the system behaviour. Accounting for this lack of knowledge (of the model parameters or the model itself) can help to design efficient experiments. In Bayesian optimal designs, current knowledge (prior to any proposed experiments) of system parameter estimates can be represented by the prior distribution p(θ) (Drovandi and Pettitt, 2013). This prior distribution can incorporate information or data from previous experimental studies, expert-elicited data, and/or subjective belief of the experimenters (Ryan et al., 2016;Drovandi and Pettitt, 2013). Thus, Bayesian optimal designs are designs that account for pre-existing knowledge of uncertainty in parameter estimates. Following the decision-theoretical approach presented by Lindley (1972), it involves the use of a design criterion or utility function u(d, x, θ), where x is the data to be observed when the model parameters are θ, and d is the experimental design to be applied.
The utility function u(d, x, θ) is of particular interest here because it describes the "worth" (based on the experimental aims) of choosing the proposed experimental design d yielding data y from a model with parameters θ (Ryan et al., 2016). This "worth" is usually focused on information gain rather than management outcomes, but the mathematical formulae used in optimal design do not require this to be the case. We will here make use of the broad range of what types of "worth" the utility function could capture, to establish the mathematical link between VoI and optimal design.
The expected utility u(d) of applying design d can be calculated from u(d, x, θ). This calculation typically requires an expectation that is taken over both the prior distribution p(θ) and the future data x. It is given by where p(x|d, θ) is the likelihood function of the observed data given that the design d is applied. The design d * that maximises the expected utility function u(d) over all proposed experimental designs d is given by: Thus, the optimal design d * would be recommended as the experimental design that should be selected.
If, however, the utility function already accounts for all possible states θ of the system (see Drovandi and Pettitt 2013, for example), then this utility function can be written as u(d, x), and Equation (12) simplifies to Recall that, in Section 2.2.2, we found that the traditional VoI metric for an imperfect measurement, EVSI, is equal to the expectation over all measurement outcomes x of the VSI, a new metric that we introduce in the present work. Comparison of this finding to Equations (13) and (14) reveals that traditional VoI calculations are a specific case of Bayesian optimal design, where (1) there is one design d representing the proposed measurement being considered in VoI analysis; (2) the model parameters θ in optimal design are equivalent to the system state s in VoI analysis; and (3) the utility function u(d, x) is chosen to be equal to the expected VSI given a particular measurement outcome x: Hence, there is a clear mathematical link between VoI and optimal design that is here quantifiable after defining VSI (Equation (9)). (Note that we could have equally chosen The link between VoI and optimal design described here also permits interesting extensions for common VoI calculations. For example, consider the situation in which there are multiple different experiments being proposed that each yield imperfect measurements. If these experiments are compared individually for their effects on decision-making, choosing the experiment with the highest EVSI is equivalent in optimal design to assigning each proposed experiment a different design d in Equation (13), and thereafter choosing the utility function u(d, x) to be equal to VSI x (d) calculated for the associated experimental design d and measurement outcome x. The second case study in the present paper provides an example of this extension (Section 3.2).

Case studies
Here we use two case studies to demonstrate the potential utility of the proposed new VoI metrics. The first case study is rather simplified and informs management actions to protect a threatened frog population from disease, whilst the second case study is more complicated and informs management actions to increase the survival rate of turtles released from captivity.
Both case studies were analysed in  for the traditional VoI metrics of EVPI and EVSI. We restrict our attention to the realistic case of imperfect measurements, so focus on comparing our new metrics to the corresponding EVSI values. For each of the case studies (Sections 3.1 and 3.2), we provide a brief description of the environmental management problem being addressed, summarise the traditional VoI metric findings, and thereafter explain how our four new metrics (∆EV x , VSI x , σVSI, and rVSI δ ) complement EVSI to provide a comprehensive overview of the potential value of a measurement being considered by a decision-maker to inform their actions.
3.1. Case study 1: Protecting frogs from disease outbreaks 3.1.1. Problem description The first case study concerns the risk of disease outbreak in the management of a threatened frog species . The frog species has a total population of 100 individuals, all of which are located at a disease-free site within a protected area. The decision-maker's goal is to maximise the total frog population within the protected area. One potential action to achieve this goal is to establish a 'new' population of the frog species by translocating some individuals to a new unoccupied site that is also within the protected area. The frog species management needs to be considered in light of two possible system states: the disease is present at the new site (state s 1 ) or absent at the new site (state s 2 ). The decisionmaker currently believes there is a 50% chance that the disease is present at the new site, i.e. p(s 1 ) = p(s 2 ) = 0.5. For simplicity, there are two possible actions being considered: translocate 50 of the 100 frogs to the new site (action a 1 ), or do nothing (action a 2 ). If nothing is done, the total frog population will remain as 100 individuals because the disease is not present at the site where the frog population currently resides. If the frogs are translocated and the new site is disease-free, the total population is expected to increase to 135 frogs. If the frogs are translocated and the disease is present at the new site, the total population is expected to reduce to 55 frogs.  If the test is negative for disease at the new site, which has a 60.5% chance of occurring, the management action will change -from do nothing (action a 2 ) to translocation (action a 1 ) -and the measurement has an expected value of 17.2 frogs (VSI x = 17.2). In this case, the expected frog population after translocation will be 117.2 individuals, so there is an increase in the expected frog population after measurement (∆EV x = 17.2). On the other hand, if the test is positive for disease at the new site, which has a 39.5% chance of occurring, the management action will not change -no frogs will be translocated (action a 2 ) -so the measurement has no value (VSI x = 0). In addition, the frog population will remain as 100 individuals, so there is no expected change in the total frog population (∆EV x = 0). If this outcome is described in the context of risk aversion, there is a 39.5% chance that the measurement yields no value (rVSI 0 = 0.395). This finding could be worrying for a riskaverse decision-maker because it is a reasonably high probability of a low-value measurement outcome. However, this finding is hidden when reporting only the traditional VoI metrics.
This simple example demonstrates that there is utility in reporting the new VoI metrics proposed here alongside traditional VoI metrics when a decision-maker is choosing whether to perform an additional measurement (or not) prior to action. We also note that, in this case study, ∆EV x = VSI x for all measurement outcomes x (Table 2); this possibility was noted in Section 2.2.2. In the second, more complicated, case study we will show that this is not true in general.
3.2. Case study 2: Improving survival of turtles released from captivity

Problem description
The second case study concerns a species conservation project that wishes to reintroduce a population of captive-bred turtles into the wild . The recommended age range of released turtles is between 3 and 5 years old inclusive. Prior knowledge suggests that post-release survival of turtles may be age-dependent; there are three pre-existing hypotheses for how the survival of turtles depends on their age at time of release from captivity. Briefly, these three hypotheses are that there is no negative post-release effect of reintroduction on turtle survival (state s 1 ), the post-release effect decreases with turtle age (state s 2 ), or the post-release effect increases with turtle age (state s 3 ). The three potential management actions that the decision-maker is considering to implement are to release a very large number of captive turtles when they are three years old (action a 1 ), four years old (action a 2 ) or five years old (action a 3 ). The decision-maker wishes to maximise the survival probabilities of wild turtles at six years of age, as a proxy for long-term survival of wild turtles: how these probabilities depend on the system state s and chosen management action a are described in   (Table S3.2). For each trial release, there are eleven potential measurement outcomes -between zero and ten turtles inclusive will survive at the end of one year (i.e. x ∈ {0, 1, 2, ..., 9, 10}). A binomial probability distribution can be used on a single turtle's survival rate (Table S3.2) to calculate the probability of each of the eleven potential measurement outcomes x for each of the three proposed experiments.

Outcomes from the traditional VoI metrics
We here briefly summarise the relevant outcomes from traditional VoI metrics for this turtle reintroduction case study. First, if the decision-maker is risk-neutral and does not perform any trial releases prior to choosing a management action, releasing a large number of 4-year old turtles is the best action. This will yield an expected survival probability for turtles at 6 years of age of 62.0% (according to the metric EV uncertainty ). Second, doing any of the three proposed trial releases will improve the expected survival probability for turtles at 6 years of age, by differing amounts. If the trial release is performed with 3-, 4-or 5-year old turtles, the expected survival probability for turtles at 6 years of age increases to 62.4%, 64.1% or 65.4% respectively (EV less uncertainty calculated for each experimental design). Thus, the value of each trial release is an increase in expected survival probability for 6-year old turtles of 0.4%, 2.1% and 3.4% respectively (EVSI calculated for each experimental design). The best experiment based on maximum EVSI is therefore a trial release of ten 5-year old turtles.
Note that the choice of which trial to perform, based on which trial has the largest EVSI, is mathematically identical to choosing the design with maximum utility if this were treated as an optimal design problem (Equation (13)); thus this case study highlights the mathematical link between VoI and optimal design described in Section 2.3.

Outcomes from the new VoI metrics
New metrics we introduce that are independent of measurement outcome (σVSI and rVSI δ ) complement the conclusion based on maximum EVSI that the trial release consisting of ten 5-year old turtles in the best choice of the three proposed trials. First, the new metric σVSI demonstrates that the one-year trial of releasing 5-year old turtles is the only experiment of the three that does not have an uncertainty in its value for decisions (σVSI) that is higher than its expected value for decisions (EVSI; see Table 3). This indicates that the range of potential value gains, from the one-year trial of releasing 5-year old turtles, is mostly constrained to positive values, so the decision-maker can be confident that the outcome of this trial will likely be valuable to inform their selection of a management action. Second, the risk of zero information value, evaluated using the risk metric rVSI δ for a threshold VSI of δ = 0, rVSI 0 , is smaller by a factor of 3-10 fold for the one-year trial of releasing 5-year old turtles compared to the other proposed experimental designs (Table 3 and Figure 2).
This indicates that the one-year trial of releasing 5-year old turtles is the best experiment to choose if the decision-maker is risk-averse to zero-value measurement outcomes. and 5-year old turtles (bottom), for the second case study of improving survival of turtles released from captivity (see Section 3.2 for case study details). Bars in blue indicate that the measurement outcome x yields a positive value of information (VSI x > 0) so the management action will change; the darker the shade of blue, the greater the value of the measurement outcome for the action. Bars in light red indicate that the measurement outcome x yields no value (VSI x = 0) so the management action will not change. In this figure the symbol d i , where i = 1, 2, 3, refers to the ith experimental design or strategy d, and it is assumed that the best one design out of d 1 , d 2 or d 3 may be implemented. Notice that the probability of zero information value across all potential measurement outcomes, rVSI 0 = p(VSI x = 0) (sum of heights of all light red bars), is smallest for the one-year trial of releasing 5-year old turtles (design d 3 , bottom plot).

Experimental design d VoI metrics
EVSI ± σVSI rVSI 0 One-year trial release of 3-year old turtles (d 1 ) 0.0042 ± 0.0179 66.4% One-year trial release of 4-year old turtles (d 2 ) 0.0209 ± 0.0259 18.4% One-year trial release of 5-year old turtles (d 3 ) 0.0342 ± 0.0239 6.3% If the decision-maker is unsure of whether they should perform the one-year trial of releasing 5-year olds or not perform any trial at all, the new metrics we introduce are useful here as well. The obtained value of rVSI 0 = 6.3% (Table 3) indicates that the one-year trial of releasing 5-year old turtles has a 93.7% chance of changing the management action chosen by the decision-maker. In other words, this one-year trial has a 93.7% chance of improving the design of the large-scale turtle reintroduction. Because this means that the trial has a 93.7% chance of increasing the expected long-term survival of wild turtles, the additional metric of rVSI 0 here may be a critical piece of information that convinces the decision-maker of whether to perform the trial or not.
Next, using the measurement outcome-dependent metric VSI x , we find that only one outcome of the trial of releasing 5-year old turtles -measurement of seven surviving turtlesyields zero information gain for the purpose of improving decisions (light red bar in bottom panel of Figure 2). Thus, this measurement outcome of the trial, which has a probability of 6.3% (Table 4), is solely responsible for the non-zero value of the risk metric rVSI 0 for the trial of releasing 5-year old turtles. Similarly, only one outcome of the trial of releasing 4-year old turtles -measurement of eight surviving turtles -yields zero information gain (light red bar in middle panel of Figure 2), although this outcome has a higher probability of 18.4% (Tables 3 and S3.6) compared to the equivalent zero-value measurement outcome of the trial of releasing 5-year old turtles. On the other hand, the trial of releasing 3-year old turtles will yield zero information gain for the purpose of improving decisions if the outcome of the trial is 7, 8 or 9 surviving turtles (light red bars in top panel of Figure 2); these outcomes possess probabilities of 21.1%, 24.6% and 20.8% (Table S3.4), together yielding a risk of zero information gain of 66.4% (Table 3). Hence, the metric VSI x explains how the risk of lowvalue measurement outcomes is spread across the different potential measurement outcomes of a proposed experiment. For a decision-maker wanting to avoid low but positive measurement outcomes, the risk of low value of sample information, rVSI δ , may need to be calculated for values of δ larger than zero. Calculation of such risk metrics can be illustrated using Figure 2. Blue and red bars in Figure 2 indicate measurement outcomes x that yield positive (VSI x > 0) and zero (VSI x = 0) values of information, respectively, with the value of VSI x stated above each bar.
For example, if the decision-maker wants the trial's value to be an expected increase in longterm survival rate of wild turtles of more than 5%, the probabilities p(x) (y-axis) associated with measurement outcomes x (x -axis) that yield VSI values that are less than or equal to 0.05 can be added together to calculate the risk that this goal is not met.  Table 4); which together add to 0.66. Thus, the probability of the trial yielding an expected increase in survival rate of released turtles that is less than 5%, is 66% (mathematically, rVSI 0.05 = 0.66). Based on this information, a decision-maker that is risk-averse to such low-value measurement outcomes might reconsider whether or not they should perform the one-year trial release of 5-year old turtles.
Finally, this case study provides an example of when the metrics ∆EV x and VSI x are not equal to each other (Table 4). Such a situation can arise, for example, when new information gained from an experiment causes the estimates of the highest achievable expected value of the system to be downgraded (i.e. negative ∆EV x ) whilst simultaneously causing a better management action to be chosen (i.e. positive VSI x ).
For the trial release of 5-year old turtles, measurement outcomes of less than seven out of ten turtles surviving in the wild for one year, possess negative ∆EV x and positive VSI x (Table   4). When these measurement outcomes are obtained, the hypothesis of reduced post-release survival of older turtles is given greater weight (i.e. system state s 3 , see associated upwardpointing arrows in Table S3.3) compared to the other two hypotheses about the system. The expected long-term survival rate of wild turtles is therefore downgraded (negative ∆EV x ) because the now more strongly weighted hypothesis about the system (s 3 ) is, overall, more pessimistic about long-term wild turtle survival rates than the other two hypotheses (Table   S3.1). However, a potentially better management action will be chosen with this improved knowledge gained from the trial. According to the hypothesis of reduced post-release survival of older turtles (system state s 3 ), the long-term survival of wild turtles is greatest if they are released from captivity at the age of 3 years old (Table S3.

New and traditional VoI metrics together yield a comprehensive view of measurement value
Environmental decision-making, whether it is about choosing to monitor and/or to act, always carries some element of risk (Bennett et al., 2018); this is especially true when the consequences of an incorrect decision are ecologically catastrophic (Martin et al., 2012).
Gathering new information can therefore be a risky activity (Freixas and Kihlström, 1984), and the relationship between risk aversion and the desire of a decision-maker to measure before choosing an action can be unavoidably complicated (Bleichrodt et al., 2020). The present work does not intend to resolve the complexities of the human element of decisionmaking; rather we show how additional metrics which complement the traditional metric of VoI for an imperfect measurement (EVSI) can explicitly describe the risk and uncertainty of such a proposed measurement. The new metrics can also complement the traditional metric of VoI for a perfect measurement, EVPI, via a mathematical simplification described at the start of Section 2.2. Hence the approach to VoI analysis presented in this paper yields a comprehensive set of information for the decision-maker to determine the next step forward to best manage their system, regardless of both the monitoring activities that are available and the cognitive processes that are invoked in the decision-making process.
First, the metric σVSI describes the uncertainty in the value of information gained from a measurement, so provides an excellent complement to EVSI. We suggest that these two metrics (traditional and new) can be reported together as EVSI±σVSI (Table 3). Second, the metric rVSI δ expresses the risk that the proposed measurement will yield an information value that is viewed as not sufficiently useful for decisions (quantified as VSI less than or equal to a threshold value δ). In the specific case of δ = 0 this risk metric can also be interpreted as the risk that the information gained from the proposed measurement does not change the decision that would otherwise have been made if the measurement was not performed.
Hence, in absence of additional knowledge about the decision-maker's preferences, δ = 0 is an obvious default choice; however, if the decision-maker's tolerance to low-value measurement outcomes is known, a specific value of δ ≥ 0 can instead be used. Finally, the metrics ∆EV x and VSI x , which are both dependent on potential outcomes x of the proposed measurement, disentangle how the proposed measurement may improve the estimates of the maximum achievable expected value of the system, and improve decision-making about this system, respectively.
Whilst the traditional metrics of VoI are expectations over measurement outcomes and therefore inherently assume risk-neutrality of the decision-maker with regards to measurement outcomes (Yokota and Thompson, 2004), our additional metrics can account for the risk tolerance of the decision-maker to low-value measurement outcomes. Acknowledging such risks, by explicit calculation of their probability of occurrence via the metric rVSI δ , may be sufficient to help a risk-averse decision-maker to choose whether to proceed with a measurement or not. For example, in the first case study presented here, the traditional metric EVSI indicated that the proposed measurement (a test for disease at a potential frog translocation site) will yield an average saving of approximately 10 additional frogs, whilst the new metric rVSI 0 highlighted that there is a 40% chance that the measurement is val-ueless (Section 3.1). On the other hand, in the second case study, the traditional metric EVSI indicated that the most valuable proposed measurement (a one-year trial release of ten 5-year old turtles) will yield an increase in the expected long-term survival probability for wild turtles of 3%, whilst rVSI 0 indicates that there is only a 6% chance that the measurement is valueless (Table 4). In both case studies, the complementary metrics (traditional and new) provide different and potentially pivotal perspectives on the measurement's value to the decision-maker. In any case, it should be noted that VoI analysis by itself cannot influence potential measurement outcomes; however, the probability of specific measurement outcomes can be forecasted (using e.g. Bayes' theorem and conditional probability formulae), and VoI analysis can be subsequently used to forecast the flow-on effects of these potential outcomes on decision-making (Table 4).

The new VoI metrics are informative for addressing other risks of measurement
VoI analysis, when including the new metrics discussed here, can address other potential concerns of the decision-maker beyond the risk of low-value measurement outcomes, as follows. First, if the decision-maker is considering to perform the measurement only because they desire more evidence that their currently proposed action (i.e. best action under uncertainty) is the best, the metric rVSI 0 can instead be interpreted as the probability that the measurement will confirm that the decision-maker is choosing the best action. This is a novel application of rVSI 0 that may be useful for a cautious decision-maker. Second, if the decision-maker is risk-averse to undesirable system outcomes, then the value function V (a, s) used in VoI analysis can be scaled by an appropriately-chosen monotonic function of the desired system value (Hilton, 1981). In this second case, the new metrics can still be used, thus extending VoI analysis to address risk aversion to low-value measurement outcomes and low-value system outcomes simultaneously. However, care would need to be taken when calculating and reporting values of both the traditional (EVSI) and new metrics (σVSI, rVSI δ , VSI x and ∆EV x ) to ensure that their outcomes are not misinterpreted due to the rescaling of V (a, s) required for addressing risk aversion to low-value system outcomes.
In the case studies discussed in the present work, VoI analysis was considered for static problems in which there is one set of management actions to choose from at a particular time, and potential data collection activities that could inform the choice of actions at that time point only (Figure 1). In this context, static VoI analysis is particularly helpful as a fast prototyping tool when measurements are identified as valueless, since actions can then be chosen without collection of additional information. If, in contrast, the VoI analysis identifies measurements as having sufficient potential value, static VoI analysis may itself be insufficient for decision-making if there is a long term management goal for the system of interest. This issue has motivated VoI analysis approaches in a more dynamic setting, by analysing a sequence of (potential future) management actions over time that is affected by corresponding (potential future) changes in the environmental system, altogether as one large VoI problem (Xiao et al., 2019). The latter approach is able to yield a comparison of different adaptive management strategies (Chadès et al., 2017). We acknowledge that the new metrics we introduce here were designed in the context of static VoI problems; whether these metrics are directly transferable to dynamic VoI problems may be worthy of consideration in future research.

Consequences of the mathematical link between VoI analysis and optimal design
Both VoI and optimal design are decision-theoretic approaches for estimating the expected benefits of collecting further information about a system (Jackson et al., 2022). The metric VSI x introduced in the present work permits a clear mathematical link to be made between VoI and optimal design (Section 2.3). Specifically, VoI is a special case of optimal design in which the utility function being maximised is the value of sample information associated with the proposed measurement (Equation (15)). This means that techniques developed in the field of optimal design (Ryan et al., 2016) may find new applications in VoI analysis; these connections remain unexplored but potentially ripe for future work.
Whilst VoI specifies the value of a proposed measurement for the improvement in decisionmaking, optimal design may not always be used for this specific purpose. For example, a common utility function chosen in optimal design is based on the precision in knowledge of the system after the measurement (e.g. Drovandi and Pettitt 2013). In this example, optimal design specifies the value of the proposed measurement for the improvement in knowledge of the system, which is a related but not identical goal to improvement in decision-making.
These differences in objective have led some authors to argue that VoI and optimal design have unavoidably different goals (McDonald-Madden et al., 2010).
There may indeed be a linguistic resolution to this issue. Notice that VoI analysis aims to quantify the value of a measurement for improved decision-making (via an uncommon utility function within the optimal design framework), but optimal design (with a commonly chosen utility function) aims to quantify the value of a measurement for information gain. Hence, it may be that, instead of treating this distinction as VoI versus optimal design, one should instead treat this distinction as VoI for improved decision-making versus VoI for improved knowledge. If this linguistic change is made, both approaches may be referred to as optimal design and VoI simultaneously -with the chosen utility function determining whether the goal of the design aims specifically to improve decision-making or improve knowledge. This resolution is most suitable for the case of static problems (see Section 4.2) in which there is one time at which the measurement may occur and one (later) time at which the management action may occur (Figure 1). For dynamic problems that require sequential decision-making, mathematical techniques for adaptive management are likely to be more appropriate (Chadès et al., 2017).
For both case studies considered here, calculation of VoI was computationally inexpensive. However, it is well-known in the optimal design literature that a potential challenge in the application of optimal design of experiments (and therefore also in VoI analysis) is the computational feasibility for large and complex problems such as the representation of real and complicated ecosystems. Because the expected utility function for a design must often be estimated using Monte Carlo integration methods, computationally intensive stochastic optimisation schemes, such as simulated annealing (Kirkpatrick et al., 1983) or approximate coordinate exchange (Overstall and Woods, 2017) are required. While improving the computational efficiency of these schemes is still an ongoing area of research, there are some common approaches to approximate utility functions including variational inference (Bei et al., 2017), integrated nested Laplace approximation (Rue et al., 2009) and generalised estimating equations (Woods and van de Ven, 2011;Gill et al., 2022). Recent advances in variance reduction (Jacob et al., 2020), multilevel Monte Carlo (Giles, 2015;Jasra et al., 2019;Warne et al., 2022) and deep reinforcement learning (Blau et al., 2022) can dramatically improve the performance of complex Monte Carlo integrals; such methods have the potential to enable optimal design and VoI to scale to complex conservation projects.

Good decisions, fast decisions
Since decisions about environmental systems sometimes must be made rapidly (Martin et al., 2012), there may be both limitations to what information can be gathered to inform a VoI analysis, and time pressures on communication to decision-makers prior to an action being chosen. First, VoI analysis requires sufficient information to produce forecasts of system value in response to different management actions and measurement outcomes, and thus like other quantitative ecological forecasts (Adams et al., 2020), VoI analysis is data-hungry. If such information is available, VoI can also prioritise which components of data-hungry forecasting models should be improved (Strong and Oakley, 2014). When data are more scarce, qualitative approaches can instead be used (Clark-Wolf et al., 2022); such approaches are currently being developed for VoI (Rushing et al., 2020;Lawson et al., 2022;Stantial et al., 2023).
Qualitative approaches for VoI should ideally be later replaced by quantitative approaches, once more data becomes available. Second, the desire to simplify outputs of environmental analysis to one or a few indicators (e.g. Wibeck et al. 2006;Gibari et al. 2019) may favour, in the context of VoI, strategic decisions about which metrics should be reported. This goal must be balanced by the increased transparency gained from commensurate reporting of uncertainty and risk (Chess et al., 2005;Burgass et al., 2017) for environmental metrics that directly inform decision-making.  (7), the expectation of ∆EV x over all possible measurements x is: which is the definition of EVSI in Equation (6). S1.2. The expectation of VSI x over all measurement outcomes yields EVSI In Section 2.2.2 of the main text, we stated that EVSI = E x [VSI x ]. Here we show this derivation. From the definition of VSI x in Equation (9), the expectation of VSI x over all possible measurements x is: (by the law of total expectation) which is the definition of EVSI in Equation (6).
S2. Further details for Case study 1

S2.1. Further details for calculation of traditional VoI metrics for Case study 1
In Section 3.1.2 of the main text, we summarised the outputs of traditional VoI calculations for the first case study of the manuscript (protecting frogs from disease outbreaks). Here we explain how these outputs are calculated from application of Equations (1), (2), (5) and (6) to the case study information provided in Section 3.1.1. The description of these calculations here is similar to that presented in ; however, the present description is also designed to assist with understanding the calculation of the new metrics introduced in the present work (detailed in Section S2.2).
First, it is useful to tabulate the system's values V (a, s) for each system state s ∈ {s 1 , s 2 } and chosen management action a ∈ {a 1 , a 2 }, as well as the prior probabilities p(s) for each state s (Table S2.1). All of this information is drawn from the text of Section 3.1.1. Prior probabilities p(s) 0.5 0.5 From application of Equation (1) to the information shown in Table S2.1, the expected values of the system prior to any measurement, EV(a 1 ) and EV(a 2 ), if action a 1 (translocate) or action a 2 (do nothing) is chosen respectively, are: Combining Equations (2), (S2.1) and (S2.2), the expected value of the system if the best management action is carried out prior to any measurement, EV uncertainty , is: Therefore, if no measurement is carried out, doing nothing (action a 2 ) is the best action, and this yields an expected total frog population of 100 individuals.
The proposed measurement, which is a test for disease at the new site, has a sensitivity of 0.73 and specificity of 0.94 (Section 3.1.1). If x 1 and x 2 denote test outcomes that are positive and negative for the disease at the new site, respectively, the information about test sensitivity and specificity can be mathematically represented as: discrete states s i and discrete measurement outcomes x j is: This information is represented graphically in the right side of Figure S2.   (7)- (11) to the case study information provided in Section 3.1.1. We also here make use of quantities already computed in Section S2.1 for the present case study.
Combining Equations (7), (S2.3), (S2.7) and (S2.8), the shifts in expected value of the system, ∆EV x 1 and ∆EV x 2 , if measurement outcome x 1 (test positive for disease) or x 2 (test negative for disease) are obtained respectively, are:  Table S2.1 and Equation (S2.6): = V (a 2 , s 1 ) p(s 1 |x 1 ) + V (a 2 , s 2 ) p(s 2 |x 1 ) = 100 × 0.924 + 100 × 0.076 = 100, (S2.13) = V (a 2 , s 1 ) p(s 1 |x 2 ) + V (a 2 , s 2 ) p(s 2 |x 2 ) = 100 × 0.223 + 100 × 0.777 = 100. (S2.14) Then, combining Equations (9), (S2.7), (S2.8), (S2.13) and (S2.14), the values of sample information VSI x 1 and VSI x 2 dependent on measurement outcomes x 1 (test positive for disease) and x 2 (test negative for disease) respectively, are:    Table S3.3: Probabilities of system state dependent on measurement outcome, p(x|s), of the third proposed experimental design (d 3 : release 5-year old turtles for one year) in the second case study of improving survival of turtles released from captivity (see Section 3.2 for case study details). Upward-pointing arrows (↑) and downward-pointing arrows (↓) highlight the increase or decrease, respectively, in the probability of the associated system state s due to measurement outcome x, compared with the prior belief of this system state: p(s 1 ) = 0.4, p(s 2 ) = 0.2 and p(s 3 ) = 0.4, as shown in Table S3.1.   Table S3.5: Probabilities of system state dependent on measurement outcome, p(x|s), of the first proposed experimental design (d 1 : release 3-year old turtles for one year) in the second case study of improving survival of turtles released from captivity (see Section 3.2 for case study details). Upward-pointing arrows (↑) and downward-pointing arrows (↓) highlight the increase or decrease, respectively, in the probability of the associated system state s due to measurement outcome x, compared with the prior belief of this system state: p(s 1 ) = 0.4, p(s 2 ) = 0.2 and p(s 3 ) = 0.4, as shown in    Table S3.7: Probabilities of system state dependent on measurement outcome, p(x|s), of the second proposed experimental design (d 2 : release 4-year old turtles for one year) in the second case study of improving survival of turtles released from captivity (see Section 3.2 for case study details). Upward-pointing arrows (↑) and downward-pointing arrows (↓) highlight the increase or decrease, respectively, in the probability of the associated system state s due to measurement outcome x, compared with the prior belief of this system state: p(s 1 ) = 0.4, p(s 2 ) = 0.2 and p(s 3 ) = 0.4, as shown in