Flexible forecast value metric suitable for a wide range of decisions: application using probabilistic subseasonal streamflow forecasts

. Forecasts have the potential to improve decision-making but have not been widely evaluated because current forecast value methods have critical limitations. The ubiquitous Relative Economic Value (REV) metric is limited to binary 10 decisions, cost-loss economic model, and risk neutral decision-makers. Expected Utility Theory can flexibly model more real-world decisions, but its application in forecasting has been limited and the findings are difficult to compare with those from REV. A new metric, Relative Utility Value (RUV), is developed using Expected Utility Theory. RUV has the same interpretation as REV which enables a systematic comparison of results, but RUV is more flexible and able to handle a wider range of real-world decisions because all aspects of the decision-context are user-defined. In addition, when specific 15 assumptions are imposed it is shown that REV and RUV are equivalent. We demonstrate the key differences and similarities between the methods with a case study using probabilistic subseasonal streamflow forecasts in a catchment in the southern Murray-Darling Basin of Australia. The ensemble forecasts were more valuable than a reference climatology for all lead-times (max 30 days), decision types (binary, multi-categorical, and continuous-flow), and levels of risk aversion for most decision-makers. Beyond the second week however, decision-makers who were highly exposed to damages should use the 20 reference climatology for the binary decision, and forecasts for the multi-categorical and continuous-flow decision. Risk aversion impact was governed by the relationship between the decision thresholds and the damage function, leading to a mixed impact across the different decision-types. The generality of RUV makes it applicable to any domain where forecast information is used for making decisions, and the flexibility enables forecast assessment tailored to specific decisions and decision-makers. It complements forecast verification and enables assessment of forecast systems through the lens of 25 customer impact.

Derivation of relative economic value (REV) 15 This section derives the standard relative economic value (REV) metric using information in the following contingency table (Richardson, 2000). Table S1: Contingency table for the cost-loss decision problem with expenses from each possible outcome. Here C is the cost of the mitigating action, u L is the unavoidable portion of loss L from the event occurring, and a L is avoidable portion of loss from 20 the action.

Event occurred
Event did not occur The expected long run expense for decisions based on forecast information depends on the rate at which each outcome occurred over a historical period. The net expense of that outcome: outcomes Average net expense rate of outcome net expense of outcome = × ∑ (S1) 25 This can be quantified by substituting the terms from Table S1 into Eq. (S1) and noting that the rate and net expense of some elements will be zero. For forecast information this leads to A user with access solely to climatological historical average information will take protective action either always or never.
It is assumed that the user will choose the action that leads to the smallest net expense according to the climatological 30 A user with access solely to climatological information will act when a C L o < . That is, a user will either always act or never act depending on whether their particular a C L is smaller or larger than the event frequency.
If perfect information is available, then the outcome will always be either a hit or correct rejection: 35 The Relative Economic Value (REV) metric can be constructed comparing the relative difference between the forecast and perfect information relative to the climatological baseline.
Noting the following parameter which is known as the cost-loss ratio.
Substituting the cost-loss ratio into Eq. (S7) and dividing by a L leads to the following standard equation for REV (Zhu et al., 2002).
50 Derivation of equivalence of relative economic value (REV) and relative utility value (RUV) This section demonstrates the equivalence of the REV metric as detailed in Eq. (S9) and the RUV metric when 5 assumptions are applied to the decision context. We first define the general decision context using expected utility theory, the RUV metric, and then introduce the specific assumptions required to constrain RUV to REV and prove equivalence.
Thereby demonstrating that the REV metric can be considered a special case of the more general RUV metric. 55

Expected utility framework with cost-loss economic model
The von Neumann-Morgenstern expected utility for a single timestep over M states.
where m t p is the probability of state m occurring in timestep t and m t E is the outcome associated with that state. The outcome is typically but not necessarily in monetary units. 60 For a state of the world m at a specific timestep t , with damages ( ) d m , cost to mitigate the damages t C , and amount of damages avoided ( ) t b m , the outcome is given by The benefit function ( ) b m specifies the damages avoided from taking action to mitigate them, The optimal amount t C π to spend at timestep t can be found by maximising the expected utility ( ) This leads to the following expression for the ex post utility after substitutions Eq. (S11), (S12), and (S13) into Eq. (S10) is the ex post utility, t C π is the spend amount that was found ex ante, o t m is the state of the world associated with the observed flow at timestep t .

Relative utility value (RUV)
RUV is defined as follows is the expected value of the ex post expected utility from Eq. (S14) over a set of observations and either forecast ( f ), reference climatology ( r ), or perfect information ( p ).

Equivalence of REV and RUV with specific assumptions
In a cost-loss decision problem the two relevant states are "flow above" and "flow below" a decision threshold d Q . 80 Assumption 1: A step damage function with binary values of 0 and L is used to specify the losses above and below the decision threshold, To calculate the net outcome when action is taken to mitigate the loss, we substitute Eq. (S12) and (S17) into Eq. (S11) and 85 make a change of notation, which leads to the following net outcomes for the states above and below.

( )
, , min , Assumption 2: Linear utility function is assumed which implies no aversion to risk, Substituting Eq. (S18) into Eq. (S10), applying the linear utility function assumption in Eq. (S19), and simplifying for only 90 two states of the world using p the forecast probability of flow above the flow threshold leads to.
Assumption 3: Probability of flow above the threshold will always be either 1 or 0, Assumption 3 is required because the core REV formulation is only valid for categorical forecasts. When REV is used with 95 probabilistic forecasts they are quantified by converting them to categorical forecasts using the threshold-approach with a threshold which maximises REV for each value of α . We can now determine the single timestep ex ante utility for the four possible outcomes; forecast probability is 1 or 0, and an action has been taken or not. The derivation for the case where 1 p = and 0 C ≠ is as follows. Consider the avoided losses to be.
Substituting this into Eq. (S20) and noting Simplifying Eq. (S20) for the other 3 outcomes and presenting all in a tabular form leads to the following table of ex ante utility value. 105 Note that the elements of Table S2 are identical but negative to Table S1 used in the derivation of the REV. Applying Eq.
(S13) to Eq. (S20) will lead to an optimal amount t C π to spend on the mitigating action for each timestep. By considering that the probability is always either 1 or 0 due to assumption 3 and that all costs and losses are positive values we can 110 formulate t C π as follows.
Therefore, for any timestep the cost will be either 0 The ex post utility for each timestep, shown in Table S3, can be found by substituting these optimal costs back into the elements of Table S2, and letting the probability be conditioned on the state of observed flow above the threshold according to the following.
Like the derivation of REV, a contingency table is used since every timestep can be mapped to one of the 4 outcomes in Table S2. 125 Determining the expected ex post utility for the reference climatology (event frequency) information requires an additional assumption.

Assumption 4:
The frequency of the binary decision event o is used for the reference baseline.
Since the reference climatology is a single value, the decision-maker will either always take action or never take action.
Expected ex post utility for perfect information is 135 Expected ex post utility for forecast information is where h is the hit rate, m is the miss rate, f is the false alarm rate, and q is the correct rejection (quiets) rate from the contingency