Comparing Robust Decision-Making and Dynamic Adaptive Policy Pathways for model-based decision support under deep uncertainty

A variety of model-based approaches for supporting decision-making under deep uncertainty have been suggested, but they are rarely compared and contrasted. In this paper, we compare Robust Decision-Making with Dynamic Adaptive Policy Pathways. We apply both to a hypothetical case inspired by a river reach in the Rhine Delta of the Netherlands, and compare them with respect to the required tooling, the resulting decision relevant insights, and the resulting plans. The results indicate that the two approaches are complementary. Robust Decision-Making offers insights into conditions under which problems occur, and makes trade-offs transparent. The Dynamic Adaptive Policy Pathways approach emphasizes dynamic adaptation over time, and thus offers a natural way for handling the vulnerabilities identi ﬁ ed through Robust Decision-Making. The application also makes clear that the analytical process of Robust Decision-Making is path-dependent and open ended: an analyst has to make many choices, for which Robust Decision-Making offers no direct guidance. © 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).


Introduction
Uncertain changes in climate, technological, socio-economic and political situations, and the dynamic interaction among these changes, and between these changes and interventions, pose a challenge to planners and decision-makers.Due to these uncertainties, there is a risk of making an inappropriate decision (too little, too much, too soon, or too late).There is a need for approaches that assist planners and decision-makers with making long-term plans and informed policy decisions under deep uncertainty.Weaver et al. (2013) argue that exploratory model-based approaches are highly suitable for supporting planning and decision-making under deep uncertainty.In exploratory modeling, modelers account for the various unresolvable uncertain factors by conducting series of computational experiments that systematically explore the consequences of alternative sets of assumptions pertaining to the various deeply uncertain factors (Bankes, 1993;Bankes et al., 2013).A literature is emerging that adopts this exploratory modeling approach in support of decision-making under deep uncertainty (e.g.Auping et al., 2015;Bryant and Lempert, 2010;Dalal et al., 2013;Groves et al., 2014;Groves and Lempert, 2007;Hadka et al., 2015;Halim et al., 2016;Hall et al., 2012;Hallegatte et al., 2012;Herman et al., 2015;Kasprzyk et al., 2013;Kwakkel et al., 2013Kwakkel et al., , 2015;;Kwakkel andPruyt, 2013, 2015;Kwakkel et al., 2012;Lempert, 2002Lempert, , 2003;;Lempert and Collins, 2007;Lempert and Groves, 2010;Maier et al., 2016;Matrosov et al., 2013aMatrosov et al., , 2013b;;Parker et al., 2015;Pruyt and Kwakkel, 2014;Thissen et al., 2016).A substantial fraction of this literature focuses on model-based decision support for environmental systems undergoing change.
Over the last decade, climate adaptation research has increasingly focused on supporting decision-makers in developing climate adaptation strategies 1 and understanding the tradeoffs among different climate adaptation options (Maru and Stafford Smith, 2014).This research focus represents a shift from a focus on understanding climate change impacts to a solution-oriented focus on supporting climate adaptation decision-making through iterative risk management.Within the broader literature on decisionoriented climate adaptation, one strand of research has a strong analytical focus on designing effective climate adaptation strategies in the presence of a wide variety of presently irresolvable deep uncertainties (Dessai and Hulme, 2007;Dessai et al., 2009;Lempert et al., 2003;Maru and Stafford Smith, 2014;Wise et al., 2014).Because of the presence of unavoidable uncertainty, decisionmakers are advised to look for robust decisions that have satisfactory performance across a large range of plausible futures.One of the key design principles for such robust decisions is to make plans that are flexible and can be adapted over time in response to how the world actually unfolds (Haasnoot et al., 2012;Hallegatte, 2009;Kwakkel et al., 2010;Walker et al., 2013).The acceptance of uncertainty as an inevitable part of long-term decision-making has given rise to the development of new model-based tools and approaches.These include Dynamic Adaptive Policy Pathways (Haasnoot et al., 2013), Adaptive Policy-Making (Kwakkel et al., 2010), Real Options analysis (de Neufville and Scholtes, 2011; Woodward et al., 2014), Info-Gap Decision Theory (Ben Haim, 2006;Korteling et al., 2013), decision scaling (Brown et al., 2012;LeRoy Poff et al., 2015), Robust Decision-Making (Groves and Lempert, 2007;Lempert and Collins, 2007), and Many Objective Robust Decision-Making (Hadka et al., 2015;Herman et al., 2015;Kasprzyk et al., 2013).
The availability of a variety of model-based analytical approaches for designing flexible robust plans raises a new set of questions.How are the various approaches different?Where do they overlap?Where are they complementary?Answering these questions can help to pave the way for the future harmonization and potential integration of these various approaches.It might also help in assessing if certain approaches are more applicable in certain decision-making contexts than others.Hall et al. (2012) compare Info-Gap Decision Theory and Robust Decision-Making.They conclude that along quite different analytical paths, both approaches arrive at fairly similar but not identical results.Matrosov et al. (2013b) also compare Info-Gap and Robust Decision-Making.They reach a similar conclusion and discuss in more detail the complementary character of the analytical paths used by both approaches.Matrosov et al. (2013a) compare Robust Decision-Making with an economic optimization approach (UK Water Industry Research (UKWIR), 2002).In this case, the results are quite different, suggesting a need to combine both approaches.Roach et al. (2015Roach et al. ( , 2016) ) compare Info-Gap Decision Theory and robust optimization.They conclude that there are substantial differences between the plans resulting from these two approaches, and argue in favor of mixed methodologies.Gersonius et al. (2015) compare a real options analysis (in detail reported in Gersonius et al., 2013) with an adaptation tipping point analysis (Kwadijk et al., 2010).They highlight the substantial differences in starting points and suggest that both approaches could be applied simultaneously.
In this paper, we compare the Dynamic Adaptive Policy Pathways (DAPP) approach (Haasnoot et al., 2013) with Robust Decision-Making (RDM) (Groves and Lempert, 2007).The Dynamic Adaptive Policy Pathways approach has not been compared before with any of the other model-based analytical approaches.We choose to compare it with RDM as it has served as a benchmark against which other approaches have been compared.The aim of the comparison is to provide insight into the different analytical paths followed by the two approaches.What information and tools are needed, what decision relevant insights are being generated, and how different is the resulting plan emerging from the application of the two approaches?We compare both approaches using a stylized case, inspired by a river reach in the Rhine Delta of the Netherlands (Haasnoot et al., 2012).
From a conceptual point of view, RDM is an iterative process for developing a robust plan.Robust decision-making provides little guidance on how this robustness is to be achieved, resulting in some claims that RDM is intrinsically static.This claim, however, is at odds with various RDM applications that produce adaptive plans (e.g.Bloom, 2015;Groves et al., 2013Groves et al., , 2014)).To provide guidance in the development of an adaptive plan using RDM, we draw on adaptive policymaking (Hamarat et al., 2013;Kwakkel et al., 2010).In contrast, the DAPP approach primarily emphasizes dynamic adaptation over time and specifies a stepwise approach for developing such plans.This stepwise approach is more open ended with respect to how models can be used in it.To do this, we draw on earlier work on the use of multi-objective robust optimization for the design of adaptation pathways (Kwakkel et al., 2015).
Given this setup, we can already highlight some key differences.Since RDM is an iterative process where one or more candidate plans are stress-tested over a range of uncertainties, the computational costs are primarily dependent on the number of plans that are tested and the number of cases needed to provide reliable insight into their vulnerabilities.In contrast, the multi-objective optimization approach exhaustively explores the design space and is, therefore, computationally more expensive.This implies also that in RDM the design space is not analyzed with the same rigor as in the multi-objective optimization approach.
In Section 2, we introduce both Robust Decision-Making and the Dynamic Adaptive Policy Pathways approach in more detail.In Section 3, we introduce the case to which both approaches are applied.Section 4 contains the Robust Decision-Making application, and Section 5 contains the Dynamic Adaptive Policy Pathways application.We compare the results in Section 6. Section 7 presents the conclusions.

Robust Decision-Making
There are four main steps in RDM, as shown in Fig. 1.The first step is a generic policy analytic decision structuring activity that aims at conceptualizing the system under study, and identifying the key uncertainties pertaining to this system, the main policy options, and the outcomes of interest.This step often involves stakeholder interaction.The second step is case generation.In this step, the behavior of one or more models of the system under study is systematically explored across the identified uncertainties, and the performance of candidate strategies is assessed.The third step is scenario discovery (Bryant and Lempert, 2010).Using statistical machine learning algorithms, the performance of candidate strategies across the generated cases is analyzed to reveal the conditions under which candidate strategies perform poorly.These conditions reveal vulnerabilities of the strategies, in light of which they can be modified.Step two and three together are sometimes also referred to exploratory modeling (Bankes et al., 2013).The fourth step is trade-off analysis, in which the performance of the different strategies is compared across the different outcome indicators, thus providing an additional source of information that can be used in redesigning the strategy.The steps can be iterated until a satisficing robust strategy emerges.
Scenario discovery forms the analytical core of RDM (Bryant and Lempert, 2010;Groves and Lempert, 2007).The main statistical rule induction algorithm that is used for scenario discovery is the Patient Rule Induction Method (PRIM) (Friedman and Fisher, 1999).PRIM aims at finding combinations of values for the uncertain input variables that result in similar characteristic values for the outcome variables.Specifically, PRIM seeks a set of subspaces of the uncertainty space within which the value of a single output variable is considerably different from its average value over the entire domain.PRIM describes these subspaces in the form of hyper rectangular boxes of the uncertainty space.In this paper, we use a Python implementation of PRIM, which has been demonstrated to be effective even in case of heterogeneously typed data (Kwakkel and Jaxa-Rozen, 2016).In the context of scenario discovery, the input for the PRIM analysis is the set of cases, or computational experiments.The results of these cases are classified using a binary classification, indicating whether a given case is of interest or not.
To assess the quality of the identified hyper rectangular boxes, analysts can look at the coverage and density of a given box.Coverage is the fraction of all the cases that are of interest that fall within the box.Density is the fraction of cases within the box that are of interest (Bryant and Lempert, 2010).In addition, it is customary to look at the quasi-p values for each of the dimensions of the hyper-rectangular box, which indicate whether the restriction imposed by the box on each of the dimensions is statistically significant.This value is calculated using a one-sided binomial test, assuming a 95% confidence interval (see Bryant and Lempert, 2010 for a detailed discussion).
The RDM process describes the steps necessary for the identification of vulnerabilities and tradeoffs.It does not provide guidance on how to address the identified vulnerabilities in the (re) design of a strategy.In many applications of RDM, the results of scenario discovery are used by the analyst, in interaction with stakeholders, to formulate a new strategy that has a reduced sensitivity to the identified vulnerability.The lack of explicit guidance on how to address vulnerabilities identified through scenario discovery might create the impression that RDM results in static strategies (Walker et al., 2013).A more careful review of the RDM literature reveals, however, that RDM can be used to design strategies that are adapted over time in response to how the future unfolds (see e.g.Bloom, 2015;Groves et al., 2013Groves et al., , 2014;;Lempert and Groves, 2010).Adaptivity is created through a signpost and trigger system, where a strategy is modified in a prespecified way in response to a pre-specified trigger.This is in line with ideas found in Assumption-Based Planning (Dewar, 2002;Dewar et al., 1993) and Adaptive Policy-Making (Kwakkel et al., 2010;Walker et al., 2001).While RDM provides clear guidance on the detection of vulnerabilities, it is less clear on how the signposts and triggers are to be specified.The description of this in Groves et al. (2013Groves et al. ( , 2014) ) suggests that signposts are connected to the vulnerabilities, but the specification of triggers is not further clarified.
The adaptive robust policy design advocated by Hamarat et al. (2013) explicitly combines the RDM approach with Adaptive Policy-Making, and offers guidance on when vulnerabilities identified through scenario discovery are better addressed through static action or through dynamic adaptation.If a given uncertain factor is part of all the vulnerabilities identified through scenario discovery, this approach advocates a static solution.Adaptivity is introduced to cope with uncertain factors that are unique to a given vulnerability.Signposts can be derived from this.Where possible, signposts are tied directly to the uncertain factor.A similar approach of tying scenario discovery results to signposts is suggested by Bloom (2015).Triggers can than be specified using expert opinion (Haasnoot et al., 2013;Kwakkel et al., 2012), detailed model-based analyses (Haasnoot et al., 2015), or through robust optimization (Hamarat et al., 2014).In this paper, we follow the suggestions of Hamarat et al. (2013) for supporting the design of adaptive policies using RDM.

Dynamic Adaptive Policy Pathways
The Dynamic Adaptive Policy Pathways (DAPP) approach combines two bodies of literature on planning under uncertainty: work on Adaptive Policy-Making (Kwakkel et al., 2010;Walker et al., 2001); and work on adaptation tipping points (Kwadijk et al., 2010;Offermans, 2012) and policy pathways (Haasnoot et al., 2012).Fig. 2 shows the overall approach.For a more detailed description, see Haasnoot et al. (2013).
The first step in DAPP is to describe the setting, including objectives, constraints, major uncertainties, and a definition of success.Next, vulnerabilities and opportunities of the status quo are assessed by identifying adaptation tipping points: the conditions under which the status quo starts to perform unacceptably for the relevant uncertainties, using expert judgment and/or model simulations.The timing of an adaptation tipping point (use-by date) is derived from linking the use-by conditions with scenarios, or from the changing performance over time resulting from transient or semi-static model simulations.This reveals if and when policy actions are needed to obtain the desired outcomes.
Based on this problem analysis, policy actions are identified to address vulnerabilities and seize opportunities.For these policy options, one also needs to assess the conditions under which they might meaningfully be used, as well as the conditions under which they are insufficient for reaching the desired outcomes.Once the set of policy actions is deemed adequate, pathways can be designed and evaluated.A pathway consists of a concatenation of policy actions, where a new policy action is activated once its predecessor is no longer able to meet the definition of success.
Based on the evaluation of the pathways, a manageable number of preferred pathways can be identified.These preferred pathways can be improved through contingency planning, which requires the specification of 'corrective', 'defensive', and 'capitalizing' actions, and an associated monitoring system with 'trigger values' that would result in the implementation of the actions.In light of the final Adaptation Pathways Map, a plan for action can be made, which specifies the actions to be taken immediately, the developments to monitor, and the conditions under which contingency actions should be taken.For a more detailed discussion of each of the individual steps, see Haasnoot et al. (2013).
Fig. 3 shows a stylized example of an Adaptation Pathways Map.In the map, starting from the current situation, targets begin to be missed after four years.Following the grey lines of the current policy, one can see that there are four options.Actions A and D should be able to achieve the targets for the next 100 years in all climate scenarios.If Action B is chosen after the first four years, a tipping point is reached within about five more years; a shift to one of the other three actions will then be needed to achieve the targets (follow the orange lines).If Action C is chosen after the first four years, a shift to Action A, or D, or a combination of C with B will be needed in the case of a high-end scenarios (follow the green lines).In all other scenarios, the targets will be achieved for the next 100 years.The colors in the scorecard refer to the actions: A (red), B (orange), C (green), and D (blue).

The Waas case
To compare and contrast Robust Decision-Making and the Dynamic Adaptive Policy Pathways approach, we use a hypothetical case, called 'the Waas' (Haasnoot et al., 2012;Kwakkel et al., 2015).The case is based on the Waal, a river reach in the Rhine Delta of the Netherlands.The case study area is shown in Fig. 4. The river and floodplain are highly schematized, but the characteristics are realistic.Embankments bind the river.The floodplain is separated into five dike rings.A large city is situated on higher grounds in the southeast part.Smaller villages exist in the remaining area, including greenhouses, industry, conservation areas, and pastures.In the future, climate change and socio-economic developments may increase the pressure on the available space and the potential damages, so actions would be needed.
The meta model of the Waas was derived from validated models for an area similar to the Waas -the river Waal in the Netherlands.The cause-effect relations are implemented using PCRaster, a gridbased spatial analysis tool for dynamic modeling (van Deursen, 1995).The model was checked for internal consistency and plausibility of the outcomes by expert judgment.In the model, discharges arising from the transient climate scenarios are translated into water levels using discharge rating-curves, representing the relationship between river discharge and water level (stage) at each river kilometer.The water levels are translated into a 2D surface, and are compared with the dike heights derived from the elevation map.Subsequently, the model calculates the probability of dike failure caused by piping or by wave overtopping by examining the difference between dike level and water level (van Velzen, 2008).Whether the dike fails or not depends on a random number selected between 0 and 1.If that number is lower than the probability of dike failure, the dike is assumed to fail, even if the water does not overtop it.In the case of a dike failure, the water level is considered to be equal to the river water level in the whole dike ring.A water depth map is derived from an intersection of the water level (assuming a planar surface) and a digital elevation map, and is corrected for higher lying areas that prevent some lower parts from flooding.Damage due to the flooding of dike rings is calculated from the water depth and damage relations (De Bruijn, 2008;Haasnoot et al., 2011).Using these relations, the model calculates, for each land use, the flood impacts per hectare, by multiplying the maximum potential flood damage in the cell under consideration by the water level-dependent damage factor (value between 0 and 1).This yields the total damage for sectors such as agriculture, industry, and housing.Casualties are assessed using water depth, land use, and flood alarms triggered by the probability of dike failure.The costing of the various options is based on the analysis of each of the options in isolation.This is a simple approach, which overlooks interactions and scale effects that in practice might be relevant.We do not include discounting in either the costs or damages.For more details on the meta model, see Haasnoot et al. (2012).
The analysis takes into account uncertainties related to climate change, land use, system characteristics, and the effects of policy actions (Table 1).The effects of different climate change scenarios are considered through changes in the river discharge (see Haasnoot et al., 2012;Haasnoot et al., 2015 for details).Both climate change and land use uncertainty are included using pre-specified transient scenarios (Haasnoot et al., 2011(Haasnoot et al., , 2015)).Uncertainties in the cause-effect relations for the fragility of dikes and economic damage functions are taken into account by putting a bandwidth of plus and minus ten percent around the default values; for each experiment, we randomly pick a value in this interval and update the default values accordingly.This bandwidth was chosen for illustrative purposes, but was informed by discussion with experts.The three outcomes of interest are the cumulative number of casualties, cumulative flood damage in millions of euros, and cumulative capital costs of the policy options in millions of euros.We use 100 years as the time horizon.
Table 2 provides an overview of the 20 policy options that are explored.The actions include flood prevention measures such as heightening the dikes, strengthening the dikes, and giving room for the river, and flood mitigation actions such as upstream collaboration, evacuation preparation, early alarms for evacuation, additional embankments around cities, and houses on stilts.These options can be combined into packages that are executed simultaneously, or into sequences where one option follows a previous action.

RDM analysis of Waas case
The RDM process for the Waas case begins by exploring the performance of the system in the absence of any policy across 5000 computational experiments.Based on prior experience with the Waas case, 5000 experiments is sufficient for getting a fair understanding of the behavior of the system across the various uncertain factors.These experiments, which cover the uncertainty space, are generated using Latin Hypercube sampling (McKay et al., 1979).The performance of the system in terms of cumulative number of casualties and cumulative flood damage is shown in Fig. 5. Given that there are no policy options, there are no costs.
In order to identify the combination of uncertainties under which the system performs poorly, we used scenario discovery.The correlation between casualties and flood damage is 0.8, so we were able to use either outcome of interest.Here we use flood damage, and classify a scenario as being of interest if x > 50; 000 0; otherwise where x is the cumulative flood damage.The threshold of 50,000 million euros is chosen in light of in Fig. 5.In total, 972 out of the 5000 computational experiments are of interest.Table 3 shows the results from the scenario discovery.We are able to explain 91% of the 972 cases.In 90% of the cases of severe climate change in combination with any additional urbanization, cumulative flood damage will be higher than 50,000 million euros.If the system faces a severe climate change scenario in combination with increasing population density because of ongoing For river discharge and precipitation, 10 realizations for each of three climate change scenarios are considered (Haasnoot et al., 2012;Haasnoot et al., 2015).These three climate change scenarios are: no change, moderate climate change (G), and severe climate change (Wþ) (van den Hurk et al., 2007Hurk et al., , 2008)).The river discharge numbers are based on those for the Rhine river. 1e30 Land use Land use will change in the coming 100 years.Seven alternative land use scenarios are considered: slow or fast urbanization, de-urbanization, sustainable growth, more nature, urbanization followed by de-urbanization, and no change.
Slow urbanization, fast urbanization, de-urbanization, sustainable growth, more nature, urbanization followed by de-urbanization, and no change Fragility of the dikes There is uncertainty about the relationship between the relative difference between the height of a dike and the water level, and the propensity of a dike to fail.A bandwidth around the default fragility relation is considered.This bandwidth is chosen for illustrative purposes.In a real world case, this bandwidth should be grounded in evidence.

À10%e10%
Damage functions There is parametric uncertainty about the damage functions used to calculate the damage from flooding and the damage to shipping.A bandwidth is considered.This bandwidth is chosen for illustrative purposes.In a real world case, this bandwidth should be grounded in evidence.

À10%e10%
Policy Two of the candidate policy actions include upstream collaboration.The effect of this on the discharge is uncertain.A wide bandwidth is taken into account.
Other options include adding or modifying dikes and changing the way we live, which are affected by the aforementioned uncertainty about the fragility of the dikes and the damage functions. 1e1.6 Table 2 Overview of policy options.Strength of the dikes improved with asphalt coverage (instead of grass), resulting in wave-overtopping resistant dikes 9 RfR Side channel Room for the river -With extra side channels, the river is given more space after a threshold discharge is exceeded 10 RfR Small Scale Room for the river -Small scale: with extra side channels, the river is given more space after a threshold discharge is exceeded 11 RfR Medium Scale Room for the river -Medium scale: with extra side channels, the river is given more space after a threshold discharge is exceeded 12 RfR Large Scale Room for the river -Large scale: with extra side channels, the river is given more space after a threshold discharge is exceeded 13 Coop Small Upstream cooperation: maximum discharge reduced to 20.000 m3/s 14 Coop Medium Upstream cooperation: maximum discharge reduced to 18.000 m3/s 15 Coop Large Upstream cooperation: maximum discharge reduced to 14.000 m3/s 16 DC Floating Floating houses: resulting in damage functions with 10 times less damage 17 DC Dikes Fort cities: extra embankments around the cities 18 DC Mounts All cities are raised by 4 m, resulting in houses on an area of elevated ground 19 DC Elevated Building houses on stilts so that they are raised above the ground and water can pass underneath.20 Alarm Early Evacuate inhabitants in case the chance of dike failure is higher than 20% (default value is 40%), resulting in fewer casualties but potentially more false alarms 21 Alarm Education Inform and educate inhabitants on what to do in case of a (potential) flood emergency urbanization, the result is high damages and high casualties.Conversely, if the population density declines, even in case of severe climate change, there is limited reason for action.

Name
To cope with climate change and reduce flood damage in case of urbanization, a wide variety of pre-specified policy options is available.We tested dikes designed for a 1:500 flood þ0.5 m (option 2), climate dikes (option 7), room for the river (option 11), upstream cooperation (option 14), and early alarm (option 20).Of these, options 2 and 11 are the most promising.
Fig. 6 shows boxplots for these two policy options for casualties, flood damage, and costs.Dike raising has variable costs, because the design discharge is updated in light of the observed river discharge.So, the actual height of the dike changes over the course of a simulation and differs from one computational scenario to the next.This continuous updating and associated dike heightening is a standard operating procedure in the Dutch context.
Both raising the dikes and giving more room to the river help to improve the situation, but there might be room for further improvement.To investigate this, we performed scenario discovery on the results for both policy options.Again, we analyzed the correlation between casualties and flood damage and found that it was 0.8, so we focus on flood damage.We update our rule for classifying a scenario as being of interest f ðxÞ ¼ & 1; x > 10; 000 0; otherwise where x is cumulative flood damage.For these new results, it is more difficult to find a clear explanation.That is, we are unable to find a single explanation with both high coverage and high density.
We can, however, find two boxes that jointly explain over 64% of the cases of interest (see Table 4).These two boxes suggest that damages can occur on both moderate and severe climate change, in combination with urbanization.Interestingly, compared to the previous iteration of scenario discovery, the urbanization followed by de-urbanization land use scenario is no longer included as part of the first identified box.This suggests that if population density declines in the future, there is less reason for action.
The second iteration of scenario discovery suggests that neither of the individual actions is sufficient in the face of moderate or severe climate change and increasing population density.We therefore need to improve the policies.Here, we focus only on improving the dike raising strategy, and consider four alternatives: dike raising to a 1:1000 river discharge (option 3); dike raising 1:500 combined with alarm early (options 2 and 20); dike raising 1:500 combined with room for the river (options 2 and 12); and dike raising 1:500 combined with climate dikes (options 2 and 7).Of these, the combination of dike raising and alarm early (options 2 and 20) is ineffective.The boxplots for the other three options are shown in Fig. 7.
The three policy options all have improved the performance compared to dike raising 1:500 in isolation.To assess whether there is room for further improvement of the policies, we again perform scenario discovery, but now separately for casualties and damages, because they are no longer strongly correlated.For casualties, we classify a scenario as being of interest if x > 300 0; otherwise where x is the number of casualties.3773 out of the 15,000 scenarios meet this criterion.Table 5 shows the results from this  analysis.A virtually identical result is obtained when we do scenario discovery on damages, using the same classification rule as in the previous iteration.The scenario discovery results suggest that if actions are taken conditional on whether the system is experiencing a particular land use and/or climate scenario, we might be able to further improve the efficacy of the policy.
Earlier work on the Rhine (Haasnoot et al., 2015) suggests that one of the best indicators for detecting climate change is the running average over 30 years of the number of days in which the river discharge is below 1200 m 3 /s.If we use this as a signpost, and move to a 1:1000 design height for dikes only in case low flows strongly suggest a severe climate change scenario, we might be able to further improve the performance of the policy.We test this for two options.1:500, followed by 1:1000 in case of climate change signal (option 2, followed by option 3) 1:500 with climate dikes, followed by dike strengthening to 1:1000 in case of climate change signal (options 2 and 7, followed by option 3) The resulting boxplots are shown in Fig. 8.It appears that both adaptive policies are effective.We are able to further reduce costs, damages, and casualties.Further improvement might be achieved by adding actions conditional on the specific land use scenario as well.

DAPP analysis of Waas case
Adaptation pathways can be designed in various ways.Here, we use the multi-objective robust optimization approach first presented by Kwakkel et al. (2015).A similar approach for the sequencing of urban water supply augmentation is presented by Beh et al. (2015).Robust optimization methods aim at finding, in the presence of uncertainty about input parameters, optimal outcomes that are not overly sensitive to any specific realization of the uncertainties (Bai et al., 1997;Ben-Tal andNemirovski, 1998, 2000;Bertsimas and Sim, 2004;Kouvalis and Yu, 1997).In robust optimization, the uncertainty that exists about the outcomes of interest is described through a set of computational experiments (Mulvey et al., 1995), and robustness is defined over this set of experiments.We assess the robustness of candidate pathways on multiple independent objectives, avoiding the need to make a priori assumptions about decision-maker tradeoff preferences.The multi-objective robust optimization problem for the design of adaptation pathways for the Waas is Minimize F À l p;r Á ¼ y costs ;s costs ;y casualties ;s casualties ;y damage ;s damage where l p;r ¼ ½p 1 ; p 2 ; p 3 ; r 1 ; r 2 cp2P; cr2R where l p,r denotes a policy pathway, p m is a policy action, P is the set of policy actions as specified in Table 2, r n is a rule, R is the set of rules, y i is the set of outcomes for outcome i dfcosts; casualties; damageg across a set of computational experiments, y i is the mean value of y i , and s i is the standard deviation of y i .
A rule specifies the adaptation tipping point for activating the next action on an adaptation pathway.These rules are related to performance of the system in terms of casualties and economic damages.Performance is evaluated and classified into no event, small event, large event, and extreme event.We activate a new action if, in the previous five years, an event of the pre-specified level is encountered.
There is a wide literature on robustness and robustness metrics (Herman et al., 2015;Lempert and Collins, 2007;Rosenhead et al., 1973;Wald, 1945).Here, we use the mean across a set of computational experiments and the dispersion around this mean measured by the standard deviation.There are three reasons for using the mean and the standard deviation separately.First, it is easier to interpret than the signal to noise ratio as used in earlier work (Hamarat et al., 2014;Kwakkel et al., 2015).Second, the signal to noise ratio is not a monotonically increasing objective function (Ray et al., 2013).Third, it provides decision-relevant insight into the tradeoff between average performance and the deviation from this average performance.
There exist various approaches for solving multi-objective optimization problems.In this paper, we use BORG, a state of the art genetic algorithm in which the evolutionary operators coevolve with the search (Hadka and Reed, 2013).In various comparisons, BORG has been demonstrated to be among the best available genetic algorithms for solving multi-objective optimization problems (Hadka and Reed, 2013;Reed et al., 2013).
In this case, we use 200 computational experiments and calculate our robustness metrics over this set of 200 experiments.An analysis of ten randomly generated pathways revealed that adding more experiments did not substantially change the value of the robustness metrics.We evaluated the objective function 1000 times using the same set of computational experiments, for a total of 200,000 runs of the simulation model.We used ε-progress (Hadka and Reed, 2013) as an indicator for the rate of convergence of the genetic algorithm.It indicates whether the algorithm has been able to find better solutions.There was no ε-progress after a bit over 800 evaluations of the objective function.This suggests that the algorithm has converged.So, after 200,000 runs of the model, the algorithm has found a Pareto approximate set of robust pathways.
The set of eleven pathways and their scores across the six objective functions identified by the optimization is shown in Fig. 9.The optimization has produced a heterogeneous set of solutions.The pathways are sorted by mean cost, with the cheapest solutions at the top, and the most expensive solutions at the bottom.There is a clear tradeoff between costs on the one hand, and damages and casualties on the other.Cheaper solutions are less effective.There are, however, some interesting solutions that are nearly as effective as the most expensive solutions, while being substantially cheaper on average.The main tradeoff in these solutions is between the mean cost and the standard deviation of the costs.
A final step is to transform the identified pathways into an adaptation map.For this, it is necessary to analyze the timing of the adaptation tipping points.This timing is scenario dependent.Therefore, it differs from one computational experiment to the next.We generated 5000 computational experiments, which we subsequently grouped based on whether there is ongoing urbanization or not, and whether there is severe climate change or not.The number of experiments is based on prior experience with the case and is illustrative.This gives us four groups of computational experiments, where the timing of the adaptation tipping points is quite different in each experiment.For each group, we analyzed the timing of the adaptation tipping points for the different pathways, and for illustrative purposes used the 25th percentile as the timing.The resulting figure is shown in Fig. 10.
From Fig. 10, we draw several conclusions.First, the alarm early option is useful only in the de-urbanization scenarios.As can be seen in the bottom two figures, the option is never used in case of ongoing urbanization.Second, if we compare the maps, we see that many actions are needed earlier in case of severe climate change, compared to no or limited climate change, but that the layout of the map differs between the de-urbanization and urbanization scenario.Another useful feature of the adaptation pathways map is that it provides insights into the sequencing of actions, and which actions remain open if one goes down a particular pathway.So, if one starts with a small or medium scale room for the river action, we see that these actions will be sufficient for only a few years.After either of these actions, the next action is to enhance education related to evacuations.In contrast, if we start with a dike 1:500 þ 0.5 m, we see that this is sufficient for somewhere between 50 and 75 years depending on the severity of climate change.After this, there are several options open.We can either strengthen the dikes in light of the estimated worst case climate change discharge (i.e.dike 1:500 extr), or we can choose to build floating homes.

Comparison of the approaches
In this paper, we applied both RDM and DAPP to the same case in order to get insights into the different analytical paths followed by the two approaches.What information and tools are needed, what decision relevant insights are being generated, and how different is the resulting plan emerging from the application of the RDM and DAPP approaches?More specifically with respect to the object of comparison, a distinction can be made between the plan as written, the process of drafting the plan, and the actual performance of the chosen plan after implementation (Kwakkel and van der Pas, 2011;Verschuren and Hartog, 2005;Walls et al., 1992Walls et al., , 2004)).In answering the questions, we focus on the plan as written and the process of drafting the plan.

Information and tools needed
What information and tools are needed for each approach, and are there any important differences in this regard?Regarding the required information, both RDM and DAPP can use the same system model for their analysis, take into account the same uncertain factors, and consider the same outcomes of interest.Both applications require specific software.RDM requires software for the generation of cases and for scenario discovery.The multi-objective robust optimization approach used for the design of pathways also requires software for the generation of cases, and a state-of-the-art multi-objective optimization algorithm.
An important difference with respect to the tools is their computational cost.For evaluating 14 policies in the RDM analysis, 70,000 computational experiments were used.This number could have been lower if we had used (say) 2500 experiments per policy rather than 5000.In contrast, the robust multi-objective optimization approach for DAPP required 200,000 computational experiments.Introducing alternative stopping conditions, instead of a fixed number of function evaluations, might lower this.The difference in required computational resources has implications for the conditions under which each approach can be used, and has consequences for the design of models that are fit for purpose (see also Haasnoot et al., 2014 for a more in depth discussion on this topic).
Another relevant difference between the approaches is with respect to the required information.In the RDM analysis, at each iteration we had to specify what results were of interest.This implies the implicit use of a satisficing understanding of robustness using the domain criterion (Starr, 1963).That is, effectively one is trying to minimize the fraction of cases that are of interest.If one uses RDM in practice, the definition of success would be specified in consultation with the stakeholders during the scoping phase.In our application, we refined the definition of success after the first iteration.This was primarily motivated by algorithmic reasons: using the stricter definition of success in the first iteration results in an overload of cases of interest, resulting in many overlapping boxes from the PRIM analysis.In contrast, the multi-objective robust optimization approach requires the explicit specification of what is meant by robustness.In this case, we defined robustness in terms of the mean and standard deviation for each outcome of interest.The discussion with stakeholders on what is acceptable or unacceptable performance would take place only after the Pareto approximate set of pathways had been identified.
A final difference with respect to the information required for each approach is that the multi-objective robust optimization approach requires an up-front specification of the policy options, the possible rules that can be used in sequencing options, and constraints regarding the ways in which the options can and cannot be combined.In contrast, the RDM application requires only the availability of one or more policy options, and some expert judgment on how to combine or sequence options.In the final iteration of the RDM analysis, we derive an interesting signpost directly from the scenario discovery results, rather than its being defined up front, and use this for designing an adaptive plan.

Decision-relevant insights
What are the similarities and differences with respect to decision-relevant insights?A first difference is with respect to the thoroughness with which the design space of policy options is being explored.DAPP aims at developing an adaptation pathways map that contains a set of possible pathways that serves as input to a conversation between stakeholders.RDM remains silent on whether an analyst should focus on developing a single robust strategy, or whether the analysis should be used to produce a more diverse set of strategies.In the case study using RDM, we explored 14 policies, where some of the policies are refinements of previously analyzed policy options.There is thus a strong pathdependency in the RDM analysis.The robust multi-objective optimization approach to the design of adaptation pathways, in contrast, explored 1000 different adaptation pathways, and presents the optimally robust pathways in a pathways map, showing options, multiple sequences of actions, and path-dependencies of actions.
A second policy-relevant insight is that the set of pathways that emerges from the multi-objective robust optimization approximates the Pareto optimal solutions.RDM cannot offer the same guarantee that the resulting solution, or set of solutions, is optimally robust.
A third important difference in terms of policy-relevant insights is that the scenario discovery results give very good insights into the combination of uncertain future developments under which the system fails.This is of great use in identifying signposts for adaptive policies, as evidenced by the pro-active signposts that are introduced in the RDM analysis.This type of insight is absent from the multi-objective robust optimization approach, which relies instead on the upfront specification of possible signposts and triggers that are used to adapt before an adaptation tipping point is reached.

Differences in resulting plans
If we look at the plans that emerge from both the RDM process and the robust optimization approach to the design of adaptation pathways, we make several observations.First, it is possible to generate adaptive strategies using RDM, in contrast to the suggestion in Walker et al. (2013) that RDM aims at developing static plans rather than adaptive plans.To develop adaptive plans using RDM, however, we added concepts from adaptive policymaking (Hamarat et al., 2013;Kwakkel et al., 2010;Walker et al., 2001), similar to Bloom (2015).Second, the solutions emerging from the two approaches are different.RDM resulted in a single adaptive plan that specifies the short-term actions and the long-term options, including the conditions under which to use them.In contrast, the robust optimization approach resulted in an adaptation map with 11 distinct pathways.These 11 pathways can serve as a starting point for making an adaptive plan that specifies short-term actions and long-term options.
If we compare the set of solutions analyzed over the course of the RDM process to the set of solutions emerging from the robust optimization, we observe that only one solution is common to both: raising dikes to a 1:1000 discharge.This is partly due to the fact that the rules used in the robust optimization are quite different from the signpost developed in RDM.The rules used in the robust optimization approach to the design of adaptation pathways are reactive, while the signpost developed in light of the RDM analysis is more proactive.It would be interesting to include the signpost identified in the RDM analysis and one related to land use change as additional possible rules in the robust optimization.We do observe, however, that both RDM and the robust optimization approach come up with solutions that progressively raise the dikes to more stringent design discharges.In the RDM analysis, we move from 1:500 to 1:1000.One of the pathways in the DAPP analysis moves from 1:500 þ 0.5 m to 1:500 for the worst-case climate scenario.If we compare the performance of the set of pathways with the performance of the strategy resulting from RDM using the same robustness definitions as used in the multi-objective optimization, we see that the RDM solution is in the Pareto approximate set, together with all the pathways (see Supplementary material for the analysis).

Conclusions
The Dynamic Adaptive Policy Pathways approach and Robust Decision-Making are two analytical model-based approaches for supporting the design of robust strategies in the presence of irresolvable deep uncertainties.How are these two approaches different?Where do they overlap?In what respects are they complementary?We applied both approaches to the same stylized case inspired by a river reach of the Rhine Delta in the Netherlands in order to answer these questions.
Both RDM and DAPP can be used to design effective flexible strategies.RDM is less computationally expensive than the robust multi-objective optimization approach used for the design of adaptation pathways.RDM also provides actionable insight into the combination of factors that jointly determine if and when adaptation is necessary.In contrast, the multi-objective robust optimization approach used to develop adaptation pathways explores the design space much more thoroughly.This results in the identification of a heterogeneous set of robust pathways with clear tradeoffs.It requires, however, the upfront specification of options and rules for sequencing them, something that can be supported through RDM.
The robust multi-objective optimization approach is one approach for designing adaptation pathways.Other approaches could be used instead.One key direction for future research is to investigate how to modify the iterative RDM process, in which candidate strategies are modified in light of scenario discovery results, in order to develop adaptation pathways.The vulnerabilities identified through scenario discovery are, from an analytic perspective, closely related to adaptation tipping points.Specifically, vulnerabilities identified through scenario discovery are a multi-dimensional generalization of adaptation tipping points (Kwadijk et al., 2010).This insight can serve as a starting point for combining RDM and DAPP.There are various forms such a combination could take (Fig. 11).For example, one might start with the RDM cycle and iteratively develop multiple adaptation pathways.In such a combination, scenario discovery can be used for identifying signposts and triggers.Alternatively, one could start with the development of a set of pathways, for example through multiobjective robust optimization or in a participatory manner, and subsequently apply RDM to assess the performance of these pathways.
In this paper we compared two analytical approaches for developing climate adaptation strategies.There exist a variety of other approaches, including decision scaling, real options, and Info-Gap Decision Theory, which could be used for the same purpose.Given the observed complementarities between RDM and DAPP in this paper, and between Info-Gap and RDM in Hall et al. (2012) and Matrosov et al. (2013b), we speculate that Info-Gap analyses could also be of some value for designing adaptation pathways.Future work is needed to explore the commonalities among DAPP, RDM, decision scaling, and real options.dx.doi.org/10.1016/j.envsoft.2016.09.017.

Fig. 3 .
Fig. 3.An example of an Adaptation Pathways Map (left) and a scorecard (right) presenting the costs and benefits of the 9 possible pathways presented in the map.The scenario determines the speed with which a route on the map is traversed.
to cope with a 1:500 discharge, based on measurements 3 Dike 1:1000Dike height rise to cope with a 1:1000 discharge, based on measurements 4 Dike 1:500 extr.Dike height rise to cope with a 1:500 discharge, based on extreme climate change scenario 5 Dike 1:1000 extr.Dike height rise to cope with a 1:1000 discharge, based on extreme climate change scenario 6 Dike 2nd Q Â 1.5 Dike height rise: adapting to 1.5 times the second highest discharge ever measured ('rule of thumb' measure) 7 Dike Climate dikes Strength of the dikes improved by a large widening of the dikes 8 Dike Wave resistant

Fig. 5 .
Fig. 5. Boxplot for casualties and flood damage across 5000 scenarios in the absence of any policy options.

Fig. 6 .
Fig.6.Boxplots for casualties, flood damage, and costs for the two best performing policy options.

Fig. 9 .
Fig. 9. Parallel coordinate plot of the performance on the objective functions for each of the robust pathways.

Fig.
Fig. Adaptation pathways maps for four groups of computational experiments.

Table 1
Overview of the uncertainties.

Table 3
First iteration scenario discovery results for the no policy case.

Table 4
Second iteration scenario discovery results for the two best performing policy options.
Fig. 7. Boxplots for casualties, damages, and costs for the three best performing candidate policies in the second iteration.J.H. Kwakkel et al. / Environmental Modelling & Software 86 (2016) 168e183

Table 5
Third iteration scenario discovery results for the three best performing policy options.Boxplot for casualties, flood damage, and costs for two alternative adaptive policies.