Subjective risk assessment for planning conservation projects

Conservation projects occur under many types of uncertainty. Where this uncertainty can affect achievement of a project’s objectives, there is risk. Understanding risks to project success should influence a range of strategic and tactical decisions in conservation, and yet, formal risk assessment rarely features in the guidance or practice of conservation planning. We describe how subjective risk analysis tools can be framed to facilitate the rapid identification and assessment of risks to conservation projects, and how this information should influence conservation planning. Our approach is illustrated with an assessment of risks to conservation success as part of a conservation plan for the work of The Nature Conservancy in northern Australia. Risks can be both internal and external to a project, and occur across environmental, social, economic and political systems. Based on the relative importance of a risk and the level of certainty in its assessment we propose a series of appropriate, project level responses including research, monitoring, and active amelioration. Explicit identification, prioritization, and where possible, management of risks are important elements of using conservation resources in an informed and accountable manner.


Introduction
Project risks can be characterized as uncertainties that might have a negative effect on achievement of a project's objectives (Chapman and Ward 2007). Although risk is sometimes conceptualized as including both positive and negative Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. uncertainties (i.e., upside and downside risks) (e.g., Hillson 2002, Ward andChapman 2003), here we are concerned only with downside risks. For conservation projects, risks exist in both social and environmental space, ranging from invasive species outbreaks and catastrophic climate events, to community reactions, policy changes, and insecure or inadequate funding streams. As with all complex projects, the delivery of conservation outcomes is influenced by our capacity to assess the risks associated with our investments, and by our ability to manage and respond to these risks through time (e.g., Fischer and Lindenmayer 2000, Burgman  Expected outcome should be adjusted based on assessment of risk. Strategy selection should be influenced by both how robust a strategy is to risk and the costs required to adequately ameliorate risks to a strategy. Process provides a dedicated opportunity to elicit diverse input about strategies and their context

Monitoring design and investment
Overall risk rank Uncertainty in assessment of risk Identifying risks that need to be actively monitored. These might include highly ranked risks as well as low or moderately ranked risks with high uncertainty in the assessment of risk Budgeting Cost of monitoring, managing or ameliorating risks Realistic costs for risk monitoring and mitigation activities need to be considered as part of the overall cost required for a strategy or project Implementation and project management Overall risk rank Risk management actions Important risks should influence strategy implementation as well as selection, often in domains such as staff capacity, leadership or partner engagement. Risk management is an important part of project management Reporting Parrott and Meyer 2012). Despite the important link between risks and project outcomes, assessment of risk and use of this information has not featured prominently in conservation planning guidance or practice. In a recent review of conservation planning in the world's largest conservation non-government organization, risk assessment was identified as a noticeably absent component (TNC 2011). Although environmental risk assessment is a substantial field whose practice influences strategic choices for many environmentally related initiatives (Burgman 2005), these are typically assessments of risks to natural or community values (e.g., to help prioritize which risks or threats to manage; Carey et al 2007), or risks arising from the project (e.g., increased fire risk; Norman et al 2010) (but see Dale et al 2013 for an example of risk assessment applied to governance systems). Many conservation agencies and organizations also undertake risk assessments for legal, compliance and reputational issues associated with potential projects (e.g., the risk of engaging with corporate partners or undertaking controversial strategies). Although these risks are undeniably bound up with long-term conservation success, their principal focus is not on the cause of failure of the project but the potential consequences of this failure.
Some strategic conservation planning efforts have elicited estimates for the probability of success (roughly the inverse of risk of project failure) of different projects or activities (e.g., Joseph et al 2009, Obermeyer et al 2011, Carwardine et al 2012, but have not further decomposed risk into the causes for failure. Probability is a natural unit for risk and getting estimates of success likelihood is a substantial improvement over much conservation planning practice, however, we are not passive hostages in the face of risk.
Risks to project success are rarely immutable, for example, the risk of a conservation project being derailed because of community opposition can be reduced through more substantial engagement with communities. However, there will almost always be a cost associated with ameliorating any risk, where this is possible.
An assessment of risks to stated project objectives should influence conservation plans in a number of ways. These are summarized in table 1 and include, moderating the expected benefit of strategies, more complete estimates of both strategy and monitoring costs, adapting implementation to ameliorate risks, improving transparency in communication and reporting, and guiding investment in research and data collection. None of these suggested roles for knowledge about risks is novel. However, discussion of the role of risk assessment in planning has been fragmented into narrow aspects of planning or been missing entirely from published dialogues. Often comparable discussions appear under the more general banner of uncertainty (e.g., Kujala et al 2012). It has not received structured treatment as an element of systematic conservation planning (Pressey and Bottrill 2008).
Despite the broad influence that risks plays on strategic conservation decisions, there has been relatively little discussion about how to identify these risks, and still less about how organizations should prioritize investment in monitoring and ameliorating these risks. There is, however, an expectation amongst conservation funders that such things are considered (e.g., Australian Government 2013). Typically this has been the domain of a project manager's intuition. Some conservation organizations and projects do employ strategic planning methods that serve to identify risks to project success; SWOT (strengths, weaknesses, opportunities, threats) analysis is a popular example. However, a thorough review of the application of SWOT analyses revealed it typically resulted in long lists of poorly described risks with no prioritization, and as a result was ineffective as a risk management tool (Hill and Westbrook 1997). Consequently, the effort to identify and prioritize risk management in conservation is, at the very least, inconsistent and likely inadequate. Risk is generally conceived as having two dimensions: likelihood and consequence (Wideman 1992, Dale et al 2013. Likelihood is essentially the probability that the risk will materialize (although not always expressed as quantitatively as a probability), while consequence is a measure of a risk's impact in terms of deviation from expected project outcomes should the risk occur (Cagno et al 2007). Risks and the appropriate response cannot sensibly be evaluated without knowledge about both dimensions (Williams 1996). Although knowledge of both dimensions is important, comparative analysis of risks is often accomplished by combining the two dimensions to give an overall rating or rank. This is the basis of publically endorsed risk assessments in most countries (Burgman 2005).
The complexity of the social-ecological systems in which conservation takes place means that most projects must contend with continually shifting contexts (Game et al 2013b). As such, conservation projects will rarely have extensive historical data regarding the likelihood (probability) and consequence (impact) of risks to draw upon. In some cases, conservation agencies or organizations might look to comparable experiences in other jurisdictions to inform base rates, such as violation rates of conservation easements in another state, or alternatively look creatively at tactically relevant data, for example, corporate non-compliance with other regulations might be used as a base rate for the risk that a company fails to live up to its expectation under some conservation agreement. We expect, however, that most conservation project risk assessments will involve a subjective or semi-quantitative assessment of risks.
Here we describe the application of a risk ranking procedure to assess risks to the success of conservation projects as part of a conservation plan. We illustrate this application with a case study from the work of The Nature Conservancy (TNC) in northern Australia. The risk assessment was conducted in the context of conservation planning for projects in the region for the period 2012-2020. We analyse some of the strengths and challenges of using conventional risk assessment methods for assessment of risks to the success of conservation projects.

Risk in the context of objectives
Success means different things to different people. Assessing risks to success requires a datum. The obvious choice for conservation projects are the stated objectives of a project. A common understanding about what the project is trying to achieve and how this achievement will be gauged is a necessary condition for comparing the judgements of multiple people. Although clarity around objectives and the attributes used to measure it is a critical component of conservation planning more generally, it is frequently absent (Game et al 2013a). Conservation objective statements often include ambiguous and subjective terms such as 'effectively managed'. Identifying how something like 'effectively managed' will be measured and reported on can be a useful way to get clarity around its meaning. Clear units of measurement are also required in order to assess the consequence of different risks. Table 2 contains example objectives for The Nature Conservancy's northern Australia program along with the associated performance measures. In this case, all objectives were assumed to be of equal importance. There is much good existing advice on formulating and structuring objectives (e.g., Gregory et al 2012), and so we do not discuss it further in this letter.

Consequence and likelihood index
For each objective, a unique index of risk consequence was constructed, which included linguistic descriptions, a corresponding scenario in performance metric of that objective, and a score. Both linguistic descriptions and scores were based on the Risk Assessment and Management Process (RAMP) standards of the Institution of Civil Engineers and Faculty and Institute of Actuaries in Great Britain   (Lewin 2002). In a workshop setting, key project staff and organizational leaders were asked to agree upon scenarios for each objective that corresponded to the linguistic descriptions of risk consequence. An example of the consequence index with scenarios for an objective is given in table 3. Much of the group discussion around risk consequence scenarios hinged on the assumed current baseline, for example, how much of northern Australia is currently subject to early dry season burning, with the severity of consequence being heavily dependent on this starting point. A risk likelihood index was also adopted from the RAMP standards (table 4). Unlike indices of consequence which are outcome specific, the likelihood index remains consistent across objectives.

Identifying risks
Just as good alternatives lie at the heart of good decisions (Edwards 1990), even the most rigorous assessment of risks will be of limited value unless the important suite of risks is first identified. To build a list of risks to the project outcomes we conducted three activities with workshop participants. The first was to employ a technique known as a pre-mortem (Kahneman et al 2011, Schlesinger andKiefer 2012). All participants were asked to imagine that the project had gone ahead as designed in the current conservation plan but that it is now 5 years in the future and the project has gone disastrously wrong. Each person was then asked to describe to the group what they had imagined went wrong. This process served the purpose of both directly identifying risks and also triggering thought about the range of things that could feasibly go wrong. The pre-mortem was followed by free-form listing of additional risks, and then a systematic assessment of assumptions associated with the theory of change for each strategy.
The final list of risks to The Nature Conservancy's northern Australia project is given in table 5. The set of risks includes both specific events and more general downside uncertainties such as 'lack of staff capacity'. Because some of the risks identified refer to external partners and the risk assessment was a purely internal exercise, parties identified in the risks have been anonymized for presentation here. The initial list of risks suffered heavily from the linguistic uncertainties identified by Carey and Burgman (2008). For example, one of the risks was the loss of dedicated government funding for Indigenous Protected Areas. Stated like this, the risk was victim to vagueness; it is possible that some participants might consider a partial loss of IPA funding whereas others considered the complete loss of IPA funding. We would expect total loss to be more severe in consequence than 50% loss, but is also less likely to occur. To address this vagueness we separated the loss of funding for IPAs into categories of total loss and 50% loss. Obviously the actual extent of funding loss could fall at any point between none and total, but even two categories proved a satisfying solution for participants.
For some risks, vagueness or under-specificity could not be completely removed, for example the 'lack of TNC staff capacity' (meaning in this case 'not enough staff'). However, the assessment methodology used here allowed participants to be clear about the severity of risk they were considering when making their estimate of likelihood. Where possible, ambiguity in risk descriptions was removed by clarifying amongst participants what was meant by terms such as 'sustained funding'.

Subjective evaluation of risks
Six staff involved with The Nature Conservancy's conservation work in northern Australia were asked to individually assess the consequence and likelihood of each risk in the context of each objective. A generalized Delphi process was followed where initial results were reviewed by the group and then participants were asked for subsequent individual assessments of risks. Although for most risks likelihood will remain the same across objectives, for some risks such as 'lack of TNC staff capacity', the likelihood of the risk is dependent on the particular objective such that estimates of both likelihood and consequence need to be sought in the context of each objective. Risk estimates were sought from people with diverse roles in the program including senior management, project management, science, and government relations. If participants felt unqualified to judge the likelihood and consequence of a particular risk they were instructed to provide no judgement for that risk.

Results
An indication of the importance of a risk to the project overall was summarized by the mean score given for each Table 5. Risks identified for The Nature Conservancy's work in northern Australia. 'XXXX' denotes anonymization of a third party identified in the risk.
• Total loss of Indigenous Protected Area a funding in next 2 years • Total loss of National Reserve System program b funding in next 2 years • 50% reduction in Indigenous Protected Area program funding in next 2 years • 50% reduction in National Reserve System program funding in next 2 years • Biodiversity and social outcomes of conservation actions cannot be proved c • Change of XXXX government at next election • Disinterest amongst Indigenous population in being conservation stewards • Expansion of agriculture and mining outcompetes conservation opportunities • Failure (or limited take-up) of carbon market • Failure of XXXX organizations • Healthy Country Planning d does not translate to effective management • Inability to raise long-term, sustained funding for Indigenous Protected Areas risk, summed across all four objectives (figure 1). Based on this overall score, seven risks stood out as being the most significant. These included a combination of risks considered to have potentially severe consequences but relatively low probability, such as the loss of Working on Country funding, and risks considered to have high likelihoods but potentially less severe outcomes, such as the threat of mining creates a loss of confidence in growing the conservation estate, and the lack of staff capacity in The Nature Conservancy (see figure 2).
Disagreement between participants about the importance of a risk can be informative and is part of the reason for canvasing a diverse set of voices. Differences in the domains of expertise mean different levels of knowledge about each risk, for instance some participants were more familiar with government decision making whereas others were more familiar with community sentiment. However, substantial disagreement can also be suggestive of a lack of common understanding regarding the risks in question. To look at concordance across participants, we calculated the Spearman rank correlation between all pairs of participants for each objective (e.g., figure 3). Overall there was reasonable agreement between participants about the rank order of risks, with most pairs assessments being positively correlated, even if only slightly. Similarly, group discussion revealed strong common understanding about risks. There was particularly strong agreement around risks to objective #3 (sustainable funding), which we hypothesize is because participants have prior experience and feedback to draw on when assessing sustainable financing outcomes.
It is well established that individuals perceive risk differently (Slovic 1999). To explore the extent to which individuals differed in how risky they saw the project overall we looked at average risk scores for each individual. There was clearly a spectrum of risk tolerance amongst participants (figure 4), with one individual perceiving the project as generally riskier than the others (expert #6 in figure 4), and another individual consistently perceiving less risk than the others (expert #1 in figure 4).
Because of the way the risk scales are structured, those people who see the world as riskier end up having a disproportionate influence on the overall risk rank. To minimize this effect we also looked at the mean rank of risks across objectives and participants (figure 5). Looking at mean ranks rather than scores has the desirable effect of minimizing the influence of individual risk tolerance on results, but incurs  the undesirable effect of reducing insight into the relative importance of risks gained through eliciting diverse opinions. The most substantial difference in risks when looking at mean ranks rather than mean scores was that 'lack of TNC staff capacity' went from being perceived as a high risk to being perceived as one of the lowest risks ( figure 5). This suggests  that its high risk score is driven by a subset of participants who considered it a serious risk.
Our confidence in risk estimates should be partly influenced by the degree of variation in subjective risk estimates. High variation is suggestive of uncertainty in the true nature of the risk in question. The level of certainty associated with assessment of a risk is useful information because it can influence the appropriate response to the risk (see section 4). Across all objectives, the risks with the greatest uncertainty fell roughly into three themes: (1) the role of mining in potentially eroding the existing conservation estate and the loss of confidence in growing the estate that this is likely to cause (e.g., Mascia and Pailler 2011, Adams and Moon 2013); (2) the lack of TNC staff capacity and the related issues of conservation progress being slow and marginalized as a result, and planning not translating into management; and (3) whether the invasive Gamba grass becomes the dominant ecosystem threat. These groups all deal with three quite separate spheres of knowledge suggesting certainty was not systematically biased by the selection of experts.
Because a project team should be primarily concerned about resolving uncertainty in the most important risks, we found it useful to plot uncertainty (coefficient of variation in mean risk score) versus risk score. Figure 6 shows uncertainty versus risk scores for the mean risk scores across all objectives.

Discussion
All conservation projects occur under uncertainty. Where this uncertainty can affect achievement of the project's objectives, there is risk. Explicitly understanding and managing these risks, where possible, will increase the likelihood of a project achieving the stated objectives. Here we focus on the subjective quantification of risks in relation to their impact on a conservation project's objectives given a known set of strategies. Risk rarely features as part of a conservation planning process and yet the information revealed is inimical to planning. In our case study of The Nature Conservancy's work in northern Australia, the risk assessment process revealed a range of internal and external risks to achieving the project's objectives. Risks such as those identified here should influence the way we plan a conservation project. For example, we may adjust our estimation of the cost of achieving the project's goals given the need to invest resources in managing risks, or potentially adjust our expectation of objective achievement to reflect the likely impact of risk on achieving our stated goals.
We do not expect that risks can be eliminated, nor in some cases necessarily influenced, rather risk assessment methods like those demonstrated here can help conservation projects prioritize which risks to try and manage. Broadly, risk management involves four options: (1) actively ameliorate through strategic adjustment, (2) gather information to better understand a risk, (3) monitor a risk in order to respond rapidly when it occurs, or (4) do nothing. Which of these options is appropriate depends on both the overall risk rank and the uncertainty around the risk estimate (figure 7). Highly ranked risks for which there is also high confidence in the assessment (i.e., those with low coefficient of variation across experts), are likely to be the best candidates for active amelioration, where possible. In the case of northern Australia, risks in this category were centred largely on issues of funding for the Indigenous Protected Area program and funding for the employment of Indigenous rangers to manage these. Risks that are highly ranked but with greater uncertainty around the assessment are likely to be good candidates for further investigation via directed research. In northern Australia, risks in this category included issues around the expansion of mining and agriculture, and TNC staff capacity. These risks also represent places to further reduce any linguistic uncertainty that remains in assessment of these risks. Risks that are less highly ranked but for which the assessment is highly uncertain are potential candidates for tracking using the most cost-effective methods possible. In northern Australia, risks in this category included the impact of invasive species on ecosystems, loss of conservation interest amongst the Indigenous population, and failure or limited take-up of the carbon market. All of these risks might be tracked with relatively little investment, hopefully allowing enough warning of the imminent occurrence for the project to respond effectively. However, risks can also occur without warning. For example, during the course of writing, the Australian Government's National Reserve System program, which was considered an important program to help meet The Nature Conservancy's goals in northern Australia (e.g., , had its funding discontinued. This outcome was the seventh highest identified risk but the total loss of funding to this program came with little forewarning (Fitzsimons et al 2013).
In general, it is hoped that the assessments of risk made during a conservation plan like this actually prove a poor guide to the subsequent occurrence and impact of risks. This is because ideally organizations will respond to these risks by making programmatic changes that serve to reduce their Figure 7. Four responses to conservation projects risks depending on the relative importance (risk rank) and level of uncertainty around the assessment of a risk. likelihood or impact. For example, in northern Australia The Nature Conservancy has already responded to the issue of staff capacity and long-term financing, with the aim of reducing exposure to these risks. The expected lack of alignment between risk assessment and risk outcomes does not however, diminish the importance of feedback about the occurrence of risks; information valuable both to adaptively manage the current project and to influence risk assessments for future projects. There is also cause to revisit risk assessments like this one at regular intervals during a project as it is possible that important risks will be missed in the initial assessment and likely that new risks will emerge over the course of a project. Care must be taken not to create a false sense of security at the start of a project that all risks over the life of a project have been considered. Explicitly gathering information on the occurrence of predicted risks and their impact on conservation projects is consistent with repeated calls for conservation to take a more evidence-based approach to decision making (Sutherland et al 2004, Segan et al 2011. At present, reliable information on the base rate for risks to conservation projects is largely unavailable but conducting risk assessments like the one here will help identify the range of risks for which base rate information would be useful, and encourage collection of data to inform them. Determining the resources that should be invested in either directed research or monitoring of risks depends not only on the level of uncertainty and relative importance of a risk, but also on how a conservation program would respond to greater knowledge about the risk. This is the cornerstone of a concept known as the value of information (Yokota and Thompson 2004). Approximately speaking, resolving uncertainty around risks that would necessitate a significant change in strategy (either proactively or reactively) to achieve the stated objectives, is more valuable than for risks where the currently proposed strategies would still be adequate. Although expected value of information (EVI, as it is formally known) is a well-used concept in decision science, the quantitative information and analytic expertise required for its calculation has restricted its application in conservation (but see, Runge et al 2011, Moore andRunge 2012). However, just as a useful risk assessment can be accomplished rapidly using a subjective approach, the important insights of EVI might also be gleaned through a subjective assessment of how knowledge about risks would change strategic decisions in a conservation project. We see substantial value in trying to link an EVI process to subjective risk assessments such as that conducted here, especially as there is likely to be a relationship between the relative importance of a risk and its impact on strategy choice.
We used a very straightforward measure of risk uncertainty, the coefficient of variation across experts. Given the importance of risk uncertainty in determining the appropriate response to the risk it might be valuable in future to use a more sophisticated approach to understanding uncertainty. An improved process could involve seeking estimates of the range of a risk from optimistic through to pessimistic Ward 2007, Martin et al 2012). Research in expert elicitation suggests that overconfidence bias can be substantially reduced (and therefore accuracy improved) by asking for such interval judgements using a four step question approach (Speirs-Bridge et al 2010). Using this four step approach would mean asking each participant for both high and low risk estimates, their best guess, and the probability that the interval between their high and low estimates captures the true risk.
Focusing risk assessment at the level of project objectives has the advantage of forcing a broad view of strategic risks, making it akin to a foresighting exercise (Koivisto et al 2009). The downside, however, is that estimating the consequence and likelihood of high-level strategic risks is likely to be less reliable than it would be for tactical risks as experts are less able to receive feedback about the accuracy of their predictions. The challenge of accurately estimating risks is amplified by the fact that nearly all large conservation projects involve working on relatively wicked problems in complex systems (Game et al 2013b). Rather than diminishing the relevance of risk assessment in conservation planning, we have found that the structured nature of a subjective risk assessment can serve as a powerful situation analysis tool for complex conservation projects. Simply being asked to imagine what could go wrong can highlight dependences on particular assumptions that had previously been implicit or unnoticed. For example, a risk assessment that one of us ran for another conservation project, revealed a pivotal reliance on an unacknowledged assumption that increasing wealth and entrepreneurship amongst an indigenous community would not perversely accelerate environmental degradation. For complex projects, strategic risk assessment should be seen as a tool to understand what elements the project's success is most dependent on.
The aim of conservation planning is not to reduce risk per se but to maximize performance of the chosen strategies. This requires considering opportunities as well as risks. A focus on identifying and ameliorating risks should not dominate an equivalent effort on identifying and exploiting opportunities (Hillson 2002). In practice, conservation planning is far more frequently triggered by or involves the latter as opposed to the former. Risks and opportunities are rarely independent (Ward and Chapman 2003). For example, in the northern Australia case, risks such as the failure of the carbon market or loss of government funding for Indigenous Protected Areas, only exist because the project is exploiting valuable opportunities in these areas.
Our hope is that risk assessments such as the one described here, become a standard part of conservation planning. At present, risk assessment is entirely consistent with the predominant conservation planning frameworks but is still rarely engaged in. As identified in section 1, there are numerous aspects of conservation planning that should be influenced by the understanding gained during risk assessment. We acknowledge that including risk assessment as an important step in conservation planning might be viewed as adding just one more task that gets in the way of actually doing conservation. However, this sort of subjective risk assessment can be accomplished very rapidly. In another letter in this special issue, Dale et al (2013) point out how a similarly rapid, subjective risk assessment was able to reveal a great deal about the need for governance reform on the Great Barrier Reef. The relatively strict form of subjective risk assessment also forces clarity around the objectives and how achievement of them will be measured-from our experience, this alone is a significant benefit for conservation planning (Game et al 2013a(Game et al , 2013b. Conservation funders do not expect the strategies and projects they support to be fail-safe, and our experience has been that they value honesty about the risks involved. However, there are barriers to the explicit acknowledgment and presentation of risks. Effective risk assessment requires those involved to be candid about the project. Unlike most other aspects of conservation planning that almost invariably benefit from being conducted in an open and collaborative fashion, an honest risk assessment may itself pose a risk to institutional relationships and funding opportunities, especially when risks refer to the reliability of key institutions or organizations or failures of leadership. Given that building trust between organizations is frequently cited as one of the strongest outcomes of a conservation planning process (Bottrill et al 2012), it may be the case that the risk assessment component of a conservation plan needs to be conducted 'in-house'.
Explicit identification, prioritization, and where possible, management of risks are important elements of using conservation resources in an informed and accountable manner. Informed and accountable resource use, are also the ideals of conservation planning. We believe that an assessment of risks to conservation success should be considered a core part of conservation planning, and one that can be accomplished rapidly using the approach illustrated here.