A simple, sufficient, and consistent method to score the status of threats and demography of imperiled species

Introduction

The administration and monitoring of conservation programs are closely entwined. Administrators charged with conserving imperiled species must do so often under acute budget and personnel constraints (Ferraro & Pattanayak, 2006). At the national and regional scales, these decision makers need to accurately evaluate hundreds or thousands of species based on their conservation status as part of their decision on how to allocate limited resources efficiently and objectively for the greatest conservation benefit (Bottrill et al., 2008; Joseph et al., 2008). At the same time, assessing the effectiveness of large, complex conservation programs is challenging because of the taxonomic breadth of species and the variety of threats they face (see Purvis et al. (2000) for a summary of the many factors affecting extinction risk). Rarely do metrics capture necessary information concisely and consistently across all species. But such metrics are needed for wildlife managers to monitor outcomes and effectively allocate resources.

Conservation program administrators need a small, yet strategic, number of highly informative and consistent metrics to efficiently and accurately evaluate the conservation status of each species and conservation programs as a whole. This need can be characterized in three ways:

• 1.

  Simplicity. A method to monitor the conservation status of hundreds or thousands of species has to be easy to implement. First, such a method has to be able to scale to a regional or national scope and hundreds or thousands of practitioners in the field. A complex method may provide more detail and data, but is more likely than a simple method to fail at scale. Second, limited budgets for conservation place real-world constraints on the complexity of what can be implemented.

• 2.

  Sufficiency. Two fundamental components of conservation status are a species’ demography and the threats it faces (Goble, 2009; Neel et al., 2012). Separating these factors is crucial because strategies for addressing threats and demographic status can differ greatly, e.g., population augmentation may improve demographic status while threats that will ultimately undo those gains continue unabated (National Marine Fisheries Service, 2010). These two components are composed of sub-elements (e.g., population size, range, and structure are elements of demography), but knowing these details is not necessary to evaluating conservation status across a program. That is, while a single conservation status metric is insufficient, using more than two metrics is more costly and provides more detail than is necessary for evaluating a conservation program as a whole.

• 3.

  Consistency. Large conservation programs may cover a variety of taxa, from lichens to mammals, that are managed and monitored by hundreds or thousands of people. Suitable monitoring metrics need to be calculable and interpretable consistently across species and personnel for programmatic evaluations to be meaningful.

These characteristics point to the need for a small number of monitoring metrics that would (a) be easy to calculate given existing data, rather than requiring new and expensive research or monitoring programs, (b) capture the status or change of threats and demography independently, and (c) be designed to apply consistently across all or most covered species. If such monitoring metrics are available, then the effectiveness of conservation programs can be evaluated in part (e.g., by geographic region) or in whole by analyzing the scores for all species under the program. For example, we could answer questions such as: What is the status of threats across all imperiled species covered by a specific conservation program? What proportion of imperiled species are declining or improving demographically? Are some geographic regions doing better, on average, at addressing the threats to imperiled species than other regions?

The U.S. Fish and Wildlife Service (FWS) reports in their Biennial Report to Congress two possible conservation status metrics for species listed under the U.S. Endangered Species Act (ESA). The first metric has changed over the years. Until 2010, FWS reported species status using categories including “declining”, “improving”, “stable”, or “unknown.” FWS stopped reporting each species’ “status” after 2010 because they judged the conclusions were not scientifically rigorous enough (U.S. Fish and Wildlife Service, 2011). Today, FWS reports recommendations to reclassify a species’ legal status that are based on five-year reviews of each species. Recommendations may include uplisting from threatened to endangered, downlisting from endangered threatened, delisting a species, or no status change (see Article S1 for an overview of the ESA listing lifecycle). The second reported metric is the Recovery Priority Number (RPN), which is used to prioritize recovery planning for ESA-listed species. RPNs are based on the immediacy of threats, recovery potential, taxonomic uniqueness, and conflict with human activities (U.S. Fish and Wildlife Service, 1983). They are represented by a “base” number from 1–18 (highest to lowest priority), and are tagged with a “C” if there is conflict with economic activity. Thus, both legal status and RPNs contain some information about conservation status and both are used in some fashion by FWS to allocate resources and make other decisions. But the question remains whether these reported metrics are acceptable for monitoring the conservation status of species, or evaluating the effectiveness of the Endangered Species program.

There are three challenges with using the metrics reported by FWS as conservation status metrics. First, a species listed as endangered cannot be afforded more protection under the ESA, and neither Congress nor the public receives an early warning if an endangered species has continued to decline. In contrast to IUCN Red List categories that include “critically endangered” and “extinct in the wild” as options before extinction (Rodrigues et al., 2006), the ESA recognizes no classification between “endangered” and “extinct”. Second, some changes in either threats or demography may not be sufficient to trigger reclassification, but are still sufficient to warrant the attention of managers during the monitoring and evaluation stages of the recovery and resource allocation process. FWS administrators will be hard-pressed to make informed resource allocation decisions across the endangered species program without simple, sufficient, and consistent metrics of conservation status as part of the equation. Thus, on the first and second counts, recommendations for reclassification have significant shortcomings. Third, although used in conjunction with other information to guide resource allocation (U.S. Fish and Wildlife Service, 2013), RPNs are not sufficient for evaluating species status because they combine many factors, including some that are not conditional on changes of status (e.g., taxonomic uniqueness). Because the conservation status of individual species and groups of species is the ultimate metric by which conservation programs need to be evaluated, neither Congress, the Executive Branch, nor the public can accurately evaluate the effectiveness of the ESA at recovering species using currently reported metrics. Furthermore, some species can “fall through the cracks” of conservation while recovery progress for other species goes unacknowledged. This is not to say that such species receive no attention; biologists and managers in the field may be aware of a species’ plight. But regional- or national-level administrators are much less likely to know about these issues, and cannot make informed, high-level resource allocation decisions if unaware of the facts.

Here we report on a simple, sufficient, and consistent key that can be used to translate information in detailed status reviews for imperiled species into scores denoting changes in threats and demography. This key provides the type of guidance that the Inspector General of the US Department of the Interior recommended in 2003 (U.S. Department of the Interior IG, 2003). To illustrate its use, we apply the key to two scenarios. First, we apply the key to 37 species for which FWS has recently recommended reclassification and 15 species without such a recommendation. While the recommendations are largely consistent with the scores extracted using the key, we confirm that they oversimplify conservation status. Specifically, recommendations of no status change are particularly prone to masking changes in the threat or demographic status of species. We also show that RPNs are moderately correlated with threat and demographic changes, but not consistent with either. Second, to illustrate its use across a specific region, we apply the key to non-plant ESA-listed species over which FWS has primary jurisdiction in Florida. The results indicate the threat and demographic statuses of these species are mostly declining, and the repeatability test indicates the need for detailed training of those who generate the scores. Building from these results, we provide recommendations for implementing the proposed scoring key for FWS’s Endangered Species program that will improve the monitoring and recovery of species.

Materials & Methods

Results

Analysis 1: we identified 52 species across nine taxonomic groups that met our criteria for the first analysis (summarized in Table S1). In Table S2 we provide example text from selected five-year status reviews that illustrates the range of threat and demographic scores given the criteria in Table 1. The mean threat and demographic change scores were positive across all species (${\bar{x}}_{\mathrm{threats}}=0.173$ and ${\bar{x}}_{\mathrm{demography}}=0.038$), but there was considerable variation overall (IQR_threats = 0–0.625; IQR_demography = − 1–1).

We identified 27 species recommended for downlisting or delisting, i.e., improving status. Only 15 of the 27 species had positive scores for both threats and demographics. Four had threat alleviation scores of zero (i.e., no change) but positive demographics scores. Another four had demographic scores of zero but positive threat alleviation scores. No species in this category had a negative score in both categories.

Ten species were recommended for uplisting (i.e., declining), of which only copperbelly water snake (Nerodia erythrogaster neglecta) had non-negative scores for both threats and demographics. All other declining species in this category had a negative demographics score and a threat alleviation score of zero or lower.

Two of the 15 control species had negative scores for both threat alleviation and demographics, and six species had a negative score for either threat alleviation or demographics. Three control species had a score of 0.5 (i.e., moderate improvement) for either threat or demographic change, and one control species, Yellow-shouldered Blackbird (Agelaius xanthomus), had positive scores for both threat alleviation and demographics.

Status change recommendations tended to reflect the changes in the combined threat change and demographic change scores (Fig. 1A), but the consistency with threat change scores (Fig. 1B) was weaker than the consistency with demographic change scores (Fig. 1C). This was supported by the results of the multinomial model (Table 2). Models 1 and 2 had very similar, low AIC_C scores even though model 2 had twice as many parameters. Even though the AIC_C for model 1 is lower, model 2 should be preferred for application because threat and demographic parameters are separable.

The results of the linear discriminant function analysis were similar to the multinomial model analysis. FWS status change recommendations were consistent with classifications from Model 2 for 81% of species, with >88% of improving and declining species consistently classified (Fig. 2). FWS status change recommendations may have resulted in underprotection of seven species and overprotection of three species given the classifications based on threat and demographic change scores. The consistency of FWS status change recommendations was 71% under Model 1 (demography and threats scores added), 65% under Model 3 (threats only), and 62% under Model 4 (demography only).

Both threat and demographic change scores were significantly correlated with FWS’s RPNs (range of r: 0.31–0.45) at α < 0.05 (Fig. 3). The AIC_C values for Models 1–3 were within 1 unit of each other, indicating these models are approximately equally parsimonious (Table 3). Model 2, which treats threat and demographic change scores separately, had the best overall fit among the three models. However, the relative importance of threat change scores to predicting RPNs is supported by both the higher confidence of the threat parameter estimate of Model 2 and the good fit of Model 3.

Analysis 2: on average, we found negative threat and demography scores across 54 ESA-listed, non-plant species in Florida over which FWS has primary jurisdiction (Table 4). Three species had positive scores for both threat and demographic status; seven had positive scores for either threat or demographic status; and the remaining 44 species had scores of zero or lower for both status components. Three species were recommended for downlisting and were a part of Analysis 1, 37 species were recommended for no status change, and the remaining 14 species were listed too recently for a recommendation. Threat and demography scores were correlated across all species at r = 0.72 (t = 7.19, p = 3.74e⁻⁹).

We found variation in both threat and demography scores recorded by different scorers (Fig. 4). For threat scores, the identity of the person scoring was not statistically significant (F = 0.53, p = 0.71), the mean square for the person term was small (0.13) relative to the mean square for species (1.78), and the intra-species score correlation was 0.56 (95% CI [0.30–0.84]). For demography, the person term was statistically significant (F = 4.26, p = 0.008), but the mean square term was still small (0.73) relative to the mean square for species (2.34) and the intra-species score correlation was 0.70 (95% CI [0.44–0.90]). The 95% confidence interval for mean threat score across scorers did not overlap zero (−0.47 to −0.06), but did for demography score (−0.35–0.16). For comparison, the confidence intervals for both threat and demography status overlapped zero for these ten species when calculated from scores of the single scorer for all Florida species (JWM).

Discussion

Simple, sufficient, and consistent conservation status metrics are needed to ensure that managers of large-scale conservation programs can make informed decisions (Kleiman et al., 2000; Ferraro & Pattanayak, 2006). While useful on a case-by-case basis, voluminous unstructured information (e.g., all of the data in all five-year reviews) cannot be used to evaluate the performance of a large conservation program. Too little information in status metrics can lead to unintentional neglect or poor, uninformed decisions. Simple but sufficient status metrics and quantitative summaries across species are also needed for oversight by lawmakers and for the public to understand program importance and effectiveness (Sanderson, 2002; U.S. Department of the Interior IG, 2003). The ESA is widely considered the strongest law in the world for imperiled species conservation (Bean & Rowland, 1997), but currently no simple and sufficient conservation status metrics are reported for the species it protects. We developed a simple, sufficient, and consistent key for translating detailed conservation status information into two scores—one for threat changes and one for demographic changes—that can enable the necessary species and program evaluations. We undertook two separate analyses to illustrate the use of this key. First, while we found that changes in threats and demographics were often consistent with two metrics reported by FWS for a set of 52 ESA-listed species, our results illustrate the need for refined metrics. Second, we found that ESA-listed species from Florida are, on average, not faring well, and that there is room to improve the consistency of scoring threat and demographic status among individual scorers. These two examples highlight applying the main result of the paper: a simple, sufficient, and (mostly) consistent key to score threat and demographic status of imperiled species at broad scales.

Our first analysis highlights a significant problem with current status reporting methods: species not recommended for listing reclassification were most likely to be neglected because of oversimplification leading to a perception of less need for attention. Forty percent of control species (no status change) showed threat and/or demography status changes even though FWS did not recommend reclassification. Because there are ca. 690 domestic, non-plant ESA-listed species, of which the vast majority are not proposed for a status change, our results suggest that a significant number may currently be treated as “stable” but are actually declining or improving. Additional data are needed to arrive at a robust estimate of the number of mis-represented species, or all species may be scored to have a census. Seven of the 15 species for which FWS recommended no status change had negative scores for threats and/or demography. Five of the seven are endangered, highlighting how FWS’s current reporting metric cannot indicate that the status of these species is deteriorating, at least not until they are dangerously close to extinction. In contrast to the problem of masking continuing declines, we also found that improvements can be masked: four endangered species had positive scores for threat and/or demographic changes. In each of these categories, the FWS recommendation of not reclassifying hides underlying threat or demographic changes that can shape how scarce conservation resources are allocated. Future work is needed to determine the extent to which ESA-listed plants and foreign species are susceptible to this same problem.

Species recommended for reclassification are not necessarily immune from conservation neglect. While most recommended changes were consistent with the threat and demography scores, there were many inconsistencies and hidden problems. For example, three species recommended for downlisting had either a negative threat or demographic change score. Conversely, the copperbelly water snake (recommended for uplisting) had a score of zero for both threats and demography. That is, the scores for these four species reveal changes that were masked by the recommendations made by FWS. In each case, however, the authors of the reviews directly addressed the discordance between the results of the review and the recommendation. For example, the biologists for both Smith’s blue butterfly (Euphilotes enoptes smithi) and Stephen’s kangaroo rat (Dipodomys stephensi) concluded that threats had been ameliorated to such a degree that neither species was in imminent danger of extinction (FWS, 2006; FWS, 2011). In both instances, FWS reported that population size was hard to quantify because of inadequate data, but that some populations of each species were likely declining. However, FWS concluded that while threats were still present and may have contributed to lower numbers, those threats had been ameliorated or eliminated to the point that the species no longer qualify as endangered. In the case of the copperbelly water snake, the species had met uplisting criteria by the time the recovery plan was finalized (FWS 2008). At that time FWS recommended uplisting, which was recapitulated in the subsequent five-year review (FWS, 2010). Cases like that of the water snake, a species that should be classified as endangered but whose scores were zero, may appear to indicate that our proposed scoring system is insufficient; how would an administrator know that the species needs attention? We suggest that the snake might have received earlier conservation intervention if a sufficient scoring system had been in place to highlight the species’ decline. Importantly, the cases above illustrate that the proposed scoring system complements, but does not replace, detailed status reviews: two scores cannot capture all of the nuances of individual species. The details may indicate a positive threat score has been, in the words of one reviewer, a threat reduction “only from really horrible to clearly unsustainable”.

Our results indicate that FWS tended to use a combination of threat and demographic information in status change decisions across the analyzed species. It is unclear whether both components of conservation status are used for every species, yet both are needed to distinguish between threatened and endangered species (U.S. Fish and Wildlife Service, 2010). For example, five of the ten species recommended for uplisting had negative demographic scores but threat scores of zero; none of the ten species had negative threat scores and demographic scores of zero. If threat changes were given more (perhaps equal) weight in uplisting decisions, then we might expect more species recommended for uplisting would have negative threat scores. The observed pattern suggests that FWS may not recommend species for uplisting until data show diminished demographic status, which may preclude early actions that can help species avoid deeper declines. In an ideal world, mechanistic models linking threats to demographic changes would be available for all species, and decisions could be made based on those threats, before symptoms arise. However, such models exist for very few species and best professional judgment must be used. We recommend that FWS biologists critically evaluate whether they are giving due weight to threats when evaluating status changes.

Our second set of analyses, focused on ESA-listed, non-plant species in Florida over which FWS has primary jurisdiction, resulted in two insights. First, we found that the 54 species had negative scores for both threat and demographic status on average, and only 18% exhibited at least one status metric that was positive. Twenty-eight species (52%) were scored −1 for both threat and demographic status. Simply knowing this distribution of status scores may lead administrators to investigate questions about whether there are common characteristics of species (or their management) that have at least one status component improving. Can we allocate some resources from species that are improving to species that are not, without losing the gains? Although using threat status alone may lead to suboptimal resource allocation decisions (Joseph et al., 2008), the data gained from applying the proposed key are one critical type of information needed to improve budgeting for ESA implementation (Gerber, 2016).

Second, we found variation in how different individuals score threat status and demographic status, even when working from the same information. This is not unexpected given what is known about variation in expert elicitation (Morgan, 2014). The lack of a statistically significant effect for scorer for the threat component, and the high intra-species correlations (i.e., consistent relative scoring among scorers) for both components, are promising results. However, the significant effect of scorer for demographic status suggests that refinement is needed. We suspect that additional training materials would improve scoring consistency, but further investigation is beyond the scope of this paper. We also note that, if the key is applied by FWS, then it will be applied by experts who have intimate knowledge of the species in question. Our primary concern is that their additional knowledge may lead to scores based more on intuition than on critical analysis of available information in light of the criteria laid out in the scoring key. While more intensive elicitation methods may reduce inter-individual variation and unconscious biases (Morgan, 2014), the limited resources for implementing this scoring key may preclude using such methods.

The proposed scoring system can substantially improve monitoring and implementation of complex imperiled species programs by enabling approaches that are currently unavailable. For example, negative threat or demographic change scores that persist over several evaluation periods should highlight species that require attention and allow conservation managers a more precise understanding of a species’ conservation status. Over an extended timeframe, the change scores for a suite of species may provide data needed to warn of sudden state changes to the complex systems (Scheffer et al., 2009) of which ESA listed species are a part. Another extension of the scores is that sudden deviations from past scores can signal the need for prompt intervention. For example, the status of both Indiana bat (Myotis sodalis) and gray bat (Myotis grisescens) was reported as improving in each species’ five-year review (FWS, 2009a; FWS, 2009b). Those reviews coincided with the appearance of white-nose syndrome (WNS; and its agent, Pseudogymnoascus destructans) but little was known about the potential demographic effects of the new threat (Blehert et al., 2009). The sudden appearance of a negative threat change score would stand out among the positive threat and demographic change scores for previous reviews for these species. For example, during the run-up to the ESA listing of the closely related northern long-eared bat (Myotis septentrionalis) in 2013–2014, the wind-energy industry proposed funding research to find pharmacological or other mechanisms (e.g., rapid adaptation and evolutionary rescue; see Yoshida et al., 2007; Ellner, Geber & Hairston Jr, 2011; Maslo & Fefferman, 2015) to combat WNS. If the severity of the issue had been properly flagged in 2008, then it may have triggered more effort and interest by administrators to prioritize critical research earlier. If the research was successful in identifying treatments or resistance alleles, it may have been possible to reduce the dramatic demographic declines of these species. Having quantitative threat and demographic change scores is necessary to enable these or other analyses and resultant management decisions.

Is there an alternative to the proposed key that still satisfies the needs of scoring conservation status in a way that informs program administration and monitoring? Perhaps. One reviewer suggested that scoring in 0.5-unit increments was too fine-scale; the inter-individual scoring variation we observed may support that. However, we have found the 0.5-unit increments are needed in many cases where the narrative indicates a situation other than all-or-none, and think increased training can reduce inter-individual scoring variation. While much of the focus here is on one of the five scores for threats and demography, the inclusion of the “No information available” score in the key has been very useful to FWS administrators who didn’t know what wasn’t known. The lack of a standardized way of communicating data gaps is a key problem identified in the 2003 Inspector General’s report (U.S. Department of the Interior IG, 2003) and addressed here. Even using a three-level system (−1, 0, 1) plus an “Unknown” category, while keeping threats and demographic changes separate, would be amenable to statistical analysis and add critical information needed for informed decision-making.

Conclusion

The proposed scoring system provides a simple, sufficient, and consistent way to track threat and demographic changes across the many taxa covered by large imperiled species programs. While scores are generated on a per-species basis, we do not intend for these scores to inappropriately be the basis for decisions such as listing or delisting (in contrast, see Robbins, 2009; Regan et al., 2013). Instead, the scores for individual species may best be seen as important indicators of problems or successes that warrant deeper investigation. We recommend that conservation programs lacking a broad monitoring program that separates threat and demographic statuses implement the one proposed here. This includes FWS’s and National Marine Fisheries Service Endangered Species programs, for which current reporting falls short.

We expect that implementing our proposed system adds very little workload to reporting requirements already in place, but will provide program managers and the public with much-needed information. For example, ESA-mandated five-year status reviews already require a substantial investment to compile. Adding two lines in the summary section would not require much additional work. Similarly, updating a central database of threat and demographic change scores when there is a status change, or when a five-year review is submitted, would be highly informative. Eliciting expert opinion with proper guards for recognizing uncertainties (Morgan, 2014) can ensure robust estimates of threat and demographic status.

While the mean threat and demographic change scores were marginally positive across the species in analysis 1, that sample is biased toward species recommended for down- or de-listing. Based on the negative-trending scores of the no-change species examined here, the sample of 54 species from Florida, and the work of Male & Bean (2005), we suspect that mean values would be negative if data were available for all ESA-listed species. Such a result would not be surprising given the vastly inadequate funding for endangered species in the United States (Taylor, Suckling & Rachlinski, 2005; Gratwicke, Lovejoy & Wildt, 2012; Negron-Ortiz, 2014; Gerber, 2016). Using simple and sufficient status metrics may provide the evidence that underscores the need to reverse the funding shortfalls and thereby improve conservation outcomes.

Supplemental Information