Comparing Invasive Species Risk Screening Tools FISRAM, ERSS, and FISK/AS-ISK as a response to Hill et al. (2020)

The U.S. Fish and Wildlife Service (USFWS) developed an invasive species risk assessment procedure, presented in Marcot et al. (2019), consisting of an existing rapid assessment process for plants and animals (Ecological Risk Screening Summary; ERSS) and a new probability-based Bayesian network model (Freshwater Fish Injurious Species Risk Assessment Model; FISRAM), that may be used to help prioritize invasive freshwater fish species for Federal injurious wildlife listing evaluation by USFWS under the Lacey Act. Hill et al. (2020) provided a rebuttal letter of Marcot et al. (2019) but did not critique FISRAM, the main subject of that paper. They focused instead on the ERSS process and on our characterization of several other existing risk assessment models and procedures. Here we provide our reciprocal rebuttal by addressing their criticisms. Hill et al. (2020) implied that we equate the Fish Invasiveness Screening Kit (FISK) and Aquatic Species Invasiveness Screening Kit (AS-ISK) to the USFWS risk analysis process for listing species as injurious, their “apples to oranges.” However, the USFWS process for evaluating species to list is an extensive risk analysis process explained in our paper that follows laws and standards and optionally includes ERSS as a rapid screen in an early stage. Their interpretation that we compared our rigorous listing process to the risk screening tools FISK and AS-ISK was incorrect. We explain how we use expert opinion and reiterate that the information inputs for the ERSS reports, primarily climate match and invasiveness history, are good predictors of species invasion and are appropriate for a rapid screen for use in many situations. Their criticism of a lack of regional calibration of ERSS is answered by the ERSS climate-matching heat maps that show a color-calibrated continuum of climate match for the contiguous United States. We further explain our comprehensive peer review process and why their suggestion to have each ERSS report peer reviewed is infeasible. We also discuss their letter’s misrepresentation of the injurious wildlife listing process, which can use ERSS and FISRAM to advise prioritization and to provide documentation and decision support. All models described in our paper have value, and management entities should review the literature published by the respective developers to learn of their individual utility.


Introduction
We developed the Freshwater Fish Injurious Species Risk Assessment Model (FISRAM) as a new tool that the U.S. Fish and Wildlife Service (USFWS) can use in several ways to address USFWS's mission to protect native ecosystems against invasive and injurious species. We published a description of the model in Marcot et al. (2019; hereafter, also referred to as our paper) and explained how FISRAM could also be used by other entities to help determine the risk of an invasive freshwater fish of their concern. To explain why we needed the new model, we compared FISRAM to existing risk assessment models and noted why they did not exactly meet the USFWS's needs, which include a categorization of species that are injurious but not necessarily invasive. Hill et al. (2020; hereafter, also referred to as their letter) disagreed with some of our statements. The title of their letter "Comparing apples to oranges and other misrepresentations of the risk screening tools FISK and AS-ISK -a rebuttal of Marcot et al. (2019)" indicates that Hill et al. (2020) focused their comments on our paper's discussion of the use of FISK and AS-ISK, hereafter referred to as FISK/AS-ISK, within the USFWS's broader risk analysis framework used to promulgate rules for listing species as injurious wildlife. Their letter criticizes the background information that our paper provided, particularly our existing rapid screening tool, and barely mentions the primary subject of our paper, which is a research article about the development of a secondary risk screening process (FISRAM) that we developed using a quantitative probability network model. Interestingly, FISRAM was not critiqued.
The background information that their letter focuses on was the Ecological Risk Screening Summary process (ERSS; already developed by USFWS and in use) and how it can contribute to USFWS's selection and evaluation of species to list as injurious wildlife by rule promulgation under the injurious provisions of the Lacey Act (18 U.S.C. 42 (a)). Marcot et al. (2019) included that background information simply to provide context for why FISRAM was developed. Hill et al. (2020) discredits valid comparisons between FISRAM and FISK/AS-ISK, thus hampering comparisons between the strengths and weaknesses of each of these riskscreening approaches so management agencies and others can utilize the approach that best informs their risk assessment and risk management needs.
Here we discuss their letter's misrepresentation of the injurious wildlife listing process, as well as valid comparisons among risk screening tools made by Marcot et al. (2019). We also respond to some of the more substantive comments made by Hill et al. (2020). These include: (1) invasion history; (2) expert opinion; (3) regional calibration and climate matching, and (4) need for peer review. Hill et al. (2020) misstates how the ERSS reports are used in injurious wildlife listing and other aspects of listing under 18 U.S.C. 42. We disagree with several statements. First, they state that high risk species from the ERSS are then considered for listing as injurious, implying that USFWS considers all high-risk species for listing as injurious. Our paper did not state that, and that is not accurate. Figure 1 in Marcot et al. (2019) shows that a species with an outcome of "High risk" from ERSS "is advanced as a species to consider for listing as injurious." The ERSS reports are one way to prioritize which species to consider further evaluation for listing (Marcot et al. 2019) and where compiled information can be pulled from efficiently. Because the number of potential species needing to be evaluated has grown over the decades, USFWS needed a way to rapidly identify highrisk species for prioritization for management action. Therefore, our paper stated that the current FISRAM work was spurred by the need to rapidly and efficiently identify nonnative species that may become injurious. Thus, the USFWS listing process is not dependent on either ERSS or FISRAM. In fact, FISRAM was developed only for freshwater fish (with the intent and architecture to later modify it into versions for use with other taxa), but USFWS can list wild mammals, wild birds, reptiles, amphibians, mollusks, and crustaceans as injurious (Marcot et al. 2019), as it has been doing with the USFWS standard injurious wildlife evaluation criteria for risk. We modified their letter's Figure 1 to show the relationship between ERSS and injurious listing and to demonstrate the degree to which FISK/AS-ISK could substitute for ERSS or FISRAM within the USFWS's broader risk analysis process ( Figure 1). Second, their "low threshold of evidence" statement (Hill et al. 2020, p. 9) did not accurately reflect the USFWS's injurious wildlife listing process. The USFWS publishes proposed and final listing rules as required by law under the Administrative Procedure Act (5 U.S.C. Subchapter II) and uses the standard of evidence required under that law. Furthermore, their letter states there is a blanket prohibition on importation, but Corn and Johnson (2013), whom they cited, stated accurately for what purposes permits may be issued.

Injurious wildlife listing process
Finally, their statements that the "the high risk species from the ERSS are then considered for listing as injurious" being "in stark contrast with guidance on interpreting FISK/AS-ISK for which species scoring as highrisk are assessed further" (Hill et al. 2020, p. 9) completely ignore that the species are assessed further under USFWS's standard, comprehensive evaluation for injurious wildlife listings (Box 1 of Marcot et al. 2019). We believe it is evident that injurious wildlife listing decisions are not made by ERSS reports alone as currently utilized by USFWS under existing Federal law.  Figure 1 flow diagram, showing the general process for USFWS to evaluate a freshwater fish species' risk of being injurious when an ERSS results in "uncertain" output, followed by FISRAM as a secondary screen, both of which are used as optional additional decision support tools. The injurious wildlife evaluation criteria are the only criteria always used in rule promulgation. FISK/AS-ISK could be substituted as a risk screening tool for either ERSS or FISRAM for decision support. Hill et al. (2020) dismiss the comparisons between FISRAM and FISK/AS-ISK by stating that FISRAM is part of a risk analysis process and that FISK/AS-ISK is a risk screening tool designed to identify those species likely to pose a high risk of becoming invasive and, therefore, warrant additional comprehensive assessment. While it is true that FISRAM can be viewed within a broader Federal risk analysis context, FISRAM can also be viewed as an independent risk screening tool without components of risk management or risk communication (see Wyman-Grothem et al. 2018 as an example). In fact, we believe it could be appropriate to substitute FISK/AS-ISK for either ERSS or FISRAM within the Federal risk analysis context (Figure 1), depending on information available and the desired form of the results (qualitative or probabilistic), while also acknowledging that each approach has strengths and weaknesses (Table 1). Information on the background, requirements, application, and implementation of FISRAM can be found in USFWS (2019).

FISK/AS-ISK compared to the USFWS ERSS tool and not FISRAM
Our paper explained the need for Federal agencies to develop tools to aid in rapid screening of invasive species (NISC 2008). The screening of thousands of species within the organisms in trade pathway is a daunting task (Hill et al. 2017). In an initial step in delivering on this mandate, USFWS developed the ERSS process to efficiently and effectively identify vertebrate, invertebrate, and plant species from potentially tens of thousands in trade whose invasion history and risk of establishment through climate matching suggest they are a risk for invasiveness to U.S. environments. However, the ERSS process leaves many risk screening results as "uncertain" (see more below under Invasion history) and thus the need for an additional, probability-based risk screening process presented by Marcot et al. (2019). Marcot et al. (2019) provided an overview of USFWS's risk analysis process to put FISRAM into context for how results can be used to identify species for potential risk management action. It described: 1) the initial rapid screen process (ERSS), 2) the FISRAM model as a secondary screen (the focus of our paper), and 3) the risk analysis process that USFWS uses to evaluate species as injurious ( Figure 1). Our paper provided comparisons to other risk screening processes that we believe are most analogous to the FISRAM. For example, FISK/AS-ISK and FISRAM can use one or more species experts to assess the type and magnitude of potential harm from introduction into U.S. ecosystems (Hill and Lawson 2015;Hill et al. 2017;Wyman-Grothem et al. 2018). Both models have been used to inform State regulatory actions (Wyman-Grothem et al. 2018;Hill et al. 2017Hill et al. , 2014 and can obtain measures of variability around model output values (Hill et al. 2017;Marcot et al. 2019). Using additional risk-screening tools would offer many benefits because no one tool suffices for all needs, and decisions on where and when risk management actions are needed would be improved by comparing risk screen results from multiple approaches. However, we also posit that it would be difficult to utilize FISK/AS-ISK or FISRAM approach alone for all of the thousands of species moving in trade, and our paper suggested that ERSS is serving an essential role that would be challenging for either of these other models to fulfill (Table 1). Hill et al. (2020) state that invasion history and impacts are among the most uncertain and speculative data used in risk assessments. We found this statement inconsistent with a body of literature that has found invasion  (2008) defined as the over-or under-representation of risk within risk assessment. For FISRAM and FISK/AS-ISK, we posit that this potential resides more prominently with the risk assessor because some model inputs require expert knowledge that can be subjective. Whereas for ERSS, we posit that the risk perception bias resides more prominently within the process than with the risk assessor. 5 Hours if an ERSS has already been completed and days if multiple global experts are utilized to populate data in model inputs (Wyman-Grothem et al. 2018). 6 See Hill and Lawson (2015); See Hill et al. (2017). 7 RAMP is not intended for use in assessing climate match for marine species (Sanders et al. 2018 Vilizzi et al. (2019). 14 See Hill and Lawson (2015) where only one risk assessor was used to conduct the risk screen. 15 See Hill et al. 2014 andHill et al. (2017) where multiple risk assessors were used to conduct independent risk screens. 16 Marcot et al. 2019. 17 Sensitivity measures of model outputs to model parameters can presumably be performed by varying the threshold score for invasiveness with FISK/AS-ISK models.

Invasion history
history consistently among the most useful variables to predict whether nonindigenous species introduced into new ecosystems may become invasive, and many studies have been conducted showing that history of invasiveness is an important variable for several taxonomic groups in different parts of the world (Kulhanek et al. 2011;Hayes and Barry 2008). For example, Brooke et al. (1995) found that species establishing in nonnative areas were successful when introduced elsewhere. Williamson (1996) agreed with earlier research by Daehler and Strong (1993) and Lawton (1990) that previous success at invasions was a good indicator of whether an invader will succeed in a new place. Reichard and Hamilton (1997) found the single-most reliable predictor in their models for woody plants was whether a species was known to invade elsewhere. History of invasiveness has also been found to be a significant predictor of invasiveness for a variety of vertebrate taxa, such as Australian birds (Duncan et al. 2001) and in alien fishes in North America (Kolar and Lodge 2002). A proven history of invasiveness may indicate that a species has attributes that increase the risk of it becoming a successful invader in other compatible areas and is a key factor in predicting establishment success in New Zealand (Bomford 2003. Two meta-analyses on this subject have been conducted for a variety of taxa (e.g., birds, finfish, insects, mammals, plants, reptiles and amphibians, shellfish, jellyfish) in different parts of the world (e.g., Africa, Australia, Europe, Great Britain, New Zealand, and North America,) and with similar findings that invasion history can be a reliable predictor for identifying potential invasiveness (Hayes and Barry 2008;Kulhanek et al. 2011). This body of work on invasion history suggests that invasion history is a useful predictor to screen thousands of species in trade and identify those species in potential need for Federal, State, industry, or public riskmanagement action, contrary to the assertions made by Hill et al. 2020. We believe it is also important to note that Hill et al. (2020) references Leprieur et al. (2009) to substantiate their statement that invasion history is among the most uncertain and speculative data used in risk assessments; however, this referenced article was in fact a cautionary commentary that it is "dangerous and wholly inappropriate to equate a lack of data with 'no impact'" (Leprieur et al. 2009;p. 91). The ERSS process recognizes this cautionary note for false negatives by only finding low invasion history for species that: 1) have been transported outside of their native range due to substantial trade (millions of organisms) for substantial time (greater than 10 years) with no or very little evidence of establishment outside of its native range, or 2) are established outside of their native range but scientifically defensible studies conclude that there are no significant negative impacts (USFWS 2020). The ERSS process utilizes this high threshold to achieve a "low risk" characterization and pushes many species lacking data into an "uncertain" risk characterization for further evaluation. This approach is consistent with the conceptual model presented for a precautionary approach when there is high scientific uncertainty associated with no impacts for invasion history (Leprieur et al. 2009). Finally, while Leprieur et al. (2009) provide a strong caution that a lack of invasion history should not be interpreted as not being risky, they provide limited evidence that a species with a confirmed invasion history is a poor predictor of future invasions or that it is among the most uncertain and speculative data used in risk assessment, as suggested by Hill et al. (2020). Hill et al. (2020) state that Marcot et al. (2019) failed to acknowledge the extensive application of expert opinion in the ERSS, FISRAM, and injurious wildlife listing process. Marcot et al. (2019) stated clearly that the development of FISRAM utilized expert judgement and expert knowledge (analogous to expert opinion as used by Hill et al. 2020) in development of the model, such as deriving conditional probability tables, and populating the model's input values. We agree that expert species and expert risk assessment knowledge and judgment must be allowed for populating model inputs and estimating uncertainty.

Expert opinion
We believe there are important considerations when comparing different approaches (Table 1). Within both FISRAM and FISK/AS-ISK, expert knowledge is needed to populate model inputs that require interpretation and the estimation (ranking) of certainty (defined by Hill et al. 2020 as "subjective assessor opinion," p. 7; subjective input used in the following for discussion purposes). A good example that is shared between FISRAM and FISK/AS-ISK is whether predation by the evaluated species would have adverse effects within the risk assessment area. Other model inputs, such as whether the species is venomous, are objective and require no interpretation (Objective input, Hill et al. 2020). We estimate that 3 of 7 input nodes related to harm (43%; Predation, Competition, and Habitat Disturbance; Table 1) and 10 of 49 model inputs (20%; Q ID: 3.02, 3. 03, 3.04, 4.02, 4.05, 4.11, 5.01, 5.02, 5.03, 8.05 in Hill and Lawson 2015) are subjective for FISRAM and FISK/AS-ISK, respectively. For evaluations or models where empirical data are unavailable on any factor, we stress the importance of using verifiable, defensible, objective expert knowledge over more arbitrary opinion, although these terms are mixed in much of the modeling literature.
For ERSS, we believe both model variables-climate match and history of invasiveness-are consistent with objective inputs as implemented by the ERSS Standard Operating Procedures (USFWS 2020). The guidance is prescriptive and requires the risk assessor to have clear, convincing, and scientifically reliable and credible evidence from the scientific literature relating to the species' history of invasiveness to be high. A low invasion history has similar prescriptive objective guidance as previously noted, and if history of invasiveness is either data deficient or there are no known established nonnative populations, the risk rating is uncertain (USFWS 2020). Having prescriptive objective guidance is a strength of ERSS from our perspective because it allows USFWS to train staff biologists on risk assessment procedures but not require they develop expert species knowledge on the subject species to populate model parameters (Table 1), such as predicting the effects of predation by nonnatives on native species within a presumptive invaded ecosystem. This has increased the pool of agency staff that can produce ERSS reports and the number of reports that can be completed for its risk screening mandate (NISC 2008).
It is also important to note that FISRAM, FISK/AS-ISK, and ERSS each have the potential for risk perception bias, as defined by Gozlan (2008) as the over-or under-representation of risk (Table 1). For both FISRAM and FISK/AS-ISK, we believe this potential resides more prominently with the risk assessor because one-fifth or more of model inputs require subjective model inputs. ERSS reports also contain risk perception bias because results are weighted toward the precautionary principle (a risk) or toward "Uncertain Risk," requiring additional assessment. However, this bias resides more prominently within the ERSS process than with the risk assessor. This has benefits to the USFWS because agency policy can be reflected in the model, which can then drive risk tolerance rather than individual risk assessors with possible differing risk perspectives.
Moreover, FISK/AS-ISK calls for additional evaluation when a FISK score suggests a species would be invasive (high FISK score) versus those less likely to be invasive (medium and low FISK scores; Vilizzi et al. 2019). Arguably, in some instances where risk managers should be undertaking immediate preventative risk management action for imminent threats (Lodge et al. 2006), FISK/AS-ISK pushes the risk manager deeper into risk analysis for high-risk species. However, some jurisdictions and entities may wish to expedite prevention action with lower risk levels and thresholds of evidence given their willingness to accept risk and as informed by their respective authorities, responsibilities, and values. A strength of both FISRAM and ERSS model outputs is that they call for additional evaluations when risks results are not high or low (Table 1), whereas a species scored as medium risk through FISK/AS-ISK (along with a detailed synopsis) can be used to justify a State regulatory decision on species utilization, such as allowing for pond culture of Arapaima Arapaima gigas (Schinz, 1822) in Florida where risk of establishment is also predicted for parts of the State (Hill and Lawson 2015). Hill et al. (2020) criticized ERSS because they state it has no provision for calibration (we use "scalable" in this letter) and does not allow invasion history to be interpreted to the risk assessment area. This statement is only partially correct because USFWS's Risk Assessment Mapping Program (RAMP; Sanders et al. 2018) can be used to scale results to a particular risk assessment area. The climate matching procedure within FISRAM and ERSS uses a peer-reviewed algorithm developed by the Australian government into a web application (CLIMATCH; ABRS 2010; Bomford 2006Bomford et al. 2008) and utilized within RAMP. Climate matching has been used as an efficient and accurate predictor of compatible conditions that signal risk of establishment for a wide range of taxa and geographic locations. For example, Howeth et al. (2016) showed that "Climate match between a species' native range and the Great Lakes region predicted establishment success with 75-81% accuracy" (Howeth et al. 2016, p. 1). A similar result has also been reported between climate matching and establishment success for freshwater fish from around the globe (78%; Bomford et al. 2010).

Regional calibration and climate matching
Depending on the purpose and geographic scope of a risk screen, whether through ERSS, FISRAM or other risk screen, RAMP allows a target region to be set to a specific geographic scope to obtain a climate match result, such as the contiguous United States (the case with most ERSS reports), large watersheds (such as Great Lakes Basin in Howeth et al. 2016or Wyman-Grothem et al. 2018, or State jurisdiction (Sanders et al. 2018). Accordingly, ERSS has processes in place that allow for the calibration of climate match results to smaller risk assessment areas, contrary to statements in Hill et al. (2020) that results cannot be calibrated to the risk assessment area. In fact, all ERSS reports posted on the USFWS website have heat maps to allow risk managers to visually inspect areas of greatest and lowest risk of establishment as determined through climate matching. We do acknowledge, unlike FISRAM and FISK/AS-ISK, that the ERSS process does not allow risk assessors to scale the model's invasiveness variable-in the case of ERSS, invasion history-to only the risk assessment area (Table 1). We posit that ERSS, through RAMP, provides a realistic prediction of range and combined with a positive invasion history, suggesting that there is a risk of harm within that range. We conclude that the current ERSS results are appropriately scaled to the contiguous United States and provide useful risk screening results for the agency's purposes (NISC 2008). This is based on the purpose of this work in relation to the USFWS's authorities and national geographic responsibilities, as well as the body of literature supporting that an invasion history and climate match are significant predictors of invasiveness. Hill et al. (2020) criticize how USFWS conducted peer review for the ERSS process and we disagree with their assertions. The ERSS process has undergone peer review through a number of U.S. Federal government forums and processes, including the Office of Management and Budget's peer-review guidelines for influential science (OMB 2004) and within the Aquatic Nuisance Species Task Force's committees and a regional panel (Meyers et al. 2019). Their letter stated that USFWS's selection of five experts was laudable, but they faulted the process as "not a review of anonymous experts refereed by a journal's editors consistent with the gold standard of scientific peer-review" (Hill et al. 2020;p. 11). We counter that there are a variety of formal ways that Federal agencies can have documents peerreviewed that are not for journal publication, and the Office of Management and Budget's guidelines for Federal agencies for peer review describes them (OMB 2004). We stated in our paper that we abide by the Federal guidelines, and we also did that for the FISRAM process. We posted the USFWS's ERSS peer review plan online on a publicly accessible website (USFWS 2012) prior to the review and received no comments, and we posted the results of the review online (USFWS 2014). For that review, we included the names and affiliations of the five reviewers so that readers could decide for themselves if they think there is a bias, but we summarized their comments with no attributions to each comment. We assume that USFWS's posted report is how Hill et al. (2020;p. 11) knew who the "laudable" reviewers were. We believe the public should appreciate this transparency.

Need for peer review
Their letter also criticizes USFWS's lack of peer review by their standards for each of the more than 1,200 completed, individual aquatic species ERSS reports (Hill et al. 2020). We believe having a peer review for each ERSS report is an unreasonable expectation given the purpose of this work (NISC 2008). Not only would peer review of all ERSS reports require dozens of outside experts willing to review 10-20-page reports on a short turn-around, but it would require an estimated hundreds of hours of USFWS staff to compile the comments. This defeats the purpose of the ERSSs as a rapid screen, and each report should be understood as such. Moreover, an ERSS report that is cited in a proposed rule for injurious listing is included in the package that is provided to peer reviewers of the rule and made available for public comment (USFWS 2016). Our Figure 1 shows that, whether or not an ERSS report is used for determining if a species is injurious, peer review is done at the proposed rule stage, which was omitted from Hill et al. (2020). We also note that the public is encouraged to submit information, such as errors or omissions, on the ERSS reports that might help to improve the accuracy of the assessment and have provided a public forum to do so (see https://www.fws.gov/fisheries/ANS/ species_erss.html). These approaches are consistent with Federal rulemaking laws through the Administrative Procedure Act, and ensures that input received outside of rulemaking is objective and accessible to the public and does not run afoul of other Federal statutes, such as the Federal Advisory Committee Act (5 U.S.C. App.). Contrary to Hill et al. (2020) criticism that individual reports should be peer reviewed, we believe the processes put in place by the USFWS for review and public input on individual ERSS reports are appropriate given the procedural allowances and constraints of Federal agencies and objectives of this work.

Conclusion
Given the diversity and number of species moving in trade, approaches are needed that can accurately and rapidly categorize ecological risk. We assert that comparisons between ERSS, FISRAM, and FISK/AS-ISK are appropriate and that comparisons to each other are not false equivalencies. We advocate that risk managers benefit when different approaches are utilized because each bring their unique strengths, weaknesses, and biases. No one model fits all situations, and the unique characteristics of each model should be considered when incorporating a risk screening approach within any broader risk analysis framework. We posit that decisions on where and when to manage risks differ across the landscape by Federal, State, and local governments; industry; non-governmental entities; and the public. In some cases, risk screening results are sufficient to take risk management action. For Federal action through the injurious wildlife provisions of the Lacey Act (18 U.S.C. 42), these thresholds are set high as mandated under Federal statutes and authorities. Risk screening has been proposed to help prioritize-not substitute for-evaluations for those species that most warrant more in-depth risk analysis and rule promulgation under Federal law. This process was incorrectly characterized by Hill et al. (2020). We also point out that ERSS and FISRAM use the CLIMATCH algorithm through the Service's RAMP climate matching software, which allows these approaches to be scaled to different geographies and jurisdictions. We also believe Hill et al. (2020) mischaracterized the utility of using invasion history as an affirmative predictor of invasiveness and that their conclusion was inconsistent with a large body of invasive species literature. Finally, we advocate that appropriate comparisons of risk screening approaches are needed to define the strength, weaknesses, and biases of each approach so risk managers can incorporate these processes within broader risk analyses contexts to meet their risk assessment needs.