Beyond visualizing catch-at-age models: Lessons learned from the r4ss package about software to support stock assessments

Abstract

Stock assessment analysts are exploring an increasingly diverse and complex range of models while also facing higher expectations for consistency, documentation, and transparency in reports and management advice, all within a tight timeline. Meeting these goals requires increased efficiency at all steps in the assessment process from data processing, through model development and selection, to report writing and review. Here, we describe one widely used tool that has proven successful in increasing the efficiency of the assessment process: the r4ss package, which supports the use of the Stock Synthesis modeling framework. What began 15 years ago as a tool to provide simple model diagnostics, including plots showing data and model results, has grown into a large collection of R functions to support many aspects of the assessment process. We provide an overview of the r4ss features and illustrate its utility with examples from recent applications. Finally, we discuss lessons learned from the ongoing development of r4ss that can be applied to similar efforts associated with the next generation of stock assessment packages.

Introduction

Assessment of fish stocks (hereafter referred to as “stocks”) is a necessary task, largely because of mandates by federal and regional governing bodies to provide information about stock status and apply harvest control rules to inform catch limits under harvest policies. While incorporating disparate data sources into a single population model (integrated analysis) to determine stock status is routine, understanding the fit to each data set and its associated influence on the model results can be challenging (Maunder and Punt, 2013; Maunder and Piner, 2015). A standardized set of visualization tools is key to providing understanding and transparency throughout this process for stock assessment analysts, reviewers, stakeholders, and managers. For example, standardized tools allow analysts to quickly understand model results and explore new model configurations during the model development and peer review processes; reviewers scrutinize the analyses and investigate other alternatives with the aid of visualization tools, ultimately deciding if the assessment results are appropriate for use by management; and lastly, stakeholders and managers need to understand the model results, and hence, need intuitive visualization tools to inform the range of management options and decide on which management measures to take.

Visualization tools can aid analysts throughout the assessment process. For example, Richards et al. (1997) found while developing a stock assessment for Pacific ocean perch (Sebastes alutus) that visualization tools allowed them to better understand their data sets and pinpoint data features that needed to be accommodated, develop a statistical catch-at-age model well suited to the data sets, and evaluate model output more thoroughly. Stock assessments often require hundreds of model runs. Tools for quickly visualizing model results allow analysts to more efficiently select among them. As an illustration of the power of automated workflows and visualization tools, calculating residuals by hand would take hours, while visualizing patterns in residuals already plotted can take just minutes. Visualization tools can also relieve the feeling of being time-poor when conducting stock assessments (Bentley, 2015). Aside from efficiency, a thorough and standardized set of tools for visualizing model output can help catch errors such as misspecified models and aid in the report writing process, as most stock assessment reports require numerous figures and tables.

The peer review process for stock assessments (e.g., Brown et al., 2020), to determine if assessment results can be used by management bodies for decision making, benefits from visualization tools. For example, Regular et al. (2020) found that interactive data and model dashboards improved their ability to communicate with stakeholders during the stock assessment review process for a northern cod stock in the northwest Atlantic. Producing standardized figures across assessments increases ease of understanding for readers and simplifies comparisons across assessments. Often, peer reviewers are tasked with evaluating modeling decisions and model results, ultimately deciding if the assessment results are appropriate for use by management. Requests for visualizations made during assessment review processes are often expected in subsequent reviews, especially if the same reviewers may be engaged in the future, and should be added to assessment analyst toolboxes, such that they can be better prepared for future reviews. Thus, this toolbox grows with each review and helps facilitate efficient reviews, because analysts are able to quickly produce desired output before it is asked for.

The Terms of Reference (ToR) for stock assessment reviews have also coevolved with visualization tools, increasing the value of standardization. For instance, 10 years ago the ToR for groundfish stock assessments conducted for the Pacific Fishery Management Council (PFMC, 2009) had an eight-point bulleted list of general stock assessment team deliverables, while the ToR used in 2019 (PFMC 2019) had a checklist of 74 elements within 18 sections with more specificity. These ToR changes have been driven in part by feedback from reviewers seeing the benefit of new visualizations and diagnostics for individual assessments as described above. The ToR changes, in turn, lead to wider adoption of the new approaches for analysts working to meet them, a shift which is easier when the analysts can use shared tools to meet the new standards.

Effectively translating complicated assessment models and results into an easily digestible form for fishery managers and stakeholders can be challenging, especially when presenting information across a large range of stocks (Dichmont et al., 2016). Presenting assessment results in a consistent manner across stocks can lessen the communication challenge, allowing for improved discussions between analysts, stakeholders, and managers. The development and application of an assessment toolbox for use by analysts facilitates this process without creating additional workload.

Communication methods for stock assessment results are not a frequent topic in fisheries science journals, but it is an area where new ideas are rapidly developing and which deserves greater prominence in the literature. The widespread adoption of the generalized integrated analysis platform Stock Synthesis (SS, Methot and Wetzel, 2013) provided an opportunity to develop a standardized set of visualization and automation tools, given a larger pool of potential applications, users, and contributors (Punt and Maunder, 2013). Here, we discuss how r4ss, an R package containing tools for working with SS models, has improved the stock assessment development and review processes for individual analysts, reviewers, and managers over its 15 years of active development. We also highlight lessons learned from developing and using r4ss that could be applied when developing new visualization tools for a new stock assessment modeling platform.

The r4ss package grew organically from a single code script written by a single author in 2005 for use in the R statistical programming language (R Core Team, 2020) to a large open-source R package with many contributors. Before r4ss was developed, the typical workflow for SS users was examining the output text files directly or importing them into Excel where figures were generated using Visual Basic scripts or created manually for each model. The figures were time-consuming to create, had limited reproducibility, and did not provide reviewers and managers with a consistent product with which they could become familiar as modifications for an individual model were rarely generalized for the benefit of other models. The original r4ss R script became widely used by the stock assessment team at the National Oceanic and Atmospheric Administration (NOAA) Northwest Fisheries Science Center (NWFSC) and grew in complexity as members of the assessment team provided suggestions for additions. The increase in use also increased the burden associated with maintaining the code, and in 2008 the lead developer role was shifted to a postdoctoral researcher which allowed for more directed development, facilitating the growth and use of the code to function across SS-assessed stocks. Shortly thereafter, the code was put under version control and released as open source to facilitate distribution and development, to increase transparency, and to reduce the burden of maintenance on any individual developer.

Although, in the early years of its development, most of the code was written by just two people, feedback from users was essential to improving the package. In particular, conversations with participants at the annual Inter-American Tropical Tuna Commission (IATTC) Stock Assessment Workshop series (since succeeded by the Center for the Advancement of Population Assessment Methodology workshops) led to significant steps forward in the project. The initial public release of r4ss took place during the 2008 IATTC workshop (Maunder, 2008); discussions at the 2009 workshop inspired the conversion of the script into a formal R package available on the Comprehensive R Archive Network (CRAN); and a demonstration of the Javascript viewer for Multifan-CL (SPC, 2010) associated with the 2011 workshop led to the development of an HTML viewer for r4ss plots. Formatting the r4ss script as an R package brought the benefits of structured documentation for each function; making the r4ss package available on CRAN made it easier to find and install (as CRAN is the first source most users will look to for R packages). The number of authors, all of whom have made substantial code contributions, has also grown from five in 2009 to 29 in 2020. The methods used to incorporate code into the r4ss codebase have also evolved from contributors emailing files to the lead developer, to GitHub pull requests that get automatically checked and manually reviewed before merging. Although the development workflow has grown more sophisticated, the organic evolution of r4ss leaves many legacy aspects of the code and package structure, which are typical of research software (Ram et al., 2019), but would be designed differently if starting from scratch today.

The r4ss package (github.com/r4ss/r4ss) includes functions designed to work with SS input and output files (Supplement 1). The main types of functions in the package are: 1) functions to read and plot information from SS output files to visualize model results; 2) functions to automate tasks associated with SS models that are routinely performed; and 3) functions to read, create or modify SS input files. In the examples, we will focus on functions to visualize model results and automate routine tasks.