Scientist-stakeholder relationships drive carbon data product transfer effectiveness within NASA program

Carbon cycle science is at the heart of research on global climate change and its long-term impacts, as it examines the exchange of carbon between the atmosphere, oceans, land, and the impact of fossil fuel emissions on this cycling. Given the urgency of the climate challenge, NASA’s Carbon Monitoring System (CMS) requires all funded investigators to identify and work with stakeholder organizations at project inception to accelerate the transfer of the products developed by funded research into decision making systems. In this study, we contribute to the literature through the implementation of a quantitative analysis of 908 unique survey responses from funded investigators to explore the maturity of the scientist-stakeholder engagement. The paper employs multiple correspondence analysis to provide evidence to support policy options to increase stakeholder integration into research programs. Despite limitations of the dataset used, we demonstrated that multiple funding rounds, long-standing relationships between the stakeholder and scientist, and the scientific productivity of the Principal Investigator, including the ability to produce datasets and research papers on these datasets, all contribute to carbon products moving from research to operational use. The maturity of relationships between scientists and stakeholders was shown to result improved stakeholder engagement. The use of carbon products should be identified in every stage of the program, and that capacity building is needed to support both existing and newly identified stakeholders better understand and use CMS products. As federal, state, and local policy on climate adaptation and mitigation matures, the need for information on carbon will expand. Building of stakeholder-scientist relationships in CMS results in an effective generation and use of datasets to support this need and prototype ways that improved information needed for decision making can be created.


Introduction
Carbon cycle science examines the exchange of carbon between the atmosphere, oceans, land and the impact of fossil fuel emissions on this cycling. Studying the carbon cycle helps us understand the probable impact of climate change on humanity through rising temperatures and increasing carbon dioxide levels (IPCC 2018). Climate change also threatens long-term economic development (Liobikiene and Butkus 2018), food production (Ray et al 2013), and will damage urban infrastructure (Wilbanks and Fernandez 2014) necessary to support a growing human population. However, ensuring that critical information on climate is actually used in day-to-day decisions and policy making by stakeholders such as governments, businesses, and institutions requires engagement and communication between the user and the producer of the information (Cash et al 2006).
We define stakeholders as an individuals, groups or organizations that are affected by climate change, who can make policy, investment or activity decisions with carbon data, and who are end-users of the data CMS produces. Carbon data is information, analysis, visualizations and data products that inform decision makers about carbon stocks and fluxes that move throughout the Earth system across a range spatial and temporal scales. A carbon stock, or carbon pool, is a system that has the capacity to store or release carbon. A carbon flux refers to the amount of carbon exchanged between carbon stocks over a specified time. In simple terms, CMS data seeks to model and measure the movement of carbon between land, oceans, atmosphere, and living things (Hurtt et al 2022). Although we recognize that scientists are often also stakeholders of scientific information and models, we focus on non-scientist stakeholders in this context for clarity.
NASA's Carbon Monitoring System (CMS) has worked for the past ten years to prototype capabilities necessary to support stakeholder needs for monitoring, reporting, and verification of carbon stocks and fluxes (Hurtt et al 2019). The result of this sustained funding is the development of a community of practice where scientists have learned from each other about how to do meaningful stakeholder engagement, the value of this engagement, and have learned through annual Science Team meetings and stakeholder workshops about applications of CMS products (Brown et al 2020). By connecting cutting edge carbon cycle science research to stakeholders beyond the scientific community who may use the data in their decision making, NASA CMS contributes to understanding the needs of the climate data end-user community (Moser and Ekstrom 2010). For the past eight years, the NASA Goddard Space Flight Center (GSFC) science applications team has engaged both the CMS Principal Investigator (PIs) and a diverse set of stakeholders to encourage mutual understanding of data needs and functionality of the current and planned CMS data products for effective use in decision making contexts. The goal of the CMS applications efforts is to link stakeholders to CMS science products and provide a path for feedback and lessons learned for CMS PIs so CMS is more accessible and user friendly. Stakeholders closely engaged with CMS projects at the federal level include the U.S. Environmental Protection Agency, the United States Department of Agriculture (USDA) Forest Service, and National Oceanic and Atmospheric Administration (NOAA) (figure S1) (Carlo et al 2018).
Challenging the science community to identify, learn from and engage directly with potential users of their science has resulted in improved relevance and uptake of scientific products (Brugger et al 2016). The CMS program motivates new basic research while integrating the user community into data product creation and distribution, demonstrating how science and technology can be integrated into decision making (West et al 2018).
Here we use a quantitative approach to assess CMS scientist engagement with stakeholders and promote use of carbon cycle science data developed during the project. Our primary hypothesis is that by measuring specific characteristics of the CMS PI scientist and their institution, such as their experience, personal relationships with the stakeholder, frequency of communication with the stakeholder, and the period of support for the project, we can estimate the effectiveness of CMS PIs in creating useful carbon products and transferring them to support decision making. We focus on the CMS PI, the data they produce, and their engagement with stakeholders, not the stakeholders themselves.
We also hypothesize that traditional scientific measures of 'success' , such as citation of peer reviewed articles, can be related quantitatively to changes in stakeholder engagement, as measured by our impact metric difference of application readiness level (Dif-fARL). Our hypothesis regarding co-production is that when an investigator receives multiple rounds of CMS funding, this enhances the likelihood that the CMS PI will build a mature, long-standing relationship with a stakeholder (Jahn et al 2012, Brugger et al 2016. To test these hypotheses, we develop categorical variables describing each funded CMS product, and use multiple correspondence analysis (CA) to explore the efficacy of the stakeholder-scientist interaction.

Literature review
Previous research has shown that a collaborative approach to knowledge development is more likely to result in usable information than when research is conducted in isolation (Fazey et al 2014, Wall et al 2017. Co-production of knowledge or transdisciplinary research (Jahn et al 2012) lies between basic research into processes, relationships and product development typically funded by NASA's Earth Science Division and applied research focused on defining applications that can be supported with Earth Science products and guide scientific priorities (Moran et al 2015).
Extensive previous work has been done on understanding the link between applied and basic research and its use in policy and decision making. Sarewitz and Pielke (2007) conceptualized how the supply of information generated through investment in basic research could meet the need of society. Matching the 'demand' for science, particularly in support of decision-making in public affairs, to monitor and assess the impact on society that science and technology has created, to the 'supply' of basic or applied research requires constant and early interaction between the producers and users of the information (Sarewitz and Pielke 2007). The utility of scientific information does not fall directly out of the knowledge itself, but requires that the knowledge be 'socially robust' , valid and reliable in the context in which it could be used, which is attained through engagement with experts and stakeholders throughout its development (Gibbons 1999, Cash et al 2003.
We recognize that fundamentally, building relationships between scientists, experts and stakeholders who will use carbon data is at the heart of the CMS program. Engaging with stakeholders over time in ways that allow for two way-learning, the development of long-term relationships, and transformation of methods and datasets to meet the needs of stakeholders (Cook and Zurita 2019). Issues related to climate and environment are particularly thorny to resolve because they require both scientific knowledge and political and social values. Addressing these problems requires establishing and maintaining dialogs among interested parties with differing values to bring scientific expertise together with local and environmental concerns to find solutions (Ludwig 2001, Meadow et al 2015. Our research therefore focuses on the CMS science community, their ability and interest in engaging in this dialog, and seeks to better understand the context in which their research could be used. Drawing on research from the field of technology transfer, the contingent effectiveness model draws its name from the assumption that parties to technology transfer have multiple goals and effectiveness criteria that depend on who the user is and how they value the dataset (Bozeman 2000). Similar to Cash et al (2006)'s four critical functions in application of science, which include convening, translation, collaboration, and mediation, the Bozeman model provides five broad dimensions that determine effectiveness: (a) characteristics of the scientist or transfer agent who is guiding the research and product development; (b) characteristics of the method through which the technology is transferred (transfer medium); (c) characteristics of the product itself such as resolution, time step and latency (transfer object); (d) the demand environment or the need for the data in the user community; and finally (e) the characteristics of the product recipient or stakeholder ( figure 1). An assumption of the contingent effectiveness model is that there is no single way to measure effectiveness of technology transfer since effectiveness is defined by each stakeholder individually and in the context in which the data are being used (right side of figure 1). This results in highly contextualized and fundamentally incomparable 'success' criteria, which although relevant, is also difficult for a funding agency like NASA to use in evaluating the success of its program to communicate, disseminate, and encourage use of its products. This is the primary reason why we use the scientist-provided product ARL change metric as a way to determine 'success' of CMS's impact on society through describing the products' progression of use of a product within a stakeholders' decisionmaking activities (NASA 2017).
Connections between research and societal outcomes are affected by a wide array of contingency, complexity and non-linearity factors, but these factors need not prevent the use of data for improved decision-making (Lasswell 1971,

Effectiveness criterion Focus
Relation to research and practice CMS success criteria 'Out the door' Success requires at least one organization to learn about data product, without regard to impact.
Extremely common practice to simply determine if someone used the data product with no regard to impact on decision making.
DiffARL is low or zero, since the project stays in 'Discovery and Feasibility stage' , but papers are written to increase product awareness. Science impact Citation score documenting that the research or dataset description been used in a scientific literature, demonstrating 'science impact' .
Widespread success criteria for a funded research program, with many research studies and methods available to measure science impact.
DiffARL is low, since the project stays in 'Discovery and Feasibility stage'

Economic impact
Has the transfer resulted in economic benefit for institution, community or industry through its use?
Important criteria for perception of impact but can be difficult to measure without access to private data and is beyond the scope of nearly all research programs.

Not measured with DiffARL-not evaluated here
Political reward Based on the expectation of reward or impact flowing from the use of the data product, such as increased importance of fighting forest fires or regulation to reduce pollution sources identified. Widely used as a success criteria in practice, but poorly quantified or examined in the literature.
DiffARL is 6 or greater, as product moves from 'discovery' to 'application' stages Opportunity cost and return on investment (ROI) Return on investment on the part of the scientist and the stakeholder organization, particularly against other ways of using resources and time.
Concern among practitioners, rarely studied in the literature except in cost-benefit analyses.

Not measured with DiffARL-not evaluated here
Scientific or human capital Impact of engagement with stakeholder on the enhanced scientific, technological and communication skills, particularly focused on social capital and on students and other team members in both user and producer organizations.
A high priority for CMS in its mission, but poorly measured and rarely studied in the literature.
DiffARL is 6 or greater, as product moves from 'discovery' to 'application' stages Changnon et al 2000). Here we use data on product ARLs reported by the project scientist to determine how characteristics of the transfer agent, transfer medium, the product, the stakeholder and the demand environment affect the uptake of the product by the stakeholder (Bozeman 2000) (table 1). Contributing to the literature about co-production of knowledge (Wall et al 2017), we present a quantitative analysis that focuses on determining the potential causes for why some CMS funded programs were able to engage effectively, as described in table 1, and others were not. Using this framework as a guiding principle, we will examine the likely impact on our 'Effectiveness of Engagement' metric from the scientist perspective, as measured by the difference in ARL from the start to the end of the development of a product (referred to here as the DiffARL variable). Those products with that report change in ARL are the result of engagement with the stakeholder or target organization. This change will deliver a variety of CMS 'success criteria' (table 1), such as 'out the door' , increased scientific and human capital through the engagement between the scientist organization and the stakeholder organization, and science impact through the process of writing and publishing papers on the new data product.

NASA's CMS
NASA's CMS initiative was initially funded in the 2010 Congressional Appropriation, which directed NASA to start working towards a CMS and provided specific guidance on how this could be done. NASA CMS emphasizes exploitation of the satellite remote sensing resources, scientific knowledge, and modeling expertise that are major strengths of the NASA Earth Science program (Hurtt et al 2014). The approach focuses on product development and requires close communications and/or partnerships with state, local, tribal and federal government agencies and their technical experts who develop and produce carbon inventory and biomass inventories. Here we assess CMS scientists' perceptions of their stakeholder engagement to provide relevant programmatic lessons learned for NASA Earth Science Division (ESD). Improvements in the use of Earth Science data can have a societal benefit by supporting decision making by stakeholders in their efforts to mitigate or adapt to a changing climate. Improving decision support and use of NASA data products is a key goal of the NASA ESD and of CMS.
CMS requires that all funded PIs have users and stakeholder organizations included when proposing, conducting their research, and documenting their results. The focus of CMS is to iteratively develop data products in collaboration with stakeholder organizations so that the data products better inform monitoring, reporting, and verification of carbon fluxes and stocks across a variety of institutions and decisions. Inclusion of users and stakeholder organizations is now a requirement for NASA missions.

Measuring CMS impact
To test our hypotheses, we use information on each product and its use in a stakeholder environment generated through scientist questionnaire. Because these questionnaires are repeated every year and the CMS program has specific and independent stakeholder engagement activities, there are multiple evaluation points for the data to ensure its consistency and quality. Below we set out metrics we use to describe the drivers of the DiffARL metric.
The CMS PI is our transfer agent in this context. Recent research has shown that there is a great deal of learning (Ernst 2019) that occurs within the science team and stakeholder engagement meetings supported by CMS, engendering a community of practice (Wenger 2011). Here we use information on the PI institution, the number of co-investigators they have on the grant, whether the grant is one of a series that was funded by CMS and the year the project was funded as potential drivers of the maturity of the PI-stakeholder relationship (table 2). Although economic impact is an important part of assessing the value of carbon products, the data that we had available for this review did not include information on potential economic benefits of the data. Mature engagement with a stakeholder, including generating a deep understanding of the organizational context in which the product is used (VanderMolen et al 2020), may result in a product moving from a Stage 1 ARL (research) to a Stage 3 ARL (stakeholder use of the product in decision making) (Wall et al 2017).
The transfer medium describes the way the CMS carbon data is transferred to the user. Research papers and other publications are the primary way most scientists communicate their findings about the carbon cycle to others, including stakeholders, regulatory bodies and scientists. Generating knowledge in a systematic way and publishing it is widely accepted as a primary output from NASA research funding and can be instrumental in communicating results to a broad community. Other ways CMS PIs transfer their products include direct transfer from PI to stakeholder; presentations in CMS meetings; and the engagement work of CMS applications efforts. Here we use quantitative data on total number of citations on datasets in the Distributed Active Archive Centers (DAACs) as a measure of the transfer medium.
Similar to the transfer medium, the transfer object is the carbon dataset produced by the project, its size and geographic extent. The focus is on the content and form of the dataset, and its characteristics such as spatial and temporal data extent that determines whether the stakeholder can use it or not (table 2). For example, if the stakeholder is a local user in Reno, Nevada, who is making decisions on investments in urban tree canopy, having a carbon data product on forest biomass in Maryland will not improve the user in Nevada's ability to make decisions. Similarly, if the dataset ends in the year 2000 but the decision maker needs near-real time information, the stakeholder will not be able to use the data. We also use a total data size metric as a single metric to indicate how many files and resolutions are available for use by stakeholders.
The product recipient is the stakeholder or enduser organization. Here we use information provided by the CMS PI on the recipients of their datasets. We have a PI-determined assessment of the strength of the relationship with the stakeholder for each product, as a determinant of 'effectiveness of engagement' outcome variables. If the PI considers the relationship to be strong, then theoretically the ARL change has been large if the PI has a positive interaction with the product recipient. We will test this idea here.
We have variables that describe the demand for carbon datasets across all the funded research projects and stakeholders. We use here the size of the dataset, the theme in which the project is working (biomass, oceans or atmospheric flux) and the download statistics for datasets archived at the NASA DAACs. Although the demand is very challenging to determine, if a dataset is downloaded more or the paper cited more, then the scientist either has done a good job publicizing it or is working in an area with a real need or both. Table 2 summarizes the dataset used in the analysis. The data was derived from three different CMS PI surveys from 2016 to 2020 (table S2). Each PI has their own project profile section on the CMS website https://carbon.nasa.gov, where the submitted abstract, participating scientists, project description and datasets produced by each PI's project are available for each year of the CMS solicitation. The Dif-fARL variable has a total of 908 ARL observations, which are the change in ARLs from survey responses from start to end for each data product-stakeholder pair (table 2). The ARLs change through time because of the maturity of the product changes, along with the use of the product by the stakeholder, over the period of the grant. Because here we are connecting data products to ARLs and the relationship with users, we use the information from each data-stakeholder response. In 2020, we added two questions to the survey that was not previously present (strength of the relationship and first engagement with the stakeholder), which has resulted in over 300 responses regarding the relationship between the scientist and the stakeholder for each product under consideration. Finally, we use information from the DAAC where final, completed CMS data products are hosted for archiving and final distribution. The DAAC provided citations, data size and number of files in database for 98 data products archived.

Data and methods
To establish a connection between the dependent variable, DiffARL, and each of the variables described in table 1, we use a CA technique (Greenacre and Hastie 1987). This data analysis technique is based on singular value decomposition and is used to detect and represent underlying structures in categorical data. The primary goal of CA is to illustrate important relationships among qualitative variables using a graphical representation without assuming any particular data distribution and can accommodate any type of categorical variable whether binary, ordinal or nominal (Greenacre 1984(Greenacre , 1994. Here we present qualitative variables, such as the response of a CMS PI to a question regarding their relationships with stakeholders and how they interact with them, in quantitative ways. By transforming these responses into quantitative variables, we can test which aspects of the CMS program has the greatest influence on the ability of CMS PIs to increase a product's ARL.
Each graph presented has percentage of the total variance captured by the two axes for each variable examined. The more variance captured in the second dimension, the less likely that the analysis is missing elements important for understanding how the two variables are related. We also provide the total inertia value, which is defined as the total Pearson Chisquare for the two-way variance table divided by the total sum, and therefore represents the goodness of fit of the two variables to capture all the variance present in the table. In general, the higher the inertia, the better the goodness of fit the second variable has to capture all the variance.

Results
The difference between the start and end applications readiness level (DiffARL) for all 908 datasetstakeholder pairs is shown in figure 2(A). There are relatively few products with large changes in ARLs, with only 104 products, or 11% of the total having ARL change greater than 4 (see table S1 for ARL definitions). No CMS products began at ARLs at 7, 8 or 9, which denotes operational readiness. In total, 19 products have gone from conceptual ARL1 to an operational ARL9 during the project period.
Using the CA, we present in table 3 a summary of the ability of each independent variable to capture the variance in DiffARL. The results show that the ability of the scientist to address stakeholder demand for carbon products, including the topic that the product addresses, the interest and subsequent citation of the papers written by the project about the data, and the number of files archived in permanent storage by each product are important factors in explaining the maturity of the CMS data product for stakeholder use. Characteristics of the agent and the recipient are also important, however.

Transfer agent characteristics
The results show that characteristics of the scientist or PI developing the CMS product are important to explaining product maturity, particularly the period over which the PI was funded and the year the PI submitted the proposal. The categorical data created from the DiffARL and year information shown in table 4 documents an increase in ARLs each year. Figure 3 shows the CA diagram for this same data. Projects funded in 2015 and 2016 were those that produced the largest increases in 7 and 8 ARLs. Previous research showed that in-depth understanding of stakeholders' information needs was important to data use, but that this takes time and requires continuity in relationships (VanderMolen et al 2020). In CMS, NASA-supported applications personnel has helped to increase communication and engagement with stakeholders, particularly for projects that funded time within their own grants for stakeholder engagement.
Aspects of the transfer agent that seem to capture less of the variance of the DiffARL metric include the institution from where the PI is based, the number of precursor projects and the number of coinvestigators funded under the program. In a previous paper, Brown et al (2020) found that the CMS program's ability to provide consistent funding year after year, and to provide engagement and learning of both the agent and the recipient of the data (here the stakeholder) were essential elements of the program.

Importance of the transfer medium
Few of the variables examined here capture the variability in the transfer medium because we were only  able to create variables that captured datasets distributed via the DAAC and not via the CMS PIs to their stakeholders directly, such as web media, videos, and decision support systems (figure S2). A good example of PI-led data distribution is the NOAA Global Monitoring Laboratory CarbonTracker website that displays and analyzes sources and sinks of carbon dioxide around the world (Butler 2021). However, we did find that the number of dataset citations for datasets distributed via the DAAC explains approximately 75% of the variability of the DiffARL metric, as they relate to the download, use and publication about the use in the peer reviewed literature. Of the projects with data citations of less than 40, 51% were funded in 2011 or 2013, before the CMS project began investing in a broader stakeholder engagement program to support scientists working in the program.  figure 3 correspondence analysis figure, showing the number of project-stakeholder ARL changes were documented for projects that were funded in each year.

Transfer object variables
Our transfer object variables include the number of products archived, the number of files in the DAAC and the size of the data product. We show that the number of archived datasets and the number files posted at the DAAC by each PI is quite important in explaining the variance of the DiffARL (figure 4). We found that 56% of the projects with 0-3 archived datasets were funded before 2014. This result may reflect that some projects have not yet finalized their datasets but are still engaging with stakeholders. We found that three datasets are associated with the products that have increased ARL substantially.

Product recipient variables
There are more variables describing the product recipient or stakeholder engagement, including the frequency that the PI engages with them, the mechanism through which the communication occurs, when they were first engaged by the PI, the strength of the relationship as described by the PI, and the number of stakeholders were engaged for each product. Of these, the last is most able to capture variability of the DiffARL (figure S4). We found that 37% of all PIs report that they have between one and two stakeholders, whereas 10% report that they have over nine stakeholders. The analysis shows that engaging with more stakeholders is not necessarily better for increasing the maturity of each product. Of the 55% who stated that they had engaged the stakeholder 'a long time ago' , over 80% had an ARL change of 1-3 ARLs, meaning that although they might have known the stakeholder for a substantial amount of time, the relationship may not be very mature.

Demand environment
Finally, the demand environment is critical for understanding how well the CMS PI is to engage with users and increase their ARLs during the project. One of the most important variables, as indicated by the inertia factor, is the citations of the papers associated with the dataset, which shows the number of other scientists working on the subject being described and the ability of the broader community to hear about and cite the research being conducted to produce that dataset (figure S4).

Discussion
Access, awareness and availability are key to the use and uptake of products by stakeholders. We found that the hypotheses that the scientist's ability to communicate about their product via publications, and the length of time engaged with the stakeholder were key factors in their effectiveness in creating useful carbon products and transferring them to support decision making. Our finding support previous research from Jahn et al (2012) and Brugger et al (2016) that demonstrate the ability of a scientist to understand the stakeholder context is critical for uptake. Our quantitative approach revealed the importance of the production scientific articles and datasets as the foundation upon which subsequent use of the data product by stakeholders. More frequent and decision-targeted engagement with the user during the development of the CMS product increases the awareness of how the product will best integrate into the user framework and directly connects to the stakeholders' needs and decisions. Increased awareness of the product development details has a direct impact on product access and availability to the user and helps the CMS scientist connect with the most relevant organizations. The feedback from the user can help drive the access and availability of the CMS products, directly increasing the use and familiarity, and ultimately increasing the products' ARL through the life of the project.
This paper provides a method that allows for quantitative analysis of scientist surveys to explore drivers of increased product engagement. There has been substantial amount of research showing that meaningful interaction between a scientist and a stakeholder during product development should increase the use of scientific information Morehouse 2005, Arnott et al 2020b), with others finding that even with relevant information and an engaged stakeholder, there are significant barriers for scientists to engage effectively with potential users of information (VanderMolen et al 2020) ( figure S5).
Here we find that a quantitative approach can help identify characteristics of a funding program and actions that a scientist can take to increase their success in moving from basic research to application (Whitney and Leshner 2004).
As previous research has found, characteristics of the stakeholder or recipient of the CMS data are important. Our research shows that the maturity of the user relationship with the PI at the proposal stage of the project is related to how much the ARL evolves during the period of performance (figure S6). Maturity of relationship, which can be measured through letters of interest and other documentation submitted with the proposal, can be encouraged by clearly and consistently funding CMS projects that build on existing relationships. As Arnott et al (2020a) points out, funders of science are receptive to new ways of revisiting the 'social contract' for science so that coproduction of knowledge can be prioritized. Ensuring CMS scientists prioritize relationships as well as producing products and writing papers is essential.
Engaging with stakeholders frequently, providing transparency on product capabilities and limitations, and integrating feedback while creating a strong relationship with them was also found enhance change in applications readiness. Being transparent about capabilities through frequent communication reduced confusion related to access, awareness and availability, and further strengthens the user/PI trust and relationship. CMS products that were able to achieve this were also more likely to be funded in sequential years and continue to evolve their ARL. Of all products, 43% had no precursor projects and were new to the CMS program. We also find that 52% of projects with one and two precursor projects, were more likely to report an increase in ARL increase that was higher than those with no precursors.

Limitations
An important limitation of this research is the focus on using scientist survey results as a proxy for stakeholder use of data products. We are limited by the active participation of the CMS PIs in the survey, and their perceptions as they answer questions on their relationships and engagement with stakeholders. The rigor with which they apply the ARL framework to the stakeholder's use of their product is also a critical limitation. After working in CMS for several years, most PIs are extremely aware of the importance of engagement, and therefore may report a better relationship with stakeholders than is the case. To compare the scientist provided ARLs to those provided by a stakeholder, we interviewed 12 CMS stakeholders in 2021. Of the products reviewed, we found that only 36% of the stakeholders disagreed with the scientist-provided ratings by more than one ARL, but these were evenly split between the stakeholders who believed the product was more mature than the scientist provided (a higher ARL), and those that said it was less mature (a lower ARL). We recognize the complexity of assigning ARLs, which both scientists and stakeholders find challenging, and the different perspectives that a policy maker has from the developer of the product. Further work is needed on evaluating the consistency of ARL ratings across different communities.

Significance for policy and funding of carbon datasets
The CMS project provides a consistent funding stream for scientists and stakeholders who engage with them. The result has been the development of a community of practice that has a coherent engagement of carbon and decision support topics (Brown et al 2020). Annual CMS Science Team meetings, required for CMS funded project scientists, include a one-day applications workshops, where stakeholders identified to be working with projects are invited to present their projects either in a talk or in a poster. At the 2020 meeting, 12 active stakeholders presented, and noted the importance of CMS products across a range of applications including: the role of forests in climate mitigation planning, implementing urban canopy targets, wetland and mangroves carbon monitoring, and monitoring of aquatic and marine primary productivity. Stakeholders also noted remaining data needs and gaps, obstacles, or barriers to use, and other programmatic activities CMS could do better. As the CMS project continues, additional investment in stakeholder engagement has been made, including providing more opportunities for stakeholders to attend the CMS science meetings virtually, participate in surveys and interviews from the CMS Applications team to determine their challenges and needs.

Conclusions
The maturity of relationships between scientists and stakeholders can be encouraged through both relationship building before the grant is submitted and through more rigorous review of letters of support and clear expression of how the CMS scientist intends to engage with the stakeholder. There are numerous important additional applications that could be supported with CMS products as the need for carbon information grows. The ongoing user engagement continues to inform ways in which CMS data can be applied stakeholder needs.
We found that assessing product maturity with PI-applied ARLs was able to capture investments in stakeholder relationships by CMS PIs. We were able to document changes in product maturity through PI-reported ARLs, offering a potential management tool that could be used in applications programs seeking to develop datasets usable by stakeholders. The method has the potential to determine the success of the CMS program in achieving its goals of putting data into the hands of decision makers.
New ways to use carbon products should be identified in every stage of the program, and that capacity building is needed to help both existing and newly identified stakeholders better understand and use CMS products. As federal, state, and local policies on climate accelerate, the need for information on carbon will expand, as will the need for feedback from decision makers at all scales. CMS is an appropriate prototype for generating and using datasets to support this need and to continue assessing the community needs for carbon science in society.

Data availability statement
The data that support the findings of this study are openly available at the following URL/DOI: https://carbon.nasa.gov.