A systematic approach to assess climate information products applied to agriculture and food security in Guatemala and Colombia

for targeted decision-making have rarely been systematically evaluated. Here, we report on the development of a systematic and user-centered approach to assess climate information products and networks of products; and apply it to products covering the nexus of climate, agri-culture, and food security in Guatemala and Colombia. Across both countries, we assessed 28 products used for agricultural decision making, outreach, planning research, and design of emergency responses. While climate-only information products play a central role in each network, information products intended to support agri-culture and food security need to integrate information from different themes or disciplines and sources at different scales. We find that major improvements in the credibility, legitimacy, scale, cognition, procedures, recommendations, and content of most existing products are required. Brevity and clarity of language are highlighted as desirable in both countries, as well as use of trusted and publicly-available data, and non-paper-based delivery formats. The approach and methodology are valuable for facilitating the prioritization of actions for improvement and/or the development of new products, thereby helping climate services for agriculture and food security to realize their true potential.

The approach comprises four steps.
Step 1 determines the geographic and thematic scope of the analysis and identifies the information products to be included in the study.
Step 2 explores the relationships between these products, in terms of information flows among networks of products, using standard statistical analysis.
Step 3 assesses the use and usability of individual products from the perspective of users and gathers qualitative information on users' perceptions of the characteristics of an 'ideal' information product.
Step 4 provides feedback to the technicians and scientists involved in creating the products, summarizing the conclusions of the study in the form of recommendations for potential improvements to products and the network of products.
In practice, application of the approach would help guide future investments in CS and provide valuable information to producers of CS to orient their future development, for example, by filling information gaps and enhancing their usability. The practical implications derived from the application of the approach in the two case study context can be summarized as follows: 1 Knowing the position of the products in a CS network is crucial to identify measures that could be taken to enhance their effectiveness.
-In Guatemala and Colombia, Regional Climate Outlook Forums occupy a central position in CS networks and play a key role as primary sources of seasonal climate outlooks for use by the agriculture and food security sectors.
-In Guatemala, the Regional Outlook on Food and Nutritional Security occupies a less central position in the network that might be expected. This may make it more difficult for agricultural and food security stakeholders to access the climate forecasts that they need to inform decision making. This limitation could be addressed by building the capacity of stakeholders to use climate forecasts for food security management, and by strengthening linkages among stakeholders in the climate, agriculture, and food security sectors. 2 Knowledge of which products are currently used by the different groups of decision-makers can pinpoint gaps in the information available.
-In Guatemala, national-level decision-makers tend to use products that integrate national-scale climate, agriculture and food security forecasts. However, these products lack spatial and temporal detail required for application at local scales. Local decision-makers use shorter-term national-scale climate products. These results indicate a requirement for co-produced integrative local-scale products, and these are currently being developed. 3 Decision-makers' perceptions of products currently available provide information on their evolving needs and inputs for the design of technical and procedural improvements. These could include: -Optimizing timing, frequency, and delivery mechanism/ channels.
-Enabling two-way (user-provider) communication and providing downloadable source information (identified as a priority in both countries).
-Using plain language, logical structure, and visual aids (priority in Guatemala).
-Integrating of climatic, agronomic, and food security information (priority in Guatemala).
-Using and comparing of trusted sources, and providing information on methods, results, and uncertainty levels to improve credibility (priority in Colombia).
-Incorporating local feedback to improve legitimacy (both countries).
-Providing information at an appropriate scale (both countries).
-Providing recommendations linked explicitly to the timeframes of decision-making processes, for example relating to sowing or harvesting (both countries). 4 Providing feedback on the assessment process to stakeholders is essential to validate its results.
-In both Guatemala and Colombia, CS providers stated that the synthesis of user perspectives helped them to understand how their products are being used. Information on user perspectives helped justify existing plans for improvements to their products as well as identifying further improvements that could be made.
CS is a dynamic field and new actors and products are continually emerging, reflecting the increasing importance of climate information for agriculture and food security under climate change. Thus, ideally, the systematic assessment of the usability of climate information products should be integrated into an M&E scheme that is implemented regularly to capture changes and developments.

Introduction
Climate variability associated with climate change affects agricultural production and rural livelihoods throughout the world (Porter and Semenov, 2005;Sultan et al., 2005). On average, about a third of the variation in global crop productivity can be attributed to climate variability (Ray et al., 2015). Climate variability is expected to increase, leading to more frequent climate extremes, with adverse outcomes for agricultural production and food security (Osborne and Wheeler, 2013;Springmann et al., 2016;Wheeler and Von Braun, 2013). In response, investments in new climate services (CS) address this challenge by increasing the information available to policymakers, agricultural technicians, and farmers in order to enhance their adaptive capacity (Georgeson et al., 2017;Hewitt et al., 2012;Vaughan and Dessai, 2014). CS may help decision-makers to move away from reactive responses to agricultural and food security crises, towards a more proactive approach of climate risk management (Jones et al., 2017;Wilhite, 2002).
CS involve the (co-)production, translation, transfer, and use of tailored climate information to improve decision-making at multiple scales (Vaughan and Dessai, 2014). For example, at the farm scale, a CS may deliver a seasonal forecast to help farmers to select planting dates and crop varieties (Hammer et al., 1996;Soler et al., 2007); at the regional scale, seasonal to interannual climate forecasts could improve decision-making by agronomists (Adams et al. 2003;Jury 2002) and planners responsible for water resource management (Clements et al. 2013) and flood prevention (Rayner et al. 2005). Around the world, CS are often provided through information products such as bulletins, websites, and mobile phone applications that connect technical and/or scientific information to a practical problem (Fraisse et al., 2006;Georgeson et al., 2017;Vaughan et al., 2017). These information products support decision-making if they are tailored to decision-makers' needs and are relevant to the decision-making context (Lemos et al., 2012;Rayner et al., 2005;Rosas et al., 2016).
Previous assessments of CS focus on their technical effectiveness. One stream of research examines how well models predict climate and crop yields at the seasonal or inter-annual timescale (Capa-Morocho et al., 2016;Hammer et al., 1996;Ramírez-Rodrigues et al., 2016;Roudier et al., 2016). Another considers the socio-economic value of CS, including organizational aspects and policy contexts that facilitate their use (Solís and Letson, 2013;Tall et al., 2018;Vaughan et al., 2017). These studies look for evidence of behavioral change and downstream development impacts such as yield gains, improved food security, or increased incomes through CS use (Tall et al., 2018;Vogel et al., 2017). Although the need for evaluation of the usability of information products is recognized (e.g., McNie, 2012;Moss, 2016), less emphasis has been placed on understanding the roles of different information products as part of an 'information ecosystem' and on evaluating their usability systematically (Vaughan and Dessai, 2014).
Climate services may be produced by a single organization or through a process of co-production. In co-production, the emphasis in on demand-driven products, the integration of different topics, disciplines or fields of knowledge (Vincent et al., 2018), and the deliberate, collaborative product-development work among scientists, technicians, practitioners, and users (Porter and Dessai, 2017). This is an approach to the production of knowledge for decision making that differs from traditional scientific paradigms. To capture the potential contribution of co-production to the effectiveness of CS, evaluation methodologies should view the set of information products under consideration as a whole and not merely as a collection of isolated elements.
In Latin America, climate information products for agricultural and food security range from seasonal bulletins produced in the Regional Climate Outlook Forums (RCOFs) to locally-relevant bulletins, websites and mobile phone applications Fraisse et al., 2006). Most of these products are designed and distributed following a 'loading-dock' model: the information is produced by the supplier with little coordination, collaboration, or consultation with the users or other information providers (Cash et al., 2006;Vogel et al., 2017). Where users lack the technical capacities to understand and contextualize the information they receive, its use and societal benefits are limited (Kirchhoff et al., 2013;Vogel et al., 2017). However, the use, usability, salience, legitimacy, and credibility of existing products have rarely been systematically assessed (see Esquivel et al., 2018). Consequently, little is known about whether and how information products are used as sources of information for decision making, or how they could be improved (Tall et al., 2018;Vaughan et al., 2017). The assessment of climate information products is strategic for Latin American countries. By identifying gaps and improving existing products, such an assessment provides the basis for the emergence of a climate services knowledge network, incorporating a range of products and linking knowledge and know-how held by people and institutions at different locations and scales. In a knowledge network, the flow of knowledge and its adjustment to specific decision contexts depends more on interactions among stakeholders than on formal organizational mechanisms .
Here we present the development and testing of a systematic, fourstep user-centered approach to assess a set of climate information products covering the nexus of climate, agriculture, and food security, made available by a broad range of organizations in Guatemala and Colombia. The assessment approach builds on two methodological foundations: network analysis, to provide a holistic perspective on the set of information products as a whole; and user-centered evaluation to examine the products individually.
We first applied the method in Guatemala and subsequently adjusted and validated it in Colombia. In these countries of tropical America, agricultural livelihoods and food security are highly vulnerable to climate change and variability (Bouroncle et al., 2017;Magaña et al., 1999;Poveda et al., 2011;Ramirez-Villegas et al., 2012), but differ in terms of organizational context, end-users needs, and the level of user engagement in co-production of CS. Our paper has a dual purpose. We first present the results of the application and validation of the method in the two countries. Then we discuss its strengths and limitations and propose a way forward for the broad-based operationalization of our approach for the assessment of climate information products in Latin America.

Framework for the systematic assessment of climate information products in agriculture
We developed and validated an approach for assessing climate information products and their network relationships using quantitative and qualitative methods. We define information product as a publication that is periodically made available to a potential target group of users to support their decision-making. Products can contain a combination of data, information, and expert advice, and may use different communication channels, such as websites, mobile phone applications, bulletins, text messages, and radio and television programs. We define information product network as a set of information products that exchange information among themselves in a given geographic and/or thematic context.
The workflow of the approach is presented in Fig. 1. In Step 1 we identify the objectives and scope of the assessment, identify relevant information products and examine the relationships between them.
Step 2 we explore the integration of themes or disciplines of the information network and information flows between products. In Step 3, we assess the use and usability of individual products and gather product-specific feedback and recommendations for their improvement, applying a user-centered approach.
Step 4 draws on the results of Step 3 to derive recommendations for the organizations publishing the products to enhance their usability and their role in the network (e.g. as information providers or integrators). These four steps are described below in more detail.

Step 1. Create an inventory of products
As with any assessment, the first step is to determine the objectives of the process and, specifically, the geographic and thematic scope of the assessment, in collaboration with all the organizations involved. The geographic scope is defined in accordance with the geographic area (s) of interest, while the thematic scope corresponds to products these organizations generate, use, or seem relevant to their work. This is followed by a compilation of an inventory of pertinent information products, using systematic searches and following up on recommendations from experts. If the assessment encompasses products ranging from the national to the local scale, it may also be useful to include supranational products in the inventory (e.g., bulletins of Regional Climate Outlook Forums), if these are related to national or local information products. Thematically, the criteria for inclusion in the inventory can be as broad (e.g. food security) or as narrow (e.g. midsummer drought) as needed. If the initial list is very large or includes products not targeting specific users, it could be useful to reduce the size of the inventory by selecting the most relevant products, with the participation of the organization(s) involved in the assessment and support of other agencies related to the thematic scope of the assessment. This selection should be carried out applying consistent and explicit criteria to avoid biases and maintain consistency in the selection process. For example, it might be decided to eliminate products that only contain forecasts (such as a map of rainfall over the next 48 h) if this information is included in other more elaborate products (for example, including both a map of rainfall and recommendations for farmers).
In this case, the scope of the assessment was defined as climate information products containing information related to different themes or disciplines (Balaghi et al., 2010) to support decision making in Colombia and Guatemala. We carried out systematic searches using the Google engine and appropriate keywords. We complemented this search by reviewing the Web pages of governing organizations on the subject, private sector, research centers, and international cooperation agencies. We included in the inventory only products published or updated regularly that had at least one year of continuous publication, and that were available at the time of doing the search (2014-2015 for Guatemala and 2016-2017 for Colombia). We added some products, mainly locally produced, after interviews with key decision-makers of the organizations involved in the assessment. In Colombia, since the initial list was extensive, we selected the most relevant products with the help of IDEAM, the government agency responsible for hydrology, meteorology, and environmental studies. The workforce at IDEAM includes members of staff with specific responsibility for coordination with the Ministry of Agriculture for the provision of agroclimatic services. Finally, the products are described in relation to a defined set of attributes, including theme, scale, and production process.

Step 2. Analyze the relations among products
In Step 2, the relationships between the climatic information C. Bouroncle, et al. Climate Services 16 (2019) 100137 products of the inventory are established through statistical analysis of citation data. First, the products of the inventory are grouped according to the citation patterns among them. This analysis explores the extent to which products assigned to a particular topic or discipline cite products related to other themes or disciplines, to provide a preliminary indication of the structure of knowledge flows among the products, i.e. how information relevant for decision making is acquired and transferred (Cash et al., 2005). We did this by implementing a hierarchical cluster analysis of citation data. This analysis classifies items into groups based on sets of values of several variables, and is commonly used to explore the structure of multivariate observations. The results of the analysis are displayed as dendrograms, which show groups of items with different degrees of similarity in terms of values of the considered variables (Casanoves et al., 2012). In this case, the dendrograms show groupings of products based on patterns of citations among them.
To apply this analysis requires the construction of matrices of citations between pairs of products. For each information product, we reviewed all published editions over a year and counted as a citation the mention of information from another information product using a traditional bibliographic citation system or the reproduction of fragments of information (including maps, graphs, and charts). We then organized the citation data in n-by-n matrices (with n = number of products) with the rows representing the citing products and the columns the cited products (Liu and Wang, 2005). We constructed the matrices following the conventions proposed by Borgatti et al. (2002). First, the names of the products are listed in the same order in the row and column headings; second, the value "1" represents the occurrence of a relation (citation data) and "0" the absence of one. Thus, for example, a "1" in the cell at the intersection between the column headed by product A and the row headed by product B means that product "A" is cited by product "B". Note that the matrix ignores possible self-citations; so the intersections of columns and rows headed by the same product are all shown as "0". We used the R package Cluster (Maechler et al., 2018) to perform the cluster analysis using Ward's method and Euclidean distance. We excluded from this analysis isolated products, that were not cited by and did not cite other products.
Next, relationships between products in terms of information flows are Overview of the assessment approach. Research questions (left side) are presented alongside the methodological approach (center) adopted at each step of the analysis and the tools used for information gathering (right).
C. Bouroncle, et al. Climate Services 16 (2019) 100137 examined. This stage of the analysis focuses on the structure of networks of information flows. We carried out social network analysis using the citation matrices we prepared for the hierarchical cluster analysis (Hermans et al., 2017;Wang et al., 2017) to identify (a) the products that are important providers of information to other products in the network (measured as centrality, or the number of links that a product has with others) and (b) those that are important communicators of information (measured as betweenness, or the location of a products on the shortest paths between pairs of other products) (Borgatti et al., 2002(Borgatti et al., , 2009). In the network analysis, nodes in the network are considered interdependent units and the links between the nodes are resource flow channels; the network structure is defined by the relations between nodes, that indicate opportunities for flows of resources (Wasserman and Faust, 2013). We created visualizations of the network analysis and migration analysis displaying origin, destination, direction, and volume of information flow among all products (Qi et al., 2017) using the Igraph (Csárdi and Nepusz, 2006) and Circlize (Gu et al., 2014) packages in R (R Core Team, 2018) respectively.
The network diagram shows the results of the network analysis. It shows the nodes (products) and the flows of resources (information) between them. The analysis also distinguishes between different types of products (i.e. their attributes), their role in the network, and the directions of flows of information between them. In our case, the product codes represent the type of publishing organization (color) and the principal theme (shape). The size of the node symbol indicates its importance in terms of the degree of connectedness with other nodes (popularity). The diagram identifies nodes with occupy a central position in the network (i.e. those with large numbers of direct connections to other nodes), those that are important as intermediaries (by linking otherwise unconnected nodes) and those that are located on the periphery of the network (with few connections) The arrows on the lines linking the nodes indicate the direction of the information flow (Borgatti et al., 2002). The description of the network includes a measure of its density, calculated as the number of current connections in relation to the total number of potential connections between products).
The network analysis is complemented by a migration analysis, whose results are shown a circular migration plot (Sander et al., 2014), showing the direction of flows of information between each pair of nodes.

Step 3. User-centered assessment of individual products
In Step 3, the level of knowledge and the usability of the products of the inventory are assessed from the perspective of a broad range of users.
Information was gathered from users for whom climate information is relevant for their activities in the fields of agriculture and/or food security. Respondents were asked about (i) their level of knowledge of each of the products of the inventory; (ii) the usability of a subset of products, and (iii) key characteristics of an 'ideal' information product. We used a semi-structured interview for this step (see Supplementary Text S1), carried out with decision-makers from governmental, nongovernmental, international technical cooperation, and academic organizations (see Section 2.2). To define the sample, we combined a targeted selection with 'snowball' sampling, asking targeted respondents to recommend other people for an interview. The approximately 90-minute interview included questions related to the mapping of agro-climatic information flows (Blundo Canto, 2015) and knowledge-related questions derived from the literature on agro-climatic information product evaluation (Fountas et al., 2006;Stone and Meinke, 2005). The interviews were carried out in 2015-2016 in Guatemala and in 2018 in Colombia.
The interviews comprised five sections, with questions designed to elicit the following information: 1. The identity of the user, his or her organization, and position in the organization. 2. The user's area of work and the key decisions he or she makes based on agro-climatic information. 3. The user's knowledge of the products using a Likert scale ranging from 1 (=never heard of this product) to 5 (=I use this product frequently). A maximum of five products was selected from the products rated 4 and 5 by the user for the next steps. If there were more than five products used frequently by the user, the selection considered the relevance of the products for the decision-making process. 4. The user's opinions regarding the usability of the products, following the System Usability Scale (SUS; Brooke, 1996;Lewis, 2018). The SUS is a broadly used, standardized set of questions used to evaluate the usability of a product according to users' perceptions (Lewis, 2018). The test contains ten statements that alternate between positive and negative opinions about different aspects of usability (odd items positive, even items negative) (Brooke 1996, Sauro andLewis 2011). The interviewees express their level of agreement with each statement using a Likert scale from 1 to 5 where 1 equals strongly disagree, and 5 equals strongly agree. The negatively worded statement is included to reduce the acquiescence response bias (Sauro and Lewis 2011). At the end of each statement, we gave respondents the opportunity to comment on the score they had awarded (see Supplementary Text S1, Section 4). For each completed SUS test, we calculated the SUS score following the methodology described by Brooke (1996). Then, we located the median score of each product in the categorical scale proposed by Bangor et al. (2009). Products in the categories 'worst imaginable', 'poor,' and 'ok' were considered unacceptable. Those in the 'good' category were considered marginally acceptable, whereas only those in the categories 'excellent' and 'best imaginable' were considered acceptable. We also counted and transcribed the comments on each statement, and classified them according to the common constraints of climate information products proposed by Patt and Gwata (2002). 5. Qualitative information on the user's perception of an 'ideal' information product with regard to content, format, language, frequency of publication, and the factors that can limit use. We classified the answers into six categories of constraints (Patt and Gwata 2002), namely sources of information and analysis procedures (credibility); review processes (legitimacy); appropriate scale for decision making (scale); clarity of the structure, language and graphic elements (cognition); timely distribution for planning and decision making (procedures); and definition of more specific conclusions and recommendations for different topics and actors (options or choices). We added a category related to the content (main knowledge areas, namely climate, agriculture, and food security) of the set of information products.

Step 4. Generation and communication of recommendations
In Step 4, conclusions of the study are formulated in the form of recommendations for potential improvements both to individual products and the network of products. These are intended to provide feedback, based on the results of the assessment, to the technicians and scientists involved in creating the products, and can be communicated and socialized by various means, including summary reports, detailed reports, and personal meetings.
We formulated the recommendations based on the results of the previous steps of the assessment, focusing on (i) the relationship of the product(s) with other products, highlighting its role, for example, as an information provider or integrator (Step 2); (ii) the identity of the current users of the product(s) (Step 3); (iii) users' knowledge and perception of the product(s); and (iv) the main positive aspects and limitations of the products and improvements suggested by users (Step 3).

Context: climate services in Guatemala and Colombia
We applied the methodology in Guatemala and Colombia. In both countries, there is a growing availability of climate information products, designed to orient the design, target, and implementation of interventions of a broad range of organizations related to agriculture and food security. In this section, we first describe the main factors that influence the vulnerability of agriculture and food security in both countries, such as exposure to different climate processes, the characteristics of agricultural systems, and the institutional environment. Then we explain the thematic and geographical focus of our work in each of these countries.
Guatemala is one of the most vulnerable countries to climate change and variability in Latin America, with significant exposure to climatic extremes, including droughts. Cyclical drought is a significant climate hazard in Guatemala that has widespread and severe impacts on agricultural production and food security (van der Zee Arias et al., 2012). The cyclical midsummer drought is an extended dry spell during the rainy season that coincides with a critical phase in crop production (van der Zee Arias et al., 2012). In recent years there has been a tendency for this dry spell to start earlier and/or last longer (Aguilar et al., 2005;Hidalgo et al., 2017;Rauscher et al., 2008), and climate projections suggest these trends will continue (Magrin et al., 2014;Marengo et al., 2014). The agricultural sector is dominated by small-scale, resourcepoor producers, with a significant share of the population living in food insecurity (FEWS NET, 2016). Farming systems in substantial parts of the territory show limited adaptive capacity (Bouroncle et al., 2017;Sain et al., 2017). Specifically, public organizations in the agricultural and food security sectors have limited capacity to respond to droughts (Müller et al., 2019). A national adaptation strategy and a framework for the implementation of adaptation policies and instruments are still missing (UN-DESA, 2012). Guatemalaś Nationally Determined Contribution (NDC) to the United Nations Framework Convention on Climate Change (UNFCCC) and Central America's Regional Climate-Smart Agriculture strategy both recognize the importance of climate information for adaptation and integrated climate risk management (CAC, 2017;República de Guatemala, 2015). The National Meteorological Service (INSIVUMEH), the Ministry of Agriculture and Livestock (MAGA), and the Food and Nutrition Security Secretariat (SESAN) lead the production of agro-climatic and food security information, but at the highest level, there is still limited institutional support for these efforts. However, Local Technical Agro-Climatic Committees have been recently (in 2019) been established in five municipalities under the coordination of the INSIVUMEH.
Colombia's agricultural sector is also highly vulnerable to climate variability and change (Delerce et al., 2016;Ramirez-Villegas et al., 2012;World Bank, CIAT, CATIE, 2015). In most regions of the country, rainfall is strongly influenced by ENSO, with reductions in precipitation and sometimes widespread drought during El Niño (warm) years, leading to significant agricultural productivity losses, lower farmer incomes and/or higher market prices (CIAT-MADR, 2015; Cortés Bello et al., 2013;Iizumi et al., 2014). Both the public and the private sectors in Colombia are making substantial efforts to adapt to climate variability (IDEAM; PNUD; MADS; DNP; Cancillería, 2017; Ramirez-Villegas and Khoury, 2013). Notably, in 2015 the Nationally Determined Contribution of Colombia towards the Climate Agreement (UNFCCC) established a framework for climate risk management whereby both private and public organizations must co-generate climate services for agriculture through Local Technical Agro-Climatic Committees; these committees have been working for some years in several departments of the country, under the coordination of the Ministry of Agriculture and Rural Development (IDEAM; PNUD; MADS; DNP; Cancillería, 2017). In 2017 Colombia launched the National Framework for Climate Services that establishes the National Meteorological Service (IDEAM) as a link between agriculture, health and energy teams for the development of these services. Thus, the workforce at this organization includes members of staff with specific responsibility for coordination with those sectors. These efforts create a complex interplay of stakeholders that cogenerate, transfer and use agro-climate information. As a result, many information products have been created that inform decision making in the context of climate variability, from the national through to the local scale.
In Guatemala, we focused on information products that support decision-making related to drought impacts on agriculture and food security in the country's Dry Corridor. We selected the area because of the high risk of droughts; it encompasses nearly 10% of Guatemala's territory (Ministerio de Agricultura Ganadería y Alimentación (MAGA), 2010) and contains 12% of the total population (INE, 2013). Livelihoods in the Dry Corridor are generally based on subsistence maize and bean production and off-farm income-generating activities such as coffee harvesting. Maize and bean production are sensitive to rising temperatures and changing rainfall patterns (Eitzinger et al., 2012), and as farmers depend on rain-fed agriculture, extended droughts reduce yields and exacerbate food insecurity and poverty (Pereyr and Ibargüen, 2015).
The focus in Colombia was on information products that support responses to climate variability by the agricultural sector. Geographically, we focused on the departments of Córdoba and Santander since both are involved in ongoing efforts to produce CS, which, however, differ in terms of organizational setting, history, and users. Both have functioning agro-climatic committees, whose role is to generate weather and climate forecasts tailored to local conditions through the participation of farmers, the private sector, research organizations, and national and local government organizations (Loboguerrero et al., 2018). In Córdoba an agro-climatic committee was created in 2015, providing seasonal agro-climatic outlooks for small and medium-scale maize producers across the department ( FENALCE, 2018;Loboguerrero et al., 2018). In Santander, the co-production and use of agro-climatic information are more recent, with the establishment of the agro-climatic committee in 2017.

Inventory of information products
In Guatemala we found 15 information products with relevant information on drought management for agriculture and food security (Bouroncle et al., 2015), seven are related to the monitoring of climate, crops, and food security variables, while eight are related with forecasts of variables of these themes (see Table 1). An iterative approach was adopted to compile the inventory in conjunction with INSIVUMEH, MAGA, and SESAN. As explained above, these are the three leading governing organizations involved in the production of agro-climatic and food security information, and they were involved in the assessment from the outset. An online search identified products published by national and regional (supranational) organizations. The first round of interviews with key decision-makers from these three organizations provided information on further, locally published products. The majority of products identified contain national-scale information and are produced by central government organizations and international cooperation programs. Regional-scale products are produced by intergovernmental platforms and international cooperation programs. A smaller and more recent group of products addresses the need for localscale information and includes bulletins from a municipality and a municipal inter-organizational working group. The last was the only case of product generated in Guatemala through co-production between multiple organizations.
In Colombia, from an existing list of 41 climate information products related to climate and agriculture (an initial list, includes 20 additional products related with disaster risk management, fisheries and other themes, Perez Marulanda et al., 2016), we prioritized 13 for the assessment in conjunction with the National Meteorological Service (IDEAM). Three are related to the monitoring of climate, crops and agro-climate variables, white ten are related with forecasts of variables of these themes (see Table 1). As the leading government provider of climate services in Colombia, IDEAM was involved in the assessment from the outset. The selection of products in Colombia was based on two criteria: first, the exclusion of products that include only primary data (most of them produced by the IDEAM), if those data are included in another product of the same publishing organization; second, the inclusion of the products most used by agricultural stakeholders based on the knowledge of IDEAM staff. The 13 products included websites, bulletins, and mobile phone applications at local, national and regional scales that directly support agricultural decision making (Table 1). Three of the products are generated through co-production between multiple organizations (private and public), including in the local agroclimatic committees (Loboguerrero et al., 2018).

Analysis of the relationships between products
From the 15 products in Guatemala, two (GTM_N_FMH, GTM_N_PB) had no connections with any other product (that is, they did not cite other products, and other products did not cite them) and were hence not included in the cluster, network and migration analyses. In Colombia, all the selected products were included in these analyses.

Groups of products determined by citation patterns
In Guatemala, the 13 remaining products were grouped into three main clusters according to their scale of application (Fig. 2a). The first group includes a single regional product, namely, the Central American Seasonal Climate Outlook (GTM_R_CACP) of the Climate Forum of Central America. This is the result of the collaboration between the country's national meteorological service (INSIVUMEH) and the other national meteorological services of the Central American region. The second group includes the Crop Monitoring System (GTM_N_CMS) and the Food and Nutrition Security Forecast (GTM_N_FFNS). These are both national-scale products that aggregate sector-specific information from a variety of sources. The third group combines products focused on different themes targeting regional, national, and local scales.
In Colombia, the cluster analysis (Fig. 2b) defines two groups of information products according to their production process. The first group includes the co-produced information products; they are grouped because they have a similar citation pattern that reflects a range of different themes considered in the co-production processes. As in Guatemala, the last group combines thematic products prepared in a conventional way targeting users different scales.
Dendrograms from cluster analysis of information products with relevant information on (a) drought management for agriculture and food security in Guatemala available in 2015, and (b) agro-climatic information available in Colombia in 2018. The dendrograms show the grouping of information products according to their citation patterns, colors indicate the group of products for a statistical solution of three and two clusters respectively. In Guatemala the citation pattern grouped the products according to their geographical scale, while in Colombia they grouped them according to whether they were co-produced or prepared in a conventional manner.

Roles of products in the information exchange network
In Guatemala, network analysis shows that the Central American Seasonal Climate Outlook is at the center of the network (Fig. 3a), being both the main source of climate information for the network and the main entry point of regional information. It is cited by eight products as shown by the migration plot (Fig. 4a). The Crop Monitoring System and the Food and Nutritional Security Forecast display good connectedness in the network, although with less centrality (Fig. 3a), and a similar citation pattern in terms of number and diversity (Fig. 4a). These two Table 1 Information products assessed for Guatemala and Colombia. The code includes the country abbreviation, the product scale (L -local, N -national, R -regional) and the acronym of the product name. Main theme: CL -climate, CR -crops, AC -agroclimatic, FS -food security; frequency: D -daily, M -monthly, S -seasonal; type: M -monitoring, F -forecasting. products have the most potential for the integration of information from different themes in the network. Other products, focused on different knowledge areas, are located in peripheral areas of the network (Fig. 3a) and have a lower number of connections (Fig. 4a). In Colombia, the National Agro-climatic Bulletin (COL_N_NAB) is the product with the largest number of connections (Fig. 3b). The Monthly Bulletin on Climate Prediction (COL_N_MBCP) also has a large number of connections in the network (Fig. 3b). Due to the nature of the connections (i.e. mostly as an information provider, Fig. 4b), the COL_N_MBCP is in the second group of products. Members of this group of products generally make fewer citations than they receive; their products cover a range of themes including weather forecasting (COL_N_AMF), short-term (COL_N_AF) and seasonal (COL_L_AFP) agro-climatic forecasting, early warning (COL_L_EWS), and input prices (COL_N_MBI).
In both countries, the density of the networks, measured as the number of connections between products in relation to the total number of potential connections, is relatively low (13.5% in Guatemala, and 25% in Colombia).

User-centered assessment of individual information products
A total of 40 and 35 people were interviewed in Guatemala and Columbia, respectively. In Guatemala, 17 interviewees were based in the capital city, Guatemala City, and the remaining 23 were based in the Dry Corridor. In Colombia, 19 interviewees were based in the capital city, Bogota D.C., and the remaining 16 were based in Santander and Córdoba. In Colombia, most interviewees were from the central government (12), whereas in Guatemala, most interviewees were from international cooperation agencies (14). A full list of interviewees, broken down by sector and location, is provided in Supplementary Table S1.

Decision-makers' knowledge of the products
In Guatemala, 10 out of the 15 information products were significantly better known by the interviewees working in the capital city than by those working in the Dry Corridor (Table 2). These products, none of which are intended to support local-scale decision-making, were typically rated 4 or 5 by interviewees based in Guatemala City, indicating that they use the product or have done so in the past. By contrast, most of those national-and regional-scale products were rated with a 2 ("I have heard about it before") by interviewees based in the Dry Corridor. While none of the products was rated 5 ("I use it and consult it frequently") by interviewees in the Dry Corridor they tended to give higher ratings to simple and short-term national-scale climatic products and local produced products.
In Colombia, statistically similar levels of knowledge across the three groups of interviewees (Bogotá D.C., Córdoba, and Santander) were found for 9 of the 13 products assessed. These included all national-scale products, and one local-scale information product. This suggests that, unlike in Guatemala, the dissemination of national-scale information products in Colombia is relatively uniform across scales, although not all are widely known. For example, the Monthly Bulletin of Agricultural Inputs (COL_N_MBI) and the Seasonal Climate Outlook Bulletin (COL_R_CABC) are barely known in the capital city, or in Santander and Córdoba. Both of these are ranked as "I have seen it once" at best. Notably, we find that the only local-scale product that is well known by decision-makers of the capital city, Santander and Córdoba is the Agro-climatic Forecasting Platform (COL_L_AFP). This platform provides information for some 30 different localities in 9 departments across the country. Apart from the COL_L_AFP, local information products are only known in the local scale for which they were developed. For instance, the Córdoba agro-climatic bulletin (COL_L_ABC) is very well known in Córdoba, but not even heard of in Santander. The reverse is valid for the Santander agro-climatic bulletin (COL_L_ABS). Fig. 5 shows the uses of information products structured by actions and decisions mentioned by the interviewees across temporal scales (daily, monthly, seasonal) for both countries. Information products are used for three purposes, namely, (1) obtaining advice on agricultural activity planning, (2) receiving early warning of extreme climate events or food security alerts, and (3) organization planning. There are substantial similarities between the type of decision-making supported by information products in the two countries. For agricultural planning, products exist in both countries that are intended to support the (re) programming of field activities, selection of crop varieties, programming of planting and harvesting, and the sale of products. In both

Fig. 2.
Dendrograms showing the results of cluster analysis of information products with relevant information on (a) drought management for agriculture and food security in Guatemala, available in 2015; (b) agro-climatic information in Colombia, available in 2018. The dendrograms group the information products according to the patterns of citation between pairs of products. Colors indicate the group of products for a statistical solution of three cluster in Guatemala and two clusters in Colombia. In Guatemala the grouping is principally determined by geographical scale; in Columbia the grouping is principally determined by the production processes, i.e. co-produced products vs. those produced by a single organization. Products are identified by product codes; for their full names see Table 1. Bouroncle, et al. Climate Services 16 (2019) 100137 countries, most of these products contain local-scale information, reflecting the scale at which such decisions are taken. Early warnings of both flooding (daily timescale) and drought (monthly to seasonal) are given by information products in both countries. Finally, with regard to organizational planning, we note a strong focus in Guatemala toward the provision of information products for humanitarian actions that address food insecurity. By contrast, in Colombia, products are used for the design of project proposals and technical assistance programs, the formulation of research guidelines, and deciding on insurance against crop losses.

Decision-makers' perceptions of the usability of the information products
In Guatemala, the mean System Usability Scale (SUS) score for the assessed products varies between 72.5 for the Early Warning System bulletin (GTM_L_EWS) and 90.2 for the Monthly Meteorological Bulletin (GTM_N_MMB, Fig. 6). According to the qualitative scale proposed by Bangor et al. (2009), two products are 'Excellent,' twelve are 'Good,' and one is 'OK.' The highest scoring products are focused on a single theme and provide primary information. The standard deviations show a substantial variation among the scores, which suggests that the Fig. 3. Networks of information products and their relationships as sources and receivers of information in (a) Guatemala, and (b) Colombia. Each node represents an information product, an arrow represents a confirmed relationship between two products: the source and the destination. The shape of the node corresponds to the main theme of the product; the color indicates the type of publishing organization, and the size is proportional to the number of direct relationships it has with other nodes. The position of each node in the diagram is related to its function in information exchange: The closer a node is to the center of the diagram, the more important it is as a provider of information; while the closer it is to other nodes, the more easily can it exchange information with them. The diagram also identifies 'intermediary' products that have a role in transferring information between two otherwise unconnected products. Products are identified by product codes; for their full names see Table 1.
interviewees have variable experiences of using the products and that these may not be satisfying the full range of users' requirements. To some extent, the variation may also reflect the subjective nature of the categories used to assess the utility of the products.
In Colombia, the range of the mean SUS value for products is wider than in Guatemala, varying from 46.25 for the Monthly Bulletin of Agricultural Inputs (COL_N_MBI) to 92.00 for Agroclimatic Bulletin of Santander (COL_L_ABS, Fig. 6). The latter is the only product whose mean is close to the 'Best imaginable level' of 100 (Bangor et al., 2009). From the remaining products, ten are scored as 'Excellent,' one is 'Good,' and the least scored product is 'OK.' The two lowest-ranked products were also those with the lowest level of use among interviewed persons (COL_N_MBI, COL_L_EWS; Table 2). As in Guatemala, the product scores does not show any pattern related to scale, theme or publishing organization. However, we note that the top-5 ranked products (from COL_L_ABS with a SUS mean score of 92.00 to COL_L_ACB with a mean score of 80.42) are co-produced bulletins or mobile applications, that is, products where the design is user-friendly and/or that enables direct information transfer to users.
The SUS average scores are generally well supported by the qualitative assessments made by comments from the interviewees. We noted, however, that the number of comments that accompany the negative statements of the SUS questionnaire was about double the number of comments that accompany positive statements, in both countries. This suggests that the interviewees tend to use the negative statements of the SUS to express criticism and suggestions (see a summary of comments in Supplementary Table S2).

Characteristics of 'ideal' information products
We synthesized suggestions from interviewees for an ideal information product into a single set of characteristics for both countries.  Table 1. The direction of the flow of information is indicated by the arrow, pointing to the source product. A color is assigned to each information product and the colors of the arrows group information flows from the same source product (i.e. citations of that product).

Table 2
Medians of knowledge ratings for information products grouped by work location of respondents interviewed in Guatemala and Colombia. Categories: 1 -I have never heard about this product before, 2 -I have heard about it before, 3 -I have seen it once, 4 -I have consulted it before but I do not use it frequently, and 5 -I use it and consult it frequently. Different letters show significant differences (Kruskal-Wallis test), α = 0.05 (**) or 0.01 (***). The sample (n) of people from Guatemala City who evaluated local products is lower than the total number of interviewees because these products were later incorporated into the process. For full titles of products see Table 1 Table 1. Monitoring products are shown in the standard font; forecast products are shown in italics. . 6. Variation in the System Usability Score (SUS) across information products in (a) Guatemala and (b) Colombia. The thick vertical line indicates the median, boxes extend to the interquartile range, and whiskers extend to 5% and 95% of the distribution. Products are identified by product codes; for their full names see Table 1.

Fig
C. Bouroncle, et al. Climate Services 16 (2019) 100137 The 'ideal' characteristics are classified according to the constraints that they overcome following Patt and Gwata (2002) in Table 3 (also see  Supplementary Table S2) namely credibility, legitimacy, scale, cognition, procedures, choices, and content. In both countries, the most frequently mentioned desirable characteristics mentioned by interviewees corresponded to different aspects of procedure. Procedures included timely distribution, frequent publication, convenient delivery mechanism/channel (i.e., presentation in public settings, use of mobile phone applications, voice messaging, and radio), and the options of two-way (user-provider) communication and downloadable source information. In Guatemala, the second most frequently mentioned desired characteristic was cognition. Interviewees indicated that the ideal product should be in plain language, logically structures, provide a synthesis of key messages with the help of visual aids (maps, graphs, and their corresponding interpretation), and integrate various types of information (climatic, agronomic, food security). For Colombia, the second most frequently mentioned desired characteristic was credibility, including the use and comparison of trusted sources, and the appropriate explanation of methods, results and uncertainty levels. Less frequently mentioned as desired characteristics, but still considered relevant, were legitimacy and scale. Interviewees in both countries identified the incorporation of local feedback as an important process concerning legitimacy, whereas aggregation of information was the only desired characteristic identified with respect to scale. Differences between countries indicate that most limiting factors are context-specific, reflecting evolving needs for improvement as countries develop better climate services. Users consistently identified specificity (i.e. choices) in the recommendations as an important characteristic of an ideal information product. This desired characteristic is challenging to address since enhanced organizational capacities may be required in order to produce context-specific recommendations. The kind of specific recommendations that users would like to find on the information products might also require enhanced content, i.e. the incorporation of more specific types of data that are not available at present (see "Contents" in Table 3). Similarly, changes in the scale of analysis may also be required in order to provide site-and/or sector-specific recommendations.
The desirable characteristics of information products identified by the interviewees are consistent with the comments they provided when evaluating the products available with the SUS questionnaire, whether these accompanied any negative or affirmations. This is clearly seen by organizing the results of both exercises according to the scheme proposed by Patt and Gwata (2002) mentioned above (see Table 3 and  Supplementary Table S2). The results of both exercises are also consistent with the results and recommendations of previous research on the usability of climate services (Lemos et al., 2012).

Organizational feedback regarding information product assessment
Obtaining feedback from technicians and scientists involved in the generation of the products that were the subject to our assessment was an essential final step in our research. In both countries, it allowed providers to appreciate users' perspectives on the information they are providing and on the ways in which the information is being provided. All the representatives of information providers we spoke to found the network analysis a novel approach that helped them to understand the broad context in which their information products exist and how they interact with other products. They also said that network analysis could help identify thematic and functional gaps (e.g. links with local networks), which could be addressed by collaborating with other organizations. Over and above these novel features, the assessment was seen as a useful and replicable process that provided confirmation of challenges information providers already knew about and how existing products could be improved. In this sense, results from the assessment process could be used to justify planned changes in current information products. Supplementary Text S2 provides an overview of the feedback from information providers and how it was obtained.

Table 3
Desirable characteristics of information products to overcome constraints (Patt and Gwata, 2002) that limit its usability according to respondents interviewed in Guatemala and Colombia.

Constraints Characteristics
Credibility -Use of trusted sources and sources comparison.
-Explanation of monitoring and forecasting methodologies and assumptions. -Explanation of forecasting uncertainty level. Legitimacy -Incorporation of local feedback.
-Inventory of actions taken in response to the provision of the agro-climatic information. Scale -Information aggregated at first-or second-level administrative divisions of the country. Cognitive capacity -Plain language, links to a glossary when necessary.
-Logical structure with key messages at the beginning, and synthesis sections for extension agents and farmers.
-Integration of climatic, agronomic and food security variables in the analysis. Procedures -Timely distribution for planning and decision-making.
-High frequency of publication.
-Links to original data and downloadable maps.
-The apps should allow two-way communication -Appropriate delivery mechanisms (e.g., local meetings presentations, interactive maps or apps with climate forecast and crop recommendations, concise radio messages about key decisions, WhatsApp messages with the link to the products). Choices (recommendations and other information to influence decisions) -Concrete and practical recommendations based on explicit assumptions or scenarios.
-Drought impacts on coffee, livestock and drinking water, also relevant for food security.
-List of current and planned organizational responses.
-Maps with clear spatial references (e.g., areas vulnerable to drought or food insecurity).

Contents
-Climate: relative humidity observations, ENSO situation and forecast, mid-summer drought forecast, hurricane season forecast, dry spells monitoring, frost forecast, early flood warnings, maps of climate vulnerability. -Agriculture: evapotranspiration data, water requirements for crops, phenology, seed availability, pest and disease warnings, loss and damage estimates, farmer technical assistance budget, crops market prices, availability of financial instruments (e.g. loans and insurances). -Food security: population at risk and affected, acute and chronic malnutrition, access to drinkable water, food reserves, cost of the basic food basket.

Tailored climate information for agriculture and food security Guatemala and Colombia
We designed and applied an approach for assessing information products related to existing climate services for agriculture and food security. Across Guatemala and Colombia, our findings indicate that there are a substantial number of periodically published information products that specifically address these thematic areas. The products contribute in different ways to decision-making processes at both national and local scale, providing both monitoring as well as forecast information. These findings have important implications for the provision of climate services in the two countries.
A first finding, emerging from the inventories and in-depth review of the information products, is that most of them seem to be produced following a 'loading-dock' model, in which users are barely involved in the process of product design and implementation (Brasseur and Gallardo, 2016;Cash et al., 2006;Vogel et al., 2017). Exceptions in Colombia are the Córdoba (COL_L_ABC) and Santander (COL_L_ABS) agro-climatic bulletins, where the local technical agro-climatic committees enable the effective and coordinated co-production, translation, transfer, and use of agro-climatic risk management information (Loboguerrero et al., 2018). In Guatemala, the Jocotán Bulletin of Nutritional and Food Security (GTM_L_BSJ), is prepared by the Municipal Commission for Food and Nutritional Security (COMUSAN), a coordination platform involving several organizations. However, our findings indicate that these co-produced products tend to have higher usability scores compared to those produced by single organizations, even though co-produced products play a very different role in the networks in the two countries. This is an interesting finding since the product implementation process was not a principal focus of the initial usability assessment. We consider that this point should be further analyzed (e.g. through interviews with representatives of the organizations that co-produce information products).
Second, our analysis relates to the relationships between information products. In both countries, the number of connections is relatively low in relation to the total number of potential connections between products. To some extent, this relatively low connectivity among products may reflect the limited coordination among the organizations which produce them. These gaps in integration between stakeholders and information sources have previously been identified for Colombia and elsewhere (Buizer et al., 2012;Loboguerrero et al., 2018), and can be seen especially in Guatemala, where two products (Forecast Maps for 24,48 and 72 h [GTM_N_FMH]; and Price Bulletin [GTM_N_PB]) are entirely disconnected from the rest, and thus are not part of the network (and therefore omitted from Figs. 2-4) although they contain information that is not in other products. While it is possible that these products share connections in other contexts, it is clear that they are potentially useful, yet not used, to inform climate-related decisionmaking in Guatemala. In other cases, a lack of connectivity with other information products considered in this study is a result of the specialized nature of the information product and its target user group. This can be seen, for instance, in the Coffee Bulletin in Colombia (COL_-L_ACB), which, as its name indicates, mainly targets the coffee sector. This product may have different connections that would be revealed by analysis of a specialized coffee information products network. Thus, while there are opportunities to enhance organizational coordination and integration in the generation of information products, it is critical to understand the role of each product in the information network and leverage that role in any coordination initiatives.
Third, our analysis reveals that in both contexts (Guatemala, Colombia), we find a product that acts as a 'global' information provider. In both cases this global provider is a climate-only information product that virtually all other products refer to; however, in Guatemala it is a Central American product (the Regional Seasonal Climate Outlook [GTM_R_CACP]), whereas in Columbia it is a nationalscale one (the Monthly Climate Prediction Bulletin [COL_N_MBCP]). It is likely that information flow in both contexts would be more restricted if these products did not exist. More importantly, any investment in improving the credibility, legitimacy, scale, cognition, procedures, and content of these highly central products would result in improvements in all the other products that refer to them, and therefore, probably, also to improvements in decision-making processes at both national and local scales. It is notable that both these central products are produced under the leadership or with the involvement of the national meteorological services and are either a direct result of (Guatemala) or input to (Colombia) the Climate Outlook Forum process (Ogallo et al., 2008). Moreover, in both cases these products provide seasonal information, to support seasonal decision-making, and there are recently published studies documenting their forecasting skills that underpin their credibility (Alfaro et al., 2018;Esquivel et al., 2018). In contrast, the Regional Outlook on Food and Nutritional Security (GTM_R_RFAF), occupies, unexpectedly, a less central position in the Guatemalan network. Hence, a further question raised by our results is whether agricultural and food security stakeholders have access to the information they need to effectively link climate forecasts to decisionmaking processes (Buizer et al., 2012;Loboguerrero et al., 2018) or whether there is a lack of skill in integrating climate information in a way that is meaningful to agricultural decision-makers (Briley et al., 2015). This question could be investigated by researching user capacities and needs, and the extent to which the latter are met by the available information products and the information they provide.

Strengths, limitations, and performance of the assessment approach
This study outlines an approach to the assessment of CS based on a combination of network analysis and user-centered analysis. In the following sections, we briefly examine the strengths and limitations of the approach and consider how it could be improved.
The typology of evaluations of development projects, programs, and policies proposed by the Organization for Economic Cooperation and Development (OECD, 2002) is useful for delimiting the scope of potential application of our approach, i.e. what it can and cannot be used for. Our approach can be applied to explore the relevance of the CS (e.g. their use and usability by decision-makers with different objectives) and their efficacy in contributing to the flow of information between products (e.g. by revealing the complementary roles of different information products as part of an information network). In this way, our approach can contribute towards remedying deficiencies identified in these respects: one could think about targeting improvements in usability in specific information product or identifying entry points for new climate services (e.g. improved seasonal forecasts) in the information network. On the other hand, our approach does not provide information on efficiency in the use of resources (e.g., time and money required to co-produce information products), nor on the impact of products in terms of behavioral change and downstream development impacts (e.g. avoidance of losses and damage to crop harvests). Nor does the approach consider the sustainability of measures that could be taken to improve the CS (e.g. organizational and policy changes that facilitate the use of CS). Taking all of the above into account, we propose the principal applications of our approach are (i) as a diagnostic tool to identify opportunities for improved provision of CS with an information network (diagnostic assessment); and (ii) to explore the effects of actions taken by projects, programs and policies with the aim of improving the provision and usability of CS.
Our experience of the design and implementation of the assessments in Guatemala and Colombia, in addition to findings reported in the literature on the evaluation of user-centered programs and development processes focused on innovation (see for example Guijt and Woodhill, 2002;Birachi et al., 2013), suggest that the approach proposed here will be especially useful for strengthening internal selfassessment processes, oriented towards learning from experience and reflecting on the lessons learned. In any case, the success of this and other assessment approaches will always depend on the quality of the design of the evaluation process, as well as the existence of organizational structures that provide a context that is conducive to their application (Rodríguez Aríza and Monterde Díaz, 2014). Aspects of our approach that contribute towards satisfying these conditions include the iterative nature of the process, the participation of stakeholders throughout the study, the production of data that is comparable across different contexts (see previous section), and its novelty, which was confirmed in meetings that were held to share the results with CS providers in the concluding phase of the study. Staff of these organizations found it interesting to see their products analyzed from the perspective of the broader network of products of which they formed part; our presentation of these results generated lively discussions about how to improve communication flows. Furthermore, the verification of the levels of use of individual products and the SUS scores provided useful insights for CS providers on the effectiveness of their products and ways they could be improved.
The application of the approach in Guatemala and Colombia demonstrated its capacity to generate comparable results in different contexts, while the Methods section of this study provides a transparent description of a standardized method for generating comparable results in future applications.

Suggestions for the future development of the approach
A number of changes were incorporated into the methodology between the assessments in Guatemala and Colombia, specifically changes in the order of the interview questions. However, in hindsight, we also consider that some steps in the method were too time-consuming. Specifically, the detailed review of each information product to determine which other products it cited took considerable time, as few products include a reference list. Furthermore, the time required to conduct the interviews and analyze the results could be reduced by limiting the number of open-ended questions, to eliminate the overlap between the comments on the SUS scores and the identification of characteristics of ideal products. There is also scope to improve the graphical presentation of the results to make them more accessible to non-experts. Studies using the assessment approach could explore these and other opportunities to improve both the design and the application of the methodology.
The approach could be improved by making more efficient use of resources when carrying out the assessment. Where evaluations include a large number of products and are done regularly, it is desirable to automate and simplify as many elements as possible (e.g. Jarvis et al., 2015). To increase efficiency in the use of available resources, it may be possible to conduct online or interactive-voice-response surveys, instead of face-to-face interviews. While this reduces time, it may also reduce response rates, requiring the survey to be sent to a larger number of personalized contacts (Cook et al., 2000). Moreover, keeping the respondents engaged is recognized as a challenge in online surveys (Heerwegh and Loosveldt, 2008) and occasional face-to-face interaction may still be required. For the information product network analysis, automation of citation searches may be possible, including the identification of elements copied from other information products. Such automated searches become more attractive if they have the capacity to cover several countries and scales. Building infographic layouts that convey statistical and graphical results in a more friendly way will also contribute to more efficient communication of the results.
The proposed approach leaves flexibility for integrating improvements and increase the effectiveness on the assessment, either based on lessons learned or in response to specific characteristics of the context. For example, given the importance of co-production processes revealed by this study, we consider it essential to include in the assessment approach a specific procedure to generate information on this topic, using standard principles such as those proposed by Vincent et al. (2018) or Bremer et al. (2019).

Considerations in operationalizing the assessment approach
In operationalizing our assessment approach, it will be essential to ensure that the methodological procedure adopted corresponds to the objectives of the assessment. Organizations leading the assessments could decide to simplify our approach and focus on a specific aspect depending on their own area(s) of interest and availability or resources. For instance, a government might be interested in understanding how connections between products or group of products evolve over time. In that case, only the cluster, network, and migration analyses are needed, and the assessment can concentrate on a desk review of citations and references. On the other hand, a farmers' organization may be interested in knowing whether an information product is used by their target group of extension agents or farmers and whether or not these people find the information product suitable for use. In this case, only the use and usability aspects of the assessment may be required. Depending on the available resources, the cluster, network, and migration analysis can be performed with an inventory that includes all available products, and a subsequent selection can be made for the decision-makers' assessment.
Any such assessments should mimic the ideal process of elaboration of the information products and should, therefore, be carried out as far as possible in coordination and collaboration between relevant organizations. If various organizations conduct product assessments in parallel for the same geographic or thematic areas, they should coordinate their efforts to maintain a common framework for assessment and reporting and share data and results.

Conclusions
Substantial progress has been made in Guatemala and Colombia in terms of climate variability management, suggesting a transition from reactive crisis response to proactive risk management. Product networks tend to integrate themes or disciplines. However, while climateonly information products are well connected and play a central role in the identified networks, work is needed to develop applied agriculture and food security information products to enable integration of multiple fields of knowledge and perspectives into decision-making processes. Furthermore, the results of the network analysis indicate that further opportunities for information flow exist that are not yet being taken advantage of; as evidenced, for example, by the disconnection of local information products from national-scale products in Guatemala. On the other hand, the results of our user-centered assessment point to the need for major improvements to most existing products in order to respond to the requirements of current and potential users. Brevity and clarity of language are highlighted as desirable in both countries, as well as use of trusted and publicly-available data, and non-paper-based delivery formats. Opportunities to improve CS were revealed by both analytical methods applied in this study. Network analysis identified the potential benefit to the network as a whole of investing in the further development of products that have a strong influence on other products. The user-centered analysis identified a range of potential improvements to individual products. More importantly, any investment in improving the credibility, legitimacy, scale, cognition, procedures, and content of these highly central products would result in improvements in all the other products that refer to them, and therefore, probably, also to improvements in decision-making processes at both national and local scales.
Our approach helped to elucidate general configurations of networks, for example by revealing the role of regional and national seasonal climate outlooks as 'global information providers'; as well as identifying areas where there is scope for improvement, for example by improving the connectivity of the regional food and nutritional security outlook in Guatemala. It also helped to identify measures that could be taken to improve the usability and use of existing products. The results of our user-centered assessment suggest that, while sector-specific data is available and capacities exist in both countries, information management does not always meet users' requirements. Enhanced or novel products are required to address constraints identified by the broad range of users interviewed here, most notably in terms of cognition, procedures, and credibility. When generating new products or improving existing ones, it is important to include a range of users' and organizational perspectives through the adoption of more user-driven design processes, in order to fully realize the potential for transforming existing data into useful information products that address users' needs. A user-centered design process and clarity regarding the broader information ecosystem will avoid developing products that (i) are disconnected, unusable or do not end up in the hands of users; (ii) are redundant, or (iii) that do not address existing information needs.
Our results were shared with the information providers, generating a first feedback loop between information producers and users. Moreover, the fact that our analysis included users and products at multiple scales helped to capture both the broad range of existing products and a representative cross-section of users' perspectives. The use of standardized questions and a standardized metric (SUS), helped to make comparisons possible across scales and themes. This standardization will also be essential to sustain comparability across time and space when the assessment is repeated and when it is replicated in other countries. Accumulated insights into climate information products and networks may lead to the establishment of benchmarks that can guide quality management. We believe that only through systematic assessment of CS for agriculture and food security can we know whether they are effective in providing high-quality, actionable information to improve the decisions of farmers, extension agents, agricultural organization managers, international cooperation agencies, governments, and others. Such systematic understanding can then help to prioritize actions and co-develop new products. Our systematic assessment approach provides a sound basis for coordinated efforts towards this end.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.