Using co-production to improve the appropriate use of sub-seasonal forecasts in Africa

Forecasts on sub-seasonal to seasonal (S2S) timescales have huge potential to aid preparedness and disaster risk reduction planning decisions in a variety of sectors. However, realising this potential depends on the provision of reliable information that can be appropriately applied in the decision-making context of users. This study describes the African SWIFT (Science for Weather Information and Forecasting Techniques) forecasting testbed which brings together researchers, forecast producers and users from a range of African and UK institutions. The forecasting testbed is piloting the provision of real-time, bespoke S2S forecast products to decision-makers in Africa. Drawing on data from the kick-off workshop and initial case study examples, this study critically reflects on the co-production process. Specifically, having direct access to real-time data has allowed user-guided iterations to the spatial scale, timing, visualisation and communication of forecast products to make them more actionable for users. Some key lessons for effective co-production are emerging. First, it is critical to ensure there is sufficient resource to support co-production, especially in the early co-exploration of needs. Second, all the groups in the co-production process require capacity building to effectively work in new knowledge systems. Third, evaluation should be ongoing and combine meteorological verification with decision-makers feedback. Ensuring the sustainability of project-initiated services within the testbed hinges on integrating the knowledge-exchanges between individuals in the co-production process into shaping sustainable pathways for improved operational S2S forecasting within African institutions

Forecasts on sub-seasonal to seasonal (S2S) timescales have huge potential to aid preparedness and disaster risk reduction planning decisions in a variety of sectors.However, realising this potential depends on the provision of reliable information that can be appropriately applied in the decision-making context of users.This study describes the African SWIFT (Science for Weather Information and Forecasting Techniques) forecasting testbed which brings together researchers, forecast producers and users from a range of African and UK institutions.The forecasting testbed is piloting the provision of real-time, bespoke S2S forecast products to decision-makers in Africa.Drawing on data from the kick-off workshop and initial case study examples, this study critically reflects on the co-production process.Specifically, having direct access to real-time data has allowed user-guided iterations to the spatial scale, timing, visualisation and communication of forecast products to make them more actionable for users.Some key lessons for effective co-production are emerging.First, it is critical to ensure there is sufficient resource to support co-production, especially in the early co-exploration of needs.Second, all the groups in the co-production process require capacity building to effectively work in new knowledge systems.Third, evaluation should be ongoing and combine meteorological verification with decision-makers feedback.Ensuring the sustainability of project-initiated services within the testbed hinges on integrating the knowledge-exchanges between individuals in the co-production process into shaping sustainable pathways for improved operational S2S forecasting within African institutions.

Practical implications
Reliable and useful sub-seasonal to seasonal (S2S; 2-4 weeks) forecasts have huge potential to increase people's resilience to weather-related extremes.Understanding what drives changes in weather on these timescales and how well forecast models are able to capture them is an active area of research.However, applying forecasts on these timescales to decision-making contexts, particularly across the African continent, remains in its infancy.There is increasing evidence that effectively implementing such forecasts requires a collaboration across a range of disciplines and stakeholders, through a process known as co-production.Co-production brings together different knowledge sources, experiences and working practices from different disciplines and sectors to jointly develop new and combined knowledge for addressing societal problems of shared concern and interest.
This study describes an S2S operational forecasting 'testbed' which forms part of the African SWIFT (Science for Weather Information and Forecasting Techniques) project.This S2S testbed is a two-year forum that brings together researchers, forecast producers and users from African and UK institutions, to participate in a co-production process using real-time forecasts from state-ofthe-art forecasting models.Operational groups from across East and West Africa are working closely together to co-produce new, user-driven S2S forecast products to aid decision-making in sectors such as agriculture, food security, energy and disaster risk reduction.In this operational testbed, pilot products are continually being used, iterated and evaluated in real time to improve their practical application for decision-making.Specifically, examples of user-guided iterations to the spatial scale, timing, visualisation and communication of forecast information have been shown to make products more actionable.For example, in many instances annotating spatial maps by adding county boundaries or the location of large cities was enough to improve interpretation without inappropriately representing the data.
In order to ensure the sustainability of project-initiated operational services it is crucial to critically reflect and evaluate both the new forecast products and the co-production process itself.Some initial, and crucial, lessons are emerging from the S2S testbed: i) it is critical that there is sufficient resource to support the co-production process, especially in the co-exploration of user needs; ii) all testbed participants require capacity building to enable effective co-production since it requires most working outside their existing knowledge and working practices; iii) developing evaluation systems which combine measures of forecast quality with decision-makers insight is critical to the useful application of new products.Crucially, the African SWIFT project is supporting the systematic assessment of the meteorological skill of the S2S models which is key if they will be used to activate preparedness action in resource-constrained environments.
To continue to build upon the success of the S2S testbed so far, individual knowledge gained through scientific research and userengagement needs to be integrated into institutional knowledge within operational forecasting procedures.A key aspect of this is building the cross-institutional partnerships between national meteorological services and in-country research institutions.Doing so will create sustainable pathways to continue the coproduction of S2S operational products within the African institutions involved.Documenting and evidencing the new forecast products produced and lessons learnt through the co-production process will support the generation of new forecast products in other regions and institutions.

Introduction
Forecasts on sub-seasonal to seasonal (S2S) timescales (from 2 to 4 weeks) bridge the gap between short-range weather forecasts (hours to days) and longer-term seasonal outlooks (Brunet et al., 2010, Vitart et al., 2012, Robertson et al., 2014).The provision of reliable S2S forecast information has huge potential to support planning decisions and, through early warning, allow for proactive disaster mitigation and preparedness (Coughlan de Perez et al., 2016, White et al., 2017).As such, increasing the appropriate use of forecast information on S2S timescales, provided it can be reliable and useful (Jones et al., 2015), has the potential to increase people's resilience to weather extremes and have a transformative impact on livelihoods (Williams et al., 2015, Nkiaka et al., 2019).However, there is increased recognition that doing this effectively requires iterative collaboration across a range of disciplines and stakeholders (Cash et al., 2003, Vaughan andDessai, 2014) through a process known as co-production (Bremer and Meisch, 2017, Vincent et al., 2018, Visman et al., 2018).Variously defined (e.g., Bremer andMeisch, 2017, Vincent et al., 2018), co-production of weather and climate services can be considered as the process of combining knowledge from different actors to jointly develop new products and services addressing issues of shared concern (Visman et al., 2018).As such, it transforms the role of the user from recipient of information to participant in the knowledge generation process (Vincent et al., 2020).By shifting the focus of forecast development from producers to jointly developing knowledge in the decision-making contexts of users (Vaughan andDessai, 2014, Vincent et al., 2020), co-production provides an opportunity to improve the application of S2S forecasts.Coproduction also facilitates the sharing of knowledge between meteorological and climate institutions within-country, within-region and internationally.
Despite its huge potential to improve action-based forecasting, making predictions on S2S timescales is challenging because it is sufficiently long that the impact of the initial conditions has significantly reduced; but sufficiently short that slowly varying modes, such as oceanic circulations, are only just starting to influence the forecast (Fig. 1; Vitart et al., 2012).There are, however, some phenomena and processes in the atmosphere, ocean and land surface which vary on S2S timescales and provide crucial sources of forecast predictability (Vitart et al., 2015).Some of those most relevant for the African continent are the Madden-Julian Oscillation (MJO; Zaitchik, 2017, Sossa et al., 2017, Kim et al., 2018), which is the major mode of sub-seasonal variability in the tropics, sea surface temperatures (SST) associated with coupled seasonal phenomena such as the El Niño Southern Oscillation (ENSO; Hudson et al., 2011, Olaniyan et al., 2019) and the Indian Ocean Dipole (IOD; Hirons and Turner, 2018), as well as land surface conditions such as the influence of slowly-varying soil moisture anomalies (Douville et al., 2001, Koster et al., 2011).
Producing accurate sub-seasonal forecasts relies on having a model which can capture the sources or drivers of sub-seasonal predictability (e.g., MJO;Zaitchik, 2017); as well as the response of local weather to such a driver (e.g., modulation of local precipitation by MJO; Berhane and Zaitchik, 2014).For many of the drivers outlined above, and particularly over Africa, much of this scientific understanding is in its infancy and the forefront of current knowledge (White et al., 2017, MacLeod and Palmer, 2018, Vigaud and Giannini, 2019, Moron and Robertson, 2020, de Andrade et al., 2021, Endris et al., 2021).
While having a model that is able to capture the drivers of subseasonal variability and their local weather response can provide L. Hirons et al. reliable information, this is not sufficient to develop effective weather and climate services on S2S timescales (Cash et al., 2006, Dilling and Lemos, 2011, Lemos et al., 2012).In order to improve early warning systems, reduce weather-related vulnerability and build more resilient livelihoods (Williams et al., 2015, Nkiaka et al., 2019), forecast knowledge and products also need to be useful and actionable in the decision-making contexts of users (Dilling andLemos, 2011, Lemos et al., 2012).The assessment of forecast skill not only requires a technical and scientific review of forecasts against observations (Murphy, 1993), but also an understanding of users' perceptions of forecast accuracy at the geographic and temporal scale relevant for a particular decision-making process.Co-production supports this shift towards jointly developing new decision-relevant weather and climate knowledge (Bremer and Meisch, 2017, Vincent et al., 2018, Visman et al., 2018, Bremer et al., 2019, Carter et al., 2019).
However, co-production is not merely a consultation with interested stakeholder groups to find an application of existing science or, in this case, forecast information.Rather, done appropriately, it demands a new and potentially uncomfortable approach to knowledge creation.Coproduction shifts away from a 'one-way' supply of scientific information towards a 'two-way' demand-led iterative process (Kirchhoff et al., 2013, Vincent et al., 2018), with an improved collaborative partnership between producers and users (Vaughan and Dessai, 2014).
Historically, the link between developments in meteorological understanding and forecasting and their 'pull through' to effective operational forecast products has been weak (Dilling andLemos, 2011, Lemos et al., 2012).S2S forecasting, in particular, is in its infancy on the African continent with many National Meteorological and Hydrological Services (NMHSs) currently producing limited forecasting information on these timescales (White et al., 2017).However, by directly working with specific users in the co-production of weather and climate services, a wide range of international, regional and national initiatives, the GCRF (Global Challenges Research Fund) African Science for Weather Information and Forecasting Techniques (SWIFT) project, are seeking to address these imbalances in access and usability (Hewitt et al., 2012, Dinku et al., 2018, Nkiaka et al., 2019).
Drawing on data from the kick-off workshop and initial activities of a sub-seasonal forecasting testbeda forum where prototype forecast products are co-produced and operationally trialled in real-timethis study considers: a) how co-production can influence the use of S2S information in operational forecasting; (b) what key challenges there are in making the co-production of project-initiated S2S forecast products sustainable; and (c) what lessons have been learnt so far about how coproduction can increase the appropriate use of S2S forecast information, and how they can inform future collaborative efforts.
Given the lack of S2S forecast information currently being used operationally in Africa, and the huge potential of reliable S2S information on action-based forecasting (White et al., 2017), answering these key questions is crucial to advancing this rapidly developing area of research and meteorological application.This study, and the ongoing African SWIFT S2S testbed, will form a basis of evidence that the coproduction of sub-seasonal forecast products supports potentially 'useful' S2S information to become 'usable' and 'used' by forecast users (Boaz and Hayden, 2002;Lemos et al., 2012).
The data and methodology section outlines details from the kick-off workshop and testbed case studies.The results and discussion section draws on insights from this data to answer the first two research questions.Finally, the lessons learnt through this co-production process are drawn together as the main conclusions of the study.

Sub-seasonal to seasonal (S2S) data
The GCRF African SWIFT project has been granted real-time access to the Subseasonal to Seasonal (S2S) Prediction project (Vitart et al., 2017) forecast data for a period of two years starting in November 2019.The S2S Prediction project is a World Meteorlogical Organization project to promote research into sub-seasonal prediction and prediction systems and increase the uptake of sub-seasonal forecasts in user decision making.As part of this project it has produced a database of operational sub-seasonal predictions available with a 3 week lag for research purpose.To demonstrate the social and economic value of S2S forecasts the second phase of the S2S Prediction Project (World Meteorological Organization, 2018) includes a time-limited (2 year) Real-time Pilot Project which provides real-time access to forecasts in the S2S Project database to enable feedback on the value of these forecasts in a real-time decision-making context, including aspects such as the timing and communication of the forecasts.African SWIFT is one of 16 projects taking part in this S2S Real Time Pilot Initiative.The African SWIFT S2S testbed is using real-time forecast data from the operational configuration of the ECMWF model at 1.5 • resolution (Table 1).Following user feedback, weekly forecast initialisations are downloaded on Mondays with the data available on Tuesdays for product development.Model forecasts can have systematic biases which need to be considered when interpreting the forecast.These biases can be estimated by creating a set of "re-forecasts" or "hindcasts" of previous years and comparing them to an observed climatology.The hindcasts can then be used to provide a good estimate of the bias in the real-time forecast.Consistent with the operational procedure at ECMWF, the three closest hindcast initialisation dates are used to compute the climatology and subsequent anomalies for the forecasted variables.This gives a larger, more statistically robust ensemble size of 33 for each hindcast year.

Co-production approach
The approach being followed during the African SWIFT S2S forecasting testbed is based on the guiding processes and principles outlined in the 2019 Manual: Co-production in African weather and climate services (Carter et al., 2019).Informed by key literature and operational best practice (Vincent et al., 2018;Bremer et al., 2019;Visman et al., 2018), this manual drew together emerging experiences from across regions, sectors and livelihood groups to identify emerging consensus about the process and principles that have proved effective in enabling co-production of climate services to support specific decision-making contexts.Specifically, all the activities within the testbed have been and will continue to be guided by the ten principles identified as underpinning effective co-production: improve transparency of forecast accuracy and certainty; tailor to context and decision; deliver timely and sustainable service; build trust; embrace diversity and respect differences; enhance inclusivity; keep flexible; support conscious facilitation; communicate in accessible ways; and ensure value-add for all involved.
Furthermore, in order to develop reliable and actionable tools for decision-making on sub-seasonal timescales, activities in the S2S forecasting testbed will be structured around the six building blocks of coproduction (Carter et al., 2019).A brief description of the six building blocks are provided below:

B1. Identify key actors and build partnerships
In order to form an effective basis for co-production, it is essential to develop equitable, trust-based relationships.

B2. Build common ground
Develop a shared understanding, across actors, of the intention and desired outcomes of the co-production process.Consider competing priorities, managing expectations and capacity development.

B3. Co-exploring needs
Cementing the relationships and understanding between actors.Creating a space where jointly defined issues can emerge.

B4. Co-developing solutions
Solutions to the identified issues are developed through collaborative knowledge exchanges with contributions from a variety of expertise from across the actors.

B5. Co-delivering solutions
Requires agreement about how to communicate the collaborative outputs to ensure they are accessible; that cultural considerations have been taken into account; and that all contributors are appropriately acknowledged.

B6. Evaluate
Importance of scheduling regular time for reflection and monitoring.Additionally, each co-production building block should be evaluative to enable regular review, ongoing feedback and highlight lessons learnt.
These building blocks have been numbered here for clarity but, by design, should not be limited to occurring sequentially in this order.For example, as stated in B6, each co-production building block should have an evaluative element.Furthermore, the entire co-production process is iterative so many, if not all, of the blocks will occur more than once.

S2S forecasting testbed: structure, kick-off workshop and case studies
African SWIFT is using a two-year forecasting testbed as the context for co-producing new forecast information on S2S timescales.The S2S testbed is made up of representatives from the African SWIFT partner organisations and can be thought of as having 6 "O"perational groups (O1-O6; Table 2).These are comprised of pan-African (ACMAD 1 ; Niger) and regional (ICPAC 2 ; Kenya) climate centres, as well as the NMHSs and partner Universities in the four partner countries: Ghana (GMet, KNUST) 3 , Kenya (KMD, UoN) 4 , Nigeria (NiMet, FUTA) 5 and Senegal (ANACIM, UCAD) 6 .Activities are also supported by the African SWIFT UK organisations (NCAS, UKCEH, UoR and UoL) 7 .By supporting operational forecasts through national and international collaborations, and specifically between NMHSs, universities and users in partnering countries, the African SWIFT testbed is aiming to foster the collaborative partnerships required for successful co-production (Vaughen and Dessai, 2014).

Kick-off workshop
A kick-off workshop for the forecasting testbed was held at the ICPAC headquarters in Ngong, Kenya.The intense, week-long workshop was held in November 2019 to coincide with the start of two years of access to real-time S2S forecast data.The workshop brought together forecast users (10), operational forecast producers (11) and researchers (7) from 7 African countries (Cameroon, Ghana, Kenya, Nigeria, Niger, Senegal and Uganda) across the 6 operational groups in Table 2, as well as researchers (10) from the UK, totalling 38 attendees.The forecast users represented the agriculture, food security, energy and disaster risk management sectors.All these sectors are recognised in the GFCS as having huge potential to benefit from improved S2S forecast products in their decision-making.See Tables S1 and S2 in supplementary material for details of kick-off workshop participants and affiliations.Following the kick-off workshop regular testbed-wide and operational group virtual interactions have continued between forecast users (8), operational forecast producers (14) and researchers (16).
Data is drawn from the kick-off (KO) workshop and initial activities described here.Details of each activity are given below, including the main corresponding co-production building block.Activities are labelled and referred to as KO1-KO8 for clarity of discussion in the results section.
KO1: Pre-kick-off workshop user questionnaires were distributed to the six identified key users from each operational group (Table 2; see S3 in supplementary material) to identify how S2S information could potentially influence decision-making in their sector and specific decision-making context (B1, B2).
KO2: A setting expectations discussion group aimed at coexploring how forecast users (10), forecast producers (11) and researchers (17) viewed their own and others' roles in the co-production process (B2).Individuals were asked to articulate their own expectations of the co-production process within an operational forecasting testbed, as well as reflect on the potential expectations of other actors in the process.KO3: 4 sector-themed discussion groups for Agriculture (4 users, 3 producers, 3 researchers), Food Security (1 user, 2 producers, 6 researchers), Energy (2 users, 2 producers, 4 researchers) and Disaster Risk Reduction (4 users, 3 producers, 4 researchers) to co-explore sector specific S2S needs (B3).These groups also discussed misunderstood terminology which they had encountered (B2).
KO4: Timelines of decision-making were developed within each operational group to co-explore the annual context into which S2S information would be added.These outlined the timing of dry and wet seasons, important sector activities, key user decisions, and current available forecast information (B3; see S4 in supplementary material for an example).
KO5: Country-level network maps were produced within the SWIFT partner countries of Kenya, Ghana and Nigeria to co-explore the organisational governance context into which S2S information would be communicated (B5; see S5 in supplementary material for an example).
KO6: Bi-annual questionnaires are distributed to forecast producers and forecast users from each of the six operational groups.The forecast producer questionnaire aims to understand if and how the new testbed products are being incorporated into operational procedure, as well as capture how they have been iterated based on user feedback (see S6 in supplementary material.The forecast user questionnaire aims to understand how the new testbed products are being used in the decisionmaking context they were designed for (see S7 in supplementary material).Results will be reported from the first round of responses (B6).
KO7: Operational co-production action plans were developed within each operational group to serve as a memorandum of understanding formalising the relationships for continued co-production during the two-year testbed.These are based around the iterative coproduction building blocks (B1-B6), and include formalising the responsibility within the process for maintaining collaboration for product development, communication, and evaluation (see S8 in supplementary material for an example).
KO8: Extensive notetaking by the testbed facilitating team throughout the week's activities, including making observations of the co-production process and having informal discussions with participants (B6).

Case studies
As well as drawing on data from the kick-off workshop, results from two case studies are used to support the findings of this study (B4).Case study one is between a pan-Africa climate information producing institution (ACMAD) and a central Africa regional climate centre (CAPC-AC; O1, Table 2), hereafter referred to as "CS: CAfrClim".Case study two is between a national meteorological service (KMD) and a range of substate urban flood risk decision-makers (O4; Table 2), hereafter referred to as "CS: KenFloodRisk".These case studies are named after their user application and have been purposely chosen to represent the diverse collaborative partnerships that exist in the forecasting testbed.The pre-testbed status quo and co-developed solutions for each case study are described below: Case study 1: CS: CAfrClim Pre-testbed status quo: Forecasters at CAPC-AC provide sub-seasonal forecasts to support NMHSs for the Economic Community of the Central African States (ECCAS) which comprises of eleven nations including Angola, Burundi, Cameroon, Central African Republic, Chad, Democratic Republic of Congo, Republic of the Congo, Equatorial Guinea, Gabon, Rwanda, and Sao Tomé and Principe.Current forecast products used at CAPC-AC include sub-seasonal predictions from National Centers for Environmental Prediction (NCEP) Climate Forecast System (CFS) version 2 (Saha et al., 2010) and National Oceanic and Atmospheric Administration (NOAA) Subseasonal Experiment (SubX) project (Pegion et al., 2019).Alongside predicted precipitation totals, diagnostics used to inform forecasters include precipitable water, outgoing longwave radiation, tropospheric wind shear, and MJO (Zaitchik, 2017) characteristics.The combination of all of these diagnostics from different modelling centres are used to produce a forecast of precipitation anomalies (Fig. 2).These anomalies are classified into five categories and combined with the likelihood of the precipitation anomaly occurring.Colours and numbers are used to highlight regions of substantial precipitation anomalies.Currently, producing Fig. 2 requires a high level of scientific expertise to subjectively combine different information sources.
Co-developed solution: Within the testbed, a collaborative coproduction partnership has been formed between researchers at  UKCEH and NCAS and forecast producers at ACMAD and CAPC-AC to jointly identify how access to real-time sub-seasonal forecast data can contribute to their weather and climate services.An additional, objective bulletin of extreme precipitation events has been added using the ECMWF 51-member probabilistic forecast.To produce this product, the number of ensemble members with a weekly-accumulated precipitation anomaly within specified thresholds are counted.Figs. 3 and 4 show the iterative versions of the probability of weekly-accumulated precipitation anomalies in each threshold.
Case study 2: CS: KenFloodRisk Pre-testbed status quo: Prior to the S2S forecasting testbed, participants from the ForPAc project, in collaboration with KMD, conducted a stakeholder workshop in Nairobi County to identify the format of subseasonal forecast products that users find most useful for their decision-making contexts.The one-day workshop had 18 in attendance; 16 (13 forecast users and 3 forecast producers

Results and discussion
Co-producing sub-seasonal forecast products with forecast users, forecast producers and researchers is an ongoing iterative process.It is time-consuming and involves many actors working in a way or context which is (potentially) novel to them.Whilst the forecasting testbed is ongoing, this section draws on data collected from the kick-off workshop, insights from extensive notetaking of the co-production process itself and case study examples to answer the first two research questions.The third question on lessons learnt from the co-production process will be addressed in the conclusions.

Co-production within an S2S context (a) How co-production can influence the use of S2S information in operational forecasting
Co-production has the potential to increase the use and uptake of S2S information in operational forecasting.An emerging theme from the pre-kick-off forecast user questionnaires was the desire for improved sub-seasonal forecast information on dry and wet spells within a rainy season.Specifically, having access to such forecast information could influence the "timing of planting, crop choices and varieties" as well as "prevent wastage" in the agriculture sector (O3, O5, O6).In the energy sector S2S information on dry and wet spells has potential to allow for "appropriate scheduling of our [power] generation from the hydropower plants" (O4).While the disaster risk reduction and food security sectors highlighted its potential for improving longer-term strategic planning (O1, O2; KO1).This supports the notion that improved access to S2S weather information has huge potential to improve early warning systems, reduce weather-related vulnerability and build more resilient livelihoods (Williams et al., 2015;Fig. 1).However, to realise this potential, it is recognised that forecast products have to be both reliable and actionable (White et al., 2017) in the decision-making context of users (Dilling andLemos, 2011, Lemos et al., 2012).
For a forecast to be reliable it needs to exhibit meteorological skill.Researchers in the testbed have conducted a thorough assessment of the pan-African skill of S2S models (de Andrade et al., 2021, Endris et al., 2021) and shown that the ECMWF forecast system has the highest skill in predicting precipitation in weeks 1-2.While skill was lower for weeks 3 and 4, probabilistic forecasts were shown to have reasonable skill in wet regions during particular rainy seasons (e.g., East Africa; March -May rains).Furthermore, a reduction in forecast quality was observed when the influence of large-scale drivers such as the MJO, Indian Ocean Dipole (IOD) and El Niño Southern Oscillation (ENSO) were removed (de Andrade et al., 2021).Therefore, we know that there is predictability in the ECMWF forecasting system at these timescales, but the fact that the skill can be dependent on the model (Endris et al., 2021), driven by the large-scale environment, and not geographically or temporally uniform (de Andrade et al., 2021), provides a significant meteorological communication challenge.Particularly because users highlighted terms such as "large-scale driver", "probabilistic" and "predictability" as all requiring further explanation.To address this forecast producers moved away from terms like "predictability" and "large-scale drivers" and instead talked about their confidence in the forecast (KO3).Effectively communicating probabilistic forecasts, which still have reasonable skill out to week 3 and 4 in some regions, remains an ongoing challenge.Specifically, more research is required to understand the implications that times and regions of lower forecast skill have on the co-production process and products.
Communicating the technical information in the forecast is not the only challenge, the source or legitimacy of the information was also deemed crucial for it to be reliable from a user's perspective.Users rightfully challenging "can I trust it?"and "is it better than what I already know?when receiving new forecast products (KO3, KO8).As a project piloting the use of real-time sub-seasonal information in operational forecasting these are very legitimate questions which will take careful monitoring, evaluation and learning (see Key challenges section) and can be more thoroughly answered in future studies which synthesise the forecasting testbed results and experiences over the entire two years.
For a forecast to be actionable it needs to be relevant to the decision a user is trying to make.Drawing on data from the kick-off workshop and the co-developed solutions outlined in CS: CAfrClim and CS: Ken-FloodRisk, there are many examples of user-guided iterations which have made testbed forecast products more actionable.Firstly, related to the spatial scale of the forecast product.Many users communicated interest to access forecast products on a different spatial scale to those they first received (KO3).For example, in CS: CAfrClim there was a request to reduce the domain size and focus on the central African nations relevant for the users' decisions (Fig. 3 to Fig. 4).In CS: KenFloodRisk, for relevance to local flood risk forecasting, the request was for a location specific forecast for Nairobi city as opposed to an area averaged version.
To do this the closest model gridbox for Nairobi was chosen for analysis (36.5 • E, 1.5 • S; Fig. 5).Feedback from forecast users has also revealed that information from new forecast products is used "to further downscale the forecast to usable and farmer friendly formats" to improve their interpretation (KO6).These requests for scale-related adaptations have been implemented, however, it is important to identify the appropriate scale for skilful sub-seasonal forecasting and the potential limitations in further downscaling the forecast information (Young et al., 2020).In many instances annotating spatial maps by adding county boundaries or the location of large cities was enough to address these issues without inappropriately representing the information (KO8).
Secondly, a forecast can only be actionable if the product arrives in time.The ECMWF data is available on Mondays and Thursdays, with download and analysis time factored in this would mean forecast products becoming available on Tuesdays or Fridays.Following consultation with the key testbed users (KO3, KO4), it was agreed that weekly forecasts, rather than bi-weekly, were sufficient to avoid "information overload" (KO8), and that Tuesdays would leave more 'working days' to make and implement decisions.Therefore, weekly forecast initialisations were continued on Mondays.Subsequent feedback from forecast users has reinforced that the forecasts were delivered with "perfect timing" for their decision-making purposes (KO6).That said, changes can be made in the analysis stage to improve the appropriate application in a specific context.For example, in CS: CAfrClim the analysis of weekly precipitation probabilities were chosen to begin on the Monday of each week to enable better integration of the product into existing forecasting practices.This is important because it avoids confusion and contradiction when users are faced with different forecast information from different sources.Further exploring the timing of the rainy seasons in different countries and the relevant sector-specific decisions which would need to be made provided a framework for when the important parts of the calendar year were in different regions and sectors (KO4, KO7; S4; Kenya Red Cross, 2019).For example, agricultural and food security forecast users explained the importance of "acquisition of inputs (crop types, seeds etc) and land preparation" decisions ahead of the main rainy season (February in O2 in eastern Africa; March and April in O3 in southern and northern Ghana, respectively; K04).Later feedback from forecast users revealed that new S2S forecast products were indeed "helpful as [an] advisory service to farmers on setting dates/times for land preparation activities, including procurement of inputs such as manures, seeds, fertilizers etc" (KO6).Furthermore, these timelines of decision-making identified potential gaps in existing forecast provision.Such decision-making calendars have been used in previous studies to identify key entry points where seasonal information with a longer lead time can support existing drought early warning systems (Mwangi andVisman, 2020, Audia et al., 2021).In this context, the real-time S2S data allowed the provision of weekly updates to complement existing monthly and seasonal forecasts (O2; KO4, KO7).
A third theme of user-guided iteration to improve application was in the visualisation of forecast products.For example, in CS: CAfrClim the use of grey shading for land regions with zero probability of substantial rainfall anomalies was applied based on feedback to improve the clarity of the product.Furthermore, during the stakeholder workshop in CS: KenFloodRisk users were shown the same sub-seasonal forecast information in five different forms.Interestingly, a box and whisker plot, which is often favoured by researchers for displaying probabilistic information, was ranked lowest by users for usefulness (Figure D in S9).Ranked highest were spatial maps (Figure A in S9) and bar graphs (Figure E in S9) of weekly precipitation.The former is already being routinely produced as part of the testbed and the user-guided iterations of the latter are shown in Fig. 5.These findings show visualisation is a key factor influencing user interpretation, a finding common with the representation of longer-term climate information (Daron et al., 2021).It also re-emphasised the need for the co-production of forecast products with users, rather than individual groups perpetuating the status quo.
A final, related, theme for making forecast products more actionable is their communication.For example, how a plot is labelled does not always make it clear to all groups what it shows.Nuances in terminology such as 'validation date', 'forecast date' or 'initialisation date' created confusion (KO8) and highlighted the need for clarification and better communication of existing and new forecast products.Communication improvements can be very specific to a particular product, for example in CS: CAfrClim the thresholds used in Fig. 4 are user-defined and designed, for consistency and clear communication, to match the existing bulletin (Fig. 2).Or, in CS: KenFloodRisk the difference between the model hindcast and the observations (Fig. 5(c)) caused confusion and mis-interpretation of the information so showing a bias corrected version (Fig. 5(d)) improved clarity considerably for users.In the case of CS: KenFloodRisk having direct access to the real-time data allowed the provision of a bias-corrected forecast, which took account of any systematic biases in the model, and enabled users to directly compare the forecast with local observed rainfall.The requirement for biascorrecting or calibrating a forecast model is a complex communication challenge in itself and takes considerable resource to apply in individual product applications.
As well as the specific information shown in the product, communication challenges also relate to the context into which the forecast products will be delivered.This was explored through a country-level network mapping exercise (KO5; see S5 in supplementary material for an example).One observation from this exercise was that in Kenya, following devolution, there are Directors of Meteorological Services appointed for each County that play a key role in supporting county government decision-making including through downscaling of weather information and development of services to support county-specific livelihood priorities.This is not currently the case in Ghana and Nigeria where forecast information is shared at a national level with the sector ministry (e.g., Ministry of Food and Agriculture; MOFA) which then disseminates to a regional level through MOFA.The advantages of the Kenyan model of County-level meteorological experts was recognised and the potential for implementing a similar structure elsewhere was discussed (KO5).Investigating the impact of these organisational structures on the uptake of S2S forecast information should be a focus of future research.
The kick-off workshop activities and case study examples given in this study highlight that co-production can be used to increase the appropriate use of S2S information in operational forecasting in Africa, provided they are reliable and actionable.Specifically, it has been L. Hirons et al. shown that forecasts can be made more actionable when users are included in discussions of scale, timing, visualisation, communication and evaluation of products.Table 3 provides a summary of activities and findings, across the building blocks of co-production, during the African SWIFT testbed kick-off workshop.
Collaboratively co-producing forecasts in this way may increase their appropriateness and uptake, however, it comes at a cost.It is time consuming and requires more personal commitment from individuals within institutions than other modes of knowledge production (KO8; Lemos et al., 2014).To account for this the S2S testbed identified a small subset of key users rather than a larger group in which the equitable, trust-based relationships required would be harder to build.Whilst the small number of, largely technical, forecast users was a function of the initial S2S testbed design, it does present limitations in scaling up testbed approaches and outputs to further their impact in wider and more remote regions.To address this issue, operational best practices and lessons learnt from the co-production process will be documented so they can inform future research agendas and operational procedures.

Key challenges
(b) What key challenges there are in making the co-production of projectinitiated S2S forecast products sustainable Many key challenges have already been encountered during the coproduction process within the African SWIFT S2S testbed.These are discussed below along the three emerging themes of process, sustainability and evaluation.

Process
It has been a considerable challenge incorporating widely differing views of co-production in the context of a real-time, operational testbed.These differing views were clear at the outset of the S2S testbed (KO2).Expectations of the users focused on the tailoring of products and the improved timing of forecast delivery.Meanwhile, expectations of forecast producers and researchers centered around the examination of forecast skill, as well as the improvement of trust and use of their science.All actors had the expectation that the co-production process would increase the use and uptake of S2S forecasts.This discussion revealed that some testbed participants take a descriptive view (Bremer et al., 2017) -that co-production embeds the existing power structures, even after collaborative interactions between researchers, forecast producers and users.This was particularly apparent in the comments of one forecast producer who suggested that users would be consulted, but not co-conceptualise new products: "our intention is to come up with [our own] working tools.I think that [the users'] contributions could only guide us to tailor our deliverables to something similar [to what we originally conceived]."This view of co-production does involve user interaction, but only within the existing power structures and not with joint ownership of the process (KO2, KO8).Other participants take the normative view (Bremer et al., 2017) -that co-production is a deliberate aim of the participating actors to engage in a process which, by including new perspectives, increases knowledge for use in decision making.This view was held by many users who expressed an expectation to have a better understanding and ability to communicate the forecast products, through their involvement in the co-production process (KO2).In light of these differing views, it was important to formalise the respective roles of forecast producers and users in an operational co-production action plan which served as a memorandum of understanding between groups involved (KO7, S8).
While all groups had expectations to increase the uptake of S2S forecast information (KO2), both descriptive and normative views of coproduction were envisaged as pathways to do so.Understanding these different views of co-production, and the existing power structures in which they are held (Daly andDilling, 2019, Turnhout et al., 2020), will help in addressing some of the barriers, at both individual and institutional levels, to the uptake of forecast information and provide insight into how it actually works in practice (Lemos et al., 2018).Specifically, Daly and Dilling (2019) argue a normative approach to co-production can only be transformative if the inequalities in partnerships are addressed and all actors, particularly scientists, reflect on their own practices to improve their productive engagement with other knowledge systems; a process which will take considerable time and personal commitment (Lemos et al., 2014).

Sustainability
A major challenge within the S2S testbed is ensuring resources to Ongoing access to the real-time S2S forecast data (Vitart et al., 2017) is contingent on evidencing the impact of new forecasting tools.Critically, such evidence involves not only reporting on meteorological forecast skill but also on the co-production process itself by incorporating user feedback (KO6).Documenting this process will help integrate practice with theory (Lemos et al., 2018) and identify where resources can be best directed to support effective ongoing coproduction.Furthermore, the S2S testbed, and African SWIFT project more generally, has a focus on strengthening in-country collaborations between NMHSs and research institutions (Table 2).Many previous projects have worked with one of these entities, however, the S2S testbed approach builds cross-institutional collaborations required to support the resource for ongoing action-based forecasting.
Co-production within the S2S testbed relies on participants working within new methods of knowledge production.Whilst valuable new skills are being acquired by individuals, documenting these skills, ensuring shared ownership of the process and providing training are imperative to enabling knowledge to become institutional rather than remaining individual.Furthermore, systematic monitoring and learning is required in order for NMHSs, particularly, to demonstrate the value of new services and on this basis justify requests for increased investment from national and sub-state governments, as well as further international investment.While potentially beyond the scope of the current S2S testbed, future solutions to ensuring project-initiated services can continue could include sustainable business models through public-private sector partnerships, as other projects have sought to support (e.g., Ouédraogo et al., 2018;Ouédraogo et al., 2020).

Evaluation
Evaluation is a cross-cutting theme of the entire S2S testbed coproduction process.While sources of predictability on S2S timescales present huge potential in aiding action-based forecasting (White et al., 2017), it is important to remember that forecasting on these timescales is still in its infancy (e.g., Moron and Robertson, 2020).It is therefore critical that any provision of new forecasting tools coincide with a systematic assessment of their skill (de Andrade et al., 2021, Endris et al., 2021).The challenge does not stop there, however; communication of forecast skill in new tools and products is arguably as important for its appropriate use in decision-making.The responsibility for improving communication lies with all groups in the co-production process.In the context of this forecasting testbed, responsibilities for maintaining collaboration within the iterative feedback process have been formalised through operational co-production action plans within each group (O1-O6; KO7).Ongoing user evaluation is captured through the bi-annual forecast user and producer questionnaires (KO6).Initial results from these have demonstrated that there is still a recognised need for strengthening decision-makers understanding and confidence in using probabilistic forecasts.However, there is an equally strong need to enhance the understanding and capability of forecast producers and researchers to systematically and effectively communicate the key technical concepts of their forecast products.Developing strategies which address both of these challenges will be a focus of the ongoing African SWIFT testbed.

Lessons learnt and conclusions
(c) What lessons have been learnt so far about how co-production can increase the appropriate use of S2S forecast information, and how they can inform future collaborative efforts To ensure project-initiated services can continue, relies on evidence that new forecasting tools are beneficial to decision-makers.Here, three main lessons are highlighted focussing on the three key challenge themes identified in the previous section.These lessons continue to guide S2S testbed activities and can inform other future collaborative efforts.
• Ensure sufficient resource to support co-production Co-production is a complex process with different types of interaction amongst different groups at different points in the process.It is not a simple solution to overcoming barriers to the uptake of forecast information (Lemos et al., 2018).Rather, it requires considerable investment of time and resources from all participants to develop an understanding of each other's contexts and knowledge systems to then jointly define and seek to address shared issues of concern.Recognising that coproduction is resource-intensive, the S2S forecasting testbed has highlighted key points and approaches for maximising the benefits of coproduction.Across the process, in-depth engagement of users (immersive co-production) has been found to be particularly vital in the coexploring needs and evaluation building blocks (B3 and B6), while their engagement can be less intensive (more consultative), but still effective in this context, during the co-development part of the process (B4).Specifically, later stages of the process suffer if sufficient resource is not invested in the early relationship-building stages.The iterative feedback process has been far more effective with groups who have been engaged from the outset of the co-production process and attended the kick-off workshop.This study has applied a particular co-production framework (Carter et al., 2019), however, there remains a wide variety of co-production approaches (e.g., Bremer and Meisch, 2017;Vincent et al., 2018) and the efficacy and resource demands of each still requires more thorough assessment and metrics for measuring their effectiveness developed.

• Support capacity building in all groups
The root of many challenges within the S2S testbed co-production process is effective communication.The responsibility for addressing communication challenges does not lie with a single set of actors, but is shared across all groups.From experience within the S2S testbed, focused capacity building activities substantially improve communication.For example, there is a need to invest sufficient resources into strengthening decision-makers' understanding of key forecast concepts and enhancing forecast producers' and researchers' capability to effectively communicate these technical concepts.Such capacity building activities can contribute to creating the common ground required to enable effective co-production to take place (B2, B3).Some studies suggest the inclusion or creation of boundary organisations, with specific expertise, could provide a resource-efficient solution to doing this (Lemos et al., 2014).Other initiatives have specifically sought to strengthen the capacities required to support effective engagement in meteorological agencies and research institutions themselves, rather than relying on external organisations (Visman and Tazen, 2019).The co-production approach adopted within the S2S testbed has, by combining expertise and resources from NMHSs and in-country research institutions (Table 2), gone some way to strengthening engagement capacities amongst partners.

• Include decision-makers in forecast evaluation
In order to enable useful monitoring that can support the ongoing process of learning and product-development, it is critical that combined evaluation systems are developed.That is systems which combine meteorological forecast skill evaluation with a consideration of how the forecasts have benefitted, and could better support, key weathersensitive decision-making processes.The former in isolation only reveals forecast reliability, whilst the latter in isolation only indicates how useful the forecast product can be.However, it is clear that to develop appropriate action-based S2S forecast products and tools evaluating both reliability and usefulness are essential (White et al., 2017).Developing a combined evaluation system is not an easy process and itself necessitates co-production: requiring different knowledge sources and practices from all groups working together to develop shared understanding about different and complementary evaluation processes and jointly agree metrics or indicators of 'success' that can track changes in areas of respective primary interest (Visman et al., 2018).
The GCRF African SWIFT S2S testbed provides an exciting real-time, operational context to explore how co-production can improve the appropriate use of S2S forecast information in decision-making.While information on these timescales has huge potential to support early warning systems, and strengthen resilience to weather-related risks, it will only be able to do so if the forecast information is reliable and relevant to users' specific decision-making processes.Ultimately, the success of the S2S testbed will not be defined by advances in scientific understanding, but by the resulting socio-economic benefits to which knowledge-exchange and learning that occurs between individuals and institutions as part of the co-production process can contribute.Demonstrating the tangible benefits resulting from co-produced climate services will support establishing sustainable pathways for the coproduction of weather services on sub-seasonal to seasonal timescales in Africa will ensure that these efforts live beyond the lifespan of this project and can shape future operational procedures and further collaborative efforts.Engaging with users of climate services offers ways of collating the data to demonstrate these tangible benefits.

Fig. 1 .
Fig. 1.Schematic showing the sources of predictability and decision-making applications across timescales from weather forecasting (blue) to climate projections (orange).Subseasonal to seasonal (S2S) timescale shown in grey.(For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 3 .
Fig. 2. CAPC-AC sub-seasonal early warning information based on precipitation forecast for week commencing 26th March 2020, issued on 19th March 2020 (7-day lead time).Weekly precipitation forecasts are given in mm/day.Colours indicate the level of risk based on the risk matrix of Impact (rainfall thresholds) versus probability (expert judgement).Numbers from 1 to 10 indicate the probability associated with each threshold.Red indicates very high risk of well below normal precipitation (e.g., loss of life and animals, food and water shortages); orange indicates high risk of below to well below normal precipitation (e.g., increase risk of disease, air pollution and locusts); yellow indicates low risk of below normal precipitation (e.g., reduced dam water levels, increased tensions in the population); grey indicates normal precipitation; green indicates above normal precipitation (e.g., economic losses, migration of people and animals); and dark green indicates a very high risk of well above normal precipitation (e.g., flooding & landslides, loss of life and animals, increase risk of disease).(For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 5 .
Fig. 5.The four major iterations in the development process for Nairobi weekly rainfall: from the initial user-guided KMD concept (a); the first production of a weekly forecast bar chart (b); the amalgamation of the forecast, hindcast and observation data (c); and the latest iteration which includes the bias corrected weekly forecast and observation data (d).Iterations (b), (c) and (d) are all for the Nairobi 1.5 • grid box (centred at 36.5 • E 1.5 • S), with weekly ECMWF sub-seasonal forecasts initialised 30th March 2020, with a 5 day lead, valid from Saturday 4th April to Friday 1st May 2020.

Table 1
Table of specifications for the ECMWF S2S real-time data.

Table 3
Summary of initial testbed, and kick-off workshop, activities under each coproduction building block.the project-initiated services developed.The new forecasting tools and products can only continue to be operational after the S2S testbed if; the real-time S2S data continues to be available; new knowledge is tied to institutions rather than dependent on individuals; and efforts are taken by operational forecasting agencies to systematically produce the new products.Meeting the first requirement is reliant on the S2S testbed participants evidencing the impact of new forecasting tools in users' decision-making, whilst solutions to the latter two are arguably rooted in capacity building and training of all groups. continue