Advancing climate services for the European renewable energy sector through capacity building and user engagement

The development of successful climate services faces a number of challenges, including the identification of the target audience and their needs and requirements

sensitivity to climate drivers. Having clearly identified the main subject focus, six categories of users and potential users were specified: (i) transmission and distribution system operators, (ii) energy generation companies, (iii) national and regional departments/agencies, (iv) international associations and coordination activities, (v) policy and operational research organisations, and (vi) commercial climate service providers for the energy sector. The latter category turned out to be a more important group than anticipated at the start of ECEM, illustrating that not all users are just 'end users' and that commercial activities are an essential component of the market adding value to the climate services chain. Having a clearly defined target audience made it easier to manage expectations and to delimit the scope of the service. This was particularly important within the constraints of the proof-ofconcept contract, which aimed to develop a pre-operational, rather than a fully operational, service.
The final evaluation and feedback on the ECEM products and outcomes was generally positive, indicating that ECEM has helped to close the usability gap and delivered data and information that is credible, legitimate and salient. The ECEM team attributes this success to having had a good idea of the main target and structuring all user engagement activities to address this target. From the outset, the team of six institutions (including universities, a national weather service and an energy company) had a clear vision of what they wanted to achieve and a plan of the main direction of travel. This was based on their understanding of the potential new opportunities which climate services offer the energy industry and previous experience in producing data sets likely to be relevant for this community. The team did not, however, attempt to impose their vision on external stakeholders but instead devised an interactive and evolving process of stakeholder engagement. This meant that the team was flexible and dynamic and able to respond and adapt to users' expectations and use cases. Whether or not the team fully achieved its stated ambition of 'co-production' can be debatedas can the meaning of such terms in practice. Nonetheless, it is concluded that having a good process of stakeholder engagement led to both good outputs and good outcomes.
The proof-of-concept climate service developed by ECEM encompasses three inter-related elements. The first and most obvious 'product' is the C3S ECEM Demonstrator (http://ecem. wemcouncil.org/). This interface allows users to visualise, explore and download climate and energy data sets on three different timescales (historical, seasonal forecasting and climate projections) aggregated to the country or, in some cases, subcountry, level. The second element is high-quality climate and energy data which can be used to inform an end-user problem with or without further tailoring by, for example, a commercial service or consultancy, or by in-house experts in an energy company. Data consistency and quality were highlighted as essential requirements of a climate service in the stakeholder evaluation and feedback. These data are embedded within the Demonstrator and can be downloaded in text format which was identified as the 'first choice' for much of the target audience. However, ECEM also identified a group of more technical users who are primarily interested in the data alone. For this group, a facility was provided to directly download by FTP large volumes of underlying gridded data in NetCDF format. The third element is 'know-how' about how to use climate information as actionable information. This was underpinned by a range of documentation and guidance embedded within the Demonstrator, including Key Messages and Event Case Studies, as well as a programme of webinars and workshops/symposia.
All three elements (the Demonstrator, data and 'know how') were assessed as important by ECEM stakeholders, although different individuals or organisations naturally had different priorities in terms of what was most important for them. Similarly, some Demonstrator users were more interested in climate than energy data, or in say the historic period rather than seasonal forecasting or climate projections. Thus, the Demonstrator was designed with multiple entry points -also recognising that decision making is not a linear process. At the same time, one of the greatest strengths of the ECEM team was its capacity to produce consistent and high-quality data for all three timescales, for both energy and climate, and then to integrate all into a single interface to enable seamless and transparent decision making across a business. The Demonstrator can therefore also be viewed as a 'shop window', which may attract a user in search of a particular product -but who may then explore other products on offer.
A key element in ECEM has been the compilation of a robust energy database, consisting of energy demand as well as generation from wind, solar and hydro power, together with information on installed generation capacity, for all EU countries. Such a database has recently been made available by the European Network of Transmission System Operators for Electricity (ENTSO-E) under an EU regulation, but these data present some issues and inconsistencies, which made their use difficult in ECEM. The point here is not to criticize the ENTSO-E data, rather to emphasize that significant progress needs to be made on energy data completeness and homogenization in order to develop operational, fit for purpose climate services for the energy sector.
In practice, it was not so easy to treat seasonal forecasting information, which requires displaying more frequently updated data, in the same way as the historical or projection information. Issues relating to the generally limited and specific skill over Europe, probabilistic formats and user expectations and use for operational decision making also emerged. During the final symposium, for example, many users were most interested in seasonal forecasts because they had less existing awareness and understanding of these elements and saw the Demonstrator as vital to understand what they are and how to potentially use them. Thus the expectations of these users were rather different to those for whom 'Data is King'.
Through a carefully planned process of iterative engagement with stakeholders, ECEM has raised the capacity of both providers and users to deliver and use climate services for the European renewable energy sector. The emerging community of practice is now focused on spin-off activities encompassing development of an operational service (C3S Energy -https://climate.copernicus. eu/operational-service-energy-sector) as well as commercial products and services, and research, for example on the improved tailoring and assessment of the added value of seasonal forecasts.

Introduction
The development of a market for climate services in Europe is the motivation for substantial investments in regional, national and private sector initiatives such as the European Union's European Roadmap for Climate Services (E.C., 2015). The context for these initiatives is the recognition of the need to better understand both the supply (i.e. provider or "push") and demand (i.e. user or "pull") perspectives together with an aspiration towards co-design, co-production and co-evaluation (Schuck-Zöller et al., 2017;Street, 2016;Vaughan and Dessai, 2014). A strong motivation for these investments is to mitigate threats, for example potential increases in energy costs due to the higher balancing charges due to less predictability in the system. At the European level, the Copernicus Climate Change Service (C3S -https://climate. copernicus.eu/) was launched in 2015 to lead and co-ordinate development of climate service infrastructure and underlying data provision. Amongst the first C3S activities to be funded were a number of Sectoral Information System (SIS) projects with the aim of producing proof-ofconcept climate services. Two of the first SIS contracts targeted the energy sector: CLIM4ENERGY (http://clim4energy.climate.copernicus. eu) and, the focus of this paper, the European Climatic Energy Mixes (ECEM -https://climate.copernicus.eu/european-climate-energymixes) contract which ran from November 2015 to March 2018 .
The energy sector is sensitive and potentially vulnerable to weather and climate, both in terms of climate variability and climate change (Ebinger and Vergara, 2011;Johnston et al., 2012;Schaeffer et al., 2012;Troccoli, 2018). This is particularly the case for renewable energy (namely, wind, solar, hydropower) generation and electricity demand. There is growing recognition, for example, that five-years of wind power output where new farms are being installed is not sufficient to understand extreme events, and thus longer historical series are needed for both climate and generation (Cannon et al., 2015). Shifting energy policy towards decarbonisation is catalysing the growth of renewables across Europe. Hence this weather/climate sensitivity is likely to increase with the increasing contribution of renewable energy to electricity generation, and increasing electricity demand, for air conditioning for example .
ECEM was developed in response to these factors, which are affecting the need for climate services, as reflected in the emerging demand from the energy sector (Bruno Soares et al., 2018). The importance of the energy sector in terms of climate services is also reflected by its selection as one of the 'exemplar' sectors for the Global Framework for Climate Services (WMO, 2017). In the case of seasonal forecast information, the earlier EU-funded EUPORIAS project (Buontempo and Hewitt, 2018), concluded that although uptake of this information is generally fairly limited across Europe, the energy sector does include some so-called 'early adopters', including Electricité de France (EDF) (Soares and Dessai, 2015). In general, however, the wider use of such climate information by the energy sector was limited at the start of ECEM.
ECEM also started against a backdrop of growing evidence of the potential to produce skilful seasonal forecasts for the European energy industry and other sectors (Clark et al., 2017;Doblas-Reyes et al., 2013;Palin et al., 2016), as well as studies indicating the potential impacts of anthropogenic climate change on European energy demand (Damm et al., 2017;Wenz et al., 2017) and solar , wind  and hydropower (van Vliet et al., 2016) generation. Like its sister SIS contract CLIM4ENERGY, ECEM was innovative in aiming to bring together information on both these timescales, i.e. seasonal forecasting -a season ahead, and climate projections -out to the middle or end of the century, together with historical information for the last 40-50 years.
Rather than attempting to address the whole energy sector, ECEM deliberately focused on the particularly climate/weather sensitive issues of electricity demand and renewable generation and two related challenges which were highlighted in the original proposal: • To describe the ways in which electricity supply and demand over Europe are affected by the spatial and temporal variations of their climate drivers • To produce scenarios that demonstrate how different energy supply mixes can meet demand at the European scale, particularly given the projected high level of highly climate-sensitive renewable energies.
Implicit in these issues and challenges is the objective to provide consistent climate and energy data which is readily applicable to the decision-making context. These focal points also helped to constrain the target audience and potential users of the ECEM proof-of-concept climate service and thus, right from the start of the contract, determined the specific organisations and individuals with which the project team sought to engage in order to achieve its ambition of co-production and co-design. Thus organisations such as investment banks involved in the risk analysis and market pricing of electricity were not explicitly targeted, although they too have a need for climate information.
The target audience of ECEM was rather different from that of CLIM4ENERGY which focused on a number of application case studies each involving specific end users. Thus the process of stakeholder engagement was also rather different in the two projects. The CLIM4ENERGY approach provided scope for more in-depth and specific tailoring for each case study, but could be considered less extendable towards a public operational climate service (see Section 5).
The development of the new and improved climate and energy data sets powering the ECEM proof-of-concept is described elsewhere (Bett et al., 2018a(Bett et al., , 2018bDe Felice et al., 2018;Dubus et al., 2019;Jones et al., 2017;Saint-Drenan et al., 2018;Troccoli et al., 2018). Here the focus is on the process of engagement with users and stakeholders, particularly in the context of development of the C3S ECEM Demonstrator. The Demonstrator (http://ecem.wemcouncil.org/) comprises an online interactive tool to display, explore and download climate and energy data for three timescales (historical, seasonal forecasting and climate projections) along with supporting user documentation and guidance. Throughout this paper the term 'stakeholders' is used to refer to both the ECEM project team and external stakeholders i.e. external users and potential users of the Demonstrator and other ECEM outputs (see Table 1).
According to the original proposal, ECEM aspired to 'produce a relevant and useful user-driven and scientifically robust end-to-end proofof-concept climate service for the energy sector' and thus sought 'a close engagement of knowledge co-production with users to assess their priorities and requirements … with reiteration and feedback as the project develops'. While the extent to which these ambitions were being achieved was reviewed on an ongoing basis throughout the project, this was not done within a formal assessment framework. Furthermore, the ECEM consortium did not specifically discuss what was understood by the terms 'co-production' or 'co-design'. For the purpose of this paper, co-production is considered as a sustained collaborative process between scientists and decision makers for the production of useful, actionable and socially robust knowledge (Beier et al., 2017;Meadow et al., 2015;Taylor et al., 2017). Meadow et al (2015), for example, define co-production of knowledge as 'the process of producing usable, or actionable, science through collaboration between scientists and those who use science to make policy and management decisions'. As well as outlining the emergence of the concept of co-production over the last decade or so they highlight the potential benefits that come from framing and answering research questions together, including a greater sense of ownership over the final product.
This paper thus represents a retrospective reflection on ECEM structured around a somewhat clearer understanding of 'co-production' and some of the climate service challenges and research issues identified in the recent literature. These research issues include three priorities with respect to the Global Framework for Climate Services identified by Vaughan et al. (2016): 1) understanding users' needs, contexts and capacities; 2) improving communication; and, 3) tailoring of information. Addressing these research priorities should help to improve the credibility, legitimacy and salience of the information and products underpinning climate services (McNie, 2013;Vaughan and Dessai, 2014) and thus help to close or at least narrow the so-called usability gap (Lemos et al., 2012). In this context, it is helpful to distinguish between evaluation of the actual process of co-production and evaluation of the outputs and outcomes (Schuck-Zöller et al., 2017). Other challenges include evaluating the quality and effectiveness of stakeholder engagement (Gardiner et al., 2018;Hewitt et al., 2017;Schuck-Zöller et al., 2017;Swart et al., 2017) and developing and sustaining communities of practice (Street, 2016;Vincent et al., 2018). In relation to development of the climate services market, the European Roadmap for Climate Services identifies three challenges: enabling market growth; building the market framework; and enhancing the quality and relevance of climate services (E.C., 2015).
Using these issues and challenges as a structural framework, the self-assessment and reflective process described in this paper addresses the following questions: • How successful was ECEM in its aim of co-design/co-production? • Did the process of stakeholder engagement lead to higher quality outputs/products and outcomes than might be expected in the absence of such engagement?
• To what extent has ECEM contributed to capacity building and market development within the European energy sector climate services community?
The paper is structured as follows. Section 2 on Methods describes how the consortium was built and developed to fit the scope and focus of ECEM and the timeline of iterative engagement and Demonstrator development as well as the engagement, feedback and evaluation mechanisms used. Section 3 outlines the main outputs of ECEM focusing on the Demonstrator and identification of what is distinctive about it and the embedded data, as well as its evolution given the design process. Section 4 discusses the engagement processes and outcomes focusing on the three questions above. The concluding Section 5 highlights the distance travelled from the start to the end of ECEM and outlines how the work and expertise is being taken forward.

Methods
The ECEM project team (listed in Table 1a) included research organisations and individuals who already had some expertise in processing and assessing climate and energy variables on the three target timescales of historical, seasonal forecasting and projections. The majority of team members already had some experience of working with the renewable energy sector and varying levels of understanding of the types of data and climate information required by the target audience for decision making (see Section 1). The team also included an organisation, the Institute for Environmental Analytics at the University of Reading, with the expertise in visualisation and online portal development required for the Demonstrator. The team members most closely involved in designing and implementing the stakeholder engagement activities were able to draw on the experience of, amongst others, the earlier EUPORIAS project Dessai, 2015, 2016) and involvement in climate service network activities such as the Climate Services Partnership and its series of International Conferences on Climate Services. The team included organisations involved in both public and commercial operational weather services (e.g. the UK Meteorological Office, MO) and three founding Directors of the World Energy and Meteorology Council including its Managing Director (who was also the ECEM project co-ordinator).
While the diversity of users and the need to tailor climate services for specific sectors is widely recognised (Bruno Soares et al., 2018), and motivated the C3S SIS initiative, it is also the case that decision making needs are diverse within a particular sector. The nature of the relevant decision making also differs depending on the timescale of the information provided (see, for example Figure 7 of Bruno Soares et al. (2018)). Seasonal forecasting timescales are relevant for operational, trading and maintenance activities, for example, while climate projection timescales are relevant for the wider vision and strategy of an organisation, such as making decisions around infrastructure investment. The focus of ECEM on these two timescales, along with historical (the last few decades), was dictated by C3S. Thus weather forecasting and subseasonal forecasting, although relevant for operations, were explicitly excluded due to the C3S remit. Initialised decadal predictions were not considered as these are not yet available, for example through C3S. The omission of some timescales may, however, appear rather arbitrary for those organisations who employ a number of different timescales in different sections of the business (Bruno Soares et al., 2018).
The scope and focus of ECEM was thus constrained by the wider context of the C3S SIS programme as well as by the resources and time available for development of a proof-of-concept climate service. In view of these constraints, a decision was taken at the proposal stage to focus on the country scale rather than attempting to go straight to higher spatial resolution. Flexible methodologies for data and Demonstrator development were however devised which would make it relatively easy to move to higher spatial (and temporal) resolution in the future, should the necessary energy data be available. Some sub-country information, for example, was eventually produced by ECEM -see Section 3. The lack of a European-wide, homogeneous and sufficiently long energy dataset was indeed a major challenge in ECEM. These issues relating to scope and focus were important in terms of managing user expectations throughout the project and in the identification of appropriate target users.
Considering all the above issues, one of the early tasks of ECEM was to use the strong existing networks and expertise of team members accrued over many years to identify the target categories of external stakeholders (i.e., potential users) and relevant organisations in each of these categories. Six such categories were identified as listed in Table 1b along with examples of organisations in each category who actively participated over the course of ECEM (e.g., as workshop and webinar attendees, and Advisory Board members). The six categories include energy generation companies such as EDF. EDF R&D was also part of the project team demonstrating that the same organisation can act as both a provider and user of climate services (Table 1). As noted in the Introduction, EDF are considered as one of the 'early adopters' of climate information in the energy sector and the EDF R&D team plays an important role in providing climate and climate-based energy information (services) to other parts of this large organisation. The EDF R &D team itself can also be considered as a user of climate data provided by other members of the ECEM team. Fig. 1 illustrates the relationship between the project partners and the external stakeholders (Table 1) and the development of these stakeholder relationships over the course of the 29-month project. The ECEM level of achievement is discussed in Sections 3 and 4, while the advances in terms of Technology Readiness Levels (TRL -left-hand arrows in Fig. 1) (Heder, 2017) are discussed in Section 4 together with the progression from a disconnected community of potential users towards a stronger and more coherent community of users and target users (right-hand arrows). Here, the importance of new users and champions who became more actively involved with the project team over the course of the project is noted. A specific example of how active participation in ECEM is leading to further downstream development and commercial application of some of the ECEM outputs is presented in Box 1. Fig. 2 shows the intertwined timeline of Demonstrator development and stakeholder engagement highlighting the three ECEM stakeholder workshops and two C3S Symposia on Climate Services for the Energy Sector that were jointly organised by ECEM and CLIM4ENERGY. These workshops and symposia were the key engagement events used by ECEM but were complemented by a range of other mechanisms designed particularly to maintain contact and momentum in between these face-to-face events ( Table 2). how they could interact with the system. The carousel (i.e. rotating break-out group) sessions during the second workshop focused on the Demonstrator wireframe and climate and energy data requirements but were complemented by a participatory exercise designed to better understand the different types of users and their potential decision-making applications, as well as to inform preparatory work on the ECEM event case studies. This exercise addressed the following question: "What aspects of climate do you see as posing the greatest risk to maintaining secure supply-demand balance in the European power sector, both now and into the coming decades?" By the time of the first Symposium in February 2017, corresponding to the third stakeholder workshop, the Demonstrator Wireframe was at an appropriate stage of development for hands-on evaluation and testing by stakeholders. The Symposium therefore included a two-hour hands-on session which started with a brief walk through the Demonstrator menu structure and functionalities. Participants were then free to explore the Demonstrator for themselves with the ECEM team on hand to answer questions about the Demonstrator itself and the data currently incorporated. The temporary link to the Demonstrator provided to participants remained live for three weeks after the symposium to support and encourage further testing and feedback.
The Demonstrator continued to be developed in an iterative approach with frequent input from stakeholders and prospective users. As well as the stakeholder workshops, symposia and webinars, one feedback mechanism that became particularly useful during the final stages of development was a feedback form link on the Demonstrator itself (Table 2). By the time of the fourth stakeholder workshop in June 2017 the Demonstrator was sufficiently advanced to support both 'Exploring the Demonstrator' and 'What can the Demonstrator do for you?' sessions. In the latter, external stakeholders worked with ECEM project team members to develop their own tailored data assessments or mini case studies, many of them focusing on specific extreme events identified by the external stakeholders and how their frequency might change in the future.
At the end of October 2017, a close to final version of the Demonstrator (the Proof-of-Concept) was made available for the subsequent five-month period of pre-operational evaluation and refinement. The engagement and feedback mechanisms listed in Table 2 continued to be used during this period -including a second and final Symposium in March 2018. In addition, the evaluation mechanisms listed at the end of Table 2 were designed and implemented during this period. Table 3 summarises the climate and energy data incorporated in the ECEM Demonstrator, along with its functionality and embedded documentation and guidance. The climate and energy variables include those proposed in the first vision of the Demonstrator (see Section 2 and Fig. 2) but the need for these was repeatedly endorsed by the external stakeholders and by the EU-funded ConnectingGEO project (http:// www.connectingeo.net/). Sea level pressure was added primarily at the request of ECEM team members in order to facilitate exploration of modes of climate variability such as the North Atlantic Oscillation which in turn influence both spatial (e.g., from northern to southern Europe) and temporal (e.g., year-to-year) variability in the energy variables (Brayshaw et al., 2011;Clark et al., 2017;Thornton et al., 2017). Snow depth was added at the request of stakeholders including team members who required it for the calculation of hydropower generation. Due to concerns about data quality, however, this variable was only provided for historical timescales and not for projections (and was not available in the case of seasonal forecasting).

Results: outputs and products focusing on the C3S ECEM demonstrator
Although it was originally proposed to restrict the spatial scale to the country level (see Section 2) and the team sought to manage users' expectations in this respect, the desirability of having higher spatial resolution information was acknowledged and in most cases it was also possible to provide information at the level of "clusters". These are 96 regions identified by the eHighway 2050 project based on NUTS3 regions (see https://www.entsoe.eu/outlooks/ehighways-2050/).
In terms of functionality, the Demonstrator was designed to provide an appropriate balance between online visualisation and exploration, and data download, recognising that getting this balance right is an important and complex issue for climate service development. During the first ECEM stakeholder workshop (Fig. 2) participants generally gave higher priority to data access than to visualisation. This was summarised in a key workshop message as 'Data is King'. It was therefore decided to incorporate an 'advanced' data download facility within the Demonstrator. However, this prioritisation tended to reflect the generally greater technical expertise of this workshop's participants who included a number of consultants and intermediary organisations. Indeed, the preferred format of the majority of end users is for data to be downloaded directly from maps and time-series displayed in the Demonstrator in text format (CSV files), but a link is also provided to an FTP site where 'advanced' users can download data at 0.5 degree latitude/longitude resolution in NetCDF format. These gridded data were aggregated to provide the country/cluster values provided in the Demonstrator itself.
In subsequent workshops and engagement activities, stronger requirements for flexible and easy-to-use visualisation capabilities were expressed, reflecting the broader range of potential users involved in these later events. Thus in the final stages of Demonstrator development (Fig. 2) more effort was channelled into the visualisation functionality including, for example, improved user control of map colour schemes. It was also decided to further support users by providing some pre-prepared graphs for seasonal forecasting skill scores and climate projections along with short guidance documents -see below. The pre-prepared graphs were constructed to deliver faster plotting as the plots include input from multiple data files and to illustrate the importance of adequately considering projection uncertainty. Hence these projection plots automatically include the multi-model mean, an ensemble range and individual model outputs for two different emission scenarios (Representative Concentration Pathways, RCP, 4.5 and 8.5 -(van Vuuren et al., 2011)) as well as observed data (Fig. 3). At the same time, users retain the flexibility to construct their own projection plots.
Particular consideration, especially in the later stages of Demonstrator development, was given to producing a comprehensive suite of documentation and guidance targeted at a range of users from general users and those new to climate and/or energy data through to The three user-motivated Event Case Studies illustrate the type of more detailed analyses that can be undertaken using the ECEM Demonstrator and the underpinning data, focusing on extreme weather events that have had a major impact on the European energy sector. The first Event Case Study focuses on conditions during winter 2009/10 which saw high power demand due to extremely cold temperatures across much of northern Europe, while the second considers how the frequency of such events might change over the coming decades due to climate change. The third Event Case Study considers the impact of summer heat waves on electricity supply and demand in Spain. Each of the Event Case Studies follows the same structure. The first page provides a summary in terms of: Boosting Decision Making; Scientific/ Technical Advances; and, Key Lessons. The subsequent pages take a narrative approach from the initial starting point or event through the analysis and lessons learned, also illustrating 'How can we use the demonstrator to learn more?'.
The stakeholder engagement highlighted the different types of energy sector user and the different types of decision that a user may be focusing on at any particular point in time. Thus the Demonstrator supports a number of different entry points to the data and information provided (Fig. 4). A user can choose to start exploring energy data rather than climate data, or the seasonal forecasting rather than historical or projection timescale. They can focus on the map view, or on timeseries plots for individual countries/clusters. A policy maker might go straight to the Key Messages or Event Case Studies in the help and information section, whereas a consultant might go straight to the ftp data link. These multiple entry points provide flexibility and recognise that decision making is not a linear process, hence a more modular approach is appropriate (Haße and Kind, 2018;Laudien et al., 2018).

Documentation/guidance Description
Using the Demonstrator Getting started, how to navigate, outline of functionality etc.

Methods and Assumptions
Overviews of products and introduction to different types of data for the non-expert. Cross-cutting issues include guidance on the use and interpretation of the data and some of the underlying assumptions.

Key Messages
A series of six Key Messages for the European energy sector based on the analysis of data in the ECEM Demonstrator. Non-technical -two pages.

Pre-prepared graphs
Examples and guidance on the more complex graphs provided, including how to interpret uncertainty ranges for climate and energy projections and the seasonal forecasting historical skill scores. Non-technical -two pages.

Variable Fact Sheets
A series of factsheets which provide metadata for the climate (seven) and energy (four) variables. Technical documents -five to eight pages.

Event Case Studies
Three case studies based on actual extreme events which illustrate how the Demonstrator can be used by the energy sector to enhance understanding and support decision making. Non-technical -four pages.

FAQs
Covering ECEM and the data sets -based on questions raised at workshops and webinars. Glossary Climate and Energy terms. About Underlying software, version number, changes and updates, known issues. Tutorials Text-based map and graph tutorials available at start-up and from Help menu. Link to webinar videos and slides.
On the other hand, the number of choices available to users may be daunting. Thus good guidance, such as the 'Getting started' part of the Demonstrator help menu, is essential. The ability to download data in text or NetCDF format extends the power of the Demonstrator beyond a 'simple' visualisation tool. It gives the user flexibility to integrate the ECEM data sets with their own datasets and statistical/modelling tools. This is something else which emerged strongly from the ECEM workshops, webinars and Advisory Board discussions. Generation data, for example, is provided as capacity factors as well as in terms of power or energy, also depending on a prescribed future energy mix taken from the five eHighway2050 scenarios (http://www.e-highway2050.eu/). A user can, however, download the capacity factors and apply them to their own or any alternative energy mix scenarios.
From the early stages of engagement with potential users of the ECEM Demonstrator it was evident that access to high-quality climate and energy data was a priority. Thus it was important to assess the quality of the ECEM outputs, focusing on the Demonstrator and the Fig. 3. Example of a pre-prepared projections graph from the ECEM demonstrator. Summer energy demand in Greece shown as power (MW) with socio-economic trends removed (see http://ecem.wemcouncil.org/pdf/ECEM Preprepared Graphs Energy 20180220.pdf for details). credibility, legitimacy and salience of its embedded data. These issues are all important to build trust in the data so that decisions can be made in the business context. With respect to salience, the majority of user requirements in terms of climate and energy variables were largely met (see Table 3), with only a few exceptions (see Section 4). The main aspect that users would like to see improved relates to higher temporal and spatial resolution.
Consistency was also identified as a key user requirement in terms of data credibility and salience and was achieved, so far as possible, with respect to a number of aspects of the data. In particular, they are consistent in terms of spatial and temporal scale and coverage, format and underlying assumptions between countries/clusters, variables, energy and climate, time period and climate/energy scenarios.
The quality of the data in terms of credibility and legitimacy was maximised by using the most reliable input data available and through processing such as bias adjustment of the climate data used to calculate energy variables. This is documented in underlying journal papers Dubus et al., 2019;Jones et al., 2017;Saint-Drenan et al., 2018;Troccoli et al., 2018) and summarised in the Variable Fact Sheets. As well as evaluation and checking by the individual data providers, all ECEM team members were able to review and assess datasets in a 'test' version of the Demonstrator before being made public. A number of issues were thus identified, resolved and updated data sets provided. Where specific issues could not be resolved they were documented in the 'Known issues' section of the Demonstrator help menu.
A question on data quality was asked as part of the final short user survey (see Table 2). This question was deliberately left somewhat ambiguous and it was evident that some respondents interpreted quality as referring to the spatial and temporal resolution and the types of variables provided (salience) -whereas others highlighted the reliability and provenance of the data (credibility and legitimacy). A few respondents wanted data at the city or catchment scale, for examplesee the end of Section 4 for a brief discussion on 'unmet' needs. Nonetheless it is encouraging that two-thirds of the 75 respondents 'agreed' or 'strongly agreed' that the quality was appropriate for their needs. Rather than this ad hoc and post hoc approach to assessing data quality, ECEM could, for example, have developed a specific data quality matrix of energy industry requirements -which might be different from those of the scientific community.

Discussion on processes and outcomes
One of the stated objectives of the assessment reported here (see Section 1) is to consider how successful ECEM was in its aim of coproduction/co-design. Such aspirations have become increasingly common over the last few years in the context of applied climate research and climate services (Beier et al., 2017;Taylor et al., 2017;Willyard et al., 2018). In general, however, terms such as co-production are used without a specific definition or in-depth discussion of their practical meaning (Meadow et al., 2015) or how to evaluate success (Durose et al., 2018). This was the case for ECEM, which nonetheless devised an evolving and iterative process of stakeholder engagement from the early stages of Demonstrator development (Fig. 2). Concepts such as scientific and social robustness, which are considered to underlie co-production of knowledge (Taylor et al., 2017), were not explicitly used within ECEM.
Similarly, although ECEM strove for meaningful engagement with users and potential users, the level of engagement sought was never explicitly defined. If the three broad categories of engagement proposed by Hewitt et al. (2017) are considered, then ECEM certainly went well beyond passive engagement and information provision (category one). Interactive group and dialogue-based activities (category two) were widely used in the workshops (Table 2) and, to a large degree, the engagement was active and led to the development of focused and targeted relationships (category three) (Hewitt et al., 2017). The engagement was also sustained and synergistic, which is considered one of the defining characteristics of co-production (Laudien et al., 2018;Meadow et al., 2015). It is therefore concluded that ECEM moved a long way towards co-production within a relatively short period even though it takes time for people to appreciate and uptake climate information, and to grow a strong stakeholder base.
Another objective of this post-hoc assessment is to evaluate the 'quality of relationship' between the ECEM developers and users (Gardiner et al., 2018), and to consider whether the process of stakeholder engagement led to higher quality outputs and outcomes from ECEM than if a purely consultative approach (Meadow et al., 2015) had been followed. The issue of data quality (i.e. quality of outputs/products) is discussed in Section 3. Here, the issue of quality of outcomes (Schuck-Zöller et al., 2017) is considered, including evaluation of the Demonstrator functionality, usability and usefulness. The 76 respondents to the short user survey (Table 2) undertaken during the preoperational evaluation period (Fig. 2) were generally positive in this respect, i.e. they 'agreed' or 'strongly agreed' with the survey statements. Over 80% of respondents were positive that the Demonstrator is 'easy to use' and 73% were positive that the documentation and guidance provided 'will help you to make best use of the C3S ECEM Demonstrator'. It was particularly encouraging that 70% of respondents had already 'identified some specific features of the C3S ECEM Demonstrator and datasets which will facilitate my work'. Feedback from this survey and the other feedback and evaluation mechanisms used in ECEM (Table 2) support the conclusion that ECEM has successfully narrowed the usability gap (Lemos et al., 2012) by developing a proofof-concept Demonstrator that is user friendly in terms of both tool functionality and supporting documentation. It is unlikely that such positive feedback would have been obtained without iterative user engagement from the start.
The third objective here is to assess the extent to which ECEM has contributed to capacity building and market development within the European energy sector climate services community. An indication of the movement in these respects is given in Fig. 1. The process of collaboration between the ECEM partners and external stakeholders, and the relationship between this sustained engagement and co-production, are discussed above. Through this process, new users and champions, have emerged. It is estimated that the Demonstrator attracted an average of around 60 new users per month during the later stages of ECEM. Box 1 provides one example, that of Transvalor, to illustrate how a commercial product can be developed from a public climate service.

Box 1: Working with commercial providers -the example of Transvalor
Transvalor is a commercial energy service provider (Table 1b), based in the south of France. Transvalor and ARMINES (an ECEM partner) had had some previous contact and interaction due to a shared interest in solar radiation and their close geographical location. A representative from Transvalor attended the fourth stakeholder workshop and the ECEM-athon in 2017 (Table 2). For the ECEM team, it was very informative to have time for detailed discussion with a new user, particularly with respect to climate/ energy projections, including terminology and the complexity of information and uncertainty. It was agreed that it was more appropriate for Transvalor's customers to access ECEM outputs including the Demonstrator and data sets via Transvalor, after appropriate processing and repackaging, rather than directly from ECEM. A strategy by which Transvalor could gradually introduce customers to the concepts of climate projections was devised, based around the existing SoDa website (http://www.soda-pro. com/). Transvalor are now receiving more inquiries from customers concerning future conditions. Transvalor also reviewed and substantially improved the Key Message on solar radiation changes (see Section 3). During the ECEM-athon discussions, Transvalor identified a requirement for additional processing of the ECEM radiation data sets for input to in-house modelling tools, and this work was subsequently undertaken by a MINES ParisTech PhD student. This example (others include that of Naturgy and wind/hydro generation) illustrates how commercial products and services can be built upon a 'public good' free service such as C3S (see Section 5).
The goal of ECEM -consistent with the objectives of the C3S SIS (see Section 1) -was to develop a pre-operational tool (the Demonstrator) and to implement a proof-of-concept service, i.e., one bringing together climate and energy data in a useful and usable way for the energy sector . This goal was successfully achieved (Fig. 1). Inevitably the C3S ECEM Demonstrator lacks some of the characteristics of an operational service. In particular, real-time seasonal forecasts are not yet embodied -only information about historical performance of the forecasting systems. Although the team continues to be responsive to feedback (Table 2) there is not an active 'help desk', typical of an operational system, responding to inquiries within service level agreement time frames. Nonetheless ECEM has moved a considerable way towards an operational system which is reflected in a retrospective assessment of Technology Readiness Levels (TRL) as adopted by the EU for the Horizon 2020 work programme (Heder, 2017). The arrows down the left-hand side of Fig. 1 indicate the estimated TRL at the start (TRL 2-4) and end (TRL 5/6) of ECEM. The starting point of individual team members varied somewhat but in general all had already studied and tested the proposed methods and main data sources in different contexts (i.e. TRL (2) technology concept formulated, (3) experimental proof of concept, and (4) technology validated in lab). By the end, the technology had been validated (TRL 5) and demonstrated (TRL 6) in a relevant industrial environment.
In order to progress from TRL 2-4 to TRL 5-6 a number of technical challenges had to be met. These challenges included bias adjustment and validation of historical ERA-interim solar radiation and 10 m/ 100 m wind data (Jones et al., 2017) and calibration/validation of generation models particularly for wind power (Dubus et al., 2019), solar power  and hydro power . Sourcing appropriate data for the energy models was also challenging (Dubus et al., 2019). Availability of energy data was an issue for calculating generation and demand, together with issues about how to incorporate non-climatic influences such as day of the week and technological/social trends (Dubus et al., 2019).
As a proof-of-concept climate service, one of the challenges of ECEM was to link meteorology/climate expertise to energy expertise. At times, this introduced some tension between the kinds of data that are most useful to (a) the end user and (b) the researcher. The choice was made, for example, to account for evolving trends and detailed factors such as weekends and public holidays when modelling demand data. This makes the data more directly relevant and familiar to end users, but harder to work with from a research perspective even when the research itself seeks to inform end users.
Whereas the meteorological community has adopted a data sharing and common format approach since its infancy, the energy sector situation regarding data is very different and less coordinated. Little regulation and few standards exist, which make it extremely difficult to gather a harmonized dataset at the European level. EU regulation 543/ 2013 imposed a new paradigm, but is very recent, and much work is still needed to reach the necessary level, even though considerable progress has already been made. Good quality and accessible data are essential to both build and validate the energy models. The major challenges in energy data are: • The lack of standards (data formats, accuracy, quality, availability of metadata) • The relatively short duration of available time series (for instance, hourly energy generation from wind, solar and hydropower for all EU countries are only available since 2015) • Inconsistencies in datasets (for instance, the net generation capacity reported on the ENTSO-E Transparency Portal (https:// transparency.entsoe.eu/) is not coherent with the generation data for some countries for solar and hydropower) The compilation and preparation of the energy data used in ECEM required a very significant amount of time and effort. In order to improve this important aspect of energy climate services development, it is essential that the above challenges are addressed in a long-term approach. This certainly requires a concerted effort, involving EU and national regulation authorities, network operators, energy companies and climate services developers. On top of ENTSO-E's Transparency Portal after EU Regulation 543/2013, several initiatives have developed to progress on data issues. These include for instance the Open Power System Data Platform (https://open-power-system-data.org) and the World Energy and Meteorology Council Special Interest Group on Data Exchange, Access and Standards; but a higher level commitment is needed, as research groups do not have the legitimacy to impose rules on the whole sector.
Addressing these challenges has substantially increased the technical capacity of the scientific team. At the same time, ECEM has helped to build the capacity of external stakeholders through ongoing exposure to relevant terminology and data, and participation in ECEM workshops and webinars. The range of Demonstrator documentation and guidance (Table 3) was also designed to build the capacity of users, whether new to climate data or more experienced.
As well as building technical capacity, the ECEM activities have led to improved individual and collective understanding of the needs of different users within the renewable energy sector including their decision-making contexts and capacities (Vaughan et al., 2016). Thus the growing technical capacity has facilitated the production of credible and legitimate information, while bringing together researchers and potential users from the start of ECEM has helped to build trust and to ensure that the tailored information produced is also salient (Vaughan et al., 2016;Vaughan and Dessai, 2014). With regular communication including face-to-face during workshops/symposia/tailored meetings, open discussion and willingness to adapt the Demonstrator and other outputs to reflect emerging user needs, it can be argued that ECEM has contributed to the emergence of a community of practice (Vincent et al., 2018). This community of practice is well placed to contribute to the ongoing development of the climate services market (see Section 5). It is inter-disciplinary in that it includes experts in climate, energy, observations, seasonal forecasts and climate projections as well as communications and end users. It could, however, be extended to include, in particular, more expertise in social and decision-making science.
Whilst concluding that ECEM was effective in stakeholder engagement, it is nonetheless acknowledged that more energy industry stakeholders could have been involved in testing the Demonstrator during the pre-operational phase (Fig. 2) and that climate services are still not widely accepted or used in the renewable energy sector. It is also possible that the generally very positive responses to the user survey discussed above reflect, in part, disengagement of any dissatisfied users. Deeper engagement and capacity building could have been achieved through mechanisms such as secondments or internships, either within the consortium itself, or of team members to external organisations and vice versa.
The relatively short timeframe for the development and launch of the pre-operational service (Fig. 2) restricted the amount of time available for quality control of data and development of supporting documentation, particularly case studies, key messages and journal papers, before data were made public. Not all user needs expressed could be met within the constraints of the proof-of-concept, and indeed not all user requests are necessarily feasible at all given the current state of climate science and the availability and reliability of both observed and simulated data. In such cases, the reasons were always explained to users and these needs were documented for the future (see Section 5). These include requests for higher temporal and spatial resolution, additional variables such as offshore wind and streamflow, and additional Demonstrator functionality such as trend and variability analysis.

Concluding remarks
Through a carefully planned process of iterative engagement with stakeholders, ECEM successfully developed and launched a pre-operational climate service for the European renewable energy sector, i.e., the C3S ECEM Demonstrator (http://ecem.wemcouncil.org/). This is an interactive visual on-line tool which allows users to view and explore energy supply and demand profiles, and climate variables, for each European country and subnational clusters, in map and time series format ) (see also Table 3). ECEM partners and external stakeholders from six targeted categories within this sector (Table 1) worked together (Fig. 2) to develop and evaluate ( Table 2) the Demonstrator and to build capacity on both sides. Alongside the tangible outcome of the Demonstrator and its embedded data, thus raising the TRL, the ECEM activities have supported the emergence of a community of practice (Fig. 1).
Members of this community of practice are now recombining in a number of different groupings to work on new initiatives, building on the ECEM and CLIM4ENERGY expertise. These include Horizon2020 research and innovation projects such as SECLI-FIRM (http://www. secli-firm.eu/) and S2S4E (https://s2s4e.eu/) which are assessing the value of seasonal forecasts for the energy sector, MEDSCOPE (https:// www.medscope-project.eu/) which is focused on the Mediterranean, and Plan4Res (https://www.plan4res.eu/) which is using the ECEM historical and projections data to assess the impacts of climate change on EU energy systems. Having developed the proof-of-concept, selected partners from ECEM and CLIM4ENERGY have since come together in C3S Energy to work towards delivery of an operational service (https:// climate.copernicus.eu/operational-service-energy-sector), which will be fully and transparently integrated with the Copernicus Climate Data Store (https://cds.climate.copernicus.eu). Reflecting this next step towards operationalisation, C3S Energy is co-ordinated through the World Energy and Meteorology Council (see Section 2) rather than an academic institution. It is expected that C3S Energy will raise the TRL to 7 (system prototype demonstration in an operational environment) or 8 (system complete and qualified) (Fig. 1). The C3S-funded operational service will provide a free and public service, i.e. it will be for the 'public good'. It will still leave considerable scope for service companies from both the private (e.g. consultancies) and public (e.g. National Meteorological and Hydrological Services) sectors to use C3S Energy outputs for the development of commercial products and services. Such commercialisation would raise the TRL to 9 (actual system proven in operational environment).
In terms of the challenges underpinning the European Roadmap for Climate Services (E.C., 2015;Street, 2016) it is easier to argue that ECEM has enhanced the quality and relevance of climate services for the energy sector than to assess how it may have enabled market growth or helped build the market framework. Research on the market potential for climate services, such as undertaken in the EU Horizon 2020 EU-MACS (http://eu-macs.eu/) and MARCO (http://marco-h2020.eu/) projects, is still in its infancy (Hoa et al., 2018). When the ECEM proposal was written and submitted, however, it was the case that there were few potential users actively or openly expressing a need for such a service. During its course, ECEM has gradually engaged more and more of the energy community including potential service providers (see Table 1b), making them aware of climate services and information, and thus driving demand for further services.

Funding
This work on the European Climatic Energy Mixes (ECEM) contract was supported by the Copernicus Climate Change Service (C3S), a programme being implemented by the European Centre for Medium-Range Weather Forecasts (ECMWF) on behalf of the European Commission (grant number: 2015/C3S_441_Lot2_UEA).