Introduction to the Evaluation

Finland is committed to implementation of the United Nations (UN) 2030 Agenda for Sustainable Development and its goals, with a national policy to evaluate the implementation of Agenda 2030 once during every four-year electoral term. Finland was the first country in the world to conduct an impartial and independent evaluation of the national implementation of Agenda 2030. The evaluation, called PATH2030, was funded as part the government’s 2018 analysis, assessment, and research activities (www.tietokayttoon.fi/en).

In its implementation plan for the 2030 Agenda, the Finnish government was committed to a comprehensive evaluation of the national implementation efforts of the agenda. The purpose of the evaluation was to support evidence-based decision making and knowledge management. This is important because efficient implementation of the universal 2030 Agenda requires an entirely new kind of comprehensive approach and coherence in the policies of the administrative branches. By producing information on the nation’s sustainability work for all administrative branches, the evaluation results could be used for coherent and long-term sustainable development policy and activities. The evaluation also considered the status of sustainable development in the foreign policy sector; specifically, the government sought to assess how Finland’s foreign policy in all administrative sectors promotes the achievement of the Agenda 2030 goals (Prime Minister’s Office, Finland [PMO], 2017a, b).

The objective of the evaluation was to examine the state of sustainable development in light of national sustainability indicators, key sustainable development policies and objectives, and national implementation of Agenda 2030. The evaluation was expected to produce concrete recommendations on the future directions for Finland’s sustainable development policy, taking into account different timespans and levels of ambition, and proposing future evaluation approaches. Thus, the evaluation results would also provide learning.

In its timely execution, the evaluation aimed to strengthen the knowledge base for updating the Finnish 2030 Agenda implementation plan after the parliamentary elections in 2019, and to give input into the preparation of the new Government Programme. It provided content for social policy debate preceding the parliamentary elections and it produced information on the sustainability work of ministries and relevant stakeholders. Further, the evaluation could serve as an input for Finland’s next voluntary national review (VNR), a component of the United Nations’ implementation process for Agenda 2030, intended in part to facilitate the sharing of successes, challenges and lessons learned.

The Finnish evaluation mainly examined the national-level implementation during the period following the adoption of the 2030 Agenda in early 2016. However, it also considered other significant public instruments at the national level, such as the Society’s Commitment to Sustainable Development launched in 2013 (PMO, 2013) and the supporting Commitment2050 tool, “The Finland we want by 2050” (PMO, 2016); the government report on development policy (PMO, 2017a; VNS, 2016); and the government implementation plan of the 2030 Agenda (PMO, 2017b; VNS, 2017).

Preparations

Initial discussion of the evaluation began in 2017 between ministries and other implementing parties and planning began in 2018 under the leadership of the Prime Minister’s Office. The assignment highlighted the involvement of stakeholders. Accordingly, the evaluation approach was developmentally oriented and strongly participatory by nature. It emphasized comprehensive participation of key actors and stakeholders in sustainable development policy. To promote learning and sharing, evaluators collected data using interactive workshops, interviews, forums, and surveys.

The Prime Minister’s Office chaired the cross-administrative steering group with representatives from the ministries of environment, finance, foreign affairs, agriculture, and forestry. A broader advisory group also was nominated to ensure wider perspective of relevant experts and stakeholders and to strengthen the use and usefulness of the evaluation results. The Development Evaluation Unit of the Ministry for Foreign Affairs had an expert role in the evaluation steering group to comment on the evaluation design, methodology, and reporting. Conducting the evaluation was an multidisciplinary team with members from three Finnish organizations: think tank Demos Helsinki, the Helsinki Institute of Sustainability Science (HELSUS), and the Finnish Environmental Institute (Syke).

The implementation of the evaluation was an intensive process, with launch taking place in August 2018 and the results published just 7 months later, in March 2019. The evaluation title, PATH2030, describes the road map toward transformative policy as put forth in Agenda 2030 (Berg et al., 2019). The publication is part of the implementation of the 2018 Government Plan for Analysis, Assessment and Research (see www.tietokayttoon.fi/en).

Dissemination

The results of the Finnish national evaluation of Agenda 2030 delivered an overall picture of the progress and the status of implementation. Moreover, the document enhanced awareness of the role evaluation can play in the implementation of policy goals and underpinning the Agenda’s Sustainable Development Goals (SDGs).

Sustainable development policy is a broad subject to evaluate, with no single, right way to produce such a vast, national-level evaluation because much depends on the context of the country and there are many variables to consider. However, Finland wanted to share its experience and serve as a motivator for other countries to produce policy-level, strategic evaluations of Agenda 2030 goals. After publication, the evaluation provided input to several proceedings and motivated further international cooperation.

We presented the PATH2030 evaluation in 2019 at the Third International Conference on Evaluating Environment and Development in Prague, Czech Republic, organized by the Independent Evaluation Office of the Global Environment Facility (GEF), the Earth-Eval Community of Practice, and the International Development Evaluation Association (IDEAS). The VNR report was presented to the UN High-Level Political Forum on Sustainable Development in New York in 2020.

In this article, we present the evaluation’s approach and discuss the key results and their usage. We also reflect on the evaluative lessons learned and future options. To aid in understanding the focus of the evaluation, we also briefly introduce the coordination model of sustainable development policy in Finland.

Implementation of the Evaluation

Focus of the Evaluation: Sustainable Development Policy and the Coordination Model

The PATH2030 evaluation focused on Finland’s sustainable development policy and cross-administrative foreign policy. It examined the coordination model of sustainable development in Finland, presented in Fig. 1. The model covers stakeholders, networks, and documents that support sustainable development policy at the national level. These are central to the coordination, management, monitoring, and continuous improvement of Finland’s sustainable development policy, providing, for example, perspectives of young people and research.

Fig. 1
figure 1

Finland’s sustainable development coordination model

Full size image

Since 1993, the Finnish National Commission on Sustainable Development has acted as a coordinating body at the national level. The Prime Minister serves as chair and commission members represent broadly various sectors of society, from political decision making to ministries, research institutes, interest groups, and nongovernmental organizations. The work is supported by a secretariat at the Prime Minister’s Office and by the Coordination Network of ministries. This network acts as a link between the various administrative sectors and national sustainable development policy. The Expert Panel on Sustainable Development, which has been in operation since 2013, consists of independent experts and researchers from a range of fields. In 2017, the Agenda 2030 Youth Group, coordinated by the Finnish Youth Cooperation Alliance, was established under the Finnish National Commission on Sustainable Development.

Finland’s first national strategy for sustainable development was published in 1998 (Ministry of the Environment, 1998) and the second in 2006 (PMO, 2006). In 2013, the Finnish National Commission on Sustainable Development issued a new strategic statement called the Society’s Commitment to Sustainable Development (PMO, 2013).

In the PATH2030 evaluation, the key concept is sustainable development. This concept has no single formulated definition, but it is commonly understood as development that meets the needs of the present without compromising the ability of future generations to meet their own needs. For this particular evaluation, central documents on sustainable development policy in Finland guided the conceptualization of sustainable development as a political objective; this process also helped in analyzing the coherence of concepts across those key documents (Berg et al., 2019).

Sustainable development policy, in its broadest sense, refers to all policies that affect the achievement of Finland’s sustainable development targets. It may also, therefore, refer to policies that have not been included in the scope of sustainable development in previous declarations. For instance, this scope might include measures to combat climate change or to prevent the growth of societal inequality, or measures of economic policy that increase or decrease the total consumption of natural resources or the rate of employment (Berg et al., 2019).

The evaluation looked at the operational model of sustainable development, which refers to a comprehensive set of cross-administrative policies with the official mission of promoting sustainable development. Examples of these policies include budget reviews from a perspective of sustainable development and the integration of sustainable development as part of the strategies, measures, indicators, and evaluation of ministries (Berg et al., 2019).

The evaluation also studied foreign policy in all sectors of government. This refers to the Finnish government’s aim that the nation, as a global partner, supports the sustainable development of developing countries through various means of foreign and security policy, such as trade policy and development policy (PMO, 2017a). According to the Government Programme, Finnish development policy emphasizes strengthening the business activities and tax bases of developing countries (PMO, 2015). The PATH2030 evaluation report included these government priorities, which are also linked to the UN processes that preceded the 2030 Agenda.

Evaluation Questions

The ultimate purpose of the evaluation was to create preconditions for coherent and long-term sustainable development policy and strengthen the knowledge base of the implementation plan for the 2030 Agenda. To achieve this, the evaluation needed to cover complex phenomena and the manifold policy context of the 2030 Agenda. Therefore, our evaluation team, steering group, and supporting group worked intensively to handle the comprehensive tasks required in the call for proposals and to define the main evaluation questions. With regard to assessing impact and effectiveness, we also had to take into account the short time span.

In the end, the final main evaluation questions related to:

  • the state of sustainable development in Finland in light of indicators

  • the main goals and means of the development policy

  • challenges and strengths of sustainable development policy

In relation to foreign policy, the evaluation explored:

  • the links and coherence between the different administrative branches of foreign policy and the sustainable development goals (focusing on international tax policy and trade policy)

  • the different ways Finland’s foreign policy can contribute to achievement of goals across all administrative branches

Carrying Out the Evaluation: Approach, Methods, and Process

The evaluation approach derived from the tradition of developmental evaluation, but it also relied on a theory-based analytical tool adapted for this evaluation. Theory-based assessment aims to understand both preconditions and mechanisms of implementation, and we paid attention to the theories of change behind the impact pathways (see Stame, 2004, 2006; Weiss, 1997a, b). Based on a desk study, the evaluation evolved around four central target areas:

  1. 1.

    The status of sustainable development

  2. 2.

    Theory of change behind sustainable development policy

  3. 3.

    Policy measures

  4. 4.

    Foreign policy

The 4I’s framework (Brockhaus & Angelsen, 2012), in which sustainable development policy is analyzed through institutions, interests, ideas, and information, served as a key analysis structure for the evaluation. Using this framework, the evaluators analyzed how, at the institutional level, societal structures limit or promote development, how the interest of stakeholders gain a voice, and how different interest groups participate in the decision-making process. At the level of ideas and ideology, evaluators identified ideologies and explored how ideas have been accepted in politics. The evaluation team also studied the type of information that was used to support and guide policy. Table 1 presents the framework that was applied to the evaluation (Berg et al., 2019).

Table 1 Analytical framework of the PATH2030 evaluation
Full size table

The intensive evaluation process had three phases that formed the basis for conclusions and recommendations:

  1. 1.

    Analysis of the current state of sustainable development policy

  2. 2.

    Evaluation of the current state and its challenges and opportunities

  3. 3.

    Validation and development of the preliminary recommendations via interviews and workshops and within the steering group and the broader advisory group.

The evaluation team also liaised with the international evaluation community through EvalPartners and EVALSDGs to share ways of working and advice.

The evaluation explored the theory of change for Finnish sustainable development policy first and foremost by focusing on the most central documents—policy instruments for sustainable development—and the views of the representatives of ministries and other experts. The analysis sought to clarify not only the official goals but also the ways in which key actors perceive them (Berg et al., 2019). The evaluation team consulted the international SDG index (Sachs et al., 2018) and Finland’s national sustainable development indicators.

The developmental aspect assured that the evaluation process was inclusive and included several participatory elements. The process gave different stakeholders in sustainable development policy the opportunity to contribute to the evaluation and learn from each other’s experience during the evaluation process. The evaluation used multidisciplinary methods and cross-sectoral data collection to acquire a wide, diverse set of material. The data consisted of both indicators and documents, with expert insight collected through questionnaires, interviews, roundtable discussions, and workshops. A total of 130 experts took part in the workshops, and we interviewed 80 bilaterally. The questionnaire produced 238 responses. The data collection process also included an international workshop held with partner organizations Stockholm Environment Institute (SEI) and Sustainable Development Solutions Network (SDSN). Figure 2 illustrates the process of the evaluation.

Fig. 2
figure 2

The process of the PATH2030 evaluation. (Source: Berg et al., 2019)

Full size image

Key Messages from the Evaluation

The PATH2030 evaluation stated that in the future, Finland should focus on substantive issues where the nation still faces particular challenges or opportunities to progress. The evaluation found that Finland’s strengths in implementation of sustainable development include societal stability, competence, and capability to mediate conflicts of interest. The biggest challenges lie in addressing climate change, the state of the environment, unsustainable consumption, and social inequality.

Although the evaluation concluded that sustainable development is broadly accepted in Finnish society, at the policy level, Finland is still missing clear common vision and a plausible plan for achieving policy goals. The evaluation also recognized that the theory of change for sustainable development needs to be clarified in terms of objectives and measures, and in use of indicators. Despite this, Finland’s policy has succeeded in generating ownership and inclusiveness of sustainable development, but the evaluation noted room for improvement in policy coherence and capacity to produce transformative change. The evaluation suggested increasing proactive measures and enhancing coherence of sustainable development policy. At the institutional level, sustainable development has not yet been sufficiently integrated into all government sectors and management systems. Sustainable development is reflected quite well in strategies, but various ministries could include more management-level Agenda 2030 discussions. The evaluation also found that the systemic use of indicators and other data and knowledge in decision making and societal learning could be strengthened (Berg et al., 2019). Overall, the evaluation provides a wealth of information on the state of sustainable development and different solutions.

A key finding concerning Finland’s foreign policy was that promoting sustainable development in cross-sectoral foreign policy has largely remained the responsibility of the Ministry for Foreign Affairs, especially its Department for Development Policy. Hence, the evaluation noted a need to enhance the impact and coherence of cross-administrative sustainable foreign policy (Berg et al., 2019).

Table 2 presents a summary of the evaluation’s analysis of strengths and challenges (Berg et al., 2019).

Table 2 Summary of analysis: strengths and challenges of Finland’s sustainable development policy
Full size table

The evaluation recommended that sustainable development should become the basis of future Government Programmes, that a roadmap for how to reach the goals should be created, and that, for example, the indicators and organization of sustainable development and the role of the expert panel on sustainable development should be revised. Promoting sustainable development in cross-sectoral foreign policy should be a central part of the Government Programme (Berg et al., 2019).

The evaluation also explored how the monitoring and evaluation of the SDGs could be organized in the future. It recommended strengthening the monitoring of Agenda 2030 by improving the usability of indicator data and by creating a more clearly visualized indicator system, which could serve as a broad-based, topical benchmark for discussion. More usable data would promote monitoring the achievement of goals and developmental trends. The evaluation also suggested that a systemic, cross-administrative evaluation system would help more systematically assess the impact of Finland’s sustainable development policy and form a basis for the long-term follow up of the results. The systemic national level evaluation tool could increase knowledge of interconnection of activities between different administrative and policy sectors. In relation to decision making, the tool could strengthen the use of monitoring information and impact assessment as part of the policy for implementing the 2030 Agenda (Berg et al., 2019).

The Use and Usefulness of the Evaluation

At the national level, the results and recommendations of the Path2030 Evaluation were widely shared with stakeholders. Several discussions took place with political parties, four parliamentary committees reflected on the results together, and the evaluation team presented the results and organized workshops in several ministries and with the Finnish National Commission on Sustainable Development. The evaluation’s timing was ideal in relation to the election and ongoing negotiation for the Government Programme in Finland. The Programme is very much built on the basis of Agenda 2030, as recommended by the PATH2030 evaluation. As a result, the Government decided to include a roadmap to achieve the SDG in its new 2019 Programme, in the form of a report on the global 2030 Agenda for Sustainable Development. The report, submitted to parliament in October 2020, serves as Finland’s national implementation roadmap for achieving the objectives of the 2030 Agenda (VNS, 2020). The report presents the concrete actions the government is taking to promote the achievement of each of the 17 SDGs in Finland and globally.

After publication of the PATH2030 evaluation, Helsinki hosted an international seminar on evaluation to connect the SDGs with national priorities. The event was jointly organized by the Ministry for Foreign Affairs of Finland, IIED, EVALSDGs, and UNICEF. The purpose of the joint seminar was to support country-led initiatives to evaluate national performance against Agenda 2030. Government representatives, evaluation specialists, and civil servants from more than 20 countries attended the seminar and participated in workshops on the role of evaluation in facilitating national policy dialogues and the evaluative challenge of the multidimensional characteristics of the SDGs. The main objectives were to learn from one another, share good practices, and produce an initial set of principles for SDG evaluation. Bringing people together helped identify learning needs and informed postworkshop knowledge products.

The seminar inspired emerging international cooperation that led to a coproduced handbook, Evaluation to Connect the National Priorities with SDGs (D’Errico et al., 2020). Although the 2030 Agenda has clear guidelines for follow-up, review, and reporting, the workshops in the seminar revealed a demand for clarity and support around how to conduct evaluations in a country-led process. The four organizations decided to meet this need together, collaborating on a guide to country-led SDG evaluations that would raise awareness of the role evaluation could play at the national level in the context of the 2030 Agenda. The PATH2030 evaluation was introduced and the guide was launched in New York at the UN’s annual High-Level Political Forum on Sustainable Development. This event is the main intergovernmental platform for discussions about sustainable development, including the sharing of knowledge, learning, challenges, and success.

The guide has become a very timely resource for evaluation commissioners, managers, and professional evaluators who are seeking to create tailored plans and approaches to SDG evaluation in their country context. Local and national evaluators can now benefit from support in using SDG evaluations to improve policies and programs closer to home by applying tailored approaches. Rather than offering a one-size-fits-all model, the guide supports building successful evaluation around existing national context, underpinned by the principles of Agenda 2030.

When preparing evaluation approaches, commissioners, managers, and evaluation experts have to consider complex system interactions and interventions within Agenda 2030 policy implementation. Evaluation commissioners often struggle with how to prepare for an effective and useful evaluation in their country context. In the guide, experts emphasize the importance of identifying from the outset who will use the evaluation results, how they will use them, and why. After identifying this, commissioners can consider the different ways they could use evaluation to learn about SDG implementation and relationships with existing strategies, policies, and programs.

Connecting national priorities with SDGs—including 17 goals, 169 targets, and 232 indicators—is a challenge, and knowing and ensuring that the implementation is on the right track is not easy. The international expert group that prepared the guide reflected on some of the key issues in selecting the methodology and designing the evaluation setup for an SDG evaluation. They highlighted that successful evaluation usually draws on evidence from various sources. Therefore, the guide recommends integrating monitoring systems and indicators as part of the evaluation, while giving specific evaluation methods the most robust role.

The experts observed that one of the greatest challenges is assessing integration in the context of multiple SDGs. Thus, the methodology and methods chosen must support evaluators in drawing overall conclusions from multiple findings. Selection of appropriate methodology depends on the evaluation questions, objectives, and use of the results. Participation of the key stakeholders in designing the evaluation approach also quite often contributes to choice of methodology, which increases the usefulness of the evaluation results (D’Errico et al., 2020).

The chosen evaluation approach and methodology have implications for utility. For a successful evaluation, keeping the scope manageable is important, as is limiting the number and diversity of the evaluation questions. For example, in the case of Finland’s PATH2030 evaluation, the extremely complex task led to the original evaluation questions in the call for proposals being very challenging and complicated, and too numerous. After some radical revisions, the questions were clarified and simplified to better serve the purpose and usefulness of the Path2030 evaluation (Berg et al., 2019).

Below are some practical lessons learned for improving the utility of an SDG evaluation, drawn from reflections on the Path2030 evaluation.

  • Plan the framework carefully and be focused.

  • Keep the evaluation questions short and clear.

  • Try to keep up participatory spirit—but make clear that all expectations cannot be met.

  • Allocate enough time for the entire evaluation process.

  • Write the report such that readers can readily understand the complex process and its results. Visualize and simplify the complex issues.

  • Formulate the recommendations carefully to make them clear and easy to understand and adapt.

  • Focus on the opportunities, not only the challenges.

  • When communicating the results, connect to the actual challenges and focus on the most important priorities.

In national evaluations, a participatory approach enhances the usefulness of evaluation results. In organizations, designing inclusive processes is a crucial precondition for evidence-informed learning and decision making.

Evaluative Lessons for the Future

Inclusiveness and Participation

Discussing how Agenda 2030 evaluation approaches and processes can contribute to national progress on sustainable development is interesting and extremely important. The PATH2030 evaluation of Finland’s sustainable development policy is an example of conducting an evaluation in a very participatory manner. The starting point for the evaluation was the tradition of developmental-oriented evaluation (see Patton, 1997, 2011), which is not a common approach in policy-level evaluations of SDGs. The evaluation focused more on systems and processes rather than only on end results. The developmental orientation was intended to serve both sharing and learning purposes and to bring together different interest groups and administrative sectors. Another important justification of the approach was that the collaborative methods would help in gathering valuable but undocumented data to identify a comprehensive status of Agenda 2030 implementation. Joint workshops revealed and produced both in-depth knowledge about how the indicators had been applied in practice and data used in the complicated context of decision making. Developmental evaluations are often long lasting, but this was not the case in Finland’s exercise. Despite the intensity and short duration, the evaluation’s developmental aspects were not undermined; on the contrary, we learned numerous useful lessons.

Although the Path2030 evaluation did not follow the most traditional principles of developmental evaluation, its success encourages us to apply this approach even more consistently in policy-level evaluations. The evaluation convinced us that using participatory methods consistently is possible—and worthwhile—even in a limited time span, and especially when the focus is on such a complex system. It is important to recognize the evaluation processes and structures that expand partnering, boost utilization of results, and lead to learning and transformative change. We have learned that recommendations become clearer, more concrete, and more realistic when they are formulated with the participation of stakeholders, civil servants, or policy makers who are the ultimate users of the evaluation results. Implementation of recommendations is effective if the evaluation contributes directly to ongoing reform or an organization’s development process. And if the timing is not right, evaluation may not affect the policy reviews, strategies, or implementation of policies as planned.

Learning Throughout the Evaluation Process

Several proceedings and dissemination events after the publication of the Path2030 evaluation have increased the utility of the evaluation results. Enhancing learning does not mean only disseminating results. It also requires attention both to learning throughout the evaluation process and to learning from results after publication. From a learning perspective, it is more important that an evaluation is valid and fit for the purpose in the particular context than that it rigidly fulfills the requirements of comparability of the results. Locally designed evaluations (enabling ownership) to meet local conditions have proven to have a positive effect on learning and development, as long as other quality assurance elements are embedded in the evaluation system to ensure as much confidence and trust as possible. Many studies have shown that designing inclusive processes is a crucial precondition for evidence-informed learning and decision making in organizations or in country contexts. Therefore, allocating time for sharing and reflection throughout the evaluation process is important (e.g., Mayne, 2010, 2011; Palenberg et al., 2019; Räisänen & Räkköläinen, 2013, 2014; Räkköläinen, 2011; Vähämäki et al., 2011; Vähämäki & Verger, 2019; Young, 2019). Table 3 summarizes lessons learned that relate to supporting learning throughout the evaluation.

Table 3 Learning throughout the evaluation process
Full size table

With regard to evaluation of SDGs and Agenda 2030 implementation, an important step is encouraging countries to develop their own monitoring, evaluation, and learning approaches and practical tools based on understanding of the variable needs in different contexts. Learning throughout the evaluation process should be considered beginning with the evaluation design and given special attention in the methodological choices. Concerning methodological development, one option at the local level could be to pilot the so-called real-time evaluation approach, which has its origin in developmental evaluation tradition (e.g., Cosgrave et al., 2009; Herson & Mitchell, 2005; Jamal & Crisp, 2002; Polastro, 2011). Real-time evaluation is normally associated with emergency response or humanitarian interventions because it is designed to provide immediate (real time) feedback to those in charge of interventions, programs, and projects. This feedback is usually provided during field work so that immediate improvements can be introduced and put into practice in timely manner. Real-time evaluations are often joint exercises that enable shared learning opportunities and enhance mutual accountability between different actors.

The COVID-19 pandemic has presented many obstacles to SDG evaluation—and all development evaluations. Conducting evaluations that capture the multidimensional characteristics of the SDGs is even more challenging when activities are restricted. The pandemic caused data collection problems in many countries, which may lead to a major evidence gap and make verifying the impact of the 2030 Agenda even more difficult. COVID-19 will also affect not only the way recommendations are formulated, but how they can be put into practice and linked to SDGs. The pandemic will have long-term effects on economies, jobs, livelihoods, and poverty; therefore, the evaluation approaches during this period should be strategic and forward looking. More solid national evaluation policies may also be necessary. Strengthening evaluation capacity will be instrumental in stimulating national-level ownership in evaluation and the use of evaluation results for transformative change toward sustainable development goals.