Partially due to prominent societal and policy developments like the Paris Climate Agreement, the Green Deal, and the European Union (EU)’s recent attention for societal missions, there is a rise of innovation and transition policy strategies dedicated to addressing ambitious societal goals [1]. Some of these strategies take a systemic approach by focusing on a broad range of technological, sociological, and economic factors that influence the possibilities for societally desirable innovations to become successful [2]. This not only goes beyond implementing individual interventions targeting particular system bottlenecks [3], but also extends past so-called “systemic instruments” that address various actors and multiple system functions at once [4]. Instead, systemic approaches concern overarching policy strategies for continuously monitoring and addressing problems that might occur when systems develop and transitions unfold [5,6].

We, thus, understand systemic innovation and transition programmes as integrated strategies that aim—through coordination structures and activities—to achieve alignment between a set of policy instruments that target different parts of the system (e.g., its systemic problems as suggested by [7–9]) in which it aims to intervene, thereby jointly creating conditions conducive to achieving a so far unmet societal want or need. Examples include the Dutch Topsector approach for strengthening the competitiveness of 9 technoeconomic domains [10] and the Flanders Circular Strategy for addressing barriers that hamper the emergence of a circular economy [11].

While the urgency and legitimacy of such policies mainly stem from societal developments, this most recent “transformative” innovation policy paradigm [9,12] also draws inspiration from the literature on sustainability transitions. Besides regarding this as flattering to the transitions research community, we stress that this brings responsibilities regarding delivering concrete advice on how to evaluate systemic innovation and transition programmes. If transition thinking is increasingly influencing policy making, it is inevitable that this will also lead to questions on how to determine policy success. A crucial issue in the next era of transition research will most likely be how to distinguish good programmes from bad ones, to prevent bad programmes from undermining further spread of transition-inspired policies (and make the current upsurge short lived). With systemic innovation and transition programmes gaining popularity, it becomes all the more urgent to develop evaluation approaches that allow us to tell whether and how to improve or terminate unsuccessful policies.

This is a challenging endeavour. Indeed, in a recent survey of the literature on transformative innovation policy, Haddad and colleagues [13] found that transitions- and mission-oriented innovation policy scholars have argued that evaluating systemic innovation and transition programmes requires evaluators to not only assess programmes against a broader set of relevant impacts and system-level transformative outcomes, but also account for interactions between instruments, engage (and manage conflicts between) a broader set of stakeholders, and achieve coordination between different scientific and technological fields, policy domains, and sectors (see also Amanatidou and colleagues [14] and Luederitz and colleagues [15] for an earlier discussion on the same topic). Moreover, evaluators should recognise that tracing causal mechanisms in complex and inert sociotechnical systems is difficult [10] and that policies can generate unanticipated system dynamics [16].

So far, however, the debate on assessment of systemic programmes has remained rather fragmented—in terms of communities (innovation scholars, transition scholars, policy agencies, auditors, and evaluators) as well as content. Several attempts have been made recently to develop new analytical approaches to evaluation [15,17,18], but often without referring explicitly to other similar frameworks and methods. As a result, it is difficult to get a good overview of how different concepts and frameworks complement or substitute each other. We therefore argue that it’s time to have a more extensive, structured, and creative debate on how to assess systemic innovation and transition programmes. To move forward in this endeavour, the model in Fig 1 provides some guidance by identifying 6 tentative categories of evaluations (A to F).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Generic theory of change for systemic innovation and transition programmes, with evaluation categories A to F.

https://doi.org/10.1371/journal.pstr.0000008.g001

We map these categories on an axis ranging from formative evaluations, which can help us to understand why and how policies work (or not), to summative evaluations, which serve to attribute observed outcomes to policy effects. The latter allow for uncovering causal mechanisms, which also can feed into more refined summative evaluations as it sheds light on what (intermediate) outcomes to measure. In light of recent suggestions to focus on the complementarities between accountability and reflexivity (e.g., [19]), it seems advisable to combine summative evaluations with more frequent formative evaluations suitable for rapid learning and policy adaptation.

For both types of evaluations, a coherent assessment scheme for comparative analysis of systemic programmes would be useful—whether for drawing inspiration from elsewhere (in formative studies) or for obtaining a counterfactual (in summative studies).

For formative evaluation, there is an ongoing but poorly converging search for theoretical frameworks for assessing systemic programmes. This emerging literature mainly addresses 4 of the categories in Fig 1 in that they present frameworks for assessing (A) the presence of a legitimate intervention rationale, such as system or transition failures [9]; (B) the quality of governance processes and structures [20] or policy mixes [21–23], e.g., in terms of consistency and coherence; (C) the match between policy interventions (within and across policy domains) and identified system weaknesses or bottlenecks [10,24,25]; and (D) improvements in the system’s performance, e.g., in the form of strengthened system functions [26] or transformation processes and outcomes [13,27]. Since these categories are complementary and all have the potential to affect policy impact, further research should focus on either reconciling different frameworks or providing clear guidance on when to use which one. One line of research is the use of novel methods like participatory tools that allow for engaging different stakeholders in the identification of impact processes [28] or the assessment of transformational policies [18]. This creates increasing demands on evaluators to mobilise and empower relevant stakeholders from different fields, sectors, and policy levels, coordinate and align potentially divergent perspectives on problems and possible solutions (cf [29]), and manage conflicts of interests [9].

Regarding summative evaluation, the good news is that some systemic programmes have measurable end goals, like an amount of CO₂ emissions reduction (category F in Fig 1). The bad news is that ascribing actual societal impacts to developments in the underlying system is notoriously difficult and understudied [30]. Rigorous attribution-oriented evaluation approaches might therefore focus on intermediate goals and examine policy effects—including the effects of interacting policy instruments [7]—in the form of relevant structural changes (such as new firms or networks in a system—category E) or underlying innovation and transition processes (e.g., via network analysis techniques for studying knowledge sharing in the system—category D). A particularly salient feature of systemic policies, in this respect, is that they might aim to shift the system’s focus to a particular societal problem or set of associated solutions. Evaluation of the effects of providing such solution directionality would require methods for assessing the momentum of investments in prioritised solutions, vis-à-vis developments in other solutions. We therefore encourage scholars to explore new data collection and analysis methods, like semantic techniques using project descriptions for gauging whether supposedly improved systems evoke activities that are in line with current priorities. Some inspiration can be found in recent attempts to map research and development (R&D) projects on Sustainable Development Goals (SDGs) [31] or societal missions [32].

To conclude, we emphasise that the adoption of transition thinking in emerging systemic policy approaches is merely an initial success. Real victory will only come from obtaining convincing evidence of the actual quality and impact of such policies, which can be used as a basis for policy learning and enhancing their social accountability.

This calls for further development of evaluation approaches for systemic innovation and transition programmes, as well as establishing a learning culture that is open to different ways of thinking about policy assessment. Probably the most fundamental shift concerns a deviation from evaluations of programmes in which there is a clear view from the outset of the linkages between policy inputs and envisaged impacts (e.g., in the form of a Theory of Change [33]) to evaluations that have to start with uncovering the very change mechanisms through which policy interventions might stimulate (or hamper) transformations of complex sociotechnical systems. In fact, in the case of wicked challenges, one of the purposes of systemic programmes can be to facilitate societal learning regarding which systemic problems prevent progress towards an unmet societal demand and how these problems can then be tackled [34]. However, while this type of learning requires policy experimentation, it does not follow that systemic programmes should not be held responsible for policy inefficiency or for lacking impacts. On the contrary, all evaluations are, in the end, supposed to shed light on policy effects and how they accumulate and interact. The 6 evaluation categories proposed above are intended as a first basis for collecting complementary pieces of evidence in this regard. Further research is needed to validate and possibly extend the model as well as to identify and select appropriate empirical methods (including also less conventional alternatives like outcome harvesting [35] or process tracing [36]) and reveal what lessons can be drawn from deploying and combining them.