Beyond Objectivity? Storytelling and Reflexivity as Expert Work

Abstract

This chapter focuses on an analysis of the role of storytelling and reflexivity in further strengthening and legitimising quantification in global education governance. It examines two specific empirical examples that show, first, how data visualisations in education and sustainable development are changing in order to accommodate the construction of a more democratic and inclusive governing space; and second, how policy and expert actors themselves use reflexivity as a way not only to understand and think about their daily policy work, but also to create spaces of alignment and consensus. In that way, both storytelling and reflexivity can be seen as working instrumentally, enhancing and further embedding the work of ‘governing by numbers’, rather than displacing it.

1 Introduction: Stories and Reflexivities in Global Education Governance

In the introductory chapter of this book, I discussed how the pressures to make global public policy more inclusive, open and democratic have profoundly affected how experts work in these settings in order to achieve change. As the infrastructure of global education governance becomes more and more extensive and multi-polar, the efforts to quantify and commensurate vastly different geographical and political spaces stray further and further away from constructing ‘perfect’ statistical environments; what is more, apart from the technical difficulties that amount, the sustained effort to produce national and local statistical systems with the input from and for the benefit of the local communities make the work of International Organisations’ (IOs) experts more difficult, facing challenges both upstream and downstream, with many of them feeling that they have an impossible job at hand.

Moreover, although data-led governance has often been depicted as technocratic and ‘de-dramatized’, since the mid-2000s the ‘behaviour change agenda’ (Jones et al., 2013) emerged as a catch-all term for adopting a behavioural science approach that understands the production of public policy as combining cognitive and non-cognitive elements that include ingrained biases and affective ‘moods’. Scholars have thus paid attention to how these novel forms of affective expertise ‘re-humanises’ the policymaking process, by establishing the need for policymakers to design policies which are emotionally informed. There is burgeoning literature on the affect theory and the ‘emotional turn’ (Hoggett & Thompson, 2012). Similarly, feminist geographers have written persuasively about the ‘new enthusiasm for an emotionally attuned approach to government’ (Pykett et al., 2016, p. 1). Therefore, we see increasing tendencies to envelop number with stories that can help to better make sense and produce affective responses to the issues at hand.

Another more recent direction of contemporary public policy literature is the increased focus on democratic innovations. (Bua, 2019, p. 282). Though still in its infancy, this scholarship has suggested that the horizontal, non-hierarchical character of the ‘governance turn’ (Bevir, 2011) in fact coincided with profoundly exclusionary and undemocratic practices, such as the colonisation of public policy by private interests (Lee et al., 2015), the depoliticization of social issues (Streeck, 2013) and the centralising and managerial tendencies of contemporary public administration (Gamble, 1994). The proponents of democratic innovations suggest that inclusivity and a consideration of a variety of evidence-bases, including non-quantitative ones, can enhance and expand the epistemic basis of decision-making. Similar with other fields, education governance has been guided by developments in the field of participatory governance and democratic innovations in an effort to explore the potential of data storytelling to create common, democratic spaces of deliberation through the use of visualisations.

Thus, the key question that arises in this chapter is how education experts try to remain faithful to their epistemic credentials as purveyors of factual knowledge, while also using instruments like stories and visuals in order to persuade actors to participate in these contentious processes. In trying to answer this question, this chapter turns to an examination of storytelling and reflexivity as key tools in the production of expert knowledge in this new era of ‘democratising’ global education monitoring. In this context, reflexivity is not only a thought process that is necessary for the work of expert actors in order to continuously make sense of what their work involves, to justify its failings, or to explain their moral predicament. Such a view of being reflexive to the applied and contextual character of producing knowledge for policy became a key feature in knowledge production work from the 1990s onwards, as it has been manifested in the Mode 2 literature. More recently, however, we see the workings of these ‘softer’ persuasive instruments, such as stories and reflexivities not only as an epistemological endeavour but also as a key political resource: as this chapter will show, expert actors use reflexivity as a political tool in their efforts to construct consensus and mobilise the participation of countries and their representatives in monitoring agendas and frameworks.

Therefore, although previous literature has seen reflexivity as an essential tool for scholars to address the ways that reflexivity may affect the research we do, here I am interested in the ways that reflexivity is a necessary tool for analysing the work of experts, as well as the socio-material manifestations of expert work, such as the case of data visualisations. The following section will draw upon the theorisation of the relationship of quantification with qualification, a key process as I will show, in understanding how the work of numbers is enveloped and made sense of through the application of value judgement. The chapter will then move to the discussion of two empirical examples from the field of the global governance of education: first, the use of interactive data visualisations as storytelling devices, and second, the role of instrumental reflexivity—i.e. the ways actors may practise reflexivity as a way of creating consensus and achieving commensurability.

2 Expert Reflexivity: STS and the Work of Qualification

What do I mean by a relationship of stories and visuals with the production of evidence-based policy? Examples of such interactions abound: in fact, the more collective, critical and fundamental the policy issues, the bigger the influence of stories and visuals in contributing to the shaping of policy problems (Mesch, 2013). From the historical cases of the American civil rights movement, to feminist art, all the way to the global challenges of sustainability, migration and public health, stories and visual imagery have always been actively present in shaping the formation of new public arenas. Although the contemporary hegemony of data-led decision-making is rooted in rationality as the Enlightenment’s promise of knowing and governing the human condition, the rise of the modern scientific age never rejected stories and images as a way of observing, recording and transforming the world. On the contrary, science and art have always been closely entangled. Science was not merely about producing universal and objective knowledge; it was also spectacular, in the literal sense of the term: theatres of science were venues for instruction, but also entertainment and social recognition (Blatchford & Blyth, 2019). Although political science and policy studies have developed an interest in narrative and visual approaches for the understanding of the relationship of knowledge with policymaking fairly recently, the field of Science and Technology Studies (STS) has always seen visual practice as a key medium via which new forms of knowledge, methodologies and engagements can be generated.

STS originated in the 1960s when critical debates on the societal role and impact of scientific and technological innovations emerged. From Edinburgh’s ‘Strong Programme’ of the ‘70s and ‘80s (Bloor, 2013), to Latour and Woolgar’s ethnographic study of the implicit, tacit knowledge and embodied skills that scientists develop (1979), and to Mol’s ‘ontological multiplicity’ (2002), the centrality of story/image-making in STS is not coincidental: it is routed in STS’s research agenda that focuses on technology and materiality; boundary work; subjectivity and the senses; and embodied, situated and enacted forms of cognition (Benschop, 2009). Indeed, John Law suggested that ‘knowing and its methods are materially complex and performative webs of practice’ (Law, 2017, 47). Law gives examples of art/science collaborations as ‘hybrid knowing spaces’ that work through performance, text and simulations in achieving knowledge ‘that might be otherwise’ (Law, 2017, p. 48).

Further, the STS concept of ‘qualculation’ (Callon & Law, 2005) demonstrates the very fine balance between calculation and judgement. According to Callon and Law (2005), complex decision-making requires both acts to be performed together. There are no instances where a mere calculation can give the answer to a wicked issue, given that even the very act of calculation itself requires the application of judgement. According to Moser and Law, qualculation ‘is a way of drawing attention to the fact that the two (which are habitually treated as being different in kind) both become possible—indeed they are only possible—because they array and manipulate appropriate elements within a single relevant frame in order to achieve an outcome or a conclusion’ (2006, p. 66).

Perhaps the body of scholarship that theoretically guides this analysis most powerfully is the emergent field of Art and Science and Technology Studies (ASTS) (Borgdorff et al., 2020). STS has worked closely with artistic research, a field of studies that explores the production of knowledge and research through or in artworks. According to Borgdorff et al., ‘artist-scholars in this field focus on the knowledge, understanding, and experiences enacted in creative processes and embodied in artistic products such as artworks, compositions, and performances’ (2020, p. 1). Here I draw on Caroline Jones’ and Peter Galison’s (2014) Picturing Science, Producing Art, which richly demonstrates how art and design are deeply entangled with socio-technical worlds. The authors suggest that ‘what much of the focus on “art” and “science” as discrete products ignores are the commonalities in the practices that produce them. Both are regimes of knowledge, embedded in, but also constitutive of, the broader cultures they inhabit’ (Jones & Galison, 2014, p. 2).

Such observations need not only be made by artists and scientists however: similarly, METRO fieldwork showed how education policy experts persistently describe their daily business as resembling artistic practice more than scientific work: they suggest that their day-to-day job ‘is more of an art, than a science’ (METRO interviewee). Although METRO focused on the role and effects of quantification in global governance, this—albeit cliché—phrase led to a reconceptualization of education expertise, in order to include other instruments and tools that facilitate the production of knowledge for governance. Borrowing on STS, what are the boundary knowledge spaces onto which policymakers build upon in order to make decisions? When/how does a data visualisation, for example, become an object of beauty and information? When does the photograph of a drowned child in a Mediterranean beach become an artwork and a leverage point for policy change? When does a film, like the recent Don’t Look Up (2021, by Adam McKay), become popular both as black comedy and as a stark warning to our common predicament? Exploring the production of stories and visuals as a space of knowledge and policymaking capitalises on the decades long study of the entanglement of art and science by STS in order to examine if and how policymakers turn to art to develop techniques, values and skills that may be less tangible than statistical data, but still a crucial, yet so far unknown, part of the policy repertoire.

Therefore, in order to make sense of the production of education expert knowledge, I will now turn to the theoretical underpinnings of the concept of ‘qualification’ (Reinicke, 2015), the process via which actors make value judgements on the basis of the decisions and choices they are confronted with. These value judgements might not necessarily take into account pre-conceived categorisations, classifications or even other expert advice. In fact, such value judgements are seen as being made continuously, given the infinite world of commodities and services available: selecting a lawyer is, for example, a decision perhaps not only based on the value of the services that may be on offer, or on the ranking of the local solicitors’ performance, but also on other values, too, such as trust, personal acquaintance, fame or respect.

In other words, decisions on many aspects of everyday life are not only dependent on statistical knowledge (that tends to standardise in order to reduce multiple values in a specific value: the process of quantification). Rather, they are based on judgement of the decision’s (or the good’s) values (the process of qualification): this is a process that, instead of standardisation, requires a process of ‘individualisation’ (Callon, 2002, p. 267). Translated into the context of global education governance, and despite the prevalent focus on analyses of quantitative expertise as a process of commensuration and standardisation (including the author’s), experts in the field are continuously confronted with the very specific (‘individualised’) challenges and values of local populations. Although making judgements is an inherent aspect of the production of quantification, the process of qualification denotes more than that: it is the process whereby certain measurable and standardised values (in the statistical sense) are being consciously opened up to assigning certain political values to the good in question (or they establish new ‘orders of worth’, following Boltanski & Thévenot, 2006).

In the increasingly dispersed governing space of the global education policy field, such a distinction between quantification and qualification, albeit thin and transient, is crucial to understanding the ways experts negotiate their epistemic capital with the political values on the ground, as well as their own personal ones as they go about their day-to-day work. To clarify—and return to the analysis earlier in this section—my analysis is not confined to the tensions of Cochoy’s ‘qualculation’ (2008) that all quantification practices involve judgement. Calculation does not grow on trees, as Callon and Law suggest (2005): it requires time, money and effort and the sociology of quantification has given us persuasive accounts of the judgements inherent in all quantitative practices (Strathern, 1987). Rather, the focus in this chapter is on storytelling and visualisations as the socio-material devices that often assist experts and policy actors to be reflexive and persuasive, so as to engender trust, optimism and confidence that datafication of education governance is the only way to achieve educational success and equity.

This, in some ways, is the reverse process of quantification, via which values are ascribed to value: within the field of global education governance, qualification has become a key component of consensus-building and of increasing participatory and inclusive decision-making practices. Post-PISA and the global acceptance of the orthodoxy of datafication in education policymaking, this is the complex task education experts are asked to deliver, as they try to always match global processes of commensuration with local struggles over priorities and political ideas.

The concept of qualification, as I will show, helps establish and analyse the role of personal and collective values in the struggle over establishing conventions of worth: facilitated via stories and images, reflexivity is a key resource, both at identifying and codifying values at the level of the individual expert/actor (values that make their work, however utopian, worth doing), as well as at the level of working through local political values and agendas and trying to ‘marry’ them with the more top-down global goal-setting. Here, the METRO project found that in the context of the production of global metrics, reflexivity is not only a thought process that is necessary for the work of experts to continuously make sense of what their work involves, to justify its failings, or to explain their moral predicament. Rather, reflexivity is also a key political resource: experts use reflexivity instrumentally and as a political tool in their efforts to construct consensus and mobilise the participation of countries and their representatives in monitoring agendas and frameworks. The remainder of the chapter will explain the socio-materiality of visualisations and stories in enabling reflexivity, as well as the instrumental use of reflexivity by experts in order to emerge as the trusted partners of the Global South.

3 Playing God: Education Data Visualisations and the Art of World-Making

We know by now that the visualisation of measurement facilitates the understanding of complex information sets and supports interpretation and sense-making; more than that, there is increasing realisation that visualisations prompt engagement with calculative technologies (Gatzweiler & Ronzani, 2019; Quattrone, 2017). In particular, the properties of data visualisations endow them with an aesthetic appeal that affects how users interpret them, appropriate, and make meaning with them (see Espeland & Stevens, 2008; Kornberger, 2017). For these reasons, scholars have called for a reconceptualization of data visualisations in the digital age, contending that the visualisation of ranked performance can act as a persuasive and robust ‘judgement device’ (Begkos & Antonopoulou, 2020). For example, recent research shows how higher education is influenced by the rankings’ visual formats, which ‘allow platforms to at once display cascades of inscriptions in a pleasant, aesthetic manner and further complicate the numerical-ordinal basis of traditional ranking systems’ (Decuypere & Landri, 2020, p. 12).

The interpretative flexibility of data visualisations makes them prime sites to explore some of their generative effects (Pollock & D’Adderio, 2012), including generating the possibilities for reflexivity, as I will show below. Visual elements are not only important because they support calculations but also because they offer interpretative clues that cognitively and aesthetically engage with the users of data (Espeland & Stevens, 2008; Quattrone, 2017). Thus, data visualisations are critical to knowledge brokerage, as they enable the communication of research findings to different discourse communities and play important roles in the legitimation and dissemination of data production (Allen, 2018). As I will explain, increasingly, data visuals not only substantially assist with the communication of data: they further enhance the data’s influence, as they facilitate a diversity of interpretations, translations and ultimately the reflexivity of those working with them.

3.1 ‘No-One Left Behind’—Data Storytelling as Reflexivity-Making?

Each of us walks around with a bunch of stories in our heads about the way the world works. And whatever we confront, whatever facts are presented to us, whatever data we run into, we filter through these stories. And if the data agrees with our stories, we’ll let it in and if it doesn’t, we’ll reject it. So, if you are trying to give people new information that they don’t have, they’ve got to have a story in their head that will let the data in.

This section will examine storytelling as an increasingly popular form of visualisation in the education and development world. Storytelling in public policymaking has emerged in recent years as a powerful tool for policymakers and researchers to communicate complex messages in order to reach larger audiences. Either used as a knowledge brokering tool in negotiations among policy actors or weaponised as an advocacy medium in activism, visual storytelling uses the essential elements of story-making across time and space: it is comprised by main characters, a setting, a plot and a moral, in order to help make causal relationships apparent and to frame ‘facts’ and data within particular narratives.

Crucially, the aim of visual storytelling, as I will see below, is less about communicating specific data fast. Rather, it relates to the making of larger frames of political values, where data, numbers and performance monitoring via country rankings, are only one of the building blocks of data ‘world-making’. Instead of rational and objective, visual storytelling is wholly intepretivist in nature and function. Despite the appearance of an objective rationality purported by numbers, stories are meant to be used as tools of reflexivity and data translation. Their function is to construct the narrative frame within which a carefully selected data pool can offer objective comparative country and regional performance. At the same time, however, the comparison is carefully massaged and shaped in a way that a main problem is addressed, key challenges discussed and—usually—some solutions offered.

Data storytelling is particularly interesting for the analysis of knowledge production for governing. Instead of concealing the inbuilt biases and assumptions that all objectivity-making requires, it does precisely the opposite. That is, it works with people’s engrained world views and attempts to shape and reshape them by pressing towards the making of new political problems and political values. As the analysis below will show, although the basis of the Left Behind visual is the ranked comparison of African countries and world regions, data and the graphs are simply the setting of the story; the characters, the plot and the moral message are the ones at centre stage. This is not ‘facts versus values’ evidence-making; the effect is, in fact, almost antithetical to the cold rationality of statistical numbers. Data storytelling uses facts for value-making, and in doing so exploits the subjective and contingent nature of knowledge-making.

Left Behind¹ focuses on girls’ education in Africa. It was produced for the UNESCO Institute for Statistics by Function, a data visualisation studio based in Montreal. Its sources primarily draw upon administrative data from UIS. The visual focuses on the gender inequality problem, and in particular the non-participation of African girls in education (Fig. 1).

Fig. 1

Front webpage of Left Behind visualisation

The data visualisation follows very closely the main features of a story; in fact, by using an introduction, as well as specific separate sections, the visual resembles closely the familiar feel and structure of a book. Its title page is very minimal; it offers a title and a subtitle with the background image of a girl reading, while sitting on the ground and leaning back on a wooden structure. More so than the actual image, the colour palette used for the image immediately travels the audience to the dry, hot, dusty African plains. The image therefore follows a very common stylistic feature found in art; that is, it creates a sense of exoticism. In doing so, through the subtle connotations which align this one with numerous other stories about worlds distant from the West, the image has already served towards framing this story within well-known and classic art historical framings of picturing the ‘Other’. These are not just any schools, any girls or any countries: this is Africa.

Against a slightly hazy background (a feature that continues in the whole visualisation), the title fonts are simple, medium-sized and white. There is a certain softness and stillness in the image, as we enter the world of the little girl reading. Despite the crisis in gender equity in education in Africa, the image travels us without any judgements or flashy messages. The title page offers the destination and the focus, while simultaneously creating the sensation of a slow, earthy, hot land where kids still play outside barefoot. The introductory section is structured in a very similar manner: questions (‘What would your life be like if you only had 5 years of schooling?), answers (‘For some African girls, this is the most education they can expect, and they are the lucky ones’), and statements of crisis and hope (‘Across the region, millions of girls are out of school and many will never set foot in a classroom’, ‘The world has renewed its promise to the millions of girls who have been left behind’). All the text is presented sentence by sentence as one scrolls through the visual, with the background images of girls in classrooms, in the same light creamy, dusky colour hues.

The rest of the visualisation is structured in the format of book chapters, always introduced with a title page (01. The Last Mile, 02. Barriers, 0.3 Persistence of Illiteracy among Women, 0.4 Poor school conditions, 05. More Teachers needed, especially women). Each ‘chapter’ presents relevant data in maps or graph formats. The different pages and graphs are all interactive—they do comparisons of African countries or world regions over time or in ratios. The interactive graphs and maps can be manipulated by viewers through simple movements of the mouse over them. There is nothing extraordinary about these graphs; they follow the common characteristics of contemporary visualisations, following simple lines, laconic explanatory text and modern design.

What is, however, much more interesting when one has a closer look is that all the data charts, maps and graphs are very carefully chosen and put together: some compare selected African countries (depending on the question, these countries are different every time but they are usually low in number). As a result, similar to the image, the data discussed is also fairly minimal, perhaps just a snapshot. Some graphs compare Sub-Saharan Africa with other continents; and others just focus on simple ratios, between literate and illiterate women. Although all data can be accessed by clicking on the black rectangular box at the bottom right of the page, what is striking in every one of these graphs is the careful selection of comparative country or regional data. Although there is clear ranking of countries depending on how well or badly they perform in relation to gender equity, the ranking as a visual, quick and blunt manifestation of best and worst performance is completely abandoned here. Although there are better and worse country cases (this is the function of any graph and therefore of these graphs, too), the comparison here only serves as an illustration of the wider political problem of gender inequity—this is further enhanced by the persistent alternating of country comparisons with world comparisons (Fig. 2).

Fig. 2

Snapshots of Left Behind visualisation (01)

An analysis of those data visuals immediately reveals a range of differences and similarities: there is a balance of change and stability. Clarity is paramount. There is no alarmism, although negative performance is being reported, too. Although the main character remains the same (ie. African girls, women or teachers), the plot is very carefully crafted in order to move from setting the context (0.1 The Last Mile: ‘there are good news…but the gender gap persists’), to a discussion of all challenges (in ‘Chapters’ 2, 3, 4) to the relatively uplifting final section on the necessity to have a larger women teacher workforce. Finally, despite what otherwise would have been read as a major inequity crisis, the data visualisation ends the story with nothing less than a ‘happy ending’: ‘The good news is that the international community has not forgotten these girls’. The intention here is for the visual not to paralyse, but fill its viewers with optimism and positive resolve to tackle the problem; and although the text suggests that the SDGs have pledged to decrease inequality, it asks the viewer to also ‘have their say’ (Fig. 3).

Fig. 3

Snapshots of Left Behind visualisation (02)

This is perhaps the first step in constructing actionable knowledge: enlist one’s audience not only to read and understand, but also to share their experience of the African girls’ education story and mobilise others. Interestingly, the visual does not do any bullet-point language, like most traditional print reports do. While it offers a plethora of interactive information, allowing comparison of performances and progress over time, and although it digests data through some short statements in every page of the analysis, it finishes off with a simple question (Fig. 4): ‘What do you think it will take to leave no girl behind?’

Fig. 4

Left Behind last page

This question in many senses is at the crux of this chapter’s argument: rather than finish off with a definitive memorable statement, or a killer graph, apt for describing the severity of the issue, Left Behind ends with inviting the viewer to think for themselves; that is, to weigh the evidence offered and contextualise the issue within their own story-worlds and experiences. Needless to say, this does not mean that careful selection of data and arguments has not taken place here, and that all interpretations and questions are open: quite the contrary. It is precisely because of the meticulous orchestration of text, image and data, as well as the precise crafting of the plot, that this kind of engagement can be invited. In reality, the question is primarily a rhetorical one: these are the multiple worlds that data visualisations fabricate, worlds into which specific and precise policy facts do not matter as much as the reflexive possibilities data (and especially an effective visual data story) can open up.

Left Behind is an illustrative case of the power of numbers, combined with images and storytelling, to communicate and persuade. The next section describes other tools for enhancing and further spreading the legitimation of quantitative expertise: this is reflexivity and its instrumental use by actors who are in charge of processes of engendering trust and collaboration between local politics and actors with international monitoring agendas.

4 Instrumental Reflexivity and Expert Work

As the previous section discussed, data visuals facilitate reflexive practice, allowing for multiple translations of the numbers presented in them, as they work alongside images, colours and stylistic features that work with users’ interpretive repertoires. In this section, I will move this analysis one step further, in order to explain how reflexivity has become a key resource, not only in the interpretative work that numbers require, but also as a political instrument: it is being foregrounded and used as the main means of constructing and maintaining relationships of trust between experts and countries. In this way, experts can instrumentalise reflexivity for political action (for a broader discussion of the different uses of reflexivity in global public policymaking, see Bandola-Gill et al., 2023).

In order to illustrate the ways in which experts mobilise instrumental reflexivity, I focus on one empirical example exploring the expert work of the UNESCO Institute of Statistics (UIS), not despite, but because of their explicit and intentional reflexive accounting of the challenges of producing quantification for the benefit of countries in the global South.

In order to contextualise the case, we need to understand that the history of the construction of the SDG4 is one of struggle. As already discussed in previous chapters, the two main opposing camps were, on the one hand, the ‘Education For All’ (EFA) movement, and on the other, the process of work undertaken as part of the Millennium Development Goal education indicators. For reasons of brevity and in order to avoid repetition, I won’t develop this history here, but simply state that the two groupings had very conflictual views about the best measurement approach in education to be undertaken: EFA pushed for a diverse set of goals that would acknowledge a broader, humanistic approach to education, whereas the MDGs education experts wanted to find a much more specific and measurable set of instruments, favouring a utilitarian view of education and focusing on key metrics such as literacy and numeracy.

Therefore, in the face of the threat of an education-specific goal being excluded from the SDGs due to the inability of the two groups to find common ground, a solution was found and the worst was avoided: the compromise led to the production of the SDG4. Nonetheless, even if the contestation seemed to temporarily abate, it never really went away. On the contrary, the continued challenges of meeting the SDG4 goals and constructing a solid set of indicators to do so have intensified the struggle and conflict in the field. It is in this space of clash that UIS managed to emerge as the reflexive and trusted international organisation, distinct from others who are seen as more technocratic and representing interests of the Global North. Having worked in countries of the Global South for decades, the UNESCO Institute of Statistics (UIS) was the expert organisation with long-standing links and relationships with the relevant countries, as well as the ability to use data failings (and often of their own making- UIS had had some serious measurement project failings in the past, for detailed analysis see Fontdevila 2021) in order to advocate for the notion of accepting the production of ‘good-enough’, (rather than precise) data, and the political (rather than purely technocratic) uses of target-setting for coalition-building and agenda-setting.

However, how did UIS manage to maintain their position as a data producer alongside powerful others, while at the same time appear to be at the side of struggling countries? First of all, UIS adopted a much more practical rather than ‘perfectionist’ approach to the production of global metrics in education. Instead of advocating for a single measurement tool (like the other IOs did), they focused their efforts towards accommodating the use of different assessments and harmonisation methods. In contrast to other actors, such as the World Bank or the OECD that would have been much stricter in the choice of method (with a preference for their own instruments), the UIS developed more of a ‘patchwork’ approach: they recombined several already available and legitimate models, recognising openly the limitations of each and emphasising the potential for complementarity. Following this more pluralist method, they appeared a lot more accommodating in their data demands, while acknowledging the challenging circumstances that many countries face, trying to collect both commensurable data for the top-down demands, and dealing with the day-to-day requirements of producing governing data. Due to this particular stance that UIS adopted, many interviewees recognised it as perhaps not a data superpower, but as the trustworthy actor that recognised the unequal character of the data production market and thus the difficulties of creating an inclusive space, with the emphasis on the principle of country ownership.

Second, perhaps more importantly, UIS, primarily through its outspoken Director, Silvia Montoya, publicly discussed the imperfect character of global learning data, as well as the political nature of the indicator process. In doing so, she used a highly reflexive approach, emphasising the epistemic challenges of comparison across highly different contexts, as well as the need to find better approaches to coping with missing data and the deep inequalities in statistical capacity of the participating nations. Under Montoya’s openly reflexive leadership, UIS nurtured types of approaches for the collection of data that are hybrid, and brought together different types of assessments, insisting that the different data alternatives are not mutually exclusive but reinforce one another. More importantly, this incremental approach went against selecting one specific method as technically superior to others, and thus was politically much more in tune with countries and their specificities. Thus, not only a middle-way forward was found, but countries also felt respected for their context-specificities and were not sidelined:

There has been significant growth and improvement in the field of learning assessment across the world. Yet today, it is impossible to provide a global perspective of what children are learning… We must be pragmatic. As explained in previous blogs, the best measures and methodologies in the world will amount to little if countries cannot produce them. We must therefore take a pragmatic approach, which may mean mixing the options. This stepping-stone approach was widely endorsed by stakeholders attending the June meeting. They understand the political stakes, the technical issues and the need to find a balance between pragmatism and accuracy…

We need to recognize that SDG 4 indicators are barometers – showing which countries (and, for equity’s sake, ideally which segments of which countries) are making progress and which countries need help. Instead of aiming for the most technically rigorous methodologies, we may better serve the world by taking a pragmatic approach to producing the global measures while helping countries improve the quality and use of their national data. (Montoya, 2017; my emphasis)

As is apparent in the above quotation, UIS used reflexivity instrumentally to reaffirm and strengthen its authority in the education measurement realm as the only trustworthy, ethical and transparent expert broker. Montoya’s reflexive account is not limited to an assessment of the epistemic limitations of the monitoring exercise. Instead of approaching the construction of indicators as a purely technical exercise, despite its apparent limitations, the UIS openly discussed the political nature of the debate as well as the vested interests that shaped it (for example, its director exposed the inefficiencies of the ‘learning assessment market’ in two influential blogs in 2019²). UIS openly admitted that there is no perfect way of doing this kind of work and that technical rigour would have to go hand in hand with a more pragmatic approach: this way, reflexivity became the prime instrument for the organisation to bolster its credibility and create minimum consensus in the field. As a consequence, the notion of ‘good-enough’ data gained centrality, as the political choices and judgements were not hidden but in fact, displayed publicly and used repeatedly in talks and all sorts of public fora.

Thus, similarly to the use of data storytelling, the concept of instrumental reflexivity describes the considerations experts engaged themselves in in the cases where the epistemic qualities of quantification (objectivity, de-contextualisation, universality) were in tension with the political goals of measurement. Here, not only experts did not avoid exposing the political nature of numbers, but also even went as far as to mobilise and instrumentalise it, in order to achieve their goals. Of course, one has to take into account the interdependencies, competitions and collaborations between IOs in order to get a fuller picture of how IOs interact and assume different, complementary identities as they work collaboratively: while some may take the high ground and defend their authority by sticking closely to its objectivity and trustworthiness, others choose to benefit from getting their hands dirty and muddle through political contestations and imperfect numbers.

5 Experts’ Reflexivity in Global Education Governance: The Role of Visuals and Stories

Although policymaking has always been imbued with visual messaging, the visualisation of political communication became particularly central with the rise of data-driven governance. As we have experienced during the last decade, the acceleration of datafication of contemporary policymaking has closely been accompanied by the rise of data visualisations as a key mode of not only political communication, but also policymaking itself (cf. Bekkers & Moody, 2014; Amit-Danhi, 2021).

As the empirical analysis of the Left Behind data visualisation showed, data visualisations are effective communication tools, as they have the ability to minimise the complexity of represented issues and summarise them for multiple audiences. As argued by Falisse and McAteer (2021), the success of specific data visualisations relies on their ability to simultaneously summarise complex information and contextualise them within the broader policy context. This quality positions them as ‘boundary objects’ (Star & Griesemer, 1989) mediating and brokering between different communities; in the case of Left Behind, they work at the inter-spaces of policy production, accommodating the interests and needs of both IO experts and local policy communities. Thus, visualisations have become vital passage points in complex socio-technical systems, as they are located in-between different forms of networks (Rose et al., 2014).

As I have shown in previous work, data visualisations do not only target but also can outline and constitute groups of stakeholders around issues, as they act as ‘alignment devices’ that orient diverse actors towards a common goal (Bandola-Gill et al., 2021). In the case of Left Behind, national performance appears to be of less importance than the need to create alignment and consensus around the need to reduce gender disparities in education; the visual promotes reflexivity around these issues, represented in such a way as to avoid the ‘winners and losers’ older narrative of comparative performance and instead enhance the ideas around universality and common purpose. Indeed, although the underpinning rhetoric of data visualisations is one of political neutrality and technocracy as they are deemed to be representing ‘facts’ (Kennedy et al., 2016), their effects are in reality more political: data visualisations are effective tools of persuasion (Pandey et al., 2014) and they carry this function through multiple means. On the one hand, data visualisations are the most explicit sites of the politics of visibility and invisibility of numbers and data (Espeland & Lom, 2015; Espeland & Yung, 2019)—they make some aspects of the visualised problems evident, while others disappear. More importantly, and as we have seen in the Left Behind visual, they offer not only description but also the possibility of reflection and interpretation (Bekkers & Moody, 2014a; 2014), as well as emotion (Kennedy & Hill, 2018; Lefsrud et al., 2020). Data visuals are performative—they do not just reflect the represented phenomena but construct them and their fields of practice. Even though this process is often considered implicit and almost automatic, at times it can be consciously mobilised by actors aiming to advance their political agendas (Fileborn & Trott, 2021).

Similarly, experts working in IOs shared faith in numbers to bring transformative change, but were also acutely aware that their work is mostly political (Bandola-Gill, 2021). More importantly, they were happy to reflect on, discuss and share the challenges of their day-to-day work. Providing expert advice was seen as a process that required a specific set of qualities that did not limit themselves to quantitative expertise. On the contrary, expert qualities needed to be a lot more diverse, empathetic, creative and adaptable: they involved an understanding of data but also of the local contexts; humility and perseverance in the face of limited funding and the diversity of interests and value-systems; an ability to foresee change and place themselves at the best possible place to tame it; and finally, the skill to transform a perceived obstacle (the lack of precise data) into a valuable instrument for advocacy and consensus-making (the concept of ‘good enough’ data) (Grek, 2020).

In this context, the co-existence of these different epistemic orders (related to the quality of data and the politics of measurement), political orders (related to bringing actors ‘on board’ and producing contextualised measures) and value orders (related to different ethical priorities and cross-cultural ways of working) meant that the work of the experts went beyond just ‘producing numbers’. This multifaceted navigation between different priorities required them to mobilise different styles of knowing. Reflexivity, therefore, emerges as one of the new skills in the expert arsenal. As such, it is both an epistemic practice (as traditionally discussed in the literature on the topic) but also a practical and strategic tool that can be mobilised in the context of complexity.

Approaching reflexivity as practice allows for unpacking its core elements. First, it helps analyse not only the practices of experts themselves but also the socio-material tools that support and promote reflexivity for wider audiences and users—in this view, reflexivity in education governance becomes a shared resource rather than individual endeavour only. In this regard, and as I have shown in this chapter, the visualisation of data is key, as data visualisations appear to perform a significant function: they work towards the political goal of aligning policy priorities towards specific global challenges, many of which might look similar, yet, they can also be adjusted because contextual and regional specificities and trajectories render them different. This multiplicity and ‘adjustability’ does not take away from the authoritative nature of the data. On the contrary, it further reinforces data credibility by making them relevant to all without antagonising lower-performing countries. Therefore, we observe how socio-technical devices materialise instrumental reflexivity, as not only as a practice applied by experts themselves, but also as a way of creating the conditions for promoting data reflexivity for users, too: a certain kind of ‘world-making’ that only metrological realism could have enabled and promoted. In so doing, data visualisations allow their users to create acceptable narratives around both their own specific country performance and the common global sustainable development agenda. Data visualisations retain the illusion of the political neutrality of their producers foregrounding an ‘issue-based’ message (for example, gender equality) rather than focusing on augmenting competition and peer pressure to achieve individual goals (e.g. benchmarking country-level performance). As such, data visualisations create reflexive spaces that align actors with diverse interests and interpretations of performance by allowing for the coexistence of multiple, often contradictory interpretations of data.

Further, as we have discussed in detail in previous chapters of this book, IO experts work in an institutional and political context that is shaped by ever-expanding infrastructures of measurement (Merry, 2019) and pressures to decolonize global governance (Rottenburg, 2009). This context is not only highly fragmented by the growing number of actors, but it is also increasingly participatory and requires constant mediation and brokerage (Grek, 2020). Furthermore, an increasingly pressing issue is the development of measurement approaches that avoid alienating low-performing countries by continuing to pressurise them to conform with ‘best practices’ from elsewhere. International organisations and other key global players purport to design their measurement programmes following equity paradigms, where all the countries—and especially the developing ones—are seen as leading on tackling the global challenges (Best, 2014).

Against this backdrop, the work of reflexivity ‘softens’ the rigid measurement of data and facilitates the promotion of participatory paradigms of global governance. Data, apart from objective, has to be reflexive, and therefore produced in a way that navigates political pressures while communicating the urgency of the global problems as truly global—affecting the entire international community. Using either data storytelling or applying reflexivity instrumentally and strategically, expert actors can entice participation in measurement programmes (Desrosières, 2015; Le Grand, 2003, 2007), while keeping competitive behaviours—that would be dysfunctional in the global policy space—at bay (e.g. a sense of zero-sum competition, gaming, cherry-picking or the manipulation of data—see Espeland & Sauder, 2007; Merry, 2016; Slager & Gond, 2020). Arguably, the global ‘need’ for quantification and performance measurement has never been as perceivably legitimate as it has been since the introduction of the SDGs framework. Simultaneously, there has never been as much attention paid to how global performance measurement may be a form of ‘southering’ (Grotlüschen & Buddeberg, 2020) that presents developing countries as regions of persistent deficit, under the surveillance of Western institutions through different forms of quantification (Arora, 2016).

It is within this context that ‘old’ formats of presenting data (graphs and tables) do not seem to be suitable to capture this multiplicity of values and needs. The qualities of interactivity, engagement and trust are essential in heterarchical and polycentric settings such as the global governance space, where different ‘hierarchies and orderings intertwine and reproduce, none of which can claim to be dominant or even to be fixed’ (Esposito & Stark, 2019, p. 15). Since no single order shared by all exists in such a space, expert work seeks to achieve more ‘equitable’ and politically acceptable solutions by ‘softening’ the data’s appeal through acts of reflexivity; this is achieved via multiple means and it might entail visuals and stories, as well as the inclusion of diverse methodologies in an effort to push for epistemic justice (ostensibly, at least) rather than division. Instead of ordering data according to performance, reflexive practices increase the visibility of areas of concern and potential intervention without seeking to ‘shame’ explicitly any country.

Technological developments during the last decade have greatly enhanced the possibility to offer more intricate analyses of global data, as well as allow the use of data in diverse and more creative—and strategic—ways. For example, through the application of interactive tools, data visualisations are not simply communicated to the user; rather, although the messaging remains focused and clear, the user is also given the tools to engage with the digital interface. By ‘playing’ with the interactive formats, the users can see the multiplicity of data, and choose to work with aspects of it that interests them more. What emerges at the end of this ostensibly playful interaction is a message that does not seek to identify out a clear ‘loser’ because—rhetorically—there is none. By offering multiple views of the data, as well as a range of other visual and stylistic tools, ranked countries become de-individualised and move towards being aligned. The interactive formats METRO explored are a clear departure from the traditional visibility that is perpetuated by more conventional data visualisation whose argumentative power and appeal are tightly linked to their capacity to communicate ‘winners and losers’ almost at a glance (Bevan & Fasolo, 2013; Pollock & D’Adderio, 2012; Wedlin, 2006). On the contrary, data storytelling in the context of global education governance capitalises on more subtle qualities: similar to what is often expected from experts themselves, visuals invite engagement, they afford personalization, and seek to adapt to individual preferences and priorities.

Indeed, this is the central role of reflexive numbers in this context: data has to be clear enough to point to problems and inspire collective action, however, without shaming lower-performing countries. The instrumental use of reflexivity in this context actively facilitates this multiplicity of interpretations and fuels the messaging and its political acceptability. The inherent multiplicity does not take away from the authoritative nature of the visual and the data it carries. On the contrary, it further reinforces the credibility of data by making them relevant to all without antagonising lower-performing countries. In global governance, expert work uses reflexivity to re-adjust country monitoring, while offering a steer towards a very specific and discursively ‘universal’ set of goals.

This point on the inherent multiplicity of data visualisations leads to explore them as ‘world-making’ tools. As argued by Latour (1986), visualisations stabilise specific versions of reality; they can make impossible things realistic and make possible objects more probable than others. In the METRO findings, we explored how data visualisations not only allow for the exploration of multiple aspects of the data but also enable customisation that allows the user to choose different value dimensions in accordance to their own preferences. They are conceived and designed in a way that allows expert—but also their users in the broader sense—to be reflexive and create their own knowledge and interpretations.

Further, having interviewed over 80 experts working in international organisations in the fields of education, poverty and statistical capacity development, METRO analysed the reflexive accounts of these actors’ day-to-day business, as they went about describing and justifying their work. Reflecting on my own expectations of what these accounts might entail, we anticipated that interviewing them would require more intensive probing to get them to explain the limits and challenges of quantification and the types of political work required to successfully implement metrics. The project team did not underestimate them and never thought that their technical expertise would not allow them space to be analytical; our surprise did not relate to the fact that they were thoughtful and eager to reflect on what their work involves. What did surprise the research team was the extent to which, time and again, many of these actors treated the interview space as a cathartic zone, where they would freely share their exasperations at being asked to achieve the unachievable, but also a space where they would share their conviction that measuring inequality was the only available means not only to know, but also crucially to raise awareness of the injustices communities—in the global South in particular—have to endure and overcome. Reflexivity therefore was not only a thought process experts exercised as part of the encounter of the interview; instead, they described it as a tool in their day-to-day job, as they were tasked to insert meaning to their work of numbers, to persuade and to build relationships of trust and reciprocity. In other words, apart from a focus on experts as the ones holding the epistemic capital to know global education governance by naming and measuring it, statisticians, IO experts, and national and local decision-makers reflect on their practices that produce the monitoring system and wonder—other times despair—over how much or how little real-world effects their work has.

Thus, reflexivity appears to be doing a lot more of the heavy lifting of quantification than the literature has so far discussed. As the case of UIS has shown, reflexivity is not merely a process of self-appraisal by experts, as they make sense of their work in an internal dialogue between their personal values and aspirations and their activities on the ground. More than self-reflexivity, I showed the ways actors used the process of opening up the black box of number-making not only to us as researchers, but also with those in the field—including colleagues, collaborators and even policymakers. Experts purposefully put reflexivity to work, in order to, on the one hand, explain and justify choices as they muddle through trying to establish some order in the messy realities of quantifying complex problems, and on the other, as they actively attempt to imbue data production with the political values of inclusivity. Thus, they purposefully apply processes of qualification, as almost the reverse process of quantification: in their efforts to engage and co-opt communities, they need to—momentarily, at least—move away from the rationality and objectivity of commensurability, in order to open up these numbers to contextualisation and even contestation.

Although seemingly antithetical to the production of quantification as the process through which multiple values come together and are expressed through their representation by a single value (the one that can then represent multiple realities and thus be commensurable), qualification is a sine qua non to quantification. This is not simply because judgement is inherent in every single decision, no matter how large or little, over the making of numbers (i.e. what the concept of qualculation denotes). Although such considerations are important, one of the most startling METRO findings was that expert actors were happy to go as far questioning their own authority by opening up a debate about numbers, where theirs and their interlocutors’ political and personal values would acquire almost the same weight as the data itself: in casting light on the ways that reflexivity becomes an essential element of the performativity of qualculation, Skeggs eloquently suggests that ‘values will always haunt value’ (2014, p. 1).

I am not claiming here that experts at any point during their accounts of their work in the field questioned the use, validity and significance of their numbers’ work; quite the opposite, as their quotations eloquently show. What instrumental reflexivity shows is that in order to make quantification work, experts need to re-attach political values to numbers, and thus allow them to take on new meaning and be translated in ‘useful’ ways in the field. Hence, qualification becomes the socio-material process via which new qualities are attributed to measured values in order to become locally malleable and stabilised, pre-arranged and re-arranged in order to suit local needs. This is the process of attributing new qualities to standardised values that have already been commonly accepted. As this chapter showed, calling one’s data practices as purer than another’s, promoting data collection as a ‘duty of care’ towards communities, assembling different data sources to suit local preferences and needs in a bid to look more democratic and ethical, are all acts of politically (ear-)marking numerical work as a lot more than simply numbers that represent reality as is. Thus, reflexivity becomes a useful instrument in the everyday political struggles that experts fight, not only to collect data ‘values’ from the field, but crucially ‘to establish what value is’ (Graeber, 2001, p. 88).

Therefore, and to conclude, unlike the predominant focus in the literature posing reflexivity as almost ‘hidden’ and happening on the level of the individual (Porter, 2020; Scott, 2008), I showed how reflexivity is mobilised, either through socio-material work (data storytelling and visualisations) or at the micro-level of expert practices (instrumental reflexivity). Thus, reflexivity is a key resource in pushing for datafied governance, especially in a context of increased emphasis on democratisation and decolonisation: reflexivity allows the assignment of political values (values with ‘heart and soul’) back to the measurement of statistical values, in order to enlist participation, facilitate inclusion and thus further enhance quantification as the only available means to know and govern education.