Assessing global competence in PISA 2018 : Challenges and approaches to capturing a complex construct

International large-scale assessments such as the Programme for International Student Assessment (PISA) yield comparative indicators of student achievement in various competence domains. This article focuses on global competence as a suggested cross-curricular domain for the PISA 2018 study. The measurement of global competence is related to a number of challenges, which are elaborated, described and discussed. As these challenges have so far not been sufficiently targeted, Germany, among several other countries, has decided not to assess global competence in the upcoming PISA cycle. In conclusion, propositions are made regarding viable options to capture global competence in international comparative studies so that established quality standards can be met.


Introduction
The globalizing world is shaped by a dynamic change that is visible in several areas: in trade, digitalization, the production of goods, or in the labour market (e.g.Boix Mansilla and Jackson, 2013;Coatsworth, 2004).Such change carries with it numerous challenges to society, for instance poverty, disparity, climate change, resource depletion, demographic change and migration.Concepts like the Grand Societal Challenges refer to these issues (see the Framework Programme for Research and Innovation by the European Commission, 2015).The German Council of Science and Humanities (Wissenschaftsrat) and the Fraunhofer Society mention similar aspects in their Great Societal Challenges (Wissenschaftsrat, 2015; also Kallerud et al., 2013) and 'emerging questions of our age' (Fraunhofer-Gesellschaft, 2014: 7).A highly complex set of requirements is thereby imposed on today's societies and their citizens.Related objectives are, among others, reducing poverty worldwide, guaranteeing inclusive, fair and valuable education for all and stabilizing peaceful, inclusive societies in terms of sustainable development (Sustainable Development Goals, United Nations, 2014).Experts in the fields of education, politics and society (e.g.Reimers, 2009;Wagner, 2008) equally judge that so far schools are insufficiently engaged in preparing children and adolescents for tackling the challenges mentioned and, respectively, for actively pursuing the related objectives.Indeed, neither these challenges nor the concept of a global society are new.For example, Parsons (1937) already describes actions in society as a complex of relationships, which he calls a 'web' or a 'tangle'.Luhmann (1997) goes one step further and describes global interrelations as a global society related to complexity and uncertainty.Recently, constructs and approaches that exactly address these aspects have been in great demand.Several countries and educational systems intend to include competencies to deal with the challenges mentioned above in their curricula (e.g.Sweden or India, see Boix Mansilla and Jackson, 2013;Reimers and Chung, 2016).Furthermore, a number of educational systems are currently elaborating guidelines for scaffolding such competencies in different school subjects.For example, Germany has developed the 'orientation framework for global development' (Appelt and Siege, 2008).Organizations such as the Organisation for Economic Co-operation and Development (OECD) and United Nations Educational, Scientific and Cultural Organization (UNESCO) accentuate global competency or global competence (OECD, 2016a(OECD, , 2016c) ) and global citizenship (UNESCO, 2014) and emphasize the education systems' need to deal with global challenges.In the upcoming cycle of the Programme for International Student Assessment (PISA 2018) of the OECD (2016a), global competence is projected to be assessed as the so-called cross-curricular competence.
Against this background, this article examines the challenges related to assessing the construct of 'global competence' in the context of large-scale student assessments such as PISA.The OECD (2015) claims to unite the concepts of global citizenship, intercultural competence and knowledge on globalization in the idea of global competence.While earlier versions of the OECD's assessment framework on global competence particularly emphasized the close relationship of the construct with intercultural competence, the current version is far more focused on critical thinking and reflecting global topics (OECD, 2016c).This article starts with a definition of global competence based on the current state of theory-building, followed by a description of the different versions of the assessment framework.Building on this, specific challenges related to the assessment of global competence are elaborated and their relevance for an international comparison is worked out.These currently unresolved challenges have eventually led to the withdrawal of Germany (and more than 30 other countries) from participating in the assessment of global competence in PISA 2018.The article concludes with prospects and propositions as to how global competence could be adequately assessed in order to master challenges in the context of an international comparative large-scale assessment.
This article was written in early 2017, just before the PISA 2018 Field Trial began for the first countries in March.On the one hand, it reflects the complex processes of test development in international large-scale student assessments, and on the other hand, it argues which criteria could be useful for preparing the assessment of a construct like global competence.Since, at the time of writing, PISA 2018 was still being prepared, we refer to several unpublished versions of documents used as a reference for assessing global competence in the context of PISA 2018.These documents are referenced accordingly.
Both authors are involved in PISA 2018, in their roles as national project manager and as member of the international consortium for developing the context questionnaires.This article does not present a contribution to the discourse on how the construct of global competence should be defined.Rather, it takes on the perspective of assessment in the context of international comparative educational monitoring.The readiness of the construct for assessment is thus examined, i.e. the state of theorybuilding and operationalization of global competence as a precondition for assessment in the context of an international educational study.

Global competence: A relatively young construct
As a concept, global competence has been used in common language for several decades (see Lambert, 1994).In the scientific context, however, it is still considered to be a relatively young construct.It is only in recent years that pertinent scientific contributions have been published.Influential approaches have been proposed by Reimers (2009) or Boix Mansilla and Jackson (2011).In Germany, global competence is discussed especially with regard to education for global and sustainable development (see Appelt and Siege, 2008;Lang-Wojtasik and Scheunpflug, 2005;Rost, 2005).In contrast, English-language research has focused more on an individual's aptitude to communicate, and this aptitude is frequently differentiated into intercultural communication, linguistic and cultural competences or a behaviour which, within a specific society, counts as 'acceptable and intelligible' (Fantini et al., 2001: 8).
Accordingly, a unanimous definition of global competence cannot yet be identified.Scientific theory-building for the construct of global competence is in this regard relatively young and undeveloped.Furthermore, the question as to which global topics are to be considered dynamic settings for globally competent attitudes, mindsets and behaviours lacks a broad consensus at this point.There is, however, a general consensus on the assumption that global competence comprises the three dimensions of knowledge, skills and attitudes and can be described accordingly (Reimers, 2009;OECD, 2016a).Different theoretical approaches vary in the selection of these subdimensions and what names they are given, but there is considerable overlap.For example, Reimers' (2009) approach focuses on a positive attitude towards cultural differences, on foreign-language skills and knowledge about globalization.In its most recent publication on global competence, the OECD (2016a) defines a set of skills and attitudes as components of the construct, that is: analytical and critical thinking, knowledge about and understanding of intercultural and global topics, as well as openness and respect for cultural diversity.While global competence has only quite recently been a topic in academia and policymaking, numerous related concepts possess a considerably longer and broader research tradition: intercultural competence (Deardorff, 2006), global citizenship (Gundling, 2003;Oxfam Development Education Programme, 2006;UNESCO, 2014), intercultural communication (Chen and Starosta, 1996;Gudykunst and Mody, 2002) or intercultural sensitivity (Bennett, 1993).For example, intercultural competence is defined as an effective and adequate communication and behaviour in intercultural situations (Deardorff, 2006), and global citizenship (UNESCO, 2014) is defined as the knowledge, understanding, skills and values that students need in order to effectively engage for their own well-being and the well-being of others (Oxfam Development Education Programme, 2006).In the context of global learning, which is devoted to the principle of sustainable development, adolescents shall be equipped with competencies allowing them to meet their needs without putting the needs of future generations at risk (see Hauenschild and Bolscho, 2015;Schreiber and Schuler, 2005).Current research on global learning and education for sustainable development comprises, on the one hand, situational empirical studies focusing on educational opportunities for development in post-war societies (e.g.Krogull and Scheunpflug, 2016;Mojab, 2010;Ruggie, 2017).On the other hand, a number of studies exist that combine research and the practice of global learning and these contribute to the development of proficiency models (see Asbrand, 2008;Reddy, 2008).In particular, studies focusing on reconstructing how adolescents develop their mindset of orientations towards global questions identify important essentials (see Asbrand, 2014).Examples of approaches towards describing such competencies are Bögeholz's (2007) proposed competence of systematic decision-making in complex situations of sustainable development and Rempfler and Uphues's (2012) model describing the competence of understanding complex (global) systems.

Assessing global competence in PISA 2018
In addition to the three literacy domains regularly assessed in PISA (mathematics, reading and science), one cross-curricular competence per cycle is part of the study (see OECD, 2013aOECD, , 2013b)).This competence is not subject-specific and hence more comprehensive than the three standard domains.It yields results to questions such as 'to what extent are 15-year-old students able to plan their actions, deal with frustrations in learning or solve problems in teams with other students'.A central criterion in selecting the cross-curricular competence for an upcoming PISA cycle is that it provides information on the degree to which students are prepared for fully participating in their society and lifelong learning (OECD, 2005).Aspects such as team spirit or considering the consequences of one's own behaviour are discussed as central issues in this context.In the most recent rounds of PISA, in 2012 and 2015, the test mode was shifted from paper and pencil to computer-based (see Sälzer and Reiss, 2016;Heine et al., 2016), and another criterion for selecting the cross-curricular competence for future PISA cycles was introduced: it shall be innovative (OECD, n.d.).In preparation for the upcoming PISA cycle in 2018, global competence was selected with regard to the growing influence of globalization processes such as migration and the dynamic development of communication technologies in the contexts of learning, living and working.
In PISA, competences in each domain are assessed referring to an 'assessment framework'.This assessment framework is developed by international boards of experts and serves as the basis for constructing test units and context questionnaires, the latter capturing, among other things, family and school context, learning environment and the social background of students.Accordingly, an assessment framework for global competence was elaborated, which is introduced in the following paragraphs.Every assessment framework used in the context of PISA is developed along three criteria: (1) the construct of interest needs to be relevant for everyday situations; (2) it needs to represent a central domain-specific theory or a basic concept of the domain; and (3) it needs to be appropriate for capturing the developmental stage of 15-year-olds (OECD, 2016b).These three criteria are used to evaluate the assessment framework of global competence.Since this assessment framework had not been published at the time of writing this article, we refer to several unpublished pre-versions (i.e.draft frameworks), which were accessible to national project managers and members of the international PISA consortium in the course of preparing PISA 2018.
In its most recent publication on global competence, the OECD (2016a: 6) defines the construct as 'the capacity to analyse global and intercultural issues critically and from multiple perspectives, to understand how differences affect perceptions, judgments, and ideas of self and others, and to engage in open, appropriate and effective interactions with others from different backgrounds on the basis of a shared respect for human dignity'.Based on this definition, global competence is composed of such skills, attitudes and knowledge that enable a person to establish productive and respectful relationships with people from different cultures and to engage for just, peaceful, inclusive and sustainable societies.In PISA, this competence can be measured with respect to the state of the students' development and, hence, the extent to which they are prepared for acting in a globalized world with regard to their skills, attitudes and knowledge.
In each round of PISA, one of the three continuously assessed domains of reading, mathematics and science is the major domain.Accordingly, the majority of the test units used are part of this domain.The context questionnaires for students, school principals, parents and teachers contain a general part and a subject-specific part referring to the major domain.In PISA 2018, reading will be the major domain for the third time and a major part of the context questionnaires will refer to reading.Data analysis of the test units and the context questionnaires is usually carried out in separate procedures, so that results focus on cognitive aspects on the one hand (e.g. the average reading competence of students) and, on the other hand, describe related motivational constructs (e.g.students' self-concept in reading).With regard to global competence, at this point of preparing PISA 2018, two blocks of data collection are planned in relation to the major domain, that is, a battery of test units and a number of scales in the context questionnaires.
In its most recent publication on global competence, the OECD (2016a) distinguishes three dimensions of the construct: 1. knowledge and understanding of global issues as well as intercultural knowledge and understanding 2. skills, especially analytical and critical thinking 3. attitudes, e.g.openness, global-mindedness, responsibility.
The knowledge and understanding of global developments and challenges envisages, according to the OECD, students' familiarity with the most important issues that cut across national boundaries.Some examples are climate change, migration and poverty.Related contexts for assessing these issues are learning (education), work (occupation), life (social environment) and, in general, shared environments with other people who each bring in a different perspective.The second dimension, skills, refers to the students' readiness and aptitude to form and adapt their own communication and behaviour so that their interactions with others are adequate and effective.However, it remains open at this point how adequate and effective interactions are to be defined and, accordingly, assessed and compared, especially when other people bring in different perspectives and act differently in different contexts.The third dimension, attitudes, comprises, among other things, an understanding of new ideas and situations, of other people, but also of differing perspectives and backgrounds.In the assessment framework, OECD (2016c) emphasizes that the students' own thinking and acting needs to be adapted to the respective context, while these contexts refer to learning, working and living.
These three dimensions show several overlaps when composed to form the construct of global competence, reflecting the close interrelatedness of the dimensions and showing that the test units and questions capturing global competence are expected to measure several facets of the competence at once.When assessing global competence in PISA, analytical and critical thinking, as well as intercultural knowledge and understanding of global issues, are central (OECD, 2016a).However, at the time of writing this article, there is a mismatch between the description published in the OECD's latest publication on global competence and the latest, as yet unpublished, assessment framework.At this point (March 2017), the knowledge test is missing in the assessment framework and, instead, the student questionnaire comprises selfreported knowledge on global and intercultural topics.Other questions refer to the students' judgements of their own intercultural communication competence or their skill at adapting to different contexts or to take other people's perspectives.Also, the student questionnaire is designed to capture attitudes such as global-mindedness (see OECD, 2016c) and openness and respect towards people from different cultures.
Based on this theoretical foundation for assessing global competence in international large-scale student assessments like PISA, some related challenges are elaborated in the following section.

Challenges related to assessing global competence in the context of PISA
The relevance of intercultural sensibility and the readiness to tackle the challenges of a globalizing world is undoubted.However, it is questionable whether this construct of global competence, in its current state of development, is applicable to describing and empirically measuring the degree of students' preparedness for such challenges.In the following, we elaborate some specific challenges of assessing global competence and hereby differentiate between challenges regarding the character of the construct and those regarding the specific approach of test preparation in PISA.We refer to the OECD's (2016b) three criteria for developing and reviewing the theoretical frameworks in PISA: (1) relevance for everyday situations; (2) representation of an important theory related to the domain; and (3) adequacy for capturing the developmental stage of 15-year-old students.

Challenges with regard to the character of the construct of global competence
Global competence as a complex, multidimensional construct The construct of global competence, which is projected to be assessed as the cross-curricular competence in PISA 2018, is a hybrid of cognitive components and personal attitudes of the assessed students.Much clearer than in the three standard domains of PISA -reading, mathematics and science -global competence is meant to encompass personal characteristics captured via students' self-reports in the student questionnaire.At first sight, this approach seems reasonable with regard to the definition of global competence as a competence consisting of skills, knowledge and attitudes.Nevertheless, this aspect bears a first challenge.The interrelatedness of cognitive components (ideally assessing knowledge and skills) with attitudes and personal preferences is multidimensional in a highly complex way.In international comparative studies, such challenges due to complexity are usually solved by consensus (see Sälzer and Prenzel, 2013).Through discourse, common ground is sought and mostly found, meaning that the participating countries negotiate on the topics that are relevant and meaningful for a majority of the countries and adequate for the assessed cohort of students.These topics should be close to the curriculum and relevant for everyday situations.According to the complexity and the state of theory-building of a construct, negotiating to find common ground takes place with a strong reference to scientific standards.In contrast to the closely subject-related PISA domains of reading, mathematics and science, which are built on an international, research-based consensus on the contents of these literacies, global competence still lacks a consensus on an overarching definition.The definition formulated in the assessment framework may serve as a valid starting point for further differentiating and structuring the domain of global competence, but it does not solve the problem of finding a minimal consensus for the kind of content that can be assessed in an international comparison through test units and questionnaires.This aspect is related to the second criterion for theoretical frameworks published by the OECD (2016b), namely that an important theory or a central concept of the respective domain needs to be represented in the framework.For global competence, such a selection has yet to be made.The complexity of the construct can thus be considerably reduced and made accessible for operationalizing.

Construct validity
One way of establishing a test's validity is to determine the degree of congruence between a construct measured by the test and existing definitions of the construct and related theories.This aspect should also be oriented toward the criterion of representing an important theory of the domain (OECD, 2016b) in an assessment framework.The development of a test intended to be used in an extensive setting like PISA requires substantial theoretical work.For instance, it does not suffice to only partially cover the construct of interest.Valid measurement requires an exhaustive assessment of the concept (Lienert and Raatz, 1998).In order to be able to judge this exhaustiveness, a very precise theoretical framing and definition of a domain (here: global competence) is necessary.Compared with the degree of sophistication found in the domains of reading, mathematics and science in PISA (OECD, 2016b), the current assessment framework for global competence fails at least at the point where correct responses are distinguished from incorrect responses (see next passage).This distinction holds another challenge, that is, normative decisions when separating correct from incorrect responses.

Separating correct from incorrect responses
Whenever a competence test is used, a clear definition is needed to distinguish correct from incorrect responses, as well as when a student will be given credit for his or her response.According to this, one crucial element of preparing a PISA cycle is to establish coding guides when developing an assessment framework and test units.As much as the assessment framework itself, this aspect is tied to the criterion of representing a central theory for describing a competence domain in PISA (OECD, 2016b).The coding guides are, for example, related to the difficulty of a test item for the 15-year-old students assessed and how well the level of expectations for a correct response can be estimated.It may be well recognized that such classifications are more difficult for a construct like global competence than for purely science-based domains (such as mathematics or science) and cannot be derived exclusively from general or educational theories.It may even be necessary to work with normative statements in coding guides, for example with regard to political stakeholders' demands (see, for example, Rost, 2005).On the other hand, this means that classifying a student's response as 'correct', 'incorrect' or 'credit' versus 'no credit' has to be evaluated by the extent to which an individual is ready to engage in solving global challenges, both as an individual and in cooperation with people from diverse nationalities and cultures.Since the appraisal of a response as 'correct' or 'incorrect' is not self-evident and will in many cases be difficult, it is crucial to create transparency and replicability.For some of the subdimensions, this could imply refraining from cognitive assessment and, hence, from describing stages of higher or lower proficiency.For many of the test units developed to assess global competence and proposed by the OECD, however, it was not even obvious to National Project Managers how the degree of correctness of responses was to be derived and how correct responses differed from less correct or wrong ones.Delimiting correct from incorrect responses is, accordingly, strongly dependent on the test-takers' presuppositions, so that a consensus seems to be inconceivable at this point.Therefore, the OECD's recent decision to focus exclusively on cognitive components of global competence is welcomed, namely critical and analytical thinking with regard to global topics, as well as perspective-taking in global and intercultural situations.Nonetheless, the OECD (2016a) concedes that the test units and related correct and incorrect responses are still often not independent from a given cultural context.Aspects of global competence related to students' attitudes and motivation, as well as information on their learning environment and social conditions of global competence, are still intended to be captured by the context questionnaires.

Western bias and intercultural comparability
The meaning of content, situations and concepts within each domain tested in PISA can vary across countries and cultures.This potentially exacerbates the difficulty in fulfilling the OECD's (2016b) criterion that PISA items have to be relevant to the assessed students' everyday life.
Existing conceptualizations of global competence and related constructs have mostly been studied in the so-called Western context (see Deardorff, 2011).Although the idea of international comparability has become a routine concern in the context of PISA and is continuously considered by the expert groups, with regard to the currently existing draft test units, a Western bias of the developed instruments has to be assumed (see also the discussion in the next section on stereotypes in the proposed scenarios used).

Challenges with regard to the approach taken by PISA
Along with the construct-related challenges mentioned above, several difficulties need to be considered that are specifically due to the approach taken in the context of PISA.In this section, the OECD's criteria (2016b) for the assessment frameworks are relevant and will be discussed at the appropriate places in the text.

Assessment frameworks require time, discourse and consensus
The cognitive tests and questionnaires administered in PISA are based on assessment frameworks established by expert groups (see OECD, 2013aOECD, , 2016b)).Such frameworks have successfully been implemented in six completed rounds of assessment so far and they have contributed to valid, theoretically founded and easily interpretable measurements.This procedure is well suited for assessing established constructs with a longstanding and sound basis in research, such as reading.However, as soon as a topic is to be assessed that lacks international comparative research, this approach falls short, since it is impossible to refer to a theoretical base.Currently, in-depth, systematic theoretical work on the construct of global competence is lacking.This should have preceded the developmental work that is commonly practised in the PISA context (see also section entitled Outlook below).The assessment framework for global competence (OECD, 2016c), which was established over only a few months, could not compensate for a lack of thorough groundwork on the concept.The criterion of selecting central important theories and concepts for each domain in PISA (OECD, 2016b) is therefore only insufficiently met for global competence.This is also reflected in the fact that, at different stages of development in which the framework was made accessible to those in charge of running the PISA study in the different countries, the definition and elements of global competence varied fundamentally.As a result, the framework gave the impression of being rather arbitrarily designed and the inadequacy of a theoretical consolidation of the construct became apparent.

Stereotypical nature of the proposed scenarios
In order to meet the demand of appropriacy regarding the developmental state of 15-year-old students (the OECD's third criterion, see OECD, 2016b), the scenarios described in the test units have to be both suitable for this target group and authentic.A scenario outlines the setting or the context of a test unit and thus introduces the students to the test item.To exemplify this, a test unit that has since been eliminated is described in detail here: 1 The PISA student is asked to put himself or herself in a situation of being part of a project team, together with two exchange students from abroad.The PISA student is waiting for the two fellow students who are running late and is wondering [using set texts] why they may be late.The student is asked to select from pre-formulated responses such as 'People in country XY do not value meeting on time' or 'People in country XY seem to be known for not keeping agreements'.
In the context of preparing PISA 2018, the one-sidedness or bias of the proposed scenarios has often been criticized.During the first draft stages, many of the scenarios were placed in a fictitious chat-situation in order to establish global connectedness as the context of the assessment.With regard to the content, the frequent use of stereotypes was criticized.This means that the texts given to the students to choose from in the test scenario described above contained mostly judgemental, highly generalizing and stereotypical assumptions on work ethics in the countries of origin of the two exchange students.Accordingly, many PISA participants complained that the scenarios and the wording of numerous questions and response options were not appropriate for their national context.At the time of writing this article, this feedback from the participating countries has been accommodated for to some extent by the test developers and some scenarios have been excluded from the test.

Clear expectation of socially desirable responses
Another problem related to the test development of PISA 2018 is that the scenarios proposed have clearly suggested socially desirable reactions (according to Western standards) to students through the stereotyped content and response options.While it was clear in these cases which responses would be classified as correct and incorrect, one would have assessed the students' competence to discover socially desired responses rather than global competence.Keeping this in mind, the construct validity is questionable also.Ideally, this aspect should meet the OECD's (2016b) criterion of a solid theoretical basis, but given the current description of the construct it is not yet consolidated enough.
In summation, many countries participating in PISA 2018 have refrained from participating in the assessment of global competence owing to these currently insufficiently solved challenges.First of all, this expresses their criticism of insufficient validation, theory-building and the lack of intercultural comparability.For another, this decision was partly grounded in a concern that an insufficiently validated, and hence not suitable instrument, would be used to rank educational systems according to the 'global competence' of their students.At this point, it is important to know that the challenges mentioned are not unresolvable.Being able to act in international, global contexts, understanding heterogeneity and different perspectives as something natural and being able to take other people's perspectives are fundamental capabilities for future (world) citizens.Therefore, we would like to emphasize that assessing global competence is a sensible project and should be further developed.Preparatory work on such an assessment described here in the context of PISA should be understood as an example that teaches us how complex the development of a dimensioned assessment framework is.Such a framework needs to be used to operationalize a construct that is not supported by curricula that have evolved over a period of decades or other scientific publications documenting the process of theory-building.Even more than the cross-curricular competences captured in PISA so far, namely self-regulation (PISA 2000), problem-solving (PISA 2003, 2012) and collaborative problem-solving (PISA 2015), global competence represents a topic with a dynamic that can hardly be split into subdimensions, and it is strongly dependent on the respective cultural context.We conclude our article in the following section by presenting an outlook on how global competence could be assessed and measured by means of a large-scale assessment in the future.

Outlook: How could global competence be assessed in an international comparison?
To conclude, we take up the challenges described above and forecast how an adequate -that is, valid, reliable and interculturally invariant -assessment of global competence can succeed.
First, sound and comprehensive theoretical work on the construct of global competence and its foundations is needed.Such work requires time and space for discourse among experts and National Project Managers.Existing conceptualizations of global competence need to be systematically compared and examined for their definitions and understanding of globalization.It can be assumed that existing conceptualizations like the ones endorsed by the OECD (2016a) and UNESCO ( 2014) are based on fundamentally different ideas of how to define and implement globalization (see Grotlüschen, 2014).For example, is it crucial to work towards a more equitable globalization or is the main point to reach and maintain international competitiveness?Consequently, it seems only logical that apparently similar approaches will lead to quite different test instruments.In the context of preparing PISA 2018, the theoretical basis of global competence was, at least for the National Project Managers, neither transparent nor replicable when reviewing the assessment framework and draft test units.For quite a long time, it seemed to be enough for the assessment framework to emphasize the relevance of global competence, a point that is undisputed.However, problems emerged at the stage of translating the construct into test units.Presumably, this lack of transparency with regard to the theoretical foundation added to the fact that in PISA 2018 numerous participating countries had reservations about assessing global competence, since they suspected a hidden agenda.While in the assessment framework of global competence (OECD, 2016a), intercultural understanding, respect and engagement for the common good are listed, countries participating in PISA 2018 may be wondering whether the results of assessing this cross-curricular domain would be used to extrapolate who is likely to succeed in the international and competitive labour market.
Besides the definition of globalization as a reference for an approach to describe global competence, a number of open questions remain to be tackled in a systematic analysis comprising several steps: what are specific characteristics of globalization?To what extent can the Grand Societal Challenges expressed by the European Commission (2015) be broken down into topics that meet the OECD's (2016b) criteria for an assessment framework?Which personal characteristics, skills, and dispositions does an individual need to tackle the challenges listed, to address them together with others?Which of these dispositions are accessible by education and can, thus, be learned and improved?Only when these questions have been carefully addressed, can other steps follow to develop instruments for assessment and scaffolding (Reeff, 2014).
Not every assessment context requires an interculturally comparative, and hence interculturally invariant, measurement of global competence.When this is the case, as in PISA 2018, it is essential to integrate both Western and non-Western perspectives on the construct such as, for example, the South African concept of Ubuntu (see Nwosu, 2009).Establishment of an interculturally comparable measurement will remain a very challenging task and there is a need for research and discourse on the question of whether this can be eventually achieved for all components of global competence.In a first step, it is necessary to clarify which preconditions or consequences of globalization are universal and which depend on cultural or geographical context.Then, universal and interculturally comparable competencies can possibly be derived.However, it needs to be pointed out that large-scale student assessments, especially with an international perspective, are inadequate test instruments for some aspects of global competence.
The problem of cultural variance in the sense of international comparability could alternatively be tackled by a test format that puts the PISA student in the role of an observer who watches video sequences and is then asked to judge different situations.The switch from paper-based to computer-based assessment since PISA 2015 offers an ideal environment for such an approach.One example could be observing a typical conflict in a classroom.The PISA student can show his or her ability to take the perspective of others by summarizing the arguments of the conflicting parties.At the same time, this kind of task provides a frame of reference for 'correct' and 'incorrect' responses.The classification of student responses, however, does not have to be limited to the categories of 'correct' and 'incorrect', especially in a domain like global competence.An alternative approach to accrediting students' responses could be to assign them scores for subcompetencies assumed to underlie their responses and sum these scores up to a scale ('credits').For instance, a PISA student could in such a case receive scores for perspective-taking, for incorporating arguments put forward by others, for integrating different opinions of roles of actors, and for explicit reasoning and drawing conclusions (see Gehlbach, 2004).
Even though the everyday lives of 15-year-old students and the educational systems in the more than 70 countries currently participating in PISA varies considerably, a consensus has yet always been achieved on which content, processes and contexts are considered relevant for this cohort (see Sälzer and Reiss, 2016).Compromises have International Journal of Development Education and Global Learning 10 (1) 2018 to be made, the view of the majority incorporated and different aspects negotiated.With regard to the relevance of global competence for the everyday lives of 15-yearolds, one first step in finding such a consensus is to agree that the related topics of global challenges are meaningful for this cohort and their lives.Even though the PISA students are not full citizens yet, for example they cannot drive or vote, the ideas behind those challenges can be integrated in an assessment framework for 15-yearolds.For example, PISA does not intend to measure higher mathematics, but basic competences are integrated and defined in the concept of mathematical literacy, and the same could be done with global competence.The topics related to global competence contain big challenges for the global society and are thus highly relevant for 15-year-olds' everyday lives as future adults and citizens.For them, growing up in a globalized and globalizing world and being confronted with global challenges is normal.Accordingly, mastering and shaping global challenges is especially relevant for young people, since they will have to live with today's decisions in the future (BMFSFJ, 2017).In this regard, it can be useful to focus on topics that are within practical reach for the PISA students, such as, for example, individual consumer behaviour (see literature on 'Consumer Citizenship' such as Thoresen, 2005).
To conclude, we summarize that the OECD's intention to assess global competence as an innovative and cross-curricular domain in PISA 2018 is worthwhile and meaningful.Global competence is a concept with manifold aspects and dimensions that are desirable characteristics of global citizens not only in tomorrow's world, but already today.The aspects we have critiqued regarding the current content and process of developing the concept and test units for global competence show that assessing global competence by means of an international educational comparison may be a huge task, but not an unresolvable one.Mastery of such a task requires common international efforts in developing and delimiting the theoretical basis as well as in the discourse for consensus on which content is to be selected.Another hurdle is the step from the theory to the test units, which needs to consider a number of perspectives and cultural contexts, in other words global competence.In our roles as part of this process, we draw the conclusion that this task can very well be mastered, if an adequate amount of time is taken, the process is transparent and open for criticism from all participants from the beginning, and if the board of experts is only satisfied when PISA's established quality standards for assessment frameworks and test units are met.Concretely, this could be done by selecting theoretical models according to defined criteria.Ideally, such models have already been empirically verified and modified accordingly.Building on this, constructs and interrelations contained in the models can be described so that potentially existing test instruments can be identified for an empirical study.If such instruments have to be modified or newly developed, an adequate amount of time needs to be granted so that experts can involve themselves in a thorough discourse not only on content, but also on assessment.In practice, this can be done by integrating and classifying existing theoretical and empirical studies from different traditions of research, such as quantitative and qualitative studies, which have at this point been left aside within the process of test and framework development of global competence in PISA.For PISA 2018, this chance has passed, but we are optimistic for PISA 2021.

Notes on the contributors
Christine Sälzer obtained her PhD in education and is currently working in educational monitoring and teacher education.Christine studied in Germany (Braunschweig),