Can teaching philosophy in schools count towards the Research Excellence Framework (UK)?

Even though critical thinking is carried out in schools across the country, there is no attempt to take high-level published philosophical research into schools. This paper reports on a pilot where this was attempted. The findings suggest that this can be done successfully and moreover that the results can be contextualised so as to count towards “impact” in the Research Excellence Framework (UK). The paper suggests that this means philosophers have not only a way of developing children’s confidence and reasoning, but also a way of meeting the requirements on Higher Education Institutions within the UK. Subjects: Children & Childhood; Philosophy; Philosophy of Education


Introduction
Every 6-7 years, all subject areas at all Higher Education Institutions (HEIs) in the UK undergo an assessment of their research, via formal methods, to determine their research funding for the next 6-7-year period. This began, in 2001, with the RAE. There has been a recent, very substantial, change to the formal methods by which subject areas will be tested. That is, the decision to include an "impact" component in REF2014. The rough idea is that research carried out at HEIs should, in at least some cases, lead to some positive social or economic benefit to non-academic partners. How much benefit research within subject areas has generated will play a role in determining how much research funding they receive.
It is no part of this paper to discuss this revision. Instead, it offers a critical reflection of some work carried out within one very specific discipline (philosophy) and discusses how this work might constitute "impact".
ABOUT THE AUTHORS Andrew Fisher has written and published on metaethics, philosophy of religion, and the philosophy of education. He has published a number of books including a book on pedagogy in schools and has been teaching philosophy in schools for many years.
Jonathan Tallant has written and published numerous articles on metaphysics and the philosophy of time. He has published a book on metaphysics. He oversaw the Research Excellence Framework submission for the philosophy department and has taught philosophy in schools over a number of years.

PUBLIC INTEREST STATEMENT
Why should anyone outside academia care what philosophers write about? Can philosophy be of any public value? In this paper, we show that the philosophy research done at Universities can have a direct impact on the social, emotional and academic lives of children and young adults; far from merely having value in its own right, academic philosophy can enrich the wider community. Moreover, we show how such enrichment can be contextualised so as to be an attractive proposition to UK philosophy departments. Namely, such work can be successfully labelled "impact", and hence count as part of the national Research Excellence Framework.

What is "impact"?
Paragraph 25b of the Assessment framework and guidance on submissions (REF2014, 2012 gives an outline statement: The sub-panels will assess the "reach and significance" of impacts on the economy, society and/or culture that were underpinned by excellent research conducted in the submitted unit, as well as the submitted unit's approach to enabling impact from its research. (www.ref. ac.uk) Thus, each unit will be required to show how research produced within the unit has led to impacts on the economy, society and/or culture. Paragraph 141 outlines the kinds of changes that will be considered to be "impact" for the purposes of the assessment exercise.
Impact includes, but is not limited to, an effect on, change or benefit to: • an activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding • an audience, beneficiary, community, constituency, organisation or individuals • any geographic location whether locally, regionally, nationally or internationally.
Philosophy (not uniquely amongst the Arts and Humanities) faces challenges in demonstrating impact. A good deal of research carried out in philosophy is not driven to a practical end. For instance, it's not unusual to see research carried out on the question of whether there exist abstract objects-objects that are characterised as existing outside space and time and being incapable of causally interacting with any physical objects. It's very hard to see how this kind of research could lead to impact; for example, how might it lead to a change in activity for a company in the FTSE100 or improve health care policy in a particular region.
Against this backdrop, however, there is a move to include philosophy in the classroom, to bring about benefits to young children and there is growing empirical research considering these benefits 1 . There has also been some good theoretical work about the nature of philosophy in schools (e.g. Hand & Winstanley, 2009). Some of which argued that philosophy gives a significant/highly significant improvement on mathematical and reading ability (Lane & Lane, 1986). Others have talked favourably about the social and moral benefits of philosophy.
The Inclusion of philosophy in the curriculum directly impacts on the development of pupil's moral and social development as well as enhancing their capacity to become independent learners. (Fisher, 1999, p. 51) Research found that philosophy helped the social and cognitive capacities and adaptability of children and increased children's ability to relate and adapt at home and in other social settings (Doherr, 2000). Research has also shown that philosophy can raise a child's IQ by 6.5. And a longitudinal study on the topic found that: … a weekly philosophical collaborative inquiry intervention in primary schools led to prepost gain in cognitive ability which was maintained two years later on the same measure. (Topping & Trickey, 2007, p. 795) What these studies have in common is that: none of them deploy contemporary philosophical research to try to bring about the benefits described. None of these studies can then be deemed "impact" in the terms outlined by REF.
In fact, the standard model deployed by those engaged in what they call "Philosophy for Children" (P4C) is to provide space for children to talk openly and critically. Consequently, it would probably be more accurate to label it CT4C Critical Thinking for Children rather than P4C.
However, the question that interested us was whether we could draw upon published research in high-ranking philosophy journals, done by professional philosophers at a research-intensive University, to bring about an impact in a range of children's lives? Was there something inherently problematic in this proposal? Or are the supposed problems simply pragmatic?
We chose research published on: (i) moral reasoning; (ii) sport and (iii) character. Our thought was that if we could improve the confidence that children have of their own reasoning powers, then we would have clearly demonstrated a positive change in the attitude of an important audience; an audience that is a key part of any local community.
In this paper, we describe our efforts, the outcomes that resulted and make some general observations and recommendations. Section 2 describes the design of the workshops in detail. In particular, we outline how the abstract research was transformed into a lesson plan; how the workshop was run; what types of questions were asked of students. In Section 2, we also outline the data probes that were deployed and an account of the various different schools that we worked with. It also provides some background information about the socio-economic context. Section 3 outlines the data that was gathered from the work in Primary and Secondary Schools and provides some contextual information concerning why this work arose. In Section 4, we discuss the findings.
Section 5 concludes that there were demonstrable impacts to be had from engaging young children with the research in both the children's perceived ability and in terms of social impact. We recommend Philosophy Departments and units from other academic disciplines consider drawing on research and using existing connections with Schools to develop a route to impact that actively engages young children.

Workshops: background and design
Because we set out to engage young children, primarily in a classroom setting, we had to be mindful of a number of factors. For instance: we could not assume any prior knowledge of philosophy; we had to assume that we would be working with a wide range of abilities; we had to avoid complex language. Importantly, we also had to be selective in what kind of philosophical research we chose to work with. As noted, some branches of philosophical research are extremely esoteric.
Although we are confident that such material could be presented to young children in an interesting way-and in fact, we have done so, e.g. when we presented the special composition to nine-and ten-year-olds (see Note 1)-what concerned us in this project was whether or not we could make a positive difference to the way that children live their lives. Could we enable them to use philosophical research to better navigate the world around them? To this end, research that had a more obviously direct practical impact was chosen.
The main area in which we were working is the area known as Nottingham West (UK). In discussion with the local council (who engaged with the project a little later on), we were advised that the local socio-economic situation is as follows: West Area in Nottingham City comprises three wards: Aspley, Bilborough and Leen Valley, with a total population of 42,000. A significant percentage of the population is made up of young people under the age of 20 (28%, compared to a city average of 24%). The West area is predominantly White British (77%). There are approximately equal number of owner occupiers (47.9%) and rented housing (52.1%), although a higher proportion than the city average rents from the Council. 45.7% of the West Area population have no qualifications and within some neighbourhoods only 2% progress to higher-level qualifications. 40% of pupils living in Bilborough ward achieved 5 A*-C GCSEs including English and Maths in 2012. The number of people achieving level 4/5 is 8.8% compared to the city average of 17.6%.
At schools in the area, the percentage of pupils identified with special educational needs and/or disabilities is above the national average, as is the percentage of pupils who have a statement of special educational needs. The proportion of pupils known to be eligible for free school meals is also well above average.
Children enter school at levels well below the national average and some have very low speaking and listening skills. They also lack confidence and have low emotional wellbeing, which impacts on their aspirations and attainment throughout their education. (Correspondence with Charlotte Malik, Nottingham City Council) Previous experience of working in a range of local primary schools, alongside teachers and pupils, gave Fisher a very specific insight into the kinds of ways in which philosophical research might be used for the betterment of children's lives. In particular (and picking up on some of the themes from the cited correspondence), it seemed striking to Fisher (after numerous conversations with staff and pupils) that low emotional well-being was negatively impacting upon student performance.
In many ways, this link with emotional well-being and socio-economic status (SES) is to be expected. There is a well-researched "socio-economic gradient" or "socio-economic" gap relating socio-economic status to attainment in school (Perry & McConney, 2010;Sirin, 2005;White, 1982).
There is also growing evidence to show the gap remains fairly stable from the age of 7-11 years and widens at an increasing rate from the age of 11-15 years (Caro, McDonald, & Willms, 2009). And a recent meta-analysis of 30 research articles underlines this link (Van Ewijk & Sleegers, 2010).
Moreover, children from socio-economically disadvantaged families, rather than children visiting schools in disadvantaged neighbourhoods, have a particularly high risk of involvement in bullying, which in turn has a direct effect on well-being and the student's experience of school and their expectations whilst in school (Keung, 2011).
Equally, the problem is compounded by the fact that parents of children with low SES have a direct effect on their children's attainment and aspirations at school (Barnard, 2004). And that SES has a measurable effect right through to Higher Education (Sewell & Shah, 1967). The effects of SES are also wider than on academic attainment, and have a direct impact on a much broader set of cognitive abilities (Duncan & Magnuson, 2012).
Against that background, Fisher and Tallant set out to design workshops that would raise the student's confidence in their abilities in specific areas. Obviously, given the close connection between academic attainment in specific areas and confidence, it made sense to pursue a policy of looking for areas in which we could contribute to supporting pupil's progress through the curriculum.
Fisher and Tallant were acutely aware that the question "what is confidence?" is well researched and tough one to answer; wedded to sophisticated issues in psychology, sociology and educational theory. And a full discussion of the nature of this is beyond the scope of the paper. After reading the literature cited above, we were convinced that there was a meaningful notion of "confidence" that could be employed in our research context. We encourage the reader who wants to think harder about the notion of confidence to turn to some of the literature cited above.
However, as well as looking to support subject-specific improvements, we wanted to try to contribute to the pupils' development in a more holistic way; in fact, such an approach had the potential to have the maximum effect as it is about the student's self-actualisation, and hence is an enabling condition for all other areas (there are obvious advantages then to reconceptualise this approach within a humanistic learning theory (e.g. Maslow, 1968), however, such a task is beyond the scope of this paper). As such, we wanted to try to contribute to their aspirations; their goals; their moral developments.
In such areas, it's notoriously hard to measure success-particularly in the short term (see, e.g. Kleitman, Stankov, Allwood, Young, & Mak, 2013). However, we took the view that just as confidence in specific areas of learning has been demonstrated to lead to ability, using philosophical research to improve children's confidence to think for themselves about, e.g. moral issues, would help them to think through morally pressing issues more clearly. As Cam writes relating the importance of teaching philosophy to children: Whatever else we do by way of values education, we must make strenuous efforts to cultivate good judgement. When it comes to deciding what to do in a morally troubling situation, good judgment involves distinguishing more from less acceptable decisions and conduct. (Cam, 2013, p. 6) Tallant conducted a careful study of research carried out in their home Department over the last 15 years with a view to picking out items that could usefully be packaged into a short workshop structure, with a view to identifying specific tranches of research that could underpin sessions intended to either help pupils improve their performance in specific subject areas or else help students to better navigate the world around them by improving their confidence in their reasoning.
With that in mind, Tallant identified three strands of research that would be of use. We briefly describe these items of research, here, and provide reference to the research itself for the reader to pursue at their own leisure. We comment, below, on why these specific areas were chosen and how this fits in with the strategy already described. Leibowitz (2013) argues in favour of a new interpretation of Aristotle's view of morality. He argues that Aristotle's view is best read as particularist-the view that we explain moral phenomena without appeal to exceptionless principles and that we may need different explanations of (e.g.) rightness in different contexts. Leibowitz's interpretation of Aristotle is that: (1) we are already aware of which acts are right and which are wrong, (2) Aristotle's goal is to teach his students why right actions are right; that the practical goal of Aristotle's writings was to help his students perform actions known to be right in a virtuous manner (performed for the right reasons; from a stable moral character and so on). Finally, (3) Leibowitz suggests that this Aristotelian view of the result is itself a plausible moral theory. Mumford (2011) offers an in-depth discussion of the nature of what it is like to watch sport. In particular, he argues that sports people should not be role models to those who watch sport through a three-step argument. (1) Role models are a bad way of imparting moral lessons. (2) The status of role model is unreasonably conferred by others on to the athlete whether they want it or not. (3) The role of role model is too demanding for anyone to fulfil, particularly the sportsperson who is engaged in a very specific type of contest. Mumford further argues that it is intrinsic to sports, unlike games, that we have a contest, and that the explanation for our interest in watching sport is, in large part, due to the emotional high that we derive from these contests and our involvement in these narratives. We took the view that this research would be particularly accessible to those students with a prior interest in sport and watching sport. Currie (2009Currie ( , 2010 argues that character may add to the coherence of narrative by enriching the connections between events. Currie reviews the literature from empirical psychology, noting two potentially troubling findings for his thesis: there is experimental evidence that we tend to think that people's behaviour justifies attributions of character when it does not; that people behave in ways that are strongly determined by circumstances. Both tranches of evidence suggest that real people's character is far less significant than one might imagine to how they behave in the world. In turn, that threatens the suggestion that character adds to narrative: if there is no such thing as genuine character, then it is hard to see what value fictional character may have in narrative. Against this threat, Currie argues that we should view character not so much as a psychological-explanatory tool, but as a device for making vivid and coherent the interplay of other, psychologically real factors. Thus, Currie, suggests, character adds a great deal to narrative, even in the absence of a kind of realism about character in the world itself. Fictional characters need not be at all like real people in order to play the role required of them in successful narrative.

Narrative
Our reasons for including Leibowitz's research are relatively obvious. As cited above, research shows a link between thinking morally and acting in a moral way (e.g. Cam, 2013). In reference to the Aristotelian theme Cigman captures the idea well: … [there is an] ethical territory in which children have a great deal to learn. They do not enter this territory by being encouraged to hope or be happy grandly and indiscriminately. According to Aristotle, they need wise guides to cultivate their emotions, keep their infantile excesses in check, engender good emotional habits and gradually help them to understand why certain responses are deficient or excessive, why this response is more appropriate than that in this situation, and help them to take pleasure in feeling, thinking and acting well. So develops the practical reason that is at the heart of the ethical life, essential, in Aristotle's view, for well-being. (Cigman, 2012, p. 452) We hoped that by exposing the pupils to this research, we would be able to improve their confidence in moral reasoning. Specifically, we could help them see that it is not a failure to be unable to apply exceptionless moral principles. But rather, by exposing them to examples that required contrary moral principles, we would help them see that they can come to challenge the very existence of exceptionless moral principles. Indeed, the particularist approach to ethics advocated by Leibowitz lends itself especially well to discussion in the classroom because of its emphasis on thinking about contextual features, rather than moral rules. Leibowitz's research makes it clear how genuine philosophical discussion of moral problems could take place without prior knowledge or understanding of the moral theories advocated by moral generalists. In reflecting on contextual factors, pupils would be reflecting on the very considerations that are, according to Leibowitz's research, central to moral thinking. We hoped then that Leibowitz's work might give the children confidence to develop their moral reasoning in this way.
Mumford's research offered (at least) two valuable ideas. The first is that we should not treat sports people as role models-there is already research in other areas underpinning such issues, e.g. Lines, 2001. This insight could be very valuable to pupils who often look up to prominent sportswomen and men who are in the public eye. The research would help the children move away from sports men and women as unattainable exemplars, and instead build up confidence in their own abilities.
The second is about building the children's confidence in their metacognitive abilities. Why for instance, do a majority of them watch sporting programmes? Do they know, have they ever reflected on the decisions they make regarding watching sport? Indeed, what is distinctive about sports in general? In getting better at recognising and commenting on their own choices, we believe that the children would become more confident in their own ability to defend the decisions they take and the actions they perform.
Currie's research also offers two insights that we thought could contribute to the ways in which children live their lives by increasing their confidence. The first is giving them some insight into the way in which professional writers construct characters (these characters are, to put it crudely, larger than life).
The second is to come to the realisation that (real) character plays less of a role in determining our actions than one might naturally suppose. Children often come to be dominated by stereotypes and labels based on their characters that are supposed to be dictating their actions (e.g. Crocker & Major, 1989).
Currie's research would help them to see that their futures aren't in this way predetermined. The child who moves beyond simple judgements, based on character traits, would-we hoped-become more confident in how they see themselves in relation to others; and relatedly be less likely to blame individuals for their actions and more inclined to look at more nuanced and "grown up" explanation for actions, e.g. to environmental explanations, for the action in question. We were not able to test for this outcome, sadly.
It is one thing identifying the complex research which we believed would be suitable for the classroom, and it is another actually transforming the research into lesson plans for young children. Our process for doing this was relatively straightforward. Primarily, it involved talking to Leibowitz, Mumford and Currie. By explaining the project, including the structure of a lesson plan, the key lessons of the research were identified and these were then taken as the "learning objectives". The activities that surrounded these objectives where then developed using the experience of Fisher in discussion with Tallant. To check whether we were along the right lines, we then sent the lesson plans to the contact teachers who verified that they were realistic in terms of expectations, activities and timings.
The workshops were run within school time, and hence were compulsory. Through running many workshops, we were clear that a didactic lecture style would not work with the children. Instead, a more discursive approach was adopted. The general format was as follows.
First, we started with a game and then introduced a stimulus related to the topic-e.g. a video of a sportsperson winning a race, reading out a short passage from Harry Potter, a picture of someone doing something wrong, etc. We then facilitated a structured discussion in small groups relating to the stimulus; then we facilitated a general feedback session where each group shared their views; then a more focused "deeper" philosophical question was presented to the young people. For example, we might say: "in the last discussion the majority of you said that Messi was your favourite role model, in small groups now think about what qualities might be good -and bad -in a role model". We were prescriptive in setting up the discussion groups, allocating one scribe and explaining that the group had to come up with as much agreement as possible. We also found we needed to be very clear with timings and made sure the time and nature of the task was reinforced throughout. As was highlighted, the workshop designs were based on experience gained by running similar sessions and through discussions with teachers.

Primary school
At the end of each philosophy session, children were presented with a questionnaire, where (for the most part) the questions asked whether or not they felt that their confidence had improved with regards some specific topic. We chose to focus (largely, though not exclusively) upon improvements to confidence-even with the caveats noted above. Because sessions were run a varying number of times, we generated different quantities of feedback for the different sessions (and some questionnaires were spoiled). There were a number of different sessions run on sport, with different questionnaires. These were developed, in part because of the popularity of the topic, but also because the research has an obvious interest to young children with a pre-existing interest in sport.
After discussion, we decided not to run a "pre-test" 2 . A "pre-test"-to test the student's confidence in the area that we were going to test-would have produced a "bench mark" in virtue of which we might contextualise our results. However, we had three reasons for not running such a test. First, pragmatically, there was a very tight timetable which didn't allow such a test. Second, we felt that the results of the test would be threatened because in order for the young people to answer the questions, they would have needed to have some discussion of the issues. For example, to answer the question: "how confident are you in making moral decisions?" requires the young people to be able to make sense of "moral decisions"-which itself is part of one of the sessions. Third, and most important, we felt it was acceptable to take the children's own claim that their confidence has improved as true. This meant that the use of an objective benchmark would become redundant (Table 1).
We received a number of positive comments on the sessions we ran in Schools from teachers. For instance, of the sessions on character and sport, one teacher remarked: People believe that philosophy is a grown up subject, well our children had better questions and answers than most adults I know!… Both work-shops left the children with lasting and very positive memories. (Becky Lord, Brocklewood Primary School) In addition, the work on character, … not only developed my children's understanding of character and story writing but, most importantly, instilled in them a love of literature and a desire to read (Rebecca Sneath, Seagrave Primary School) We take the mixture of quantitative and qualitative data to be positive and to give a sense of the range of positive benefits that the project produced.

Secondary school
In addition to working with Primary Schools, we took up the opportunity to work with a local Secondary School: the Nottingham University Samworth Aacademy (NUSA). We ran two sets of workshop. Both-like the first-were compulsory and within school time. In the first, we focused on working with the top-set year-11 English class on themes from Leibowitz research on metaethics. The second workshop was effectively the same session on character that we ran with the primary school children, but with a year-10 English class. Here, rather than focus upon confidence, we invited the pupils to self-assess with regard to the knowledge and skills that they acquired. Our assumption was that they would be more able to engage in the metacognitive reflection required to make this assessment than the children we had engaged in the work in Primary Schools (Table 2).
Two further items of evidence are worth reporting. First, after the session, we received the following feedback from the class teacher who had been the same for all of the sessions. After the sessions, she remarked of the year-11 class: In terms of performance, I found the group much more able to move towards higher thinking levels. By showing them that there is no simple answer to issues, [author 2] showed the pupils that they need to approach situations from a range of perspectives and stances.
In their mock exams their responses were improved, in some cases they achieved a grade higher than in previous mock exams. (Clare Barlow, NUSA-our italics) This bears out the pupil's self-assessment that they had learned valuable skills from the sessionsfrom the research-and that the sessions had generated a positive effect.
In addition, NUSA have been sufficiently impressed with the results to make these sessions a standing part of their curriculum, and have made a commitment to running these, with the collaboration of [The University of Nottingham] on an annual basis, and for a range of ability groups.

Impact? A discussion of findings
Our view is that these results are important and significant. As stated at the outset, we are aware of no attempts to distil specific peer-reviewed published philosophical research into child-friendly workshops in order to bring about the kinds of positive, social impacts that traditional P4C sessions have achieved. As the data provided above suggests, it would seem that there is no obvious impediment to generating such positive social impacts.
We take it that we have shown that the activity has had an effect by changing, and benefitting, the children's attitudes towards their own behaviour and practice; simply, they are more confident about their ability to reason about the areas described. In general, this has increased the children's understanding of the educational environment they find themselves in. Furthermore, as noted above, we believe that one aspect of the philosophy sessions has been to develop the children's metacognitive ability and with it the awareness of themselves as learners and more generally.
We think-though more speculatively-that we have grounds for thinking that the research has thereby brought about a change in the performance of the children who have used the research. As noted in Section 2, there is a good deal of data that shows that improved confidence is tallied with improved performance.
We would reasonably expect, then, the noted improved attitude, awareness, behaviour and capacity to lead to an improved performance. There is already evidence suggesting this would be so in the comment from Clare Barlow above: "In their mock exams their responses were improved, in some cases they achieved a grade higher than in previous mock exams".
But it would be a mistake-and would sell short the impact of the project-if we focus purely on a child as an isolated individual. After all, each child is part of a learning community; is part of a family or group of people within Nottingham; specifically, as part of the geographic location of Nottingham-West.
We suggest then that the work regarding changing a child's self-confidence, ability, etc. will have direct impact on other relationships outside the school gates. The philosophy programme is an important social good and, in this case, it was brought about by workshops designed to showcase important philosophical research.
The work with children formed a part of one of the Department's case studies at REF2014. The overall impact profile of the unit was as follows: 10% 4*; 70% 3* and 20% 2*. Although the unit did not return this work in isolation, it was a part of a broader case study, such was the role of the work with children that we think it likely that the work carried out (and that we describe above) made some positive contribution to the overall performance of the Department at REF2014. Informal feedback from colleagues on Research Excellent Framework (REF) panels indicates that this is likely and we note, also, that the decision to return this line of work as a part of the REF2014 submission was approved by University committees that offered advice and guidance on submissions to REF2014.

Conclusion and suggestions for further research
Prompted by the REF "impact" agenda, Fisher and Tallant took current philosophy research into primary and secondary schools. Using the links and networks developed by Fisher, a suite of philosophy sessions were developed around three themes: (1) metaethics; (2) philosophy of sport; (3) philosophy of character.
These sessions were developed and delivered to a significant number of students in primary and secondary schools in an area of multiple-deprivation in and around Nottingham. This work was new because even though critical thinking is carried out in schools across the country, there is no attempt to take high-level published philosophical research into schools.
Data regarding the student's perceived confidence at performing various tasks was collected through anonymous questionnaires and teacher's feedback.
The results suggest that this specific philosophical research not only helped the students in very specific areas-such as character development in writing-but acted as an enabling condition for further areas of personal and social development. In particular, it developed student's confidence, and hence-we speculated -it will have a direct impact on the student's ability and attainment. We noted that despite the prima facie tough task of measuring "impact" within the discipline of philosophy, our project has a noted impact as measured against the REF criteria.
We suggest further work could fruitfully be carried out through a longitudinal study tracking the student's development and achievement across their school life. In particular, it would be interesting to see the development of the grades of the students who took philosophy against a control group; and how the students who engaged in philosophy end up narrating the academic choices they make.
We recognise that there are a number of limitations to this study. As cited above, further work on the notion of "confidence" ought to be carried out. In particular, how effective simple "yes/no" questionnaires are in tracking various psychological features in children (although as noted, such a claim is not without precedent).
We are also aware that for more quantitative subjects, more data and statistical rigor would be desired. And of course, there are also ongoing debates as to the value of focus groups/teacher comments and more generally qualitative data. For example, was there implicit bias when gathering teacher comments? How did the way the questionnaires were introduced affect the answers which were given? Hence, our research methodology would have to be more rigorously defended in a more extensive project.