(How) Do Students Use Learning Outcomes? Results from a Small-Scale Project

Pre-specified, prescribed or intended Learning Outcomes have been in use throughout higher education programs for over two decades. There is an assumption amongst quality assurance bodies and university program approval and review processes that students engage with them. Yet, learning outcomes may constrain learning, they may not always be understood by learners and their relevance to learning has been questioned. There is anecdotal evidence from lecturers that some students do not understand them and do not use or refer to them. This paper reports on a small-scale research project investigating how university student’s use prescribed learning outcomes in their everyday learning and when producing assessed work. No clear differences were found between higher and lower achieving students, yet there were differences between first- and third-year students. Surprisingly, some were able to achieve highly without referring to the outcomes against which they were assessed.


Introduction
Learning outcomes are descriptive statements articulating "what a student should be able to know and do at a defined stage of a programme and/or within a defined element in the programme of study" (Ellis, 2004, p. 2) or "statements of desired outcomes expressed in terms that make it clear how measurement can be achieved" (Melton, 1996, p. 409). They describe what a student should be able to do, or know, or the knowledge, skills and attitudes they should have acquired upon completion of a stage within a program of study, or completed program (Biggs & Tang, 2007;Mann, 2004;Otter, 1992). Watson (2002) suggests that they are things that a student could not previously do and are a change as a result of learning, though it is noted that this description ignores the possibility for the accreditation of prior learning. Over the last 20 years there has been a considerable shift towards the use of prescribed learning outcomes within modularized credit-based higher education programs (Holmes, 2019b), and to the extent that the majority of university programs now use them. Their importance is such that, as Adam (2004) suggests, "Learning outcomes represent one of the essential building blocks for transparent higher education systems and qualifications" (p. 3). Prescribed learning outcomes may be referred to as "intended learning outcomes" and as "pre-specified learning outcomes"; for the purposes of this paper readers more familiar with those terms should substitute "intended" or "pre-specified" for "prescribed." There is considerable evidence of tension and difficulties in the use of prescribed learning outcomes, both from theoretical and more pragmatic positions (e.g., Erikson & Erikson 2019;Holmes, 2019b;Hussey & Smith, 2002, 2003, 2008James & Brown, 2005;Sadler, 2007;Torrance, 2007). It is recognized that learning outcomes may constrain learning (Holmes, 2019b;Sadler, 2007;Torrance, 2007Torrance, , 2012, particularly serendipitous learning, as that which is not specified in advance is not credit-bearing and therefore may be ignored or deemed irrelevant by students, and prescribed outcomes assume that learning can be always accurately predicted and defined in advance (Heick, 2018). They may lead to instrumentalism towards assessment tasks, (Torrance, 2007) whereby students focus only on the learning required to achieve an outcome. This may restrict learning and create what Erikson andErikson (2019, p. 2297) identify as "a ceiling at which students can safely stop, knowing that any further achievements will not be rewarded." In a similar way, prescribed outcomes may stifle learner creativity as they are frequently articulated as threshold achievements (i.e., what a student needs to do to obtain a minimum pass) and may therefore encourage some to aim for the threshold level rather than aiming for a good mark (Maher, 2004). They may also hinder longer-term sustainable (Boud, 2000) and lifelong learning as students may come to understand that only learning that is certified and specified by others is of value or worthy (Boud & Falchikov, 2006;Torrance, 2012), with any learning outside of externally specified parameters of others regarded as being less valid, or even irrelevant.
There is an argument that their use has been misappropriated to serve the purposes of university management, staff accountability and institutional quality assurance and audit processes rather than pedagogy (Beno, 2004;Caspersen & Frølich, 2015;Furedi, 2006Furedi, , 2012Havnes & Proitz, 2016;Hussey & Smith, 2002;Jackson, 2000;Laisnigg ,2012;Nunley, Bers, & Manning, 2011). As such, their usage may have relevance in the critique of neoliberalism and of the marketization, commodification and commercialization of higher education (Barnett, 2011;Brown, 2011;Furedi, 2011;Love, 2008). EDUPIJ • Volume 9 • Issue 2 • 2020 They are regarded by some academics as a restriction on academic freedom, limiting what they can teach (Greensted & Hommel, 2014). Similarly, there is an argument that they may "kill originality and criticality" in pedagogy (Northwood, 2013, p. 137). Furedi (2003) argued that they undermine Socratic teaching as they negate genuine dialogue, because the results of the dialogue are specified in advance, and consequently they may inhibit a student's capacity to deal with uncertainty. Furedi suggests that student learning should be a journey of discovery rather than being directed to a predetermined destination. From a similar position, Avis (2000) argued that learning outcomes may reduce the agency of individual academics and their students, thereby limiting creativity and critical engagement. From a more philosophical perspective Reindal (2013) criticized learning outcomes because they cannot sufficiently describe aspects of higher education that may not be objectified, such as personal responsibility, and that they encourage objectivity rather than essential aspects of humanity and the understandings associated with this that are necessary in higher education.
Prescribed learning outcomes have been criticized because the language in which they are written may confuse some students, and they may only provide a general guide as to what is expected, with students requiring specific knowledge in order to be able to correctly interpret the specific meaning of the outcomes against which they are assessed. However carefully they are written, they can only be interpreted in the light of prior understanding of what quality or standard is appropriate in a given subject at a given level (Hussey & Smith, 2002, 2003, 2008. Students may frequently have a "limited awareness" of their meaning and relevance (Greensted & Hommel, 2014, p. 24). Erikson and Erikson (2019) similarly suggested that students have a "limited interpretative framework, which severely restricts the potential for learning outcomes to fulfill their assumed communicative functions" (p. 2301). A further criticism is that students possess different abilities and progress at different rates; therefore, they will be ready for assessment at different points in time (Melton, 1996), yet outcomes as currently used do not allow students the flexibility to submit assessed work when are ready, but are required to submit to specified assessment deadlines (Holmes, 2019b).
The use of too narrowly prescribed learning outcomes is antithetical to a genuinely constructivist pedagogy, because knowledge and meaning are individually constructed by, and are unique to, each learner (Holmes, 2019a;Merrill, 1991). For those who embrace radical constructivism (von Glaserfeld, 1984(von Glaserfeld, , 1995a(von Glaserfeld, , 1995b, there is even greater difficulty with their usage. Radical constructivists working from within an epistemology, where the meaning of language is based solely on individuated subjective experience, would argue that we cannot use words to formulate meaningful learning outcomes, because they are subjectively interpreted differently by each student. Despite these criticisms, the use of prescribed learning outcomes is fully embedded in contemporary higher education assessment processes. They are mandatory in higher education across Europe (Havnes & Proitz, 2016) and the norm in American universities and colleges (Kuh, Jankowski, Ikenberry, & Kinzie, 2014). There is anecdotal evidence from higher education teaching staff that some students do not refer to them when producing their assessed work, whilst, conversely, others may over-focus on them to the detriment of their understanding of areas of the curriculum they do not feel contribute to their demonstration of meeting an outcome through the assessment process. Yet, there are few studies EDUPIJ • Volume 9 • Issue 2 • 2020 regarding students' actual usage of learning outcomes in higher education. Understanding the factors that promote students' use of them is important for both theoretical and practical reasons. This knowledge may help to bridge any disconnect between theory and the use of learning outcomes, and may also help university policymakers and assessors to identify whether interventions may be needed.

Methodology
This small-scale qualitative research project involved undergraduate (Bachelor of Arts) first-year and third-year students (n = 20) studying in the field of Educational Studies at two universities in the United Kingdom, one of which is a research-intensive institution. The majority of students (n = 19) were female, six were classified as mature students on entry, 11 were first-years, and nine were third-year students. Data was collected via individual semi-structured interviews as part of a larger research project that investigated students' surface and deep approaches to learning (see Holmes, 2018b) and was analyzed using thematic analysis (Braun & Clarke, 2012. During the course of each interview, the students were asked three simple questions about learning outcomes in order to encourage reflection and discussion: (1) How do you use learning outcomes in your everyday learning?, (2) How do you use learning outcomes when producing assessed work?, and, (3) Would you classify yourself as being a higher-or lower-achieving student based on your attainment to date? Follow up questions were asked based on individual responses. Question 3 was included in order to ascertain if higher-achieving students made greater use of learning outcomes. It is recognized that student self-identification and declaration of achievement to date may not be a sufficiently accurate marker to allow labeling as a "high" or "low" achiever, yet, for the purposes of this project, it serves as a broad indicator of achievement, rather than a measurement. Access to actual student grade profiles was not available. Some students did not feel comfortable with, or were unable to identify themselves as being a high or low achiever, preferring instead to self-identify as "medium achievers." Three such students indicated that the fact they were studying at university meant they were high achievers, yet recognized that, whilst at university, they had not achieved highly, and consequently identified themselves as medium achievers.

Use of Learning Outcomes: in day-to-day learning and when producing assessed work
The interviewed students did not refer to learning outcomes in their day-to-day learning (0 of 20 indicated usage in everyday learning), yet typically did so when producing assessed work (13 of 20). Comments from them included, "Other than when I'm writing my work for assessment submissions I don't look at them," "I only really look at them when I'm producing an assignment," "I don't bother with them until it comes to writing an essay," "I don't see what use they are until you get to writing an assignment. I rely on what the lecturer is teaching us about the topic. They should know what the outcomes are supposed to be and teach us according to them," "I don't look at them until I sort of start getting nearer to the assessment hand-in date," and "There is no need to look at them until you start doing the assessment." These comments indicate that learning outcomes do not particularly inform nor shape students' day-to-day learning practices. In itself this is interesting, as one of the frequently-cited arguments for the use of prescribed outcomes is that they provide focus for student learning (Otter, 1992). The students interviewed indicated they used outcomes EDUPIJ • Volume 9 • Issue 2 • 2020 solely when engaged in producing assessed work. This suggests that outcomes neither constrain, nor focus, everyday student learning, yet that they act as a tool of focus when work is produced for assessment purposes. Other than when engaged in assessment tasks, these students paid little or no attention to the learning outcomes. This may support the argument that prescribed outcomes contribute to student instrumentalism (Torrance, 2007).

Differences between first-and third-year students
The students interviewed indicated that they had typically not referred to learning outcomes in the earlier stages of their university education, yet increasingly did so as they progressed from the first to second to their third and final year. The explanations they provided included having become more aware of how important the outcomes they were assessed against were to their academic success and wanted to improve grades, and that assessor feedback often referred to learning outcomes. This indicates both the students' growing awareness of learning outcomes, and their increased focus on achieving good grades in the latter stages of their degree [at both institutions first-year grades did not contribute to the final degree classification]. Representative comments from students included,

Differences between higher-and lower-achieving students
Other than two high achievers who indicated that they never referred to them (discussed later), there were no clear-cut differences between higher-and lower-achieving students. This was surprising as it was anticipated that higher-achieving students would be more likely to refer to them, both in their everyday learning and when producing assessed work. The lack of clear differences may be attributable to the relatively small dataset, or it may be the case that greater learner engagement with outcomes does not lead to higher achievement (there being many other factors affecting a student's grade for an assessed piece of work).

Students who never or rarely use learning outcomes
Four students indicated that they rarely used them, whilst three never used them. Comments from these students included, "I can't really remember if I look at them. I guess I may do, possibly, now and again, I think I did once or twice but didn ' would indicate that some students, although aware of learning outcomes, simply do not use them, but instead rely on their lecturers' explanations of the assessment requirements.
Of the 20 students, two high-achieving third-year students on track for a first-class honors degree [both subsequently did achieve first-class honors degrees] stated that they did not use and had never used learning outcomes. Their comments included, "I've never looked at them, no I've never bothered looking at them," and "No. I don't look at the learning outcomes. I've never looked at the learning outcomes." If these students were able to achieve first-class degrees without ever referring to the learning outcomes they were assessed against this raises a serious question about their relevance and the role they play in student learning. It perhaps also raises questions about the extent to which learning outcomes are used by assessors when grading work.
Of the three students who indicated never having used learning outcomes, one was a first-year student interviewed midway through their second semester, who answered Question 1 [How do you use learning outcomes in your everyday learning?] with "What are learning outcomes?" This comment raises further questions about their use. If a student can successfully complete the assessment requirements of Semester 1 without knowing what learning outcomes are, then their direct relevance to learning and assessment may be open to question.

Lack of understanding of learning outcomes/lecturers deconstructing their meaning
In total, 50% of the students (n = 10) indicated that they did not fully engage with the prescribed outcomes they were assessed against, either because they did not understand them or were uncertain about their understanding of them. Representative comments from the students included, "They are not always written in a language that us students can understand" [third-year student], "I don't always understand what they mean so I'll ask the lecturer to explain them" [third-year student], "They aren't always written in a way I can understand them, so I tend to try and rewrite them down for myself using words that I can understand and then I'll check with the lecturer that what I've written is right or not" [first-year student], "Well we don't always understand what they mean or what they mean in terms of what's needed for the assessment, so we'll usually ask the lecturer to explain what they mean," "Some of them are kind of very vague, so say for example they might say 'understand' but don't give an idea of the depth or level of understanding. So, I'll see the tutor and ask what exactly an outcome means, or ask them 'does it mean this?'" [third-year student], and "I don't really understand, most of us don't really as they are, well not really very specific, so we are unsure about what they mean, so we ask the lecturers and they explain what they actually mean," and "I sort of understand them, but I'm not always sure, so will always check my understanding with the lecturer and they'll explain them, as I'm not always sure if I've got them worked out correctly" [third-year student].
It would seem to be the case that because they did not understand the outcomes, or did not have the confidence in their own level of understanding and interpretation of them, these students, rather than using the stated learning outcomes directly, relied on their lecturers to explain their meaning and what was required of them in the assessed work. Or they would try to interpret and rearticulate them in language that made more sense to them and would then confirm the accuracy of this with their lecturer. This aligns with previous works by Hussey and Smith (2002, 2003, 2008, and also Greensted and Hommel, (2014), in suggesting that the EDUPIJ • Volume 9 • Issue 2 • 2020 language that learning outcomes are written in may not be best suited to some learners and that they may need specific interpretation based on knowledge and understanding which some students may not yet possess.
In practice this would seem to operate in the form of the lecturers deconstructing the outcomes, and reconstructing them into a more user-friendly format, clarifying their meaning so that students may then understand what is required of them. If 50% of the students from a sample of 20 do not understand the outcomes without them being deconstructed, this raises questions about some of the claimed benefits of prescribed outcomes for student learning, as well as about the style of language in which they are written. These students placed far more reliance on the individual lecturer's explanation of assessment requirements than on their own understanding of them. This perhaps suggests that assumptions that learning outcomes empower students (Ellis, 2004) may not be correct.

Learning outcomes constraining learning
One student, when asked how they used outcomes reported, "It depends, if it is a subject I am really interested in I tend to read widely and just out of interest or go a bit further, whereas for some modules you find a bit difficult, I just focus on the learning outcomes and just passing" [third-year student]. This comment indicates that, for this learner, where they found the module difficult, they focused more on the prescribed outcomes in order to achieve a pass grade.  Maher's (2004) study which suggested that some learners using them may aim for a threshold pass grade rather than aiming for a high mark. It also concurs with Erikson and Erikson's (2019) argument, in that they may act as a ceiling for learning, in these cases meeting (rather than surpassing or exceeding) the outcomes would effectively limit the grade they could achieve.

Conclusions
It is recognized that the current study has certain limitations, and the findings may not be generalizable. A student's self-declaration of being a high or low achiever is highly subjective; also, the 20-student sample size was small, plus the research was limited to two universities. As such, given the scale of use of prescribed learning outcomes throughout higher education programs, this work may not be representative, either of all students at the institutions involved, of students at other universities, or of those studying other disciplines. The current study has, therefore, only just scratched the surface of what is clearly an area worthy of further in-depth exploration. Further research with larger group sizes, that includes students from other disciplines and other universities is required, and that also has access to the participant students' actual grades.
Nevertheless, as has been found, some students do not use the prescribed learning outcomes to inform their everyday learning, some do not know what learning outcomes even are, whilst some do not understand them without their meaning being deconstructed by their instructors, and some are able to achieve highly without ever having referred to EDUPIJ • Volume 9 • Issue 2 • 2020 them. Together, these findings would indicate that the assumptions made by both academics and policymakers about the role played by and the importance ascribed to learning outcomes are open to question, and that further research into students' usage of prescribed learning outcomes is therefore required.