COMMENTARY Reflections on academics’ assessment literacy

This small-scale, mixed-methods study aims to investigate academics’ understanding of formative and summative assessment methods and how assessment literacy impacts on their teaching methods. Six semi-structured interviews and a scrutiny of assessments provided the data and results suggest that while these academics understand summative assessment, they have a poorer awareness of the implementation of well-constructed formative assessment. While the academics were able to clearly articulate the perceived benefits to students from undertaking formative assessments, they were less able to identify potential benefits for themselves as educators, so these went largely unrealized. Opportunities therefore exist for tutors to utilize the outcomes of formative assessment to improve student performance, particularly around tutor-reflection to amend future learning and teaching approaches in line with the theory underpinning summative and formative assessment methods. The study highlights the importance of considering all stakeholders when thinking about assessment literacy.


Introduction
This paper will reflect on findings from a small-scale qualitative study on formative assessment. The study was initially prompted by a university-wide structural change at the research site that resulted in a curriculum redesign across the whole institution.
Mindful of poor National Student Survey (NSS) scores in assessment and feedback over a number of years, the institutional review of the curriculum included a key strategic aim that required that 'formative assessment opportunities are designed into the curriculum so that students receive feedback before they encounter high stakes summative assessment' (Review of the Academic Framework Guiding Principles, Internal Policy Document, 2012). This encouraged academics to rethink their assessment approach, and one popular method was to implement in-class or online multiple-choice tests as part of a formative assessment strategy to enable students to monitor their progress towards a final assessment. Online tests were popular, given their ability to provide effective, rapid feedback (Yorke, 2005;Bartram and Bailey, 2010) developed with relatively low cost and that they could be reused as needed (Armellini and Aiyegbayo, 2010), and although this approach carries a fixed overhead, it is better suited for a mass approach to higher education than more traditional methods (Yorke, 2005).
The small-scale qualitative study was therefore designed to investigate how academics integrated multiple-choice tests, in particular those delivered online, into first-year core modules in a large business faculty, and what impact they had on their formative and summative assessment strategies. Following a brief review of the literature on formative assessment, the findings of the study will be discussed and then reviewed in light of current thinking about assessment literacy.

Background and context
The changing nature of higher education in terms of raised expectations of assessment methods (Di Costa, 2010), larger classes, and more diverse populations (Yorke, 2005;Bartram and Bailey, 2010) has led to an increase in research into innovative methods of assessment because educators have had to rethink their practices (Bartram and Bailey, 2010). With a move towards ensuring assessment for learning rather than only of learning (Wiliam, 2011), assessment methods have undergone significant scrutiny, with a particular focus on the relationships between summative and formative methods of assessment.
Summative assessment is widely agreed to be a form of evaluation that provides a judgement that encapsulates all the evidence up to a given point (Taras, 2005), usually the end of a teaching period (Wininger, 2005). It can have the purpose of providing a student with a grade (Bloom et al., 1971), where learning is seen in terms of knowledge, skills, and attributes (Kibble, 2011), thus enabling students to be differentiated among their peers (Wininger, 2005). Summative assessment is therefore a powerful tool, because through the provision of grades it 'creates texts -marks, grade sheets, qualifications -that reify identity' (Pryor and Crossouard, 2008: 8); for example, leading a learner to identify themselves as a first class or a 2.1 student. However, there are negative aspects of summative assessment, mostly regarding its nature of focusing too much on 'How did I do?' rather than 'How am I doing?' (Yorke, 2005;Kibble, 2011).
As with summative assessment, the notion of formative methods is not universally understood. Often, academics disagree about the key purpose of formative assessment (Bennett et al., 2012;Taras, 2010;Wiliam, 2011) and it can be seen as an 'add-on' to regular teaching and learning methods rather than as an integral part of the overall strategy (Yorke, 2005;Hargreaves, 2005).
The definition of formative assessment has been through several iterations since the seminal work of Black and Wiliam in 1998. They propose a more recent definition (2009) that focuses on evidence coming from classroom practices rather than assessment yet retains the multistakeholder perspective, including both learners and teachers, and the emphasis that decisions made on the basis of that evidence should advance the stakeholder's position in teaching and/ or learning: Practice in a classroom is formative to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited. (Black and Wiliam, 2009: 7) Nicol and Macfarlane-Dick (2006) constructed a list of good feedback practices, highlighting seven principles to help encourage students to proactively engage in monitoring their own learning through formative assessment strategies. The list emphasizes feedback that helps clarify good performance and facilitates self-reflection through high-quality information about students' learning that positively impacts upon motivation and self-efficacy. As well as these explicit student-oriented benefits, their list reiterates Black and Wiliam's (1998;2009) multiplestakeholder-oriented definition that feedback should shape both teaching and learning practices through encouraging tutor-peer dialogue around learning and also provide information to tutors to support teaching practices and improve instruction.

Academic rationale for the study
The original study was intended to be a small-scale scoping study with the objective of investigating how academics responded to the institutional requirement of embedding formative assessment in their modules with the aim of supporting summative assessment performance. Three key areas were identified for exploration through qualitative interviews, which were developed around the central themes of Black and Wiliam's (2009) definition of using formative assessment to impact upon summative assessment through the collection and feedback of evidence or data: 1. What do academics understand by the terms 'formative' and 'summative' assessment? 2. How do academics conceive of the interaction between formative and summative assessment? 3. How do academics manage the feedback process with formative and summative assessments?
Six semi-structured interviews were held with module leaders who teach on a large first-year general business programme. An exclusive focus on the first year was adopted, given that the way teachers 'approach their teaching of a general first year course may be quite different to their approach to teaching later year courses' (Trigwell et al., 1994: 77) and because it has the greatest potential impact on retention (Yorke, 2005). The nature of the programme meant that a wide range of disciplines was covered, which helped minimize potential single-subject bias. Rather, the breadth of disciplines covered enabled a range of opinions and experiences to be examined to see what variation of conceptions and approaches exist.

Reflections on findings
It was clear that our participants had good understanding of summative assessment as providing a grade, and formative assessment as enabling students to assess themselves and get feedback on their progress during the module. As the participants typically taught large modules, with an average registration of 420 students among the sample, the online assessments were seen as a way to provide feedback to a large cohort quickly and easily.
These general conceptions were in line with the definition in the literature in relation to students, but were found to be limited when thinking more holistically about other stakeholder benefits. All participants held firm that the formative assessment was put in place for the benefit of students, and as such it was their responsibility to engage with the assessment, get the feedback, and make use of it. However, a key finding of the study was that when prompted, no participant commented on any other potential benefits, even for their own teaching practices. There was no formal or structured way in which the formative assessment results were reviewed, which differed significantly from the processes adopted when considering summative results. This meant that the part of the formative assessment definition that commented on the benefits for educators from using classroom evidence to make better instructional decisions (Black and Wiliam, 2009) went largely unrealized.
On reflection, and with further examination of the interview transcripts, barriers to academic self-reflection became evident, which included time and lack of resources as the most common barriers. To illustrate this, when asked directly about monitoring the formative assessment outcomes, responses included: Online quizzes are not monitored. There's an overhead in setting it up and the staff time is proving too costly. It takes a lot of time.
The reason we use multiple choice is to try and compensate for the huge numbers we have. A major problem is with resourcing and I haven't got time to do it.
From an institutional point of view, we have reflected that while the aim of integrating formative assessment opportunities into the curriculum through the review of the academic framework was successful, at present they largely remain just that -opportunities. Until there is a systematic review by both students and staff on the outcomes of the assessments, it is hard to make the case that both groups are making better educational decisions than would have been made without the assessments, and therefore our findings question whether the assessments are in fact formative in nature. The challenge therefore becomes one of facilitating longer-term engagement with the institutional strategic aim to take the use of formative assessment beyond implementation and into full integration in assessment strategies, with regular reflection by all stakeholder groups.
With regard to how to move forward, our intention is to use this pilot study as a springboard to encourage academics to review their assessment methods, both formative and summative, identify the data each provides, and consider them in tandem. Fostering an environment where academics feel able to properly reflect on formative evidence will help improve pedagogic practices (Yorke, 2003) and reap the long-term benefits of improved pedagogic literacy (Price et al., 2011).
However, this is not a simple exercise. Yorke (2005) comments that the constraints experienced by academics, such as the emphasis on summative outcomes, increasing staff-student ratios, and the requirement for teachers also to be 'research active', mean that encouraging academics to change their practices is challenging and is 'threatening the use of formative assessment' (Yorke, 2005: 486); our findings are consistent with this line of thought.
This is clearly not making the most of the potential of formative assessment for improving learning and leads to the authors' observation that simply making formative assessment part of an institutional requirement does not automatically lead to assessment for learning. With this in mind, we look to the literature on assessment literacy to reflect further on the findings.

Assessment literacy
Academics' awareness and understanding of assessment and feedback can be considered part of their assessment literacy, defined as 'the level of knowledge, skills, and understanding of assessment principles and practice' required by a range of stakeholder groups (Taylor, 2009: 24). This definition does not differentiate between the stakeholders, and indeed different authors focus on the various stakeholders in a variety of ways. Smith et al. (2013), for example, place emphasis on students, conceptualizing their assessment literacy as having three dimensions: to understand the purpose of assessment, awareness of the process of assessment, and the ability to judge their own responses and reflect on personal strengths and weaknesses. Popham (2011) does not differentiate between stakeholders: 'Assessment literacy consists of an individual's understandings of the fundamental assessment concepts and procedures deemed likely to influence educational decisions' (Popham, 2011: 267). However, the focus is firmly on educators, with the definition explored exclusively from their perspective, drawing on accountability and the need to 'understand the nature of the instruments being employed to judge them as professionals' (Popham, 2011: 269). The shift to a student focus does occur, but is still seen through the lens of the educator, and Popham highlights how diagnostic assessment can play a powerful role in student learning due to the ability to provide information on current levels of learning. However, this role for formative assessment in providing 'instruction-enhancing tools' (Popham, 2011: 271) is only possible if teachers have a good understanding of assessment, in other words if they have good assessment literacy.

Reflecting on assessment literacy
Undertaking this research, and the subsequent reflections on the findings, has highlighted the importance of considering all stakeholders when thinking about assessment literacy. For students, knowing the purpose and processes of assessment is crucial, as is understanding how to access and make use of feedback to improve subsequent work. However, the educator's assessment literacy is equally important, and this has two significant aspects. Firstly, the educator needs to be able to develop meaningful formative assessment and associated feedback to ensure assessment for learning. Understanding the difference between assessment as an evaluation of student performance and assessment as a learning tool is crucial. However, an additional element in educators' assessment literacy is to view formative assessment as a tool to improve their own practice. This was not evident in our own findings, where educators simply made available online tests and answers for optional use by students. Indeed, as noted by Yorke, while academics routinely reflect as part of their professional practice, it tends to be focused on their subject discipline rather than on pedagogic practice, and reflection on the ways that assessment practice and theory are integrated into the curriculum is underdeveloped (Yorke, 2003). By analysing online test results, it would be possible to identify areas of weakness with a view to improving learning in identified areas. This is in line with Black and Wiliam's suggestion that feedback is 'used by teachers … to make decisions about the next steps in instruction' as well as by learners and their peers (Black and Wiliam, 2009: 7).

Impact on practice and assessment literacy
Overall, we found immense value in conducting this study, despite its small-scale nature. Although the sample was not large enough to enable any generalization, it did permit a valuable insight into academic practice at the research site. The findings prompted us to review the literature to examine why the full benefits of formative assessment went unfulfilled, leading us to consider assessment literacy as a possible lens, although the number of studies focusing on the academic perspective was startlingly small. Popham suggests this may be because most educators believe they already have a reasonable handle on what is meant when someone is identified as being 'assessment literate ' (2011: 267). Indeed, it would seem reasonable to posit that any academic who has the authority to set student assessments should be fluent and fully literate about what the function, process, and outcomes of that assessment are. The literature clearly places the emphasis on academics encouraging students to develop their understanding of what an assessment means for them -it seems time that we, as academics, start to do the same.

Conclusion
Having reviewed the literature on assessment literacy, formative assessment, and assessment for learning, it is apparent that the focus is clearly dominated by the student perspective, benefit, and challenges. Institutions and educators need to realize the value of self-reflection on teaching practices for future benefit in the same way that we encourage and expect students to reflect on the feedback on their work; academics should practise what they preach.

Notes on the contributors
Rebecca Lees is an Associate Professor at Kingston University within the Accounting, Finance, and Informatics Department, teaching quantitative methods and management science. She researches within the higher education environment, with a focus on formative assessment practices and initiatives to develop students' employability skills and behaviours. She is a Senior Fellow of the Higher Education Academy.
Deborah Anderson is an Associate Professor at Kingston University and Principal Fellow of the Higher Education Academy (HEA), teaching marketing to both undergraduate and postgraduate students. Deborah was awarded a National Teaching Fellowship by the HEA in 2014. With twelve years' work experience in marketing and a PhD in educational research, Deborah combines rigorous academic principles with a practical eye on students' future employability. Current research interests include pedagogic innovations, assessment and feedback, academic writing, and employability.