Supporting the development of assessment literacy of staff through institutional process change

This paper reflects on the work done at a large UK university to redesign assessment procedures in a way which was intended to contribute to an improvement in assessment literacy for staff. Existing practice was reviewed and showed that changes in assessment processes were needed to make the organisation of assessment more consistent and more transparent across the institution and to develop staff assessment literacy. Revised procedures were designed and implemented in order to make a clear distinction between institutional requirements for ensuring standards and recording outcomes, and academic decisions which ensured that assessment was designed to be appropriate for disciplinary requirements. procedures. The revised website had over 9400 page views in the 2013/14 academic year, a 35% increase on the previous year, and a similar increase in 2014/15, to 13050 page views. Staff development sessions have been attended by around 500 academic staff, with additional sessions organised for key collaborative partners at their own sites. In summer 2014, at the end of the first year of implementation of the new procedures, a survey of academic staff indicated that 76% of respondents (n=124, around 8% of academic staff) were aware of them. A comparison of external examiner reports for the 2013/14 and 2014/15 academic years showed that more external examiners were commenting favourably on consistency in approaches across programme teams, and fewer were suggesting changes in assessment procedures. Student satisfaction with assessment and feedback, as measured by the National Student Survey, has increased significantly: institutional outcomes have moved from below the sector average to the top quartile.


Introduction
In the UK, academic staff working in Higher Education have a very wide range of discretion in the planning and delivery of assessment: with the exception of a few courses which give exemption from certain professional qualifications, there are no standard curricula, no national examinations, no requirements about the types and lengths of assessments required of students. Academic staff are free to set the assessment agenda. Despite this flexibility, and despite great changes in other aspects of course design, assessment practices have remained relatively unchanged; in the authors' university, examinations, essays and reports represent over half of the assignment types in use.
In part, this may be due to anxiety about the responsibilities associated with assessment. Academic staff need to ensure timeliness, validity, reliability and equity. National standards and expectations for these factors are set out in a national Quality Code (QAA, 2013). Adherence to the Quality Code is assured with respect to standards and expectations via a system of periodic external review, as well as regular peer review of assessment processes and practice, which includes the employment of examiners from external institutions (Cuthbert, 2003) In addition to the external scrutiny, as Boud (2000) pointed out, assessment almost always has to do 'double duty': giving students feedback on how to improve as well as judging their performance. The UK Quality Code (QAA, 2013) makes it clear that assessment in higher education must be based on the assessment of clearly specified learning outcomes. Institutions also expect subject-based assessment to be used to demonstrate students' transferable skills to support their future employability (Knight & Yorke, 2003).
Staff may also feel constrained by the fear of reducing student satisfaction. The UK National Student Survey has consistently identified assessment and feedback as the aspect of their courses with which British students are least satisfied. Since 2005, most UK universities have had initiatives to increase student satisfaction with this aspect of their experiences. This had achieved an improvement by 2014, from around 58% of students being satisfied with assessment and feedback, to around 68%. This is still far short of their satisfaction overall with their courses, which had increased from 80% to 83% (HEFCE, 2014, fig 12).
This national focus has led to a great deal of research into the reasons for student dissatisfaction with assessment and feedback. Assessment literacy is emerging as a concept for describing familiarity with the language and process of assessment; Price et al (2012) make it clear that the term 'language' is used to denote not just the vocabulary, but also the 'grammar' of the assessment process. Someone who is assessment literate, according to their descriptions, will have clear understandings of how assessment fits into a course and be able to make critical decisions about the application of appropriate approaches and techniques to assessed tasks.
The development of this concept is reflected in national guidance on assessment. In the UK, the Quality Assurance Agency (QAA) expects institutions to make a conscious effort to develop students' assessment literacy. Whilst the QAA does not attempt to define assessment literacy, it gives examples of how it may be achieved: "Facilitating students' assessment literacy includes illustrating the way in which standards are communicated and applied within the relevant subject to enable staff to make judgements about student performance in different types of assessment task." (QAA, 2013, p14) Whilst assessment literacy is often discussed as an attribute to be developed in students, as in the QAA Quality Code example, our experience as educational developers suggested that many staff also feel hesitant and anxious about assessment. Staff queries about assessment often began with the phrases "Am I allowed to…?" or "I've been told I've got to…" There was a distinct tendency to seek out a (non-existent) rule-book about assessment, and to stay safely with the way things had always been done locally, despite institutional strategy to move towards more professionally-focused outcomes for students, and therefore more real-world assessment. An informal study of participants in a postgraduate certificate for teachers in Higher Education showed that around 70% use words like 'stressful', 'worrying' or sometimes 'responsible' when asked how they feel about marking (Marr & Forsyth, 2010).
The meanings of language used in assessment can also be unclear; Taras and Davies (2013) surveyed science lecturers' interpretations of the commonly-used assessment terms of 'formative' and 'summative' assessment. They discovered that there was very little shared understanding about these two relatively straightforward terms, and suggested that this situation may hamper effective discussion about assessment policy and practice in institutions.
Lack of consistency in the use of terms also raises the question of how we determine whether decision-making in assessment is consistently carried out. Orr (2007) suggests that "assessment is a socially situated practice that is informed by, and mediated through, the sociopolitical context within which it occurs." She goes on to provide examples of decision-making conversations where two or more markers were grading the same artefact which illustrate how markers may be influenced by a range of factors and by each other's personalities and belief systems. There have been several studies which have showed that reliability in marking is difficult to achieve (Ecclestone, 2001, Hanlon et al., 2004. The development of a more explicit approach to the purposes and processes of assessment, and better dialogue about assessment terminology and practice has been strongly advocated (Bloxham et al., 2015, Price et al., 2011, Price et al., 2007, Price et al., 2012.
This context provides the background for this paper, which describes work carried out at a large, diverse, UK university to explore why assessment practices have remained relatively static and to identify ways to improve staff assessment literacy.

Assessment Practice
The project received some funding from the JISC Assessment and Feedback programme, which required the production of a baseline report to form a basis for measuring the progress of the project. We carried out a review of existing information such as regulations, assessment statistics, and outcomes from student surveys, as well as interviews and focus groups with a cross-section of academic and professional staff across the institution.
Participants in interviews and focus groups were selected using purposive sampling: the principal criterion was to have an involvement in assessment because of their job role with selection made to get a maximum variation in the type of their involvement. Additionally, some academic staff were invited to participate on the basis of having tried to implement assessment changes or innovations.
Students were not interviewed as a part of the baseline study; this decision was taken after consultation with the University Students' Union Membership Services and Student Voice Managers. The existence of a considerable body of evidence of student opinion in connection with assessment and feedback, via the National Student Survey, internal surveys and Students' Union termly reports, was considered to be sufficient data for the baseline report, although students were involved in later stages of the project.

Staff perceptions of assessment practice
The need for clarity and consistency across all aspects of assessment was highlighted by several interviewees in addition to the issues already identified during the institutional audit process, as this extract shows: "It's not clear where the locus of responsibility lies with assessment… who is empowered to do what? What is the scope of the role of the unit tutor? Programme leader? Team? Admin staff?" (Focus group) Issues around feedback were also acknowledged by staff; in one interview, a member of academic staff said "Students have to wait too long for their feedback. The quality of feedback is varied and it isn't always intended to help students improve." Academic interviewees, particularly those in a leadership role, felt that more structured guidance would be helpful.
"Some colleagues have a lack of understanding about feedback (purpose and how to provide it) …this could be resolved by appropriate training and staff development." (Senior Learning and Teaching Fellow) The issue of engagement with feedback was also raised. Some interviewees mentioned the number of marked assignments which were never collected by students -there is a clear implication that piles of dusty assignments could be found in offices, waiting for carefully-written feedback to be retrieved by students. Previous studies (Weaver, 2006, Winter & Dye, 2004 suggest that around 20% of marked work going uncollected would be a reasonable estimate.
Interviews, and an audit of support requests to educational developers, indicated that there was a need for a simplified and consistent approach to moderation of assignment tasks and of marked work. One interviewee commented particularly on the time consuming nature of moderation processes, which were largely paper-based. There was sometimes confusion about the distinction between second marking and moderation. There was a plethora of moderation forms in use, which was confusing for staff who taught across several programmes. Course-based rules for resolving disagreements between markers of individual assignments seemed to have huge importance, even when the ensuing calculations would have no impact on the final award classification. A review of academic appeals carried out as part of the project showed that decisions were reliable and robust, but the existence of different approaches made it timeconsuming to drill down to the details of moderation activity.
There is a clear link between these comments from colleagues and the concept of assessment literacy; the perceived lack of clarity about expectations and presentation of information indicates some difficulty with what Price et al (2012) might refer to as the 'grammar' of the assessment language. To take the literacy analogy further, it seemed that people were quite capable of using language to express complex ideas, but they seemed to lack confidence in common usage, and spent too much time looking in the 'dictionary' of institutional rules to allow them to demonstrate fluency in assessment.

Reflection on Findings
The baseline report was widely discussed through institutional governance structures (faculty and university committees) and at central service providers' team meetings and networking days. These discussions were valuable in helping to validate the findings, to prioritise work on the project as well as ensuring that there was a wide knowledge of the aims of the project.
The baseline report, and other forms of oversight such as moderation and external examiners' reports and the most recent QAA institutional audit for the university (QAA, 2010) showed that assessment practice was fair and robust, but that assessment sometimes caused confusion and anxiety among staff and students.
In reflecting on the findings, we thought it would be helpful to recommend actions which focused on the development of a common language to describe the processes of designing, supporting and marking assignments would make it easier to exchange good practice, cover the management of assessment when there were professional or academic staff absences, and to review actions in the event of queries or challenges. We were well aware that such an improvement in assessment literacy was unlikely to be achieved by having more rigid policies. Bloxham's (2009) analysis suggests that time is better spent in ensuring that a programme team is confident in its assessment decisions and capable of engaging students effectively in the assessment process, this ensuing in a more effective and inherent assessment literacy by both staff and students. She highlighted the false goal of a truly objective marking system in Higher Education, suggesting that "confidence should come instead from the professional judgement of several different tutors across a large number of different assessment opportunities" (Bloxham, 2009).
Having confidence in professional judgements does not mean that assessment tasks and processes need to be identical across the university, nor even across a programme; teams need to feel confident in making decisions which reflect the programme learning outcomes. As Price et al (2011) point out, it is not helpful to try to impose a single system in a large and diverse university: "there are a range of positions that can be adopted but it is important that there is consistency between those positions if strategy and policy are to be coherent and relatively tension free." (Price et al., 2011). We needed to put in place procedures and processes which freed teaching to make academic decisions about assessment which were relevant to their programmes, and to explain those decisions in ways which would support students' assessment literacy.

The Assessment Lifecycle
In order to provide a basis for more effective discussion of assessment and feedback, we produced a simple diagram to represent what happens with an individual assignment from specification of the task (stage 1, which is done at initial approval of the programme of study) to annual review and reflection on the value and effectiveness of the task (stage 8, which is usually done after examination boards have met). This is shown in Figure 1.
This diagram proved to be very helpful in structuring discussions about the management of assessment with academic and professional staff. The idea of following an assignment task from start to finish is simple, and the process it depicts is uncontroversial. This focus on the task, rather than individual roles and responsibilities, allowed discussions to be depersonalised; all work was focused on getting the whole process working effectively in particular contexts, rather than on (perceived) individual shortcomings or problems, or on there being a 'correct' approach to any one aspect. Instead, it became possible to have discussions about the best ways to develop each part of the process in ways which were sensitive to disciplinary norms and expectations. In previous change projects, we have found such depersonalisation to be valuable; role plays and games have been used to disseminate sensitive research findings and to promote discussion about change without straying into making negative judgements on current practice, which encourage defensiveness (Hamshire & Forsyth, 2013).
The discussion of the assessment lifecycle enabled us to produce a picture of how different types of assessment were managed in different disciplinary areas, and how departments and faculties managed each part of the cycle. Various myths about institutional requirements emerged from these discussions, such as beliefs that certain types of assignment were not permitted, or that certain types of assignment had to be a certain size, or that feedback must be in a particular format. This led to further discussion, focusing on which parts of the assessment process would benefit from some institution-level changes to provide a more consistent approach to assessment management, and which should remain in the control of course teams. This distinction was characterised as a difference between procedural and academic decision-making and was intended to clarify core expectations, introduce consistent terminology, and reduce confusion about what was expected of course teams.

Development of staff assessment literacy
During the period of the project, the QAA was consulting on a new version of the UK Quality Code, including the section on assessment. The University Institutional Code of Practice (ICP) on assessment was thus due for review, to ensure that it reflected the revised chapter of the Quality Code. In the light of the findings of the project, it was decided that it would be rewritten completely, and the assessment lifecycle was used to provide an outline structure for the new ICP and its associated procedures. Some parts of the cycle required only a check on procedural changes, but the areas which involved academic decision-making were thoroughly reviewed with a view to making terminology and procedures consistent.
New procedures were drafted to improve the way information about marking, feedback and moderation was recorded and communicated to students and colleagues. The launch of these procedures was associated with a large programme of staff development and awareness-raising. In addition to a relaunched assessment website which included screencasts about key aspects of the procedures, short staff development sessions were held in departments and faculties, webinars were held monthly through the year, and members of the project team attended faculty and service management team meetings, external examiner and review panel training days, and internal conferences to explain the changes. An accredited module on Assessment in HE, which formed part of a PGC/MA in Academic Practice was completely revised to reflect the new procedures. The revised website had over 9400 page views in the 2013/14 academic year, a 35% increase on the previous year, and a similar increase in 2014/15, to 13050 page views. Staff development sessions have been attended by around 500 academic staff, with additional sessions organised for key collaborative partners at their own sites. In summer 2014, at the end of the first year of implementation of the new procedures, a survey of academic staff indicated that 76% of respondents (n=124, around 8% of academic staff) were aware of them. A comparison of external examiner reports for the 2013/14 and 2014/15 academic years showed that more external examiners were commenting favourably on consistency in approaches across programme teams, and fewer were suggesting changes in assessment procedures. Student satisfaction with assessment and feedback, as measured by the National Student Survey, has increased significantly: institutional outcomes have moved from below the sector average to the top quartile.
In the introduction, we discussed the difficulties and anxieties associated with making changes to assessment processes. The new procedures were intended to make changes easier, by making it clearly that academic responsibility is in the hands of course teams. We felt there was a need to make a subtle shift, from a culture where academic staff were complying with decisions which they may have previously believed to have been made for them by the institution (for instance, about sampling rates for moderation, or the expected format of feedback, or the sizes or weighting of different types of assignment), to feeling trusted to make appropriate decisions for their own courses. In order to do this effectively, we needed to spend considerable time discussing and refining the meanings of different parts of the assessment process.
Language learning needs time and practice in a supportive environment; full fluency only comes when you use the language in the country where it is the norm, rather than speaking it only with other non-native speakers, who may perpetuate myths and reinforce mistakes. We have found it challenging to achieve change in staff approaches to assessment. Asking people to do things in a different way felt as though we were asking them to learn new language rules. Open discussion about all aspects of assessment management has helped us to improve our approaches to supporting change and flexibility in assessment.