Exploring the processes used by students to apply feedback

Despite the recognised importance of feedback, and the effort that academic staff put into providing it, little is known about how students make use of their feedback to improve their future learning. This paper explores the processes that students attempt to use to feed-forward, and whether the use of technology might support and enhance these processes. In a small-scale, in-depth, qualitative study it was found that despite student preferences to initially engage with certain forms of feedback, efforts were made to apply any feedback regardless of format when clear links could be made to future learning. However, this was limited to superficial connections and there was no clear evidence that students attempted to make deeper connections; students generally expressed that contentor assignment-specific feedback was difficult to apply. The study established the need for further investigation into how tutors construct the feedback given and how students deconstruct that feedback.


Introduction
In our effort to understand how students engage with and make use of their feedback, this paper explores the processes that students use to remember or recall the feedback that they receive, and how they apply the feedback to their future learning.This paper further explores whether technology can both support and enhance these processes.Please note, as a clarifying point, this paper will use the term 'module' to mean an individual unit or course of study.
The importance of feedback in supporting student learning is well documented (e.g.Weaver 2006;Duncan 2007), as are a wide range of principles and models for providing good quality and timely feedback (e.g.Gibbs and Simpson 2004;Nicol and Macfarlane-Dick 2006).Sadler (2010, 535) observes that 'for many students, feedback seems to have little or no impact, despite the considerable time and effort put into its production'.Little is known about the use that students make of this feedback to improve their learning, a view supported by Handley et al. (2011, 548) who contend that 'the nature of engagement with feedback has received little direct attention in the pedagogic literature'.If this is the case, it becomes imperative to develop an understanding of how students can be helped to engage with the feedback they are given by their tutors.Handley et al. (2011) define student engagement with feedback as a process that includes receiving, using and taking action on their feedback; processes which students might do subconsciously (Parkin et al. 2012).These solitary processes involve what Moore (1989) describes as a 'learner-to-content interaction', and what Holmberg cited in Hawkridge and Edirisingha (2002) terms an 'internal didactic conversation'.Active engagement may involve visible interactions with other people or resources.Handley et al. (2011, 552) argue that 'students in higher education interact with parents, siblings, friends, and others when they try to interpret assignments and feedback, and they may reach for resources such as textbooks, assessment criteria grids, model answer guides and so on'.Overall, students do a variety of things to make sense of their feedback and, perhaps later, to apply their learning to new situations (ibid.).
Despite some students' attempts to read and engage with feedback, it is not clear whether this leads to action (Parkin et al. 2012), and there is still evidence that many students do not engage (or effectively engage) with the feedback they are given.For example, in a study by Bevan et al. (2008) only half of the students surveyed agreed that they used the feedback to go over what they had done in their assignments and almost half reported that, whilst they had good intentions to use feedback, they had forgotten about it by the time they came to undertake the next assignment.The perception that students often do not engage effectively with the feedback provided was reflected in the views of the academic staff (ibid.).This failure to effectively engage may be attributed to a number of reasons.It may be because the feedback arrives too late to be considered useful, or find their tutors' comments hard to understand (Weaver 2006).It may also be attributed to the lack of clear advice given on how to improve (Bevan et al. 2008;Higgins et al. 2001;Hounsell et al. 2008), or that students have received little training in using feedback (Sadler 1998).If students have not been prepared to connect with their feedback they may show little evidence of development or intrinsic motivation to learn (Mutch 2003).It is also evident that modularisation limits the opportunity for students to use feedback in subsequent assessments (Parkin et al. 2012;Price et al. 2010).
While tutors can provide good quality feedback to students, it is also understood that students must be able to self-manage learning (Nicol and MacFarlane-Dick 2006).The skills of self-management should be encouraged by lecturers who 'feed-forward' (Duncan 2007) to enhance the student experience by encouraging engagement and hopefully maximising learning.With 'feed-forward' the intention is to guide students to deploy feedback from one assignment into subsequent assignments, hence encouraging student learning (Higgins et al. 2001;Hounsell et al. 2008;Orsmond et al. 2005).
To help students make sense of feedback there is a need for students to engage with the marker in order to discuss the feedback.It has been argued by Nicol and Milligan cited in Carless et al. (2011) that possible ways of enhancing feedback processes involve viewing feedback more as a dialogue than information transmission.Earlier studies have shown that students value the opportunities to discuss the feedback and share experiences with their peers in order to make sense of the feedback and use the experience to improve their work in the future (Rae and Cochrane 2008).
The use of technology, such as Virtual Learning Environments (VLEs), is considered to be helpful and advantageous in providing student feedback (Hepplestone et al. 2011), although its application is rather variable despite students' comments suggesting that this means of communication would be useful (Rae and Cochrane 2008).Technology can facilitate the provision and storage of detailed, legible and timely feedback for students to access anytime and at a place of their choosing.The timing of feedback is crucial to student learning as highlighted by Gibbs and Simpson (2004, 19) who comment that, 'if students do not receive feedback fast enough then they will have moved on to new content and the feedback becomes irrelevant to their on-going studies and is extremely unlikely to result in additional appropriate learning activity, directed by the feedback'.

Methodology
In attempting to understand how students engage with feedback, this paper seeks to address the following two research questions: 1. What processes do students use to feed-forward into future learning?2. How can technology support and enhance these processes?
This paper draws on findings from a wider research project, 'Understanding student learning from feedback' (Hepplestone and Chikwa 2014).The small-scale research project took a qualitative approach and worked in partnership with seven selfselecting full-time, on-campus, undergraduate students, two females and five males aged between 18 and 22, studying various subjects and disciplines across different levels of undergraduate study, to explore the subconscious processes that students use to engage with, act upon, store and recall feedback.The subjects and levels that the participants were drawn from were: In spite of the participants representing different disciplines and levels of study from across the University, the authors of this paper acknowledged that using a selfselecting sample can cause a bias in the characteristics of participants and those who volunteer to participate are likely to be more engaged with their learning.As such, the sampling method is recognised as a limitation of the research.In addition, the study acknowledged that the sample size is small and it is, therefore, difficult to make generalisations from the findings.This said, the findings of this study do reflect previous studies conducted at Sheffield Hallam University (e.g.Parkin et al. 2012) and highlight themes and issues to explore in future studies.
The study used micro-blogging and reflective diaries to record and reflect upon every instance of feedback that the participants encountered over an 18-week period, between December 2011 and April 2012, and concluded with an hour-long end-of-study semi-structured interview for each participant.Participants were required to capture their interactions with all feedback regardless of its nature, who provided it, and the format in which it was delivered.Over the course of the study, and during the interview, the participants articulated on the processes used to engage with feedback in order to feed-forward into future learning, and how technology might help effective use of feedback.The data collected (from the methods discussed above) were analysed using a thematic analysis approach facilitated by the use of NVivo software, in order that patterns within the data were identified, analysed and reported (Braun and Clarke 2006).

Findings
While this paper focuses on the processes that students use to feed-forward, and whether such processes can be enhanced through the use of technology, the general findings from the entire data set of micro-blogs, reflective diaries and student interviews are covered by Hepplestone and Chikwa (2014).
Despite the small-scale nature of the study, factors emerged that:  influenced students' initial processes for engaging with feedback in order to attempt to understand and remember it for future use;  enabled students to actually recall and then apply this feedback to future learning; and  demonstrated how technology can support and enhance these processes for making use of feedback, such as the use of mobile technologies to access feedback and support dialogue around that feedback.
The themes identified from the findings were consistent and uniform across the sample, regardless of the demographic profile of the participants.It is worth noting that all students in the study articulated that they were able to identify and recognise feedback and its purpose.

Factors influencing elementary engagement with feedback
The study demonstrated that students employ different processes to engage with feedback in the first instance, depending on the context of the feedback and whether it was returned to students in hard-copy format (e.g.written on their original work) or given to them verbally, and this subsequently impacts on their disposition to understand and remember it for future use.The sample of students within the study generally expressed a preference for written comments, though valued verbal feedback greater where it formed a dialogue between them and their tutor.Hard-copy feedback was viewed as more structured, annotated at the point of reference, and easier to store (and access again) than verbal forms of feedback.
Written feedback is more useful because you can store that piece of paper somewhere in your files and if you ever encounter something similar to it, you can always take that bit of paper and look at it and remember, so you don't make the same mistakes again.
Each student within the study undertook different approaches to engaging with written forms of feedback, including reading over the feedback several times and in some cases reattempting the work again.They claimed that the approach they chose enabled them to understand the feedback so that they could remember and make use of this in future learning, particularly where they were able to identify connections from the feedback to future work.
I would look back at least once at a project, say it's a practical film project, watch it through from start to finish, and try and identify the connection between the feedback given and what's on the screen, and which areas and be able to see for myself what the tutors have seen, and to understand from their point of view.
Verbal forms of feedback required some form of effort by the students to record the spoken feedback or take notes, i.e. it was not given to them in a format that they could readily use.Each student participating in the study elaborated that they took one of two approaches to utilising verbal feedback: they either made notes for following up later, or recorded the feedback using their own personal audio recording devices or mobile phones for listening back to immediately or to retain for future reference.
I prefer it if I can record it [the feedback], because I always like to be able to go back to refer to it […] So if I'd booked in to see a tutor, I'd sort of ask them is it fine if I record, and the majority of time they were happy for me to do that, and I'd be able to listen back to it afterwards.
Verbal feedback was seen as most effective when it formed part of a dialogue between a student and their tutor.Despite their displeasure about the spontaneous nature of verbal feedback, the students within the study all valued a discussion from which they could learn.
I think that I prefer being shown where I have done well or gone wrong on a piece of paper and face-to-face with a tutor and then be able to discuss that, because it enables me to see my own mistakes on the piece of paper and then to talk about it and think of ways and discuss it with the tutorways of how I can improve on that and how I can maintain something that has gone well.
Generic feedback was seen as the least helpful.The majority of students taking part in the study reported that they struggled to understand how the comments related to them and their work individually, and on the whole their use of generic feedback was limited to check on progress and benchmark themselves against their peers.
The only way that I probably used it was to see whether I was sort of ahead of the field.
In spite of this, there was one student in the sample who did attempt to make sense of generic feedback and see how this could be applied to their work.This participant articulated using generic feedback to form a basis for dialogue or discussion with their tutors.
I could make use of it to some extent, but I always wanted to ask questions afterwards.Questions that went specifically back to things that I'd noticed with my own work.
Most students participating in the study reported that peer-feedback, whether written or verbal, formal or informal, and usually in relation to a specific task, has been valuable and they felt able to understand and apply this feedback to future work or learning.This gave them a richer and more varied source of feedback, than by the tutor alone.
It's been interesting to hear verbal feedback from peers rather than tutors.This feedback along with written feedback from tutors helped me to gain a wider perspective of way I can improve with my assignments.
It also emerged that students in the study look for and value opportunities to digest and remember feedback by discussing the feedback that they receive with others, including their peers, tutors or parents.
Sometimes I might remember it and set myself a target saying alright I need to do this and discuss it with them -say I'm talking to my mum, I might say the tutor said this, so that is kind of when I'm talking about it to maybe my mum or my sister or one of my peers in my class, it's easier to remember because we will mention it a few times and I try and put any feedback that I've got into use straight away, because if it's put in to use straight away then you're less likely to forget it.

Factors that enable students to feed-forward
A clear theme emerged from all seven students participating in the study that application of feedback is facilitated where obvious connections can be made between the feedback received and their future learning or work.The various 'conditions' that the sample of students reported enabling them to recall and apply feedback included:  making use of skills-specific feedback, such as time management, working as a group or structuring work; In the sense of universal elements that relate to all work, for example, narrative and things like that, we were able to ensure we structure the work properly and more effectively each time, and also in planning out our time for the projects, it was helpful because each project required the same three main processes, which was planning, execution and then adjustments afterwards, editing.They had all those elements to them, so in terms of scheduling our time, previous feedback did help us to understand how we could better plan our time, where, which areas needed more time spent on them, and also because we worked in small groups in a similar way in each one, we were able to understand how better to work with our peers in order to divide roles more effectively, and produce more effective work.
 making use of feedback specific to a particular assignment type, such as presentations or report writing; One of the first projects we did since starting this [research study] was a report, and I think I got in the 80s for that […] there was feedback in that specifically about the content of that report, but there was feedback on it as well about reports in general, and since then because we've done a lot of other report writing, a lot of those points have been relevant, and the grades have been higher.
 where feedback on a draft submission feeds into the final submission; In completing a 2500 word essay as part of one module, I was able to draw upon feedback from a previous assignment; that assignment involved research which leads onto the current one.

 when preparing portfolios;
Whilst […] selecting what work to use, I have been referring to the feedback […] and making sure I only include the best work.
 where regular tasks are set; There's assignments every other week to do, there's lab sessions to attend, so it just builds up on itself.You get to apply the things you have learned in the following sessions.
Conversely, the majority of students participating in the study made it explicit that any feedback that was module-, content-or assignment-specific was difficult to use and could not easily be fed forward into future work or learning.50 per cent of the time […] you're not going to use the feedback […] it's a very unique piece of work, and it won't be able to be used for anything else.
However, for one of the student participants, they did indicate that there is always something that can be taken from feedback regardless of whether the feedback is specific to the individual assignment or obvious connections to future work can be immediately identified.It was not articulated what processes this student undertook for internalising and taking this feedback forward.
I think that there's always something you can draw from feedback, whether you realise it at first or not.

How technology can enable students to process feedback
The students participating in the study were explicitly invited to respond to how they had experienced the use of technology in enabling them to engage with, recall and make use of feedback.In addition, the student participants were asked what they felt the potential of technology could offer in this regard.All students in the study articulated the logistical benefits offered by feedback that is delivered online via the institutional VLE, including administrative efficiencies in turning around feedback quickly, giving them convenient access to read and revisit feedback wherever they are working, and providing typed and therefore legible feedback.
The students were consistent in highlighting learning benefits resulting from feedback being returned electronically via the VLE, including the ability to check on progress and, most importantly, opportunities to respond to feedback and have a dialogue with tutors.
I prefer it to be online.I didn't mind getting it from the reception at all, but it was sort of -you get it and then you read it and if you don't understand something you would have to email them about it and they probably couldn't remember doing yours.Whereas if it's emailed or online, there's a record of it and you can reply to it.
In addition, many students taking part in the study reported on the usefulness of mobile technologies and associated applications (e.g. the institution's mobile application for the VLE) for accessing their feedback.Like with the VLE, using mobile technologies gave students quick and easy access to feedback.
If [feedback's] on my phone it stays on my phone, so I could refer to it at any time.
These students also reported that accessing feedback via mobile technologies enriched their engagement with feedback.They felt that it gave them greater opportunities to have a dialogue with their tutors about the feedback received, as well as enabling them to set targets and develop action plans.
I might use my calendar for example and say that this needs to be done and it stays on my phone or my computer and so I can look at that and say okay, that's my target date, that's what I have to do and you can send reminders and you can email yourself and stuff like that, so then in ways like that it can be quite helpful.
In addition, these students further indicated that they made use of technology, in particular the use of social media, to gain early formative feedback and to facilitate dialogue amongst their peers.
I also discussed ideas for one of our practical assignmentsan interactive animationwith one of the other students in my class via Facebook messages, sharing my ideas with them and offering insight as to how they could approach the task as they were a bit stuck on what to do with our relatively open-ended design brief for this module.

Discussion
Despite the small-scale nature of the reported study, the findings highlighted that students taking part understood the purpose in receiving feedback, and made efforts to engage with, remember, recall and apply feedback to future learning, where opportunities to do so are somewhat explicit.They also appreciated the role that technology can play in delivering feedback to them efficiently and exploited the use of mobile technologies in having dialogue around their feedback.Notwithstanding the limitations of the study in terms of sample size and self-selection of participants, there was uniformity in the data captured and findings from across the sample, and on this basis it does highlight interesting themes and issues to explore at scale in future studies.

Processes that students use to apply feedback
A core factor in whether students engage with feedback in the first instance, as identified as a result of this research, relates to the context or format in which the feedback is given or received.The sample of students within the study generally expressed a preference for written comments, though made efforts to value other types of feedback, including verbal feedback from tutors or peers, and even generic feedback (which was seen as the least helpful).Students took different approaches to engaging with feedback in order to make sense of it and remember it for future use such as repeatedly reading written comments, recording verbal feedback to listen again or make notes from, reattempting the work again, or reaffirming their understanding of the feedback through dialogue with peers, tutors and, on occasion, parents.
And although it was found that students make efforts to understand and remember the feedback they receive, this is somewhat wasted effort because the students assert that they can only apply feedback when a connection is seen between the feedback received and future learning.Similar frustrations were identified by Pokorny and Pickford (2010) when final year students perceived feedback to be poor or unhelpful if there were no opportunities to apply the feedback, particularly in the same module.This phenomenon is reinforced by Vardi (2013, 599) who claimed that students are 'unsure what to do with the feedback they receive' if there are no opportunities to act on that feedback.There was consensus from all students in the study that feedback can only be applied when at least one of the following five 'conditions' have been met, previously highlighted as important issues that support students' learning by Gibbs & Simpson (2004): 1. Feedback relates to skill development -this condition enables students to improve upon a particular skill (or set of skills) when completing a future assessment, e.g.time management or group work; 2. Feedback relates to the type of assignment -students are able to improve their technique when completing particular types of assignments in future, e.g.presentations, essay or report writing; 3. Feedback on a draft assignment can be fed into the final submission -e.g.where two-stage assignments are set or a research proposal leads into the actual research; 4. Regular formative tasks within a specific module are set -students can build on their work by seeing how each task connects together and apply the feedback between them; 5. Selecting the best pieces of work to include in portfolios -students reflect on the feedback provided to ensure their highest quality work is carefully chosen.
Where the feedback related to specific content or the assignment itself, the students reported that this was difficult to use, and in many cases a waste of staff effort, two reasons previously identified by Gibbs and Simpson (2004).Is this a result of students not being able to articulate deeper engagement with feedback, choosing only to articulate where superficial connections are explicit and visible?Such feedback and its impact on student engagement have been discussed in the literature, e.g.Hattie and Timperley (2007) argue that students are unable to apply feedback that is too task-specific to future assignments, while Higgins et al. (2002, 55) report on Ding's claim that 'if the feedback they receive does not help them to improve generic skills, but is instead focused solely on subject-specific aspects of assignments, then feedback may be irrelevant for subsequent work on other units'.Walker (2009) similarly questions why tutors are providing such unusable feedback on assignments that students are unlikely to repeat.Our findings echo comments by Handley et al. (2011, 551) in that students lack the 'self-efficacy' to make sense of the feedback that they receive and that students display elements of 'visible' and 'invisible' active engagement, where mindful reflection and engagement might be hidden by what appears only to be surface or superficial engagement.This suggests that academic staff ought to have a clear overview of all assessments within a course or programme that students will be required to do in order to: (a) write forward-looking statements that students can feed-forward into future assessments; and (b) look for evidence that students are actually applying feedback when they are reviewing and marking student work.Students should be given explicit guidance on how to learn from content-specific feedback throughout their education, extending the conscious application of feedback beyond the five 'conditions' and processes that students currently use.

How technology can enable students to process feedback
The students made use of mobile devices to access and store feedback, including recording verbal feedback for following up later.Interestingly, they also used mobile technologies to have dialogue around their feedback.For example, smart phones gave students immediate access to their feedback, and they made use of the technology to instantly respond to and ask questions and receive clarification about their feedback.The use of social media (e.g.Facebook) was also mentioned in terms of providing a convenient platform for the students to discuss and share ideas about assessment tasks they are working on.This suggests that students value feedback early, especially from their peers, in the stages of completing work so that they can apply this when writing their final draft for submission.
It is therefore clear that the technology, in relation to the institutional VLE helps with the logistical side of producing, returning, and giving students quick and easy access to feedback, but does not immediately assist students in forging those deeper learning connections between feedback and future learning, and strengthening teacherstudent relationships (Yang and Carless 2013).Students were therefore making use of their smart phones and social media to have a personal dialogue with their tutors and peers around feedback, to clarify feedback received and share ideas.This is consistent with findings in a study conducted by Beaumont et al. (2011) where feedback was seen by students to be part of a dialogic process rather than a summative event.Students made use of dialogue during the draft stage of assignments and after receiving feedback from their tutors on marked assignments.This helped them to interpret tutors' comments and apply the feedback in future learning.The work of Nicol (2010) further advocates that students would like more detailed feedback comments from tutors via a dialogue, and richer feedback via peers and the students in this study are seeking the use of technology and social media to achieve this.

Conclusion
This study sought to explore the processes that students use to understand and apply feedback.Students adopted different approaches to attempt to make sense of feedback, such as repeatedly reading comments on written work, recording and listening again to verbal feedback, reattempting work, or reaffirming understanding through dialogue with others such as peers, tutors and family.
Although students did articulate that preference for type of feedback was a factor in their initial engagement with it, it did not impact on their ability to recall or apply feedback to future learning.Students' use of feedback in future learning was facilitated where explicit connections could be made between previous feedback and future learning.Connections that students highlighted tended to be superficial (e.g.making use of feedback on essay writing, structuring reports or referencing style, undertaking a set of regular formative assessments, where feedback on draft work fed into the final submission, or when selecting the best work for portfolios), and students appeared frustrated where such links were not evident and they could not make use of feedback that they had engaged with and attempted to remember for future use.There was no clear indication that students attempted to make deeper connection between feedback and future learning.This may be a consequence of students being unable to articulate a deeper learning from feedback themselves.Students simply expressed that feedback relating specifically to the content or the assignment was difficult to use.This could be remedied by academic staff having an awareness of all assessments on the course or programme in order that forward-looking statements are written with these assessments in mind, and attributed to at least one of the 'conditions' that students claim enable them to apply feedback, so that students can make use of and apply the feedback received, with appropriate guidance.
Students recognised that technology supports the feedback process, primarily in the logistical aspects of turning around feedback quickly, giving convenient access and storage of feedback, and providing typed and legible feedback.The process was further enhanced by being able to use mobile devices to access their feedback and use the technology to have dialogue about feedback with their tutors regardless of location.Students are making use of social media to discuss and share ideas about assignments with peers, in order to feed-forward these ideas into their final submission.This indicates that students value early formative feedback, where an obvious connection can be seen when completing final drafts, including the use of regular formative assessments that give students the opportunity to apply feedback from one task to the next within the same module.The study highlighted that students often struggle to make connections between the feedback that they receive and future assignments, requiring further exploration into how tutors construct the feedback given and how students deconstruct that feedback.