Using learning diaries to evaluate and improve online information literacy and academic skills provision

The purpose of this paper is to describe a new method for the evaluation of online provision of information literacy and academic skills (ILAS), and advocate for its addition as a tool for the ILAS practitioner. The method is discussed in the context of evaluating its effectiveness for a project to investigate the value of six online tutorials for postgraduate taught students at Cardiff University. The paper presents the advantages and disadvantages of the method compared to the more commonly utilised focus group and interview techniques in this field of study. It concludes that the method has been highly effective in collecting substantial actionable information for the improvement of the development of student’s ILAS and recommends its use in similar projects.


Introduction and context of the study
A suite of online tutorials covering six information literacy and academic skills (ILAS) topics and aimed at postgraduate taught students (PGT), were developed at Cardiff University in response to student feedback requesting improved ILAS provision specifically for postgraduate taught students.The broad topics covered by the tutorials are: 1. Understanding your assessment task 2. Finding appropriate sources of information 3. Evaluating information 4. Academic writing 5. Developing critical arguments 6. Citing and referencing http://dx.doi.org/10.11645/13.1.2519Following the creation of the tutorials, the University Library Service initiated a further project to evaluate the tutorials and students' views of academic skills provision at the University more generally.It is on the evaluation of the tutorials that this paper focuses.Philippa Levy, the Deputy Chief Executive of the Higher Education Academy (HEA) stated in her foreword to the HEA paper Engagement through partnership: students as partners in learning and teaching in higher education that: 'Over recent years, higher education policy initiatives across the UK have emphasised the importance of students' active engagement in their learning, and the benefits to be gained when students play an active role in shaping and enhancing their learning experiences' (Levy, 2014in Healey, Flint & Harrington, 2014 p.4).
Furthermore, Healey et al. (2014) identify four broad areas in which students can act as partners in the developing of their learning experience: 1. Learning, teaching and assessment; 2. Subject-based research and inquiry; 3. Scholarship of teaching and learning; 4. Curriculum design and pedagogic consultancy (Healey et al., 2014, pp.7-8)The research in this paper demonstrates how students can be engaged in aspects of the first and fourth of these categories in the development of online ILAS provision.In order to involve students with the project we combined elements of focus groups, semi-structured interviews, diary study and reflective practice into a single approach that we termed 'learning diaries'.
The primary aim of this paper is to describe and evaluate the learning diary method and discuss its advantages and disadvantages when compared to the more traditional methods of focus groups and interviews in assessing online information literacy (IL) provision from the student perspective.The intent of this paper is not to criticise previous work using more traditional methods, rather it is to advocate for the addition of the learning diary method into the selection of evaluative tools available to the IL practitioner.Therefore, this paper is divided into two parts.First, it presents a theoretical discussion of the learning diary method.Second, it describes its implementation within the project and illustrates the type of feedback this method can generate.

Literature review
Research evaluating ILAS provision has two distinct, though often overlapping goals.First, there is research that seeks to demonstrate the impact on student learning, achievement, skills, or progression of a specific intervention or module and second, there is research seeking to investigate such provision's fitness for purpose, usually with the goal of improvement.
Impact is most commonly measured through quantitative methods including inter alia postintervention testing (Bailey et al., 2007), administrative data analysis (Wray, Aspland, Taghzouit & Pace, 2013) and online surveys (Pryjmachuk, Gill, Wood, Olleveant & Keeley, 2012;Kim & Shumaker, 2015;Coughlan & Swift, 2011).While some studies have utilised qualitative methods, such as focus groups, to investigate other types of impact, it is not the most common measure.For example, Groves, Leflay, Smith, Bowd and Barber's (2013) investigation of the impact of a study skills module on the epistemological beliefs of students.
In contrast, when seeking to evaluate a module's fitness for purpose (which is sometimes conducted at the same time as impact investigations), researchers tend towards qualitative http://dx.doi.org/10.11645/13.1.2519methods.Most commonly these include focus groups and/or interviews (Wray et al. 2013;Pryjmachuk et al., 2012;Willison, 2012).
Studies in this area most often have one characteristic in common, the focus on undergraduate (usually first year) programmes.Furthermore, the majority of studies examine face-to-face modules or interventions, with Pryjmachuk et al. (2012) being an exception to the latter.Therefore, by providing an evaluation of an online intervention aimed at postgraduate students utilising an innovative methodology, this study seeks to add to the existing literature.

The learning diary method
In designing this research, we had three key requirements that the methodology needed to meet.First and foremost, we wanted to provide our participants with the opportunity to reflect on both the online tutorials and their previous academic work.Most importantly this needed to include how the two relate to each other.Second, we wanted to engage with a wide variety of individuals in order to gain feedback from different perspectives to give us the best opportunity to meet the needs of different student groups.Finally, as we were aiming to identify areas of the tutorials that may have been difficult to understand, we wanted to ensure we used a methodology where participants did not feel 'put on the spot' and there was no cause for participants to be reluctant to share opinions due to fear of peer embarrassment.
The traditional approaches for this type of research in IL is that of the focus group and/or oneto-one interview and these approaches were considered.However, a number of issues were identified that were not conducive to the goals of this study.Albrecht, Johnson and Walther (1993) state that it is common in focus groups for members to discuss an issue and offer a consensus opinion to the researcher.While this co-construction of knowledge can have several advantages in an appropriate context (such as encouraging individuals to consider issues they had previously not thought of), it also holds a number of potential problems.Group opinions can be dominated by a single individual, either through persuasive ability or aggressive behaviour (Albrecht et al., 1993).
Similarly, participants can be reluctant to speak out in a group if the topic is of a sensitive nature or if they are afraid of embarrassment.This latter issue is of particular significance to IL research, and a key area of concern for this study, as students can be reluctant to admit they did not understand a topic or concept (Patterson, 2009).This is particularly important at postgraduate level where there is some expectation of existing skills, with students having previously been through an undergraduate degree.Moreover, Barbour (2018) stresses that the main advantage of holding focus groups is the ability to capture the interactions between participants, rather than simply extracting comments from individuals (p.146).As our primary focus was on the students' individual experience of the online tutorials and we wanted to encourage students to reflect on what they did not understand, a focus group was not deemed an appropriate tool.
The other common qualitative approach to this type of research, one-to-one interviews, does not suffer from the issues identified above.However, as one of the primary goals of the research was to gather opinions from a wide variety of students on six different tutorials, the sheer volume of interviews that would have been needed was prohibitively time consuming.Moreover, internet transcription services generally state that the transcription (prior to analysis) of a one hour one-to-one interview takes approximately four to six hours (a one hour focus group takes between six and ten hours), further increasing the time burden of the research methodology (or cost if transcription was outsourced).Finally, both focus groups and interviews analysing a particular learning activity can only be conducted after the learning activity (rather http://dx.doi.org/10.11645/13.1.2519than concurrently).This limits the possibility for collecting detailed feedback on particular aspects of the learning activity, for example, if a participant found the instructions for a particular exercise unclear or if they did not understand the explanation of a particular concept.This information would be missed in a focus group (or interview) if the participant forgets which exercise or concept they had a problem with in the time between the learning activity and focus group session.This latter issue is unavoidable in face-to-face class time for groups of students as it would be too disruptive to the classwork to attempt a simultaneous evaluation.However, for an online intervention designed around individual work, such as the tutorials in this study, it was concluded that an evaluation that allowed participants to reflect on the intervention and how it relates to their recent academic work, while working through the intervention, would be a considerable advantage for identifying specific issues and improvements with the tutorial suite.It was around this principle that the learning diary method was designed.
In its essence the learning diary method is asking a participant to work through an activity and while they are doing so, make notes on their experience.Diary study is considered an effective method of capturing the immediacy of an experience and recording the reactions and feelings of an individual in response to specific events (Symon 2004).While diary study is more commonly utilised for the study of events over a long term, in this case the methodology was adapted to a short-term approach in order to capture participant reactions to a specific event, the use of an online tutorial.
In order to keep them focused on the topics of interest to the researcher, participants should be provided with guidance on what aspects of the learning activity that you would like them to consider and reflect upon.This guidance can take the form of specific questions, topic areas or a combination of both.It must be carefully designed in order to elicit useful responses and to avoid the replication of previous research and general statements.However, participants should also be given the opportunity to write about any additional topics they consider important, or to make suggestions outside the planned topic areas to try to ensure key data is not missed.Moreover, as with focus groups, qualitative topic areas and questions can be combined with quantitative ratings questions if it is appropriate to do so (Silverman, 2017, p.297).For example, you could ask participants to rate how easy to use an online tutorial was.If quantitative questions are utilised it is important to consider sample size when drawing conclusions, particularly if different groups of students are participating, for example natural scientists and social scientists.Furthermore, if the diaries are written in an electronic format then this will prevent the need for lengthy transcription of data.

Implementation in the study
In this study, the prompts and questions were designed to encourage participants to both reflect on their experience of the tutorial and, where appropriate, to reflect on their recent academic experiences and practice in the context of the tutorial.Furthermore, the questions were divided into five general topic areas (plus an 'other' section) based on our areas of enquiry.The questions were: Content 1.For each topic covered by the tutorial please consider whether this information would have been useful to you when undertaking your previous assignments.Please describe any areas that you feel are particularly important or those that you feel may be unnecessary.http://dx.doi.org/10.11645/13.1.2519 2. Was there any content pertinent to this tutorial that is not included that you believe should be added?3. Do you feel that the learning outcomes outlined at the start of the tutorial were achieved (click more info for a reminder)?Do you now have a better understanding of those areas?If not, please give examples of anything that was inadequately covered or missing.
Usefulness 4. Overall, on a scale of 1-7 how important do you rate the topics covered in the tutorial for a.Your study at university b.Future employment 5. Overall, on a scale of 1-7: How relevant was the tutorial to the work you have done as part of your course?6. Please describe how the relevance of the tutorial to your course could be improved.7. Please consider how easy it would be to apply the skills and information in the tutorial to your work at the university.What could the tutorial include to make it easier?8. To what degree was the information in the tutorial new?Which parts of the tutorial have you already covered as part of your academic programme (either through lectures or the library inductions etc.) and which parts were new information?9. What are the most valuable aspects of the tutorial to you? Please explain why.Engagement 10.Did you find the tutorial engaging?If not how could we make it more engaging?11.How would you improve the tutorial to make it more interactive?Were there any specific sections that you felt were not interactive enough? 12. Did you know these tutorials were available prior to today?If so how did you find out about them?If not how could they be advertised better?
Usability and navigation 13.On a scale of 1-7 how easy to use did you find the tutorial?14.Please identify any area(s) you found challenging and why.15.Was it easy to find the table of contents?Did you feel it matched well with your expectations of the topics?Could it be improved and if so how? 16.Could the overall structure of the tutorial be improved and if so how?Any other thoughts 21.Are there any other comments, suggestions for improvement or changes you would make and why?
Participants were asked to work through a single tutorial (of the six) and were provided with an online form in which they could answer the questions.While it is not strictly necessary for learning diaries to be conducted in a timetabled workshop session, we did so in this study in order to allow participants to ask questions if they encountered any difficulties.However, a small number of participants did fill in further diaries for one of the other tutorials at home.The workshop sessions were optional but attendance was incentivised with an Amazon voucher of up to £20 depending on time spent (most participants received £10 for a one hour session).In each session participants were instructed that they did not need to answer every question and could do so only if they had a comment or suggestion to make.They were also asked to try to avoid single word answers and to give as much detail as possible.
Fifty-one students participated in the learning diaries from three different disciplines: chemistry, engineering (from the College of Physical Sciences and Engineering [PSE]) and education (from the College of Arts, Humanities and Social Sciences [AHSS]).A total of 60 diaries were completed.All students were enrolled in a full-time PGT level course.The engagement with students was conducted part way through the academic year in order to ensure participants had experience of written assignments at PGT level to reflect upon.The primary purpose was not to teach students ILAS, rather it was to collect feedback on the tutorials and to assess their capability for meeting students ILAS needs.However, it was anticipated that participants would also gain some benefit to their ILAS prior to undertaking their final assignments or dissertation.
Table 1 shows a breakdown of three characteristics of the participants which were considered during analysis.Ideally, recruitment would have ensured that the part-time, distance learning and different age groups of students would have also been included in the sample for analysis but unfortunately, this was not possible due to time constraints.This is a possible avenue for further research.

Data analysis methods
The learning diaries produced a small amount of quantitative data and initially, descriptive statistics were used to explore the data.The statistics used were the arithmetic average (mean) and standard deviation, along with response distribution analysis.Non-parametric Mann-Whitney U tests were also conducted to examine differences between groups of participants http://dx.doi.org/10.11645/13.1.2519 and the different online tutorials.As in standard statistical practice, this test was selected over the t-test due to likely violation of the assumption of normality for the sampling distribution; the central limit theorem did not apply due to the relatively small sample size (Field, 2018).Finally, numerical summary data was produced for the learning diaries to examine the overall response demeanour of the qualitative data.
The recognised technique for identifying and analysing recurrences of the same theme within transcripts of qualitative data is thematic encoding (Gibbs, 2018).Gibbs (2018, p.54) defines this as a process whereby the researcher 'identifies and records passages of text or other data item … that, in some sense, exemplify the same theoretical or descriptive idea'.The purpose of this research was to identify and classify requests for improvements to the online suite of tutorials or ILAS development policy.Therefore, thematic encoding was deemed the appropriate method for this phase of the analysis.The coding strategy utilised was, as far as is possible, an 'open coding' strategy.Open coding is a data driven approach where the researcher attempts to develop themes and codes purely arising from the data without any preconceptions on what might appear (Gibbs, 2018).Initially this involved a line-by-line coding approach within each data source.The codes were then refined using an iterative approach to group together similar concepts, thus developing an analytical framework.This approach is consistent with grounded theory, although this project was not conducted using a pure grounded theory approach as a literature review was completed prior to data collection and analysis (Gibbs, 2018).It should be noted that this is the same analysis technique commonly used for focus group and interview transcripts.

Importance, relevance and ease of use of the tutorials
Table 2 shows the mean score for each of the four ratings questions (4a, 4b, 5 and 13) included in the learning diaries broken down by four key characteristics investigated in this study and the overall score (all rated between one and seven with seven being the most positive rating).The overall rating given for each question is reasonably high at between 5.3 and 6.2.Therefore, it is reasonable to conclude that, based on feedback; the tutorials are both fit for purpose and are effective.Moreover, the table demonstrates that there were very limited differences between each group, with most reporting a mean score either the same (to one decimal point) or within 0.1.
The largest difference is between the AHSS (Arts Humanities and Social Sciences) and PSE (Physical Sciences and Engineering) students in their rating of the importance of the tutorials to future employment at 0.5, still a relatively small difference.Nonetheless, these group differences were tested for statistical significance using the Mann-Whitney U test and all returned non-significant results (all p values > 0.3) suggesting that none of the selected groups has any influence on responses to the four questions.This must be caveated with the relatively small sample size in this project, as with a bigger sample it may be possible to detect smaller differences due to the increase in statistical power (Field, 2018).However, even if this were the case the effect would remain very small.Therefore, this demonstrates that collectively the tutorials are reasonably equally effective at meeting the needs of these respective groups.http://dx.doi.org/10.11645/13.1.2519In breaking down the mean scores by individual tutorial, Tables 3a and 3b show there is more variation in mean scores by tutorial than there is by respondent group.However, due to the small sample size for individual tutorials (as small as eight for some tutorials) these data are extremely sensitive to outliers and therefore only tentative conclusions can be drawn.The relatively low score (4.4) for the Understanding your assessment task tutorial for its importance to future employment is understandable due to the tutorial's focus on university assessment.However, of more concern is the relatively low relevance to course scores (between 4.7 and 5.1) for three tutorials.While these are by no means poor scores, previous research highlights the importance of this number as it concludes that students are far more likely to utilise resources and support that they see as directly relevant to their subject area and/or course (Thornes, 2012;Baik & Greig, 2009;Durkin & Main, 2002).http://dx.doi.org/10.11645/13.1.2519

Citing and referencing
Importance of content of tutorial to your study at university 6.3 (1.2) 6.5 (0.9) 6.6 (0.5) Importance of content of tutorial to your future employment 4.9 (2.0) 5.9 (1.2)

Improvements to the tutorial design and content
Analysis of the responses to the learning diary questions identified 15 broad themes in the student comments and suggestions and these are presented in Table 4.The feedback (categorised into themes in Table 4) was used to inform improvements to the tutorials themselves.These improvements or changes can generally be divided into three types: adding new content or topics, enhancing existing content or removing unnecessary content.For example, it was suggested that new content on self-plagiarism should be added to the citing and referencing tutorial.Similarly, several students suggested additional content on databases for the finding information sources tutorial.For example, 'could give a [sic] information on web of science and Scopus including tutorials on how actually to use them and how to save references etc.' These and similar suggestions identified areas where students' needs were not being met by the current provision and appropriate new content was developed.
Suggestions to enhance content tended to focus on adding exercises to existing topics.For example, one student identified a problem with one of the tutorials, which had a particularly long video.They suggested: 'Maybe split it [the video] into 4 sections with related questions between them.It's not necessarily about being fun or engaging, its [sic] about making interactive work and breaking it up, to make the student feel it's engaging'.Similarly, another student stated: I felt that an exercise right at the start would have been much more engaging and increase motivation, setting the stage and allowing reflection on my own presumptions instead of pre-loading me with information that I then attempt to correct adhere to.
This, and similar suggestions, identified key areas in the specified tutorial where interactivity could be improved to enhance learning and appropriate learning activities were developed.
Lastly, participants identified content within the tutorials that they felt was unnecessary or not relevant to the topics.For example, one student stated: I feel that the library section of the tutorial is not needed as there is usually a lesson dedicated at the start of course which introduces the library and is subject specific and tells you where to find relevant books, journals, articles etc.
Similarly, an exercise utilised two large examples and participants suggested that only one was needed, particularly as one was hard to understand by non-subject specialists (it was a http://dx.doi.org/10.11645/13.1.2519technical biological sciences based example).This allowed such content to be removed to avoid duplication and enhance understanding.
However, in addition to specific improvements to the tutorials there were a number of key themes that provided more general guidance for the design and implementation of ILAS provision for PGT students, with a particular focus on online content (although there was some overlap).It is on these themes that the following sections focus.
Finally, participants of the learning diaries were asked what they considered to be a key area for the tutorial and what information it contained that was new to them.However, the responses to these questions were inconclusive and no meaningful conclusions can be drawn and therefore analysis of these themes have been excluded from this paper.

Advertising
Question 12 of the learning diaries, which asked students if they had heard of the tutorials prior to the learning diary sessions and if they had a suggestion for how they could be better advertised, highlighted an important issue with the tutorials, that of awareness.Participant responses indicated that only 16.7% of students had prior knowledge of the existence of the resources before the learning diary sessions (8.3% did not say either way).Furthermore, there were 52 suggestions from students on how the tutorials can be advertised more effectively, with the most common being: through the University website (including Learning Centralthe University's Virtual Learning Environment), induction, student email, and lecturer adverts (nine, eight, six and six occurrences respectively).
Moreover, 22 instances of student feedback highlighted the importance of early access to skills development support (the timing theme).They suggested that the tutorials would have been greatly beneficial immediately prior to essay work or more generally at the start of their course.For example, 'Having more tutorials like this intermittently before essay hand ins (at the point of a [sic] essay module launch) would be very beneficial' and 'This would have been useful at the start of the course, especially when our lecturers discussed writing essays at a Masters level'.

Level
There were 20 comments from students that discussed the level of study the tutorials could/should be used for.The majority of student comments in this theme suggested that content needed to be more challenging or was better suited to undergraduate students.For example, 'This should probably be a compulsory task for all first year BA degrees' was one student's view and another stated '[the tutorials] could be more relevant with "harder" content'.Furthermore, some students expressed the view that students working at postgraduate level should already be familiar with these skills.For example: 'I think the information is covered in great depth which is good.However, I feel that most of these skills should already be developed in someone at a postgraduate level'.
In contrast, there were a small number of comments from international students whose first language is not Welsh or English who suggested that the tutorials needed to be 'easier to follow' or that they found them too 'challenging'.Determining an appropriate level for PGT students is challenging due to the varied nature of the student population and so this is an area that has been identified for further investigation.http://dx.doi.org/10.11645/13.1.2519

Learning difficulties
There were a small number of instances (10) of participants in the learning diaries commenting on how the resources could better meet the needs of students with learning difficulties For example: 'The layout needs to be more user friendly for people with learning difficulties' and, 'I found the whole tutorial of [sic] very complicated and not very dyslexia friendly in terms of colour and font'.The latter quotation also highlights a relationship between comments about learning difficulties and the visual improvements theme.The majority of participants who commented on learning difficulties also made some form of suggestion for a visual improvement, either within the same comment like in the quotation or in a separate comment for example, '…Mostly though I would tweak design and font elements'.
There were also suggestions for visual improvements that were submitted by participants who did not mention any form of learning difficulty.However, when such improvements were designed and implemented the impact on students with learning difficulties or accessibility requirements was taken into account.

Subject specific
There were 42 instances of student participants suggesting subject specific exercises, examples or content would be beneficial in improving the relevance of the resources to their course.Most of the instances ( 22) were in response to the question asking them to suggest how the tutorials could be made more relevant to their course (question six in the student guidelines).However, there were also five responses to questions one and seven (useful to previous assignments and ease of application of skills).Student suggestions ranged from generic suggestions such as 'more subject specific' and 'it might be better giving examples in engineering or science aspect', to more detailed suggestions of specific content such as 'more information on referencing, especially on subject specific styles, e.g.. RSC [Royal Society of Chemistry] style referencing for chemistry based essays'.
These findings are consistent with literature in this area, which concludes that students are considerably more likely to engage with resources they see as specific to their subject area (Durkin & Main, 2002).Moreover, it supports active learning theory and skills development theory which posit that students will be able to apply their learning more effectively to their own work, and that students' skills will advance more rapidly and to a higher level if they are developed in the context of their own core subject area (Wingate, Andon & Cogo, 2011;Wingate, 2006).

Active learning
Active learning is a method of instruction that places emphasis on the importance of providing learners with exercises and tasks to do in order to practise the skills taught (Blevins & Besaw, 2011;Wingate, 2006).Using active learning principles in the design of online resources for teaching ILAS has previously been established as best practice by authors such as Anderson andWilson (2009), andBlummer &Kritskaya (2009).
Therefore, the ILAS tutorials evaluated as part of this project were originally designed using active learning principles; thus the project sought to assess the evidence for the success or otherwise of this approach within the student feedback.There are several sub-themes across three of the broader themes that evidence the importance of continuing to use an active learning model in resource development.
For example, within additional content there were various requests for more exercises, more examples and more interactivity (combined total was 42 out of 92 occurrences).Similarly, within http://dx.doi.org/10.11645/13.1.2519 the problems with the tutorial theme there were 17 instances (out of 71) of participants complaining about unbroken long passages of text, a particularly long video or slide show.Each of these indicates a preference among the participants for an even higher degree of active learning within the tutorial content.
In contrast, there were a small number of instances within the exercise improvements theme in the learning diaries that indicated a preference for fewer exercises (5) or fewer questions on the same topic within individual exercises (2).The latter can be interpreted as a request for greater variety of exercises in order to learn how to apply knowledge in different contexts and thus also supporting active learning principles.However, the former is in direct opposition to active learning principles and to the majority of the feedback received.

Evaluation of the learning diaries method
As part of our evaluation of the learning diaries method, we analysed the content of the openended questions for type of feedback.This excludes quantitative questions four (parts a and b), five and thirteen, as well as question fourteen which was specifically asking for challenging areas within the subject content, rather than of the tutorials themselves and thus was not appropriate for this type of analysis.Within the remaining questions, there was the possibility for 1020 total responses from participants and Table 5 summarises how these can be categorised.
Responses were generally classified as positive if they praised some aspect of the tutorials or responded in a positive way to a specific question asked.For example, statements such as 'it's great.Has been written in a clear and concise way' or 'Yes definitely' in response to question three which asked if the participant felt the intended learning outcomes for the tutorial had been met.
Similarly, negative responses tended to criticise the tutorials in some way or respond negatively to a particular question.Table 5 shows that there are more than six times the number of positive responses as negative, supporting the positive feedback from the quantitative questions.However, it is the number of suggestions (300) that highlights the success of the learning diaries as a tool for eliciting actionable feedback from participants for improving the tutorials.Moreover, this is only a guideline figure as participants could and often did make more than one suggestion within a single response, further highlighting the success of the method.The learning diary method successfully met the project aims for its implementation, including inter alia the wide range of student participants in a relatively small timescale, collecting detailed feedback on the tutorials and engaging students in critical reflection of the use of ILAS in their academic work.A further benefit, that was not previously anticipated, was that it was possible to easily attribute comments and opinions to individuals in the learning diaries.In focus groups individual participants competing to tell their own story or present their opinions can generate 'noise' making it difficult to determine to whom comments or opinions should be attributed (Barbour, 2018 p.19).This was particularly useful in this study where students from different groups were participating.This attribution allowed the project team to examine whether comments on a particular subject were only coming from one student group (for example, education students but not engineering or chemistry students, only international students etc.)This potentially allows for targeted advertising and provision to be established.
In addition to the limitations to the data collected by using the learning diary method already described, the researchers observed some further disadvantages to the approach.First, it is difficult to clarify points made if you are not sure what the participant means.In a focus group or interview you can interact with participants to clarify a point.However, with the learning diaries there is no system in place for this.Similarly, if a participant raises a point of interest the learning diary method has no in-built system to ask the participant to expand on that point.If this is a concern, then studies adopting this method could request permission from participants to be allowed to contact them at a later date for further information or clarification.Second, as participants were not observed while completing the diaries, it is impossible to analyse if participants missed any content in the tutorial as they would not be aware it was missed and therefore can't comment about it (unless they suggest it as an addition).A related issue is that while learning diaries can be useful for identifying specific issues around navigability, usability, and eliciting suggestions for how to improve them (as in this project), it is difficult to get a complete picture of these concepts from learning diaries alone.Therefore, it may be appropriate to combine a learning diary approach with an observational methodology.
As with any project of this type, participant recruitment was also challenging due to time demands placed on students.In this project, we initially attempted to recruit students for a faceto-face workshop outside their timetabled class time.However, when this met with very limited success, we approached academic staff on relevant programmes to see if our sessions could be added to the timetable (for example, we were added to a National Student Survey timetabled session).This proved far more successful as, while students were informed the sessions were not compulsory by academic staff and they were asked to accept a consent agreement, all students elected to stay in the sessions.Although learning diaries do not need to be conducted in face-to-face sessions, in this study we chose to do so.This enabled us to ensure that participants were aware that sessions were primarily intended for them to help develop the support they (and future students) receive, rather than specifically to learn academic skills (although this was expressed as an anticipated benefit to the sessions).Moreover, it allowed us to provide explicit instructions and to answer any queries participants may have had during the sessions, which would not have been available in a purely online engagement.Once participants understood what was required of them, a small number offered to complete further diaries on additional tutorials at home at later date and were provided with further incentives for this work.If the project team were to run a similar engagement again we would build this option into the recruitment strategy.
Finally, it must be acknowledged that some questions were more successful than others in eliciting actionable information.For example, we hoped that the questions regarding new information would highlight trends in the gaps in student knowledge but there was no discernible pattern to responses.Similarly, we had hoped that asking the students what the key areas of http://dx.doi.org/10.11645/13.1.2519 the tutorial were for them would highlight differences between disciplines.However, again no discernible pattern was present.It is possible that with a larger sample such patterns may have been visible, but nonetheless, if we were to conduct this research again we would exclude or redesign these questions.

Conclusion
The project set out to engage PGT students in aspects of the design and implementation of ILAS tutorials and the learning diaries proved to be an effective method.This new approach has allowed the University Library Service to gather substantial actionable information on its online provision of ILAS with the creators of the suite of tutorials impressed with both the breadth and detail of the feedback they provided.We have been able to identify key changes to ILAS provision including improvements to the content of the tutorials and an improved advertising and student engagement strategy.In short, it has led to a substantive improvement of online ILAS provision for PGT students at the University.
Practical tasks 17.Were the practical exercises useful?If not please give examples of which exercises you did not consider useful and why.If you have an idea of how they could be made more useful please also provide it.18. Was it easy to understand what was required for the practical tasks?If not please give examples of which tasks were difficult to understand and how you think they could be improved.19.Did the practical tasks help you understand how the information should be used in your own work?If not please give examples of what was unclear and how you feel it could be improved.20.Were there any topics where you felt a practical exercise would be useful and there wasn't one?If so please specify which and why? http://dx.doi.org/10.11645/13.1.2519

Table 1 :
Student participant characteristics

Table 2 :
Learning diary response ratings by category* *This table is based on total learning diary responses which will include some participants more than once as they filled in a diary for more than one tutorial.

Table 3a :
Learning diary response ratings by tutorial part one

Table 3b :
Learning diary response ratings by tutorial part two QuestionMean rating out of seven for each tutorial (standard deviation)

Table 4 :
Occurrences of themes in the learning diary data (top level)

Table 5 :
Summary of overall responses

Type of response Percentage of total responses (total: 1020)
Note: there is a small amount of crossover between positive and negative responses with suggestions, as for questions 10, 12 and 15 it was possible for a participant to respond with both a positive/negative and a suggestion.http://dx.doi.org/10.11645/13.1.2519