Meeting Information Literacy outcomes: Partnering with faculty to create effective Information Literacy assessment

This paper outlines the attempt by librarians at California State University Channel Islands (CI) to authentically assess the information literacy levels of first-year and third-year students, their partnership with faculty from CI’s writing and rhetoric programme in receiving a grant for this endeavour, the creation of a rubric and specialised assignment to facilitate the assessment process and the initial assessment data that came from the three-year grant period. Information gathered during our pilot study suggests that student information literacy proficiency levels can been determined by assessing student writing assignments, and that a targeted rubric is an effective authentic assessment measure in this endeavour. This study is of use to practising librarians as it highlights the efforts of the authors to partner with faculty, not only in assessing student papers, but also in creating a rubric and specialised bibliography assignment that can be used by librarians and faculty at their institutions. This article is based on a paper presented at LILAC 2011. score on these students with only minimal data. Students were shown to be either at, above or below baseline levels of ICT proficiency; no other data (such as proficiency in a particular competency) were reported. Librarians at CI wondered if a more authentic measure of IL (as opposed to a standardised test) would yield better information about the IL levels of students at CI. the attempt by librarians at CI to authentically assess the IL levels of first-year and third-year students, their partnership in a grant for this endeavour with faculty CI’s writing and rhetoric who instruct first-year composition and creation rubric specialised


Introduction
In 2005, librarians from all 23 California State University (CSU) campus libraries were asked to beta-test an assessment product from Educational Testing Service (ETS). The product was designed to assess information competency and technology (ICT) literacy in students at various academic levels in their university careers (first-years, third-years and graduating fourth-years). Librarians at CSU Channel Islands (CI) were asked by ETS to beta-test the ICT literacy assessment product with 50 random student participants from various academic levels. ETS returned score reports on these students with only minimal data. Students were shown to be either at, above or below baseline levels of ICT proficiency; no other data (such as proficiency in a particular competency) were reported. Librarians at CI wondered if a more authentic measure of IL (as opposed to a standardised test) would yield better information about the IL levels of students at CI. This paper outlines the attempt by librarians at CI to authentically assess the IL levels of firstyear and third-year students, their partnership in receiving a grant for this endeavour with faculty from CI's writing and rhetoric programme (hereby referred to as "writing faculty") who instruct firstyear students on composition and rhetoric, the creation of a rubric and specialised assignment to facilitate the assessment process, and the initial assessment data that resulted from the three-year grant period.

Institutional context
CI is a relatively new four-year university in the twenty-three campus California State University (CSU) system. The first freshman class started in 2003, and the campus was fully accredited by the Western Association of Schools and Colleges (WASC) in 2007. At CI, IL is seen as a campus responsibility-three IL standards are included in the university's General Education student learning outcomes. Meeting these General Education outcomes is a requirement for students to graduate from the university. The three IL outcomes determined by the university are:  the information literate student accesses information effectively and efficiently;  the information literate student evaluates information sources critically;  the information literate student explains legal, ethical and social issues associated with information. In deciding to include IL outcomes as part of its General Education requirements for graduation, the university adopted standards for IL that are similar to the broader Association of College & Research Libraries (ACRL) IL Competency Standards for Higher Education (2000).
The library's IL Coordinator targets specific first-year (freshman), third-year (upper division) and fourth-year (capstone) courses for library instruction each semester. As a result, approximately 1700 students come into the library for course-related IL instruction each semester; instruction librarians lead IL sessions for over 3500 students each year. At CI, the terms library instruction session and library IL session are used interchangeably.

Literature review
A review of the literature focused on three areas: authentic assessment of IL (specifically portfolio assessment), partnering with faculty to assess IL, including tutor-and student-centred approaches, and the use of rubrics as an effective authentic assessment measure.

Authentic assessment of IL
Interest in alternatives to traditional measures of IL has been growing in recent years, with many librarians developing their own tools to assess aspects of IL (Walsh 2009). Concern about students' ability to locate, evaluate and incorporate relevant information into their academic work, along with a desire from educators to highlight evidence of student learning, has accelerated interest in new methods and tools that measure IL (Brown and Kingsley-Wilson 2010). Increasingly, "performance" or "authentic" assessments that look at students' demonstration of skills are gaining popularity (Knight 2006). Authentic assessment focuses on the practical application of tasks in real-world settings and is a direct measure of the acquired knowledge and skills that students use to perform authentic tasks (Fook and Sidhu 2010). Additionally, authentic assessments examine students' analytical skills as well as their ability to integrate new learning into existing knowledge, giving equal weight to the process as much as the finished product (Snavely and Wright 2003;Jacobson and Xu 2004). Various types of authentic assessment include performance assessment, portfolio assessment and self assessment (Sharma 2007;Callison 1998).

Portfolios for authentic assessment of IL
Portfolio assessment of IL skills is increasingly popular (McGuinness and Brien 2007;Andretta 2005;Fourie and van Niekerk 1999). Portfolios can provide a multilayered view of student performance, experience and reflections (Hunt et al. 2000) and can afford instructors evidence of student growth as well as other information that can be used for both formative and summative assessments (Sharma 2007). Walsh (2009) notes that while there is considerable discussion of the benefits and drawbacks of portfolio assessment within the literature (Sharma 2007;Sonley et al. 2007;Fourie and van Niekerk 1999), discussion often focuses on assessment of specific aspects or modules of IL, not on how reliable or valid portfolio assessment is for assessing what it means to be information literate. Walsh points to the work of Scharf et al. (2007) as an example in the literature of portfolio assessment that attempts to produce a reliable and valid tool to assess IL.

Partnering with faculty to assess IL
Academic librarians are not alone in their interest in student IL skills (Oakleaf 2011). Examining student assignments such as research papers, writing portfolios and research journals can help librarians and faculty not only assess writing assignments but the research process as well (Belanger and Bliquez 2012). Literature on this topic often challenges librarians to rethink their traditional roles and to build relationships with faculty outside of the library in order to become more active partners in the educational process (Ducas and Michaud-Oystryk 2003). Rockman (2002) asserts that it is essential to collaborate with faculty to advance IL goals. At CI, librarians have faculty status, serve on university-wide committees, participate in university governance and teach credit-bearing courses in disciplines outside of the library. Because we collaborate with faculty on a daily basis, we have seen first hand the benefits of partnering with faculty to enhance student learning. Going beyond traditional library goals and viewing assessment from other departments' viewpoints creates opportunities for librarians (Mark and Borouff-Jones 2003;Brasley 2008). "Librarians and teaching faculty have many mutual goals and concerns. Both want to…help students become…critical thinkers, and self-directed, lifelong learners" (Yousef 2010). Collaboration is not only useful for improving student achievement; it is essential in improving students' IL skills and behaviours (Immroth and Lukenbill 2007). In seeking partnerships with faculty, librarians discover that faculty are also interested in critical thinking, the effective use of information and technology, the search process and collaborative reasoning (Smith 2001).

Tutor-centred and student-centred approaches to IL and assessment
'A tutor-centred approach to IL focuses on the transmission of knowledge…with tutors concentrating more on content than on student processing' (Brown 2003). In examining the work of Rosenshine and Stevens, Schug (2003) found that effective teacher-centred instructors provide students with a statement of goals, give clear, detailed instruction and explanations and provide a high level of practice for students. By contrast, student-centred IL encourages students to take responsibility for their own learning by engaging with resources independent of tutors, and to perceive tutors as learning facilitators (Andretta and Cutting 2003). Keene et al. agree, finding that the most successful IL strategies were student-centred approaches that enabled students to appreciate the relevance of evaluating resources, and helped them develop appropriate criteria for evaluation, a process which they would be likely to continue to use in future learning contexts (2010). IL instruction in higher education is necessary, because it cannot be assumed that students acquire information-seeking and evaluation skills on their own (Helvoort 2010;Brand-Grewel et al. 2005). Learning goals are achieved by active collaboration between the teacher and students, who together determine what learning means for each individual student (Schuh 2004).
Librarians at CI employ a mixture of tutor-centred and student-centred approaches to learning. In each IL session, librarians plan and teach concepts and skills to students based on the research assignment students will be undertaking in the course. Librarians impart expert knowledge and map out for students the scope and direction of the session so that students know the goals and intent of the session. Students gain skills and confidence through hands-on, active learning, searching and research exercises that allow them to locate and evaluate sources for quality and relevance. Instruction sessions with a librarian are always crafted around students' own research topics and assignment needs, as well as their individual IL proficiency levels. There is never a onesize-fits-all approach to IL instruction. It should be noted that the assessment effort addressed in our study takes a more tutor-centred approach. Student assignments were assessed by librarians and faculty using a specially devised rubric. These assignments had previously been assessed by writing faculty using their own programme rubric. We desired to examine authentic, previously graded writing assignments for evidence of IL and student learning. Our tutor-centred approach allowed both librarians and faculty to gauge student learning since both were familiar with what students had been taught in class and in IL sessions (if they had one). Both librarians and faculty were also aware of the grade students received on assignments completed after these IL sessions.

Rubrics as an authentic assessment measure
Current literature supports the trend that using rubrics for authentic assessment can provide a more meaningful picture than traditional assessment methods. In higher education, rubrics are considered a form of authentic and objective assessment (Knight 2006). Fielden and Foster (2010) reviewed of the literature on the history and use of rubrics, citing the work of Oakleaf (2007), Stevens and Levi (2005), Knight (2006) and Moskal (2000), among others. An important facet of authentic assessment is its focus on process. Authentic assessment looks at a product as well as the process that students go through to create that product (McGuinness and Brien 2007). Assessing student work allows for an examination of the process of student learning, not simply the outcome (Bussert and Pouliot 2010). Sharma (2007) found that assessing the process students undertake in gathering information and approaching problems is as important as evaluating the search results that students find. "Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes" (Pausch and Popp 1997, n.p.). Helvoort (2010) surveys the literature on the potential disadvantages associated with using rubrics, including rubric creation issues (Diller and Phelps 2008;Knight 2006) and issues related to reliability of the rater (Oakleaf 2009;Scharf et al. 2007). How we addressed issues of rater reliability is explored in our Methodology section.

Methodology
In 2006, librarians and faculty began the process to authentically assess student work. At the start of the grant, the university had only been in operation for three years. The focus was on creating and growing a university, not on assessment of students or programmes. Consequently, assessment data related to the IL levels of students at CI was non-existent.
The pilot study took place between 2006 and 2009. From the beginning, the authors sought to examine student writing as a way to authentically assess student success in achieving the university's IL outcomes. To that end, the authors sought partnership with faculty. Students in writing and rhetoric courses receive library instruction each semester, and the authors have a strong history of collaboration with writing faculty. Participants included three librarians and five faculty members. Both the authors and faculty agreed that a rubric would be the best way to authentically assess student writing; faculty already assessed student work using their own rubric based on their programme's writing and research outcomes, and librarians desired to pursue an authentic assessment of student work. The intent was to create a rubric that could be applied to all first-year writing assignments that require research, with the idea that the rubric could eventually be used to assess third-year student writing and fourth-year capstone research projects as well.
The group met often to brainstorm, examine existing rubrics from other colleges and universities and devise possible criteria for the rubric (see Appendix A). The authors and faculty agreed to align rubric criteria to the three campus GE outcomes for IL and not to any additional criteria, such as those reflected in ACRL's IL Competency Standards for Higher Education (2000). The rubric contains three proficiency levels per criteria: emerging, proficient and advanced.
The rubric was applied to student writing assignments from first-year writing courses that had IL sessions with librarians. Throughout each term, students self selected writing assignments to submit to their writing portfolio (a collection of work each student submits for a grade). At the end of each term, librarians and faculty separately applied the rubric to student portfolios using a score sheet created along with the rubric (see Appendix B. Note that COMP classes are writing and rhetoric classes). Faculty had previously applied their own programme's grading rubric to these assignments and librarians purposely had no knowledge of these prior grades.
Librarians were also unaware of the "normalising" process (a process by which raters grade sample papers together and review results as a group in order to maximise rater reliability) that faculty undertook in assessing these assignments. While both librarians and faculty applied the rubric score sheet to the same student products, they did so independently of each other during the first year. Study participants met again at the end of the semester to examine scores and evaluate the process.
During the second year, the same rubric was used to assess the same lower division writing courses. While librarians and faculty were satisfied with the rubric in terms of criteria, it was clear that some student writing assignments were easier to apply the rubric to than others (student portfolios contained self-selected writing assignments that ranged from problem/solution papers to narratives to individual or group research papers). To address this issue, a specialised reflective annotated bibliography component (see Appendix C) was created to minimise the difficulties that arose in trying to apply the rubric to a disparate array of writing assignments. It was also clear to librarians that rating the assignments separately from faculty was problematic; rubric scores tended to vary widely between faculty (who had previously assessed the assignments) and librarians (who were assessing the assignments for the first time). During the second year, librarians and faculty applied the rubric to student assignments together, eliminating confusion and improving scoring consistency.
The reflective annotated bibliography requires students to reflect upon the research assignment at hand, locate what they consider to be appropriate resources and then reflect upon each of these sources individually, using criteria stated in the bibliography prompt. By pairing the reflective annotated bibliography component with student research, librarians and faculty sought to highlight the research process undertaken by students and to discern the reasoning behind why students chose and incorporated the sources that they did. The reflective annotated bibliography is flexible enough to be paired with any research assignment, and having student reflections as part of assignments helped raters better identify evidence of IL in students' written work.
At the conclusion of year two, librarians and faculty reflected upon the effect, if any, of pairing the annotated bibliography assignment to student research assignments. Both librarians and faculty agreed that the reflective component of the annotated bibliography greatly facilitated the rating process. Raters were better able to see evidence of a student's intent in choosing or incorporating a source into an assignment, even if that intent wasn't immediately clear from examining the assignment alone.
During year three of the study, additional third-year, upper division (UDIGE) student writing assignments were assessed along with assignments from first-year writing courses. Over the course of the grant, all participating faculty were paid a small stipend (funded by the grant) to collect student writing assignments and rate the assignments with librarians.

Applying the rubric
Using the rubric score sheet, raters reported the type of research assignment, whether the assignment included the reflective bibliography component and the amount of IL instruction given (and by whom) in conjunction with the assignment. Both librarian and faculty raters had knowledge of the amount of IL instruction given to each class; this may have introduced unintended bias to the rating process. Librarians and faculty were interested in discerning IL levels of students. For their part, librarians were also curious to see if the rubric would indicate a correlation between the amount if IL instruction and the IL levels of students. Writing assignments were scored as emerging, proficient or advanced, summarized as:  Emerging: Student fails, struggles, is unaware or demonstrates little understanding;  Proficient: Student demonstrates proficiency, understanding;  Advanced: Student demonstrates thoroughness, critical thinking, comprehensive understanding.
IL instruction in courses ranged from "one-to-multiple sessions with a librarian in addition to IL instruction by the faculty member" to "no IL instruction of any kind by either the faculty member or librarian". IL instruction occurred more often in first-year courses than in UDIGE courses (see Table 1). At CI, all IL instruction by librarians is tailored to the research assignment and specific needs of instructors. A primary goal of the library's instruction programme is to provide assignment-specific IL instruction that is unique to the individual needs of students in a particular course. Variations in the amount and type of IL instruction in these courses were not taken into account in applying the rubric, nor was any attempt made to standardise the amount and type of IL instruction in these courses. Librarians and faculty wanted to conduct the assessments in authentic course environments.  Between 2006 and2009, 190 individual and group writing assignments were assessed using the grant rubric with the aim of discovering information regarding the IL levels of students whose work was part of the study. The writing assignments assessed were from first-year and third-year students, and they comprised a range of assignments, from individual problem/solution papers to group research projects. Using the rubric score sheet, librarians and faculty assigned scores of emerging, proficient or advanced for each of the rubric's three criteria: the information literate student accesses needed information effectively and efficiently; evaluates information and its sources critically; explains the economic, legal, social and ethical issues surrounding the use of information. Comp 105

Group
Group annotated bibliography and group research paper Scores ranged from 1.43 ("emerging") to 2.61 ("proficient") (see Table 2). Here, Outcome 1 refers to: "the information literate student accesses needed information effectively and efficiently" and Outcome 2 refers to: "the information literate student evaluates information and its sources critically". Outcome 3, "the information literate student explains legal, ethical, and social issues associated with information," was intentionally not included in Table 2. Raters found they were unable to assign scores for this outcome to student papers. Unless explicitly asked in their assignment to explain the legal, ethical and social issues associated with information (which no student was assigned to do), both librarian and faculty raters found no evidence of this outcome in the student writing assignments they assessed. Although Outcome 3 is one of the university's student learning outcomes for IL, librarians and faculty concluded that unless a student explicitly references legal, ethical or social implications associated with information in their papers or reflective essays, assessing evidence of this outcome will always be problematic, if not impossible. To this end, both librarians and faculty encourage students to think critically about these issues in their research and writing and incorporate facets of this outcome into their work when appropriate. This outcome is infused throughout the university curriculum, so students have opportunities to think about and interact with these issues in meaningful ways.
Scores from Table 2 show that indications of the IL levels of students can be determined by applying an assessment rubric to student writing assignments. Additionally, a positive correlation appears between the amount of IL instruction a student receives and the increased evidence of IL in their written assignments. For example, students in both UDIGE (#6) and COMP (#7) received one hour of IL instruction by a librarian as well as one additional hour of IL instruction from their professor and received "Proficient" scores of 2.02 and 2.28 respectively. Additionally, student papers in #8 (UDIGE) were rated as "Proficient" with a score of 2.25. Group #8 reflects a thirdyear, upper-division course that was co-taught for the entire term by a librarian.

Discussion
The findings from this pilot study suggest that a) IL proficiency levels in students can be shown by assessing student writing assignments with a targeted rubric, and b) IL instruction has a positive correlation to the amount of discernible IL proficiency in students' written work. While the authors expected that incoming first-year students would likely rank in the "Emerging" range for criteria involving accessing and evaluating, the authors were satisfied to see that students in composition courses that received the most IL instruction scored higher in those areas ("Proficient" scores of 2.61 and 2.28 for groups #7 and #9; see Table 2) than in courses that had little or no IL instruction. The authors were again pleased by a "Proficient" score of 2.61 for group #9: first-year composition students who received two hours of library instruction. Again, the authors had expected first-year student papers to be rated in the "Emerging" range. The authors were excited to find that the two groups that received the highest amount of IL instruction (groups #7 and #9) and also received the highest scores from raters were first-year composition courses. This trend continued with thirdyear, UDIGE students who received the most IL instruction (scores of 2.25 and 2.02 in groups #6 and #8; see Table 2).
Overall, data from these scores suggest that IL instruction from librarians coupled with IL instruction from course faculty may afford students the best opportunity to achieve IL proficiency (such as the scores from group #7 that show that first-year composition students produced papers rated as "Proficient" after two hours of IL instruction). Additionally, the data suggest that any amount of IL instruction to students increases IL proficiency over courses that provide no IL instruction to students at all (see Tables 1 and 2, groups #1, #2, #3).
For both librarians and faculty, this study affirmed the importance of examining the process students undertake when conducting research, not simply the papers students produce from that research. The reflective annotated bibliography component of the writing assignments allowed librarians and faculty to chart the steps a student undertook to complete an assignment as well as follow the student's thought process from start to finish in a way that is not usually apparent in final written work. There was an unanticipated benefit to students as well. By allowing students to reflect upon the research process as they were conducting their research, students were afforded opportunities to redirect, rethink, celebrate successes or vent frustrations that are not usually present in a traditional written assignment. The annotated bibliography component informed both librarians and faculty as to the clarity of the assignment (from the students' perspective) as well as the effectiveness of the research process the students undertook.
Findings from this study echo those addressed in the literature review: authentic assessment (such as portfolio assessment) was found to be a valid and reliable assessment measure to determine IL levels of students at CI. Additionally, the reflective annotated bibliography assignment illuminated the process students undertake when engaging in research, writing and critical thinking. Partnering with faculty on assessment was beneficial to both librarians and faculty; we share mutual interests in helping students to become information literate, critical thinkers and to be able to achieve university outcomes for student learning. Though our focus on the research process was studentcentred, our use of the rubric and subsequent assessment was more tutor-centred. The literature addresses positive aspects in both tutor-centred and student-centred approaches to assessment of student learning. A tutor-centred approach fit our needs. Our study assessed student work that had already been assigned and graded by faculty. As librarians and faculty, we were aware of the scope and pace of instruction that students received in their courses and IL sessions, and we sought to discover evidence of student learning through examination of student writing.
The authors acknowledge limitations of this study. Librarians and faculty had no knowledge of the IL levels of students prior to receiving IL instruction in the courses assessed in this study. Nor was a "pre-test" conducted to generate data that could then be compared to scores received after IL instruction. Because the authors intended to ascertain the IL levels of students at the time of the study, no prior data regarding the IL levels of students was pursued. Being able to compare new information with existing data about the prior IL levels of students would have strengthened this study and provided a stronger foundation and clearer direction for future studies. Additionally, raters knew how many hours of IL instruction were given to students prior to rating assignments, which may have introduced an unintentional bias. Another limitation of the study involves the research assignments themselves. The array of assignments that students selected for their portfolios occasionally made the assessment process difficult (a problem/solution paper was often easier to assess with the rubric than a personal narrative), though the reflective annotated bibliography assignment that was used in the second and third years of the study helped greatly in this area. The authors acknowledge that in rating group assignments, raters did not take into account student learning outcomes for each student in the group, rather the group as a whole. This potentially allowed for group scores to reflect the efforts of the best individual in the group, rather than the efforts of all individuals in the group. All of these issues will need to be addressed in subsequent studies.
The study brought an unanticipated benefit. At CI, a new portfolio assessment system (based on the assessment rubric and annotated bibliography assignment) is in place and is being successfully used by the writing and rhetoric programme. This system also supplies data that is used in the university's accreditation process and generates student characteristic information for the university. Results from this project were used in the California State University Channel Islands General Education Assessment Plan (2006) and as part of the report from the Western Association of Schools and Colleges (2007) that accredited the university.

Conclusion
Information gathered from our pilot study suggests that IL proficiency levels in students can been determined by assessing student writing assignments, and that a targeted rubric is an effective authentic assessment measure in this endeavour. Our information further suggests that IL instruction has a positive correlation to the amount of discernible IL proficiency in students' written work, and that any amount of IL instruction for students increases IL proficiency levels over courses that provide no IL instruction at all. The data highlighted limitations in our study that warrant further investigation and addressed the potential pitfalls and benefits for librarians seeking to undertake similar assessment efforts. This study highlights the efforts of the authors to partner with faculty, not only in assessing student papers but in creating a rubric and specialised bibliography assignment that can be used by librarians and faculty at their institutions, and emphasises the importance of process: the process that students carry out when they participate in research as well as the process that needs to be undertaken by librarians attempting to conduct successful assessment of student work.