Results from a Psychology OER pilot program: faculty and student perceptions, cost savings, and academic outcomes

This case study describes the library’s experience of collaborating with an undergraduate Psychology Department at Touro College to integrate open textbooks into their program. We discuss the pedagogical changes as well as explore the impact of Open Educational Resources (OER) on students’ savings, their academic outcomes and perceptions of OER. Furthermore, we highlight the successes and shortcomings in having the library as a central OER partner. To measure the results, we surveyed students and conducted a faculty survey and a focus group, in addition to analyzing the students’ final grades. This pilot program delivered strong results. The students’ perception was very positive, and faculty’s opinions on the textbooks used were mixed. Some professors felt that the textbook lacked important content, but because of its openness, they added their own content to the book. Students enrolled in OER courses performed better than those enrolled in the same courses using a commercial textbook.


Introduction
The price of textbooks has risen steadily, increasing more than 1000% since 1977 (Popken, 2015). In 2017, for the first time, prices have stopped rising and started to decrease. While speculating about possible reasons, Dr. Mark Perry credits Open Educational Resources publishers as responsible for this historical break in the trend (Perry, 2017). The influence of OER, however, goes beyond cost. Current research on its impact measures not only Cost, but also Outcomes, Usage, and Perceptions (known as COUP Framework, Open Education Group, n.d.). We briefly discuss research measuring these elements below.
Using Open Educational Resources instead of traditional textbooks has been shown to be cost effective based on many studies. Among them, John Hilton III and his colleagues have conducted several large-scale studies over the years in different programs, majors and multiple colleges (Hilton, Robinson, Wiley, & Ackerman, 2014;Hilton, Gaudet, Clark, Robinson, & Wiley, 2013;Ikahihifo, Spring, Rosecrans, & Watson, 2017).
In addition to being cost effective, studies have shown that the academic efficacy of students has been the same or even increased when utilizing OER. Jhangiani and Jhangiani (2017) citing ; Hilton, Fischer, Wiley, and Williams (2016), note in their article that "thirteen studies (with an aggregated sample of 119,720 students) that have investigated the impact of OER adoption on course performance found that 95% of these students have achieved the same or better outcomes when using OER" (p. 3). Similar outcomes have been confirmed by a 2018 study at the University of Georgia with large student samples over a period of six years (Colvard, Watson, & Park, 2018) finding improvements in grade levels as well as in completion and retention. Smaller and shorter studies have also reported positive student academic results (Clinton, 2018). § First author affiliation changed from Touro College to NYU Grossman School of Medicine after the paper was submitted.
A third measure, usage, can provide insight on how students and educators take advantage of open licensing permissions to adapt OER. This can be achieved by observing how materials are Deleted, Inserted, Moved, or Edited (known as DIME model, Open Education Group, n.d.).
The last component of the COUP framework is perception. Ample research has examined students' and faculty's perception of OER. Such studies provide both quantitative data, which is especially valued by administrators, and qualitative data, which can provide impactful stories that can be used during outreach. Cooney (2017) examined students' perceptions of the Psychology OpenStax textbook, after also reviewing the literature on this subject. As pointed out by the author, the majority of previous studies reported positive perceptions (Bliss, Hilton, Wiley, & Thanos, 2013a;Bliss, Robinson, Hilton, & Wiley, 2013b;Hilton et al., 2013;Petrides, Jimes, Middleton-Detzner, Walling, & Weiss, 2011, as cited in Cooney, 2017). Cooney's study differs from most in the sense that the author conducted interviews and focus groups with students, in addition to surveys. The findings confirm what has been previously reported by other studies, with students preferring OER to traditional textbooks. It is important to note that the students highlighted the convenience of accessing open textbooks. This observation is of interest to the present research, since Touro College also caters to urban commuter students. Hilton's (n.d.) summary of empirical research on OER discusses faculty perceptions in five studies (Petrides et al., 2011, Jhangiani, Pitt, Hendricks, Key, & Lalonde, 2016The California OER Council, 2016;Jung, Bauer & Heaps, 2017;and Watson, Domizi, & Clouser, 2017). As with traditional textbooks, faculty identify both negative and positive characteristics of open textbooks, but overall find that quality is comparable. Watson et al. (2017), for example, mention that "professors using an OpenStax textbook agreed that there were things missing from the text", but that "the flexibility allowed by open licensing was an opportunity to rethink and modify their classes, and add supplemental resources" (pp. 294-295).
This paper aims to add to the growing body of empirical data that measures cost, outcomes, usage, and perceptions of OER. Moreover, we reflect on the collaboration between faculty and library in the creation of an OER program. We conducted a faculty survey, a student survey, a faculty focus group, and analyzed the academic outcomes of the students enrolled in eight psychology course sessions using OER, compared to the outcomes of students using traditional textbooks in the same course in the previous semester.
These instruments were used to answer the following research questions: (i) how aware are faculty of Open Educational Resources, open textbooks, and licensing (copyright, public domain, and creative commons)? (ii) how much can students save when faculty uses an open textbook? (iii) how does the cost of traditional textbooks affect Touro College's students, and what measures have been taken to reduce those costs?; (iv) what are students' perceptions of open textbook quality, use and format?; (v) what are faculty's perceptions of open textbook quality, use, and outcomes?, and (vi) how do academic outcomes of students enrolled in OER courses compare to those enrolled in courses using traditional textbooks? These questions were designed to measure cost, outcomes, usage, and perceptions of OER quality, following the COUP framework (Open Education Group, n.d.).

Outreach
In higher education, librarians are increasingly more involved with OER. In addition to the librarians' expertise, it is beneficial to have the library as an OER partner because of "its centrality and visibility on campus" (Alpi, Cross, Raschke, & Sullivan, 2017, p. 71). As with many other aspects of the library profession, librarians can take many roles in OER initiatives. Dr. Bradlee and VanScoy (2019) summarized the ways librarians engage in OER. According to the most recent literature: Open Praxis, vol. 12 issue 1, January-March 2020, pp. 83-99 librarians can be involved in adoption; advocacy; curation, preservation, and repositories; content development; description, cataloging, and metadata; discovery; funding; information literacy; licensing, intellectual property, and copyright assistance; professional development; publishing; recognition of OER leaders, and also serving as information specialists in OER teams (Bradlee & VanScoy, 2019, pp. 429-430). Touro College New York is a medium sized commuter institution compromised of ten campuses in New York City, with "a mission to offer neighborhood-based programs that serve and encourage service to under-resourced communities" (Touro College, n.d.). Touro College libraries initiated a college wide OER program, took on the responsibility of advocating for OER, and later offered professional development webinars to faculty seeking to learn more about these resources. This particular study was conducted at Touro College's New York School of Career and Applied Studies.
Higher Education OER programs grow both as bottom-up and top-down initiatives, depending on the context of each institution. For this reason, it is crucial to understand how the institution works. One of the librarians involved in this initiative has worked at the college for more than a decade and recommended that in order to create a sustainable OER program, higher administration had to be a part of the program.
Based on this strategy, our first step, after gaining knowledge about the OER movement, was to meet with Touro College's deans. Although some were skeptical of the quality of OER, as it often is reported with people who are first exposed to OER, most were supportive. We believe that their support was immediate because we demonstrated how an OER initiative could help the college fulfill its mission and meet its strategic goals. The deans instructed the library to treat the initiative as a pilot, and the results are reported in this paper.
During our outreach to different departments, we contacted the chair of the Psychology Department. She was concerned with the rise of textbook prices and its impact on students' education, and expressed her eagerness to discuss the use of open textbooks. Seeking for a solution, that department had already tried different alternatives, investigating information about online vendors, encouraging students to rent textbooks, and adopting bundles that included digital or print books at significant discounts. These efforts yielded limited success.
The chair and her department discussed the possibility of using open textbooks but had reservations about OER. At the library's presentation, they were able to voice their concerns and gather the information needed about these resources. Aside from a general overview of OER, the librarians compared commercial textbooks used by faculty against similar available open textbooks. This facilitated the process, since the professors did not have to search for the textbooks themselves.
After this presentation, three professors decided to use open textbooks in place of commercial textbooks during the fall 2018 semester. Two of them adopted the textbook Psychology, by OpenStax (2014), and one adopted the textbook Research Methods for Psychology (Crump, Price, Jhangiani, Chiang, & Leighton, 2018). Touro College's Psychology department at the New York School of Career and Applied Studies was thus the first department in the institution to adopt open textbooks in multiple course sections. Naturally, the pilot requested by the deans was conducted at this department.

Methods
The methods employed in this study were designed according to the COUP Framework (Open Education Group, n.d.). By measuring the cost, outcomes, usage, and perception of OER, we believe we can provide a deeper understanding of both efficacy and perceptions OER quality in the context of our institution. We also trust that research that studies the impact of OER in all of the metrics mentioned can better inform new research and OER adoption in the future.
As a first step in outreach, we intended to gauge faculty's awareness of OER through a collegewide survey. Although this survey did not produce significant results, it informed our next outreach decisions. After the first semester of adoption, we assessed cost, outcomes, usage and perception using four instruments: textbook cost savings, an analysis of the students' academic outcomes, the perceptions of students through a survey, and the perceptions of faculty by conducting a focus group. The results are explained below. We will discuss each method in the appropriate sub-section, as well as the research questions they intend to answer.

Faculty Survey
To answer our first research question, "how aware are faculty of Open Educational Resources, open textbooks, and licensing (copyright, public domain, and creative commons)?", we created a 25-question survey (exempted from IRB review), based on the surveys of Allen and Seaman (2016), Bliss et al. (2013b), and Seaman and Seaman (2017).
Like Seaman and Seaman (2017), we also intended to assess (a) the decision process behind the selection of educational resources: what materials are selected, when, and the faculty's role in the decision. Also, what influences faculty's decision and how satisfied they are with it; (b) the proportion of students who purchase required textbooks, as reported by the faculty, and textbook prices; (c) potential barriers to adoption of Open Educational Resources, and (d) future use of Open Educational Resources.
From an initial pool of over 600 faculty, only 62 responded. The low response rate can be attributed to several factors. First, this survey was sent out to all undergraduate faculty while unbeknown to us three other institution-wide surveys simultaneously requested faculty's attention. Second, the survey was unnecessarily long, contained several matrix tables, and questions contained grammatical conjunctions. We are not reporting the results of this survey because we do not consider it to be significant.
Although we failed to reach a satisfying number of respondents, faculty's comments informed our outreach decisions: we learned that several departments regulate textbook and other materials selection for courses. Until then, we were unaware of the different departmental practices when adopting course materials. Based on this finding, we decided to approach departments directly, instead of individual faculty members.

Cost Savings for Students
In order to address our second research question, "how much do students save when using an open textbook?", we used SPARC's cost-saving projections (Nyamweya, 2018). During the fall semester of 2018, 99 students enrolled in eight course sections of one introductory and one upperlevel psychology course. When calculating cost savings, researchers can opt for one of two methods. In the first, one can calculate savings by using the cost of a new commercial textbook, multiplied by the number of students. This method does not account for the fact that most students do not buy new textbooks; rather, they can rent them, share them with fellow students, buy used copies, etc. Additionally, textbook prices vary widely in price: a chemistry textbook, for example, can cost a few hundred dollars, while an anthropology one can cost under fifty. To solve this issue, SPARC collected data from 600 courses at 120 U.S. institutions. They found that "the average price for courses using traditional materials was $134.26 and the average price for courses using OER was $17.32. Therefore, the average savings between courses that use traditional materials and those that use OER was $116.94" (Nyamweya, 2018).
Open Praxis, vol. 12 issue 1, January-March 2020, pp. 83-99 There were 99 students enrolled in psychology courses using open textbooks during the fall semester of 2018. Using SPARC's cost-saving projection, the total savings for those students was $11,577.06.

Student survey
For our next research questions: (iii) how does the cost of traditional textbooks affect Touro College's students, and what measures have been taken to reduce those costs? and (iv) what are students' perceptions of open textbook quality, use and format?, we developed a survey of 13 open and close-ended questions. These questions were based on studies from Florida Virtual Campus (2016), Jhangiani and Jhangiani (2017), and Bliss, et al. (2013a). Some additional questions were formulated by the authors. Through this survey we also hoped to determine the likelihood of future student enrollment in OER course sections.
At the end of the fall 2018 term, in which faculty used open textbooks for the first time, the librarians administered a survey to all students enrolled in the eight sections of psychology courses (six introductory, and two upper-level courses, 99 students). This survey was exempt from IRB review. We decided to deliver a paper-and-pencil survey, instead of a web-based one, as we believed that this method would result in higher completion rates, compared to an online delivery.
The printed survey was distributed to all students, who were instructed to read the informed consent, presented in a cover letter on top of the survey instrument. Using the informed consent as a script, we reinforced that their participation was voluntary, that no personal information would be collected, and that the survey would not interfere with their grade, or any other aspect of their class.
Sixty-one students responded to the survey, with a response rate of 61%. In the sub-sections below, we discuss the students' answers for the consequences of traditional textbook costs, format preference and usage, perceptions of OER quality, and the likelihood of future enrollment in an OER course section.

Consequences of Textbook Costs
Many studies surveyed students on the consequences of the prohibitive cost of textbooks (e.g. Florida Virtual Campus, 2016 and 2019). We replicated this question, and found similar results to those studies. Seventy percent of the students surveyed responded that the cost of required textbooks has caused them to not purchase them, either frequently (31.1%), occasionally (27.8%), or seldom (11.1%).
Not having purchased textbooks affects students in many ways. 41% do not register for a specific course, and 41% take fewer courses because of the price of textbooks. 42% earn a poor grade, 29% fail a course because they could not afford the textbook. Additionally, 20% drop and 23% withdraw from a course because of the price of required materials. Although this is a small sample, it shows that the reality of the students surveyed is very similar to other students nationwide (Florida Virtual Campus, 2016 and2019).
From the students surveyed, only two reported that they do not attempt to reduce the price of required materials, purchasing them at regular cost. The remaining 59 students take several measures to reduce the cost of required textbooks.
These findings are illustrated below (figure 1).

Format preference and usage
One of the concerns expressed by Touro College's faculty is that the students might not like digital-only textbooks. This is a valid concern, since this institution caters to religious students who may not use electronic devices during certain days. Based on this, we included a question asking the students how they feel about the format of the course materials. Only 3.4% responded that they like it less than print textbooks; 57.6% like it more than print textbooks and 39% are indifferent. All students but one also answered that they are very likely to register for a future course with online texts like the open textbook used in their OER courses. Half of the students reported that they printed sections of the textbook. The surveyed students predominantly used their smartphones and laptops to read their textbook. Less often, they used tablets and desktop computers, as illustrated by the bar graph below:

Quality
Two questions were included regarding the quality of the open textbooks used. The first asked students how they would rate the quality of the textbook used in the course. The participants could choose from Open Praxis, vol. 12 issue 1, January-March 2020, pp. 83-99 better, same, or worse than the quality of the texts used in my other courses. The majority of the students (68%) answered that it was better than other texts, and 32% answered that it was the same. No one reported that the quality of the open textbook is worse than the quality of textbooks used in other classes.
The following pie chart (figure 3) demonstrates the students' perception of quality: The following question asked students to justify why they thought the quality of the open textbooks was worse, the same, or better than traditional textbooks. To analyze their answers, we first coded each response using an open-coded process. In this first analysis, twenty codes emerged. Upon reviewing them, we grouped the most similar ones into the following six themes (table 1).  As the table above shows, there were five positive themes: convenience, cost, quality, access, and format. There was also one neutral theme, and no negative comments. Students determined that the online textbook was easy to access across devices, enabling them to study anywhere, and reducing the weight of their backpacks. They also judged that the cost and quality of the textbook were important factors. One student mentioned that having access to the required materials from day-one allowed the professor to continue the class without delays, since they did not have to wait for students to acquire the textbook. In addition, one student confirmed using the textbook more often, and one mentioned that not having markings from used textbooks was very positive. For four students, using the open textbook was the same as using a traditional textbook.

Future Enrollment in OER Course Sections
The last survey question intended to assess whether students would enroll in future courses using open textbooks (question reproduced from Bliss et al., 2013a): Imagine a future course you are required to take. If two different sections of this course were offered by the same instructor during equally desirable time slots, but one section used texts similar to the one used in this course and the other used traditional published texts, which section would you prefer to enroll in? Twenty-three percent of the students did not answer this question. This was the last question, and this was a paper-and-pencil survey. Having an open-ended question as the last survey question was not a good strategy, and it certainly contributed to lower response rates. We will consider switching the order of the questions in future surveys. Still, as seen in the illustration above (figure 4), the majority of those who responded (39 students) declared that they would enroll in a course using texts similar to the ones used in their OER class. Three students prefer traditional textbooks, four would base their choice on the price of the traditional textbook, and one is unsure.
The students' perceptions of open textbooks, as demonstrated above, was very positive. We expected that the money saved when using an OER would guide the students' impressions. However, the students also praised the quality of the textbook used, as well as the convenience, and the ability to access the textbook, as a group, since the start of the classes. After assessing the students' perceptions, we determined that it would be important to learn whether the professors had similar attitudes.

Faculty Focus Group
For the fifth research question, regarding faculty's perception of open textbook quality, use and outcomes, we conducted a focus group with three professors who had adopted open textbooks the previous semester. The conversation was led by a few open-ended questions from Bliss et al. (2013b). We also wanted to assess how the library can better support faculty who had already transitioned to OER.
In order to protect the faculty members' identities, their names were removed. This focus group was approved by the institution's review board.

Q1: How much time did you spend preparing for this class, compared to a previous, commercial textbook?
Professor 1 said that the times were perfectly comparable. The other two professors agreed that it took more time to transition to an OpenStax textbook, compared to the previous commercial textbook. This was justified because, according to them, the OpenStax Psychology textbook is lacking in some respects. To account for that, they had to complement the textbook, which was not needed before.

Q2: How did the students' preparedness compare between semesters? Was there any difference?
It was evident that the professors agreed that they could not really isolate the open textbook impact on the students. However, they recognize that, because the students now have access to the textbook, they are better prepared. The contrast, here, is not between the quality of commercial textbooks versus open textbooks, rather between textbooks that are not purchased because of their prohibitive costs versus textbooks all students can access freely.

I'd say a little bit. Because it seems that most of the students are actually reading the material. (Professor 2)
There is more involvement, because they are reading it on their phones as well. (Professor 3)

Q3: How often did you think that the students used this textbook throughout the semester?
The responses to this question were very similar to the previous one. Two professors stated that they could not assess how often the students were using the textbook. One professor expressed that her students were reading the assigned chapters at least once. One professor also mentioned that, because the students can print a chapter at a time, or read on their phones or tablets, they use their commuting time to catch up with the weekly readings: We have a mantra in our class, we all have to do our readings on the subway, especially in the mornings. We use it as an example of an empirical question too: it is easier to read in the mornings on the way to work, than it is to read in the evenings. When I assign a reading, I say: this is the reading for the subway for this week. (Professor 1)

Q4: On average, how would you rate the quality of this text, compared to the traditional one?
The two professors who use the OpenStax textbook agree that the quality is lower. The third professor, who uses a different textbook, thinks that the quality is good. Regarding the OpenStax textbook: The way it stands now, much lower, but we will try to improve it. They missed many things. I do not know if they purposefully deleted them, but they are very important, and everybody would agree that they should be there. The chapters are very similar, but the content is not. (Professor 3) During the conversation, faculty pointed out that they were not aware of other options for open textbooks in that same discipline. This shows that the library needs to do a better job in checking in with faculty using OER to make sure that their needs are met, to provide continuous support, and to offer options when they are available, and to promote OER repositories.

Q5: What feedback, if any, did you receive from students about the textbook used in this course?
Professor 1 reported that her students have been happy with the textbook being used. She uses a textbook written and published using a typesetting system. When the typesetting was converted to a PDF, some formatting errors resulted in mislabeled figures, for example. One of her students complained about it.
I think it's true that when you lack the big business of a publishing house… the perfect copy-editing [does not] happen. Arguably, I could have kept track of these suggestions, and changed them later. (Professor 1) The code for this textbook is available to everyone, and because it is published under an open license, anyone can make changes. Additionally, the author encourages the community to use Hypothes.is, a web annotation tool, to point out typos and errors that need to be fixed, or to suggest alterations.
The professors using Psychology OpenStax plan to adapt the textbook using the OpenStax CNX platform. Regarding student feedback from the OpenStax textbooks, the professors mentioned that the students appreciate free access to the textbooks.

Q6: Has anything changed in the way you teach this class?
One of the faculty participants used the new textbook to restructure the class. This was an opportunity to think about what content is most important for the class, rather covering all content simply because the students were required to buy the textbook. Aside from rethinking the content, this professor chose a different strategy to present the textbook to the students. She posted one Open Praxis, vol. 12 issue 1, January-March 2020, pp. 83-99 chapter at a time on the Learning Management System, so the students could focus on only that chapter.
Because I don't require a whole textbook to be bought, I have been more willing to cut chapters that I choose not to spend time on, which allows my students to focus on the chapters that I have selected… I only post one PDF at a time on the LMS. In the semester, it is better to select the contents that you think most important for how you are organizing the class… I think it helps students to be able to focus on a little less, rather than being inundated with more. (Professor 1) The other two faculty members teach an introductory course. Rather than cutting chapters, they felt the need to include more content to supplement the OpenStax textbook.

Q7: In future courses, how likely are you to use open textbooks?
Despite some quality concerns, and the need to supplement their textbooks (in the case of OpenStax), all faculty participants plan to continue using open textbooks in the future.
I almost think it is wrong not to. Unless you are teaching a class where it is simply not available. But even then, you'd feel more compelled to find online sources, or compile something in your own way… I think it's immoral to require a $100 textbook. (Professor 1)

Q8: How do you envision the role of the library when it comes to OER?
The faculty participants reiterated that they would like the library to help them navigate different options, offering opportunity to review other textbooks.

Academic Outcomes
Our sixth and last research question concerned students' academic outcomes: "how do academic outcomes of students enrolled in OER courses compare to those enrolled in courses using traditional textbooks?". In order to test if there was a difference in academic outcomes between the open textbook and the traditional textbook, we compared the grades of 180 students. The first group, which we named Traditional, had 81 students enrolled in psychology courses using a traditional, commercially published textbook in the spring of 2018. The second group, named OER, had 99 students enrolled in the same courses, taught by the same professors, but using an open textbook in the fall of 2018. The groups are compared as independent groups, because they comprise different groups of students.
We received the grades from the professors as letter values, and assigned them numerical values based on the correspondent grade percentage. We assumed that grades ranging from A+ to C-would be passing grades. All grades D and F would be considered non-passing grades (below 70).
As the summary below shows (table 2), the two independent groups (OER and Traditional) have similar distributions: the standard deviation of the two groups are similar, and the median and mean values of each group are also similar. Hence, the two groups had somewhat similar data distributions. The grades of students enrolled in OER classes were higher in the 25 th and 75 th percentiles. The comparative histograms below (figure 5) show the frequency of grades of students enrolled in courses using traditional textbooks (left), and open textbooks (right): After converting the grades to numerical values, we used a normal probability plot to assess whether or not the data set was normally distributed. In this test, "the data are plotted against a theoretical normal distribution in such a way that the points should form an approximate straight line. Departures from this straight line indicate departures from normality" (Heckert & Filliben, 2003). The two quantile plots below (figures 6 and 7) show that the data is not normally distributed, as the grades are scattered in small horizontal groups, not forming a straight line. For this data to be considered a normal distribution, it would have to be distributed along the diagonal lines indicated in each plot. To test the two independent and not normally distributed data sets, we performed a Wilcoxon Rank Sum test. The comparison between the OER and the Traditional group showed that students enrolled in classes using OER had better scores than those enrolled in classes using traditional textbooks. This comparison is statistically significant, with a p-value<0.05 (0.046).

Discussion
With the research instruments discussed above, we were able to answer our research questions related to cost, outcomes, usage, and perceptions. The first research question, (how aware are faculty of Open Educational Resources, open textbooks, and licensing?), remains unanswered, since the first college-wide faculty survey did not produce significant results. Since the survey was administered, we believe that OER awareness is growing, due to the workshops and webinars conducted by Touro Libraries.
The second research question regarded students cost savings. In one single semester, 99 students enrolled in Psychology courses at Touro College's New York School of Career and Applied Studies saved $11,577.06.
Through the student survey, we investigated how the price of traditional textbooks affected students (research question iii: how does the cost of traditional textbooks affect Touro College's students, and what measures have been taken to reduce those costs?). The reality of the students surveyed mirrored what has been reported in the literature so far: 70% of the students surveyed reported not purchasing required textbooks. Only two students (3% of students surveyed) declared purchasing them at regular cost. All other students have taken different measures in an attempt to reduce the costs of required materials.
The same survey showed that students viewed their open textbooks positively (research question (iv) what are students' perceptions of open textbook quality, use and format?). They acknowledged the convenience of using open textbooks, as observed by other researchers (e.g. Cooney, 2017), and despite institutional concerns, found that the format of open textbooks was beneficial and practical, especially since these are commuter students. Furthermore, students declared that the quality of the open textbook used was the same or better than other textbooks.
In addition to students' perceptions, we also proposed to investigate faculty's perceptions, of open textbook quality, use, and format (research question v). The two professors who used the OpenStax textbook agree that there is content missing from what they need to cover in an introductory psychology course. Watson et al. (2017) reported the same findings. However, because this is an open textbook, they do have the option to edit it and supplement the materials they need. In fact, these professors recently received an internal grant to adapt the Psychology OpenStax textbook.
The use of the textbook differed for the professor using Crump et al. (2018). Rather than adding material, the professor decided to delete some chapters and provide a more focused class to her students. Again, this shows the advantage of working with a resource published under an open license, which provides more options and more freedom to educators to design their classes with the students' learning in mind.
Regarding the students' academic outcomes, this study corroborates previous data  showing that students enrolled in OER courses performed better than students enrolled in classes using commercially published textbooks. This robust finding will allow us to continue our advocacy for an institution-wide OER program, and answers our last research question positively (vi: how do academic outcomes of students enrolled in OER courses compare to those enrolled in courses using traditional textbooks?). Equally important, these results contribute to global data on OER outcomes.
We recognize that using grade comparison to measure student learning is more complex than how we approached this study. Grimaldi, Mallick, Waters and Baraniuk (2019) analyzed previous research on the impact of OER on student learning. The authors introduced the access hypothesis, stating that "OER might improve learning outcomes relative to traditional course materials by improving access to the textbook" (Grimaldi et al., 2019, p. 9). In an experiment, an intervention should affect all participants of the experimental group. In the case of OER, this comparison should be between students who did not have access to textbooks to students using OER. According to the authors, one should always expect null results, and be cautious of reporting positive results (Grimaldi et al.,p. 9). Indeed, we do not know how many students in our control group had access to the traditional course material. An additional limitation is our sample size (n=180), which is considered to be small. Further, we are comparing different student groups, and other confound variables might be in place and impossible to isolate, such as students' individual performance, faculty grading styles, and differences in tests and class policies of which we are not aware.
Independent of these statistical implications, this pilot project exceeded our expectations in providing both positive results and important learning opportunities. We should not keep students from learning and achieving their potential only because they cannot afford expensive textbooks. Moreover, we should highlight that faculty using OER have more freedom to design their own classes, and students enrolled in those courses were satisfied because they had immediate and unlimited access to their required course materials. These results show that transitioning to an OER course is not seamless. However, the advantages are too large to be dismissed.
We recommend librarians to learn about institutional practices of textbook adoption prior to start advocating for OER. Also, when possible, librarians should offer faculty development workshops that include an overview of OER repositories. Another helpful practice is to identify institutional strategic goals that can be advanced with the adoption of OER.

Faculty-library collaboration
This pilot shed light upon some of our practices when collaborating with faculty. We changed our outreach methods to approach departments, instead of individual faculty members, which has been effective, as the results in this study show. More importantly, through the focus group described in this study, we learned that faculty were unsure about other options of psychology textbooks. Providing a selection of open textbooks simplified the search process, but also limited the faculty's options, Open Praxis, vol. 12 issue 1, January-March 2020, pp. 83-99 since they did not learn about OER repositories. In hindsight, the librarians should have facilitated a workshop where faculty could have searched for the textbooks themselves. This would have given them more autonomy to choose their textbooks, search for other options, and find resources other than textbooks when exploring different OER repositories.

Conclusion
In this study, we examined the academic outcomes and perceptions of students enrolled in two psychology courses using open textbooks, as well as the perceptions of three faculty members teaching those courses. The most obvious outcome of switching the required courses materials to OER is cost savings. However, many faculty and administrators still have concerns about the quality of such resources. With this pilot program, we demonstrated that students perceive OER positively when judging its quality, format, and use. Commuter students praised the ability of having unlimited digital access to the open textbook across devices. Faculty members had mixed opinions and, while two out of the three faculty involved in this project felt that the textbook lacked important materials, they appreciated the possibility of altering the textbook to better suit their classes' needs. Most importantly, we showed that students enrolled in OER courses performed better than those students enrolled in the same courses using a traditional textbook.