Open Textbooks: Quality and Relevance for Postsecondary Study in The Bahamas

Open educational resources (OER), are openly licensed text, media, and other digital and analog assets that are useful for teaching, learning, and research. Recent research has shown that in courses where open textbooks are assigned, students perform as well as or better than students in similar courses with commercially licensed textbooks. Despite cost savings and demonstrated effectiveness, perceptions about quality, relevance, ease of access, and other concerns persist. The objectives of this study were twofold: (a) to develop a practical and reusable measure for evaluating open textbook quality in terms of pedagogy, openness, accessibility, and relevance; and (b) to use the measure to rate the quality and relevance of open textbooks for use in higher education in The Bahamas. The study confirmed the viability of the quality measure as a practical tool to assess open resources and found that the open textbooks studied were accessible and well matched to course content, but of varying quality. More study is needed to explore ways to increase faculty adoption, use, adaption and production of OER.


Introduction
Textbooks play an important role in student course outcomes. As the cost of university and college textbooks continues to rise, however, students increasingly cannot afford to purchase them. Increasing costs can lead students to purchase fewer texts and hence do poorly in courses, take fewer courses, or drop out of programmes of study altogether (Florida Virtual Campus, 2016). While textbook costs are a burden to all students, poor students are particularly impacted as they are least able to shoulder high educational costs. In effect, increasing costs present an additional educational barrier. Simply put, when costs increase, access is reduced.
While the population of The Bahamas is spread over 700 islands and cays, most residents live in the nation's capital city, Nassau, on the island of New Providence. Access to education in general, and higher education in particular, is challenging for residents on the Family Islands -the name given to the 30 or so populated islands other than New Providence. Even when Family Island residents can afford expensive texts, mail delivery is limited, making it more difficult for them to access learning materials.
Open educational resources (OER) and open textbooks are promising alternative to commercial textbooks, and availability of OER is expanding (Wiley, Bliss, & McEwen, 2012). OER and open textbooks released in digital format seem especially suited to the islands of The Bahamas as Internet and mobile data connectivity is widespread, even in some of the more remote settlements.
Availability is not a sufficient condition for adopting open textbooks. Several factors may inhibit OER adoption including (a) relevance (Albright, 2005); (b) findability (Larson & Murray, 2008); (c) existing teaching practices (Hattaka, 2009); and (d) quality (Albright, 2005;D'Antoni 2006). Importantly, texts must be or must be perceived to be of a certain quality, to be considered as replacements for existing texts (D'Antoni, 2006). Despite cost savings, perceptions about quality, ease of use, ease of access, and match to course may discourage faculty from making the effort to find, review, and use open textbooks. At present, these explanations for the low adoption rate are speculative, as no formal study of OER and open textbook use in The Bahamas has been attempted to date.
This study focused on the question of open textbook quality. Specifically, the intention was to design and build an evaluation tool that would be easy to use and adapt to assess quality in a variety of different types of textbooks, regardless of content. This study made two unique contributions. First, it demonstrated the use of a new and easily reproducible approach for evaluating open textbooks quality on five dimensions: (a) pedagogy, (b) openness of license, (c) cost, (d) available media, and (e) relevance. Second, this study used this approach to evaluate potential OER texts for use in higher education courses in The Bahamas.

Objectives
The objectives of this study were twofold: (a) to develop a practical and reusable measure for evaluating open textbook quality in terms of pedagogy, openness, accessibility, and relevance; and (b) to use the measure to rate the quality and relevance of open textbooks for use in higher education in The Bahamas.
Specifically, the study attempted to answer the following questions: • How well did open textbooks score on a simple scale of pedagogical quality?
• How open were the textbooks in terms of their licenses?
• How accessible were the open textbooks in terms of cost?
• How accessible were the open textbooks in terms of the media delivery formats?
• How relevant were the open textbooks in terms of the degree to which they match course content at postsecondary institutions in The Bahamas?

Theoretical Framework
Textbooks can play an important role in student course outcomes. With students paying as much as $1,200 annually on average for books and supplies in the US (College Board, 2015), increasingly, students cannot afford to purchase textbooks and hence purchase fewer texts, take fewer courses, or may even drop out altogether (Florida Virtual Campus, 2016). Few local texts are available to Bahamian higher education students, and the cost of importing texts means that costs in The Bahamas are even higher. The University of The Bahamas website estimates typical annual costs of books and supplies for undergraduates at B$1,500 (University of The Bahamas, 2017). Textbook costs are a burden and can serve as a barrier. Added to which, off-site students, such as those studying online or at Family Island learning centres, are further disadvantaged, because in addition to high costs, they have to wait for books to be mailed to them.
Open educational resources would seem to be a viable and affordable alternative to traditional, commercially licensed textbooks. In addition to the cost savings of OER in general, electronic OER are particularly accessible and downloadable via the Internet, immediately lowering barriers to access for students in Nassau and the Family Islands alike. Despite over a decade in research and development of OER, increasing availability and ease of locating OER, and despite The Bahamas being one of the few countries in the Caribbean with a strategy for information and communications technology in education that promotes OER (Hoosen, 2012;Ministry of Education, 2017), OER use in The Bahamas seems a promise rather than an actuality (George, 2015). The reasons for slow adoption in The Bahamas have not been researched, but in other locations, issues have included: (a) questions of quality; (b) concerns about content match; (c) lack of awareness of OER; (d) difficulty in locating appropriate OER; (e) lack of recognition for faculty efforts to adopt, adapt, and reuse OER; and (f) complicated textbook approval processes (Allen & Seaman, 2016;Hilton, 2016;Mishra, 2017). Specifically, Wiley et al. (2012) identified several challenges to the adoption and reuse of OER: (a) discovery, (b) concerns about quality, (c) localization to meet local needs, and (d) lack of skills to adapt and remix content. In this study, open textbook quality was explored on a variety of levels. Alsagoff (2016) identified four approaches to assessing OER and open textbook quality: (a) faculty (peer) reviews, where college and university faculty review the content against a standard set of criteria, similar to how they would review a manuscript for a peer-reviewed journal; (b) faculty ratings, where faculty rate their perception of quality after having used the content for a course; (c) student ratings, where students rate their perception of quality after using the text in a course; and (d) efficacy research where researchers compare the outcomes of students using openly licensed resources versus commercially licensed resources for different sections of the same course. While criteria-based evaluation rates textbooks prior to use, the other three approaches rely on ratings after textbook use.
Examples of studies referencing faculty and student ratings consistently reported positive ratings of open textbooks by faculty and students (Bliss, Hilton, Wiley, & Thanos 2013;Jung, Bauer, & Heaps 2017;Vander Waal Mills, Gucinski, & Vander Waal 2019). Similarly, in a review of efficacy research on OER and pen Textbooks, Hilton (2016) found that OER were at least as effective as licensed materials at promoting student achievement.
This study needed a criteria-based framework to rate textbooks prior to use, as the intention was to determine textbook quality in order to encourage adoption and use. Harvey and Green (1993) identified five dimensions of quality in higher education, three of which are most applicable to OERtransformative learning, fitness for purpose, and cost efficiency. Expanding on these, Kawachi (2014) developed the four-part TIPS framework for quality assurance of Open Educational Resources, version 2.0 (TIPS 2.0), as a tool to guide faculty in creating and authoring their own OER. TIPS is an acronym for the four categories of quality the framework is designed to evaluate: (a) teaching and learning processes, (b) information and material content, (c) presentation product and format, and (d) technical systems and technology. TIPS 2.0 revised version 1.0, incorporating feedback from OER experts, online OER forums, and teacher-practitioners (Kawachi, 2014). In total, TIPS 2.0 identified 38 quality criteria around the themes of pedagogy, learner-centeredness, openness, accessibility, relevance, adaptability, and content quality (Kawachi, 2014).
Although TIPS 2.0 provided a comprehensive quality framework for evaluating OER and open textbooks, its length and reliance on content expert knowledge was not well suited to this study. For this study, a more practical two-part assessment instrument, derived from the TIPS framework, was developed to evaluate textbook quality on five dimensions-pedagogy, openness of license, cost, available media, and relevance.

Pedagogy
The pedagogical quality of each text was rated using five items adapted from TIPS: 1. The use of learning objectives to signal the intent and purpose of the learning content. Clearly articulated learning objectives can boost learning in multiple ways including: (a) helping students differentiate types of knowledge; (b) ensuring students focus on the appropriate content and skills; (c) scaffolding independent, self-directed learning; and (d) enhancing metacognitive skills (Eberley Center, 2019).
2. The use of unit learning activities to engage and stimulate the learner. Effective learning activities can deepen learning by encouraging active as opposed to passive learning (Fink, 2003).
3. The use of diagrams and charts to enrich the content and facilitate retention. Decades of research has demonstrated that students learn better from text and images together than from images alone (Mayer, 2008;Nilson, 2010).
4. The use of practice exercises to deepen learning. Practice exercises give students the opportunity to apply the knowledge and skills they have learned in different contexts (Fink, 2003). 5. The use of pre-and post-assessments to deepen learning and help learners monitor their progress. Self-testing before, during, and after learning is not only an effective way to monitor progress, but also an effective tool for learning itself (Rohrer & Pashler, 2010).

Openness
Openness of license was rated according to a scale shown in Figure 1. Licenses are ranked inversely according to the limits on sharing of the resource-the more the limits, the lower the ranking (Green, 2014). the litmus test distinguishing OER from non-OER shareable resources is whether the license allows for repurposing and derivation. Neither of the NoDerivative (ND) licenses qualify as OER. Because all textbooks evaluated in this study were located in OER repositories, they were expected to be openly licensed in one form or another.

Accessibility: Cost
This study opened with reference to the increasing costs of commercially licensed textbooks and the impact those costs have on the accessibility of higher education. From an economic and social justice point of view, OER textbooks represent an improvement over commercially licensed texts (Hodgkinson-Williams & Trotter, 2018;Wiley, 2017). Affordability must therefore be one of the key criteria by which the accessibility of textbooks is judged. Lower cost texts were rated more highly. Because all textbooks evaluated in this study were located in OER repositories, they were expected to be either completely free or free in some format.

Accessibility: Media
There is disagreement in the literature about whether OER must be electronic or otherwise. Electronic distribution facilitates sharing, re-use, and repurposing. Moreover, electronic formats can be edited to enable accessibility for visually impaired learners (Butcher & Moore, 2015). HTML and PDF formats are particularly portable as they can be accessed on any electronic device. On the other hand, at least one definition of OER suggests that they must still be available in suitable formats (presumably print) to facilitate access for those learners who still do not have access to the Internet (Cape Town Open Education Declaration, 2007). Accessibility is also rated according to the number of media formats in which the OER are published.

Relevance: Match
Relevance was rated by comparing textbooks to local higher education course content. Course outlines were subdivided into topics and then topics were subdivided into the specific learning objectives. Each learning objective was compared with the open resource to determine how completely, if at all, each learning objective was covered in the text.

Methodology
Textbooks from two open textbook repositories were identified for evaluation: OpenStax CNX (https://cnx.org/) and BCcampus OpenEd (https://open.bccampus.ca/). Senior undergraduate students were recruited as research assistant coders (at the time of the study, the higher education institution of the study had few graduate students, none of whom were available for the study). The research assistants participated in two training sessions, one on the process of coding for each of the research instruments. Included in both sessions were brief introductions to OER in general and the use of open textbooks specifically. For each instrument, two coders were anonymously assigned to evaluate each textbook. Scores were compared for consistency. Where coders disagreed, these ratings were reevaluated to arrive at a mutually acceptable coding. All coding was done online using document sharing facilities of Google Drive.

Instrument One (Pedagogy and Accessibility)
See Appendix A for a copy of the complete form for evaluating textbook pedagogy and accessibility.
Pedagogy. Two coders rated each text on a variety of quality items suggested by the TIPS framework: (a) use of learning objectives, (b) unit activities, (c) diagrams and charts, (d) practice exercises, and (e) unit pre-assessments and post-assessments. Item ratings were totaled to give an overall quality score for each textbook. Textbooks with significant disagreements were noted and reevaluated by a third coder. Coders could add optional comments as well. Media. Coders recorded each of the formats in which the resource was available: print, HTML, PDF, ePub, iBook, Kindle, and other. Accessibility was ranked according to the number of different formats available.

Instrument Two: Relevance
See Appendix B for a copy of the complete form for evaluating textbook relevance.
Course match. Textbooks were identified that might be suitable for specific courses at a local higher education institution. Course outlines were used to detail specific topics, subtopics and learning objectives to be taught. Textbooks were then evaluated on how completely they covered each learning objective from the course outlines using a three-point scale (2=completely, 1=partially, 0=not at all).
Ratings totals were given as a percentage of the highest possible score. Coder totals were compared and where significant differences were noted, textbooks were re-evaluated by a third coder.

Data Sources
Textbooks from OpenStax CNX and BCcampus OpenEd were identified that targeted higher education academic courses that were likely to have content matching to local higher education courses. Technical books were not included.

Textbook Identification
In the two repositories, a total of 71 college textbooks were identified that were likely to be a good match with university or college courses. Some titles were cross-listed, resulting in some degree of duplication.
In all, 30 texts were listed in both repositories, meaning that a total of 41 unique textbooks were identified. Table 1 shows how these textbooks were classified by discipline.

Pedagogical Quality
Scores for pedagogical quality ranged from 1 to 9 out of a possible 9, with modal scores of 4 and 7 and a

Cost
Of the 41 textbooks evaluated, 40 were free and the remaining one cost between $25 and $49.99, though some texts added additional charges if students wanted to access printed copies. Nonetheless, all but one of the texts were free and the one that was not free, was still less expensive than commercial textbooks. Again, as expected, almost all the textbooks were completely free or free in some format.

Media
All textbooks were available in multiple media formats, with 31 of 41 available in three or more formats.
All textbooks were available electronically, either as PDF downloads or HTML or both (see Figure 4).

Course Match
From the total sample, 17 textbooks were selected to be evaluated against 14 different courses. Courses were matched against as many as three different texts. Similarly, the same text could be matched against more than one course. In total, 29 unique pairings of textbooks and courses were evaluated. Two coders rated each text using the form shown in Appendix B. Where there were disagreements, the author recoded the text to resolve the disagreement; re-coding successfully resolved all the disagreements. The degree of match for each of the courses is shown in Table 2. In general, the textbooks were well-matched to the courses. The mean degree of match was 74%, but this measure of central tendency may be misleading as two texts were particularly poorly matched to their respective courses (6% and 14%, respectively.) In general, textbook content was well-aligned with courses at the tertiary institution being studied, as demonstrated by the distribution of textbooks by degree of match ( Figure 5.) The modal category is 80 to 89% match and a majority of textbook-course pairings (22 out of 29) fell within the top three categories.

Discussion
This study set out to explore the quality of potential open textbooks on a variety of levels.
In particular, open textbooks were evaluated on measures of pedagogical quality, openness, cost, available media, and local relevance.
Pedagogical quality varied across the spectrum. Scores clustered around two modes, namely 4 and 7, indicating that texts exhibited consistent differences in instructional design. As noted earlier, almost all texts scored highly on certain key items: (a) learning objectives, (b) diagrams and charts, (c) learning activities, and (d) practice exercises. Those texts scoring higher made more consistent use of postassessments. Few texts employed pre-assessments. This heuristic approach of evaluating texts using instructional design principles needs to be balanced by comparing quality ratings to faculty and student ratings of open textbooks as in other studies (e.g., Jung, Bauer, & Heaps, 2017;Vander Waal Mills et al., 2019) and to the results of efficacy studies (e.g., Hilton, 2016). Unsurprisingly, all but one of the texts were free in some format or another (although some charged for hard copies). Even the text that was not free, cost in the $25 to $49.99 range, still considerably more affordable than commercial texts typically costing between $100 and $300 each.
As noted earlier, all texts were available electronically and in multiple formats. This is not surprising as it costs very little to migrate documents between different electronic formats. By publishing under an open license to begin with, OER publishers demonstrated a commitment to accessibility by publishing in as many formats as possible. Perhaps a more interesting question is why all OER publishers do not distribute in as many formats as possible?
Given the uniqueness of postsecondary courses, one would normally expect much variation in the degree to which texts matched courses. The high average degree of course match in the dataset (mean 74% and modal grouping 80-89%) was somewhat of a surprise. One explanation is that both databases had a large proportion of texts for introductory general courses that would not rely heavily on local content, for example mathematics, natural science, and business studies texts. This high degree of course match might be the strongest factor that could influence faculty to evaluate and potentially adopt texts for their courses.

Significance and Conclusion
This study set out to create a practical and easy-to-use tool to assess the quality of open textbooks and use that tool to evaluate the suitability of those textbooks for higher education courses in The Bahamas.
The quality assessment tool proved fit for purpose but could be strengthened by comparison with other measures of quality as described below. Comparison of quality scores with content expert ratings would give further insight into the effectiveness of this quality measure. Variation in measures of pedagogical quality may reflect different approaches to textbook design, particularly if authors are viewing texts as tools to encourage self-directed learning on the one hand or instructor-led learning on the other.
Unsurprisingly, when the assessment measure was used to evaluate texts in two OER repositories, all texts scored highly on openness and cost. Both of these measures seem more suited to deciding whether a resource qualifies as OER rather than evaluating the quality of the resource. On the other hand, Wiley (2017) argued that because education is the sharing of public goods of knowledge, skills, and attitudes, openness of learning content is a fundamental and core value. Copyright law and commercial licensing can have the effect of restricting public goods and turning them into private goods. In this interpretation, the first measure of quality must be openness, as this measures how well the resource is aligned with educational core values (Wiley, 2017).
Educational costs are increasing and as a result, students have difficulty financing higher education. This study can also be useful in raising awareness of suitable open textbooks for higher education use, in that it addresses several common barriers to use and adoption. Potential suitable textbooks were found to be readily available, inexpensive or free, and to match well with local higher education course content. Moreover, as most of them are licensed under the CC BY license, faculty are free to use, excerpt, adopt, and adapt these materials for their instructional needs.
The study is also useful because it demonstrates a new, easily reproducible method for evaluating the suitability of OER and open textbooks for adoption in courses. Next steps might include: • public campaigns explaining the findings to increase awareness of OER by faculty and students; • an exploration of faculty awareness of OER and their use; • an exploration of faculty capacity to use, reuse, adapt, and create OER; • capacity development workshops and training on how to find, use, reuse, and create OER; • an exploration of how faculty presently select textbooks for courses; • validation of the quality tool used in this study by:

Textbook Evaluation Form 2 (Relevance)
One form to be completed for each course. List texts to be applied to that course as Textbook 1, Textbook 2, Textbook 3, and so on.