Quality and Cost Matter: Students’ Perceptions of Open versus Non-Open Texts through a Single-Blind Review

Although prior research has examined student perceptions of open materials, research investigating students’ perceptions of open versus copyright-restricted textbooks through a direct, experimental approach is lacking. To better understand how students perceive open textbooks outside the context of the classroom, we examined students’ perceptions of unfamiliar open and non-open (copyright-restricted) psychology textbooks. Forty-four introductory psychology students reviewed chapters from two open textbooks and two traditional/copyrightrestricted textbooks and then ranked the textbooks from most to least favourite. Students rated each chapter on several quality measures, including layout structure, visual appeal, ease of reading, and instructional features. Next, bibliographical information and cost were revealed, and students re-ranked the textbooks accordingly. Before knowing the bibliographic information and cost, students were more likely to prefer the two traditional textbooks. There after, they were more likely to select the open texts. Students often referred to textbook price as a determining factor for their change.


Introduction
As the use of open materials in classrooms becomes increasingly common, educators and researchers are calling for closer examination of the quality of open materials, including open textbooks. In contrast to traditional publishers' textbooks, open textbooks allow users to reuse, revise, remix, redistribute, and retain the material for free (Wiley, 2014). Open textbooks are typically offered online free of charge, and printed copies are typically available for a nominal charge. Open textbooks can save students money while potentially increasing their performance through increased accessibility to materials, and students generally provide positive feedback about the open materials they use (Hilton, 2016).
In the open textbook literature, researchers often follow the COUP framework, which examines the quality of open textbooks by assessing cost savings, outcomes, use, and perceptions (Bliss, Robinson, Hilton, & Wiley, 2013). Using this framework, the main dimensions of open materials can be examined. The last dimension-student perceptions of textbook quality-was the focus of the present study. Some significant limitations have been noted of several current studies investigating student perceptions to open textbooks, including potential bias from instructors using the text. In his review of the literature of open textbooks, Hilton (2016) concluded that future research should be conducted in less-biased settings, suggesting that "They (students and teachers) could blindly [without knowing which textbooks are OER] evaluate the textbooks on a variety of metrics including their ease of use, accuracy of information and so forth" (p. 588). At least one study has carried out a blind comparison of open and traditional textbooks (Clinton, Legerski, & Rhodes, 2019) through a between-subjects design (where students only reviewed either open or traditional texts). To allow students to draw direct comparisons between the texts, we carried out a within-subjects design, something that has not yet been done.
In the present study, to minimize bias from others (namely faculty and classmates) on students reviewing the textbooks, we recruited students from an introductory psychology classroom to blindly review chapters from two unfamiliar open and two unfamiliar non-open textbooks on the first day of class. Students provided their feedback on the chapters' readability, organization, visual appeal, and add-ons (e.g., website links) and ranked their textbook preference from most to least favourite. Because it was a blind review, students did not know the bibliographic information at first. After students had conducted their initial review, the information for each text was revealed, including author, publisher, publishing year, location, available format (print and/or ebook) and price for each format in which the text was available. The researchers then asked students to rank their textbook preference again. By doing so, it was better understood how additional bibliographic information (including price) affects students' perception of open and traditional textbooks, compared to the content and quality of the textbook. This is to our knowledge the first study that required students to compare open and traditional textbooks through a direct, single-blind study where students were unaware whether the textbooks were open or non-open. Further, the study was carried out through within-subjects experimental design to minimize the effect of extraneous participant and other (e.g., classroom) variables that could affect the measures of perception.

Literature Review
Existing research suggests that students who use open textbooks generally consider them equal to traditional texts (e.g., Feldstein et al., 2012;Gil, Candelas, Jara, Garcia, & Torres, 2013;Petrides, Jimes, Middleton-Detzner, Walling, & Weiss, 2011;Pitt, Ebrahimi, McAndrew, & Coughlan, 2013). In Hilton's (2016) review of nine studies, it was concluded that "roughly half of students found OER to be comparable to traditional resources, a sizeable minority believed they were superior, and a smaller minority found them to be inferior" (p. 588). In conclusion, much of the research on open textbooks and OER suggests that students (and faculty) view OER positively (Hilton, 2016).
A top reason students have cited for liking open texts is cost. For example, Petrides et al. (2011) found that two-thirds of students preferred OER compared to a traditional statistics textbook, citing cost, as well as ease of use, as important factors. Similarly, Lindshield and Adhikari (2013) found that students using a digital textbook in a Human Nutrition course did not like to purchase a textbook. This is not surprising, considering students oftentimes do not purchase textbooks because of cost (Jhangiani & Jhangiani, 2017).
Reasons other than cost have been cited as causes for students' preference to use and enjoy open access materials. Feldstein et al. (2012) found that of 315 students taking a course in the business department at Virginia State University, two-thirds preferred OER materials to traditional materials. The majority of students explained that they found the OER were "easy to use" (95%) and up-to-date (78%). Other reasons include convenience of accessing online materials (Bliss, Hilton, Wiley, & Thanos, 2013;Lindshield & Adhikari, 2013). These findings were consistent with those from Grissett and Huffman (2019), who found that students perceived cost, weight, and convenience as the biggest advantages of open versus traditional books. Other studies have found that students prefer or are willing to use an open textbook after using one (e.g., Illowsky, Hilton, Whiting, & Ackerman, 2016).
Open Praxis, vol. 12 issue 1, January-March 2020, pp. [101][102][103][104][105][106][107][108][109][110][111][112][113][114] As highlighted, oftentimes cost has been cited as a driving reason why students prefer the texts that they do (Petrides et al., 2011). To better understand what factors, other than cost, drive student attitudes toward OER, researchers must continue examining these additional factors through empirical methods. Researchers must be cautious because the instructor may influence the students' perceptions (intentionally or unintentionally), which could potentially skew the results in favour of OER. In fact, in his thorough review of the research on OER, Hilton (2016) states that many studies on OER are affected by "the limitations of student perceptions and the potential biases of teachers involved in the creation or adoption of OER" (p. 581). Thus, when considering the factors that influence students' perceptions on the materials they are using, more research is needed to understand the textbook qualities that students value.
With at least two exceptions (Clinton et al., 2019;Woodward, Lloyd, & Kimmons, 2017), most research on student perceptions towards open materials to date has focused on texts that students are currently using. Familiarity with the text may influence students' attitudes toward the text; therefore, it is important to introduce texts that students have not previously used. Woodward et al. (2017) asked two graduate students to examine eight project management textbooks, two of which were open, and the other six were copyright-restricted. Their study focused on the process of qualitatively analyzing open versus non-open texts, not necessarily on the actual reviews themselves. Clinton et al. (2019) asked students to blindly review either open or non-open textbooks, taking a knowledge assessment afterwards and reviewing the texts, similar to the present study; however, in the current study, students blindly reviewed a set of texts and answer a series of questions about the quality of each textbook. By conducting a blind review, students could take an unbiased view of the texts they were reviewing. The study was carried out on the first day of an introductory psychology class in a further attempt to lessen students' familiarity with any psychology text. Bibliographic information was revealed at a strategic point in the study to capture students' pre-and post-knowledge perceptions to better understand the role several qualities, in addition to cost, have on students' perceptions of textbooks. These findings could provide additional insight into what students value in open textbooks.
To emphasize the importance of their responses and to increase student engagement with the review process, we asked students to help the faculty select a textbook to use next semester. The purpose of our study was to examine the role cost, content, quality, and design (e.g., images, study aids, organization) have on students' perceptions of textbooks. Our specific research questions were as follows: 1. Do students perceive the quality (e.g., content, organization, visual appeal) of open and nonopen introductory psychology textbooks differently? 2. What do students value when making their textbook selections?
The study involved within-subjects experimental design, where students reviewed both open and nonopen textbooks (these were counterbalanced across participants) and gave their feedback for each. The study aim was to increase internal validity by eliminating extraneous variables to understand the factors that drive student textbook preference.

Participants
Participants were forty-four students (21 females, 22 males, 1 did not report) in an introductory psychology course at a regional, four-year university in the United States.

Test Materials (Textbooks)
The textbooks used in our study were purposively selected due to their popularity and representative cost for their respective textbook type. For the open textbooks, we selected Introduction to Psychology (Anonymous, 2015), published by the University of Minnesota Library, and Psychology (Spielman, Dumper, Jenkins, Lovett, & Perlmutter, 2014), published by OpenStax. Both textbooks are free of cost and are available online in downloadable and print forms. For the non-open textbooks, we selected Mastering the World of Psychology, 5th ed. (Wood, Wood, & Boyd, 2014), published by Pearson and Understanding Psychology, 10th ed. (Feldman, 2010), published by McGraw-Hill. Mastering the World retails for $199.40 for a new print version of text, and Understanding Psychology retails for $92.34 for a new print version of the text. Both non-open textbooks also provided eBook options and students can find rental options and used books to buy. We do not claim that the two non-open textbooks are representative of all traditional textbooks in terms of quality but rather are an example of a text that would be used in an introductory psychology course. The same notion applies to open textbooks that we selected.
Students were provided with a packet containing black-and-white copies of select areas of the textbook: The Table of Contents (to provide students with an idea of the chapter titles and content) and the introductory chapter on the topic (Chapter 1). Each set of textbook information (Table of Contents and the first chapter) was stapled with a sheet of white paper on top with the typed title of the textbook. We did not include the actual image of the textbook cover to avoid potential bias. It is worth emphasizing that the open textbooks used were also available in hard copy format. Although open textbooks often have interactive qualities that non-open textbooks may lack, the books selected in this study were virtually identical to the format of non-open textbooks. This decision was intentional in order to minimize potential uncontrolled differences between textbooks.
Students also received a questionnaire (listed below) to complete at the end of each textbook review: "Now based on what you have seen, please rate the following qualities for [name of textbook] on a scale from 1-5, with 1 = Not at all; 2 = Almost not at all; 3 = Neutral; 4 = Somewhat; 5 = Very." Q1. Easy to understand Q2. Suitable for my learning style Q3. Interesting to read Q4. Comprehensive-the book covers the topics that I need to know Q5. Well supported by research.
"Further, please rate how much you agree with the following statements about [name of textbook] on a scale of 1-5 with 1 = Strongly Disagree to 5 = Strongly Agree." Q6. Images and visual aids help me understand the concepts. Q7. Visual aids are appealing. Q8. Images are distracting. Q9. It would keep me reading. Q10. Study aids are provided and are helpful. Q11. Research information is well used to explain the materials. Q12. Based on the table of contents, the content covered in the book is interesting to me. I can't wait to get started.

Procedure
To make the task more goal-specific and meaningful to the students, on the first day of class in an introductory psychology course, students were told that the instructor needed help selecting a textbook for the next semester. After attaining informed consent, students were asked to provide feedback about their textbook preferences to assist with the instructor's textbook selection. Students were given a sealed packet containing black-and-white, stapled copies of four textbook chapters and three sets of questionnaires, which were completed at different stages of the study. Students looked over the chapters in pairs, meaning they shared the four chapters with a classmate, yet reviews were completed independently. The study took approximately 40 minutes to complete and took place within the classroom. The true purpose of the study-to directly compare open versus non-open textbooks and to further understand what students value when selecting a textbook-were not revealed until the end of the study during the debriefing process. Students were not expected to read the material in its entirety but were instead instructed to "look over the material as though you were flipping through a textbook for the first time. Read a few of the passages, look at the pictures, look at the format of the material". After the instructions were provided, students were given 10 minutes to review all four chapters. Students shared the textbook chapters with a partner to cut down on paper. Students were asked not to talk during the review except to exchange chapters. Textbook chapters were counterbalanced across all participants, so that each participant received a different order of textbooks to review. Students were notified once the 10-minute review time had ended.
After reviewing the four chapters, students answered the questions provided. The first question asked students to rank their textbook preference from most favourite to least favourite and to "Please explain below what influenced your decision to place the textbooks in this order." Next, students were asked to rate each of the four textbook chapters on a set of twelve qualities (Q1-Q12). Students had access to the textbook chapters while completing their reviews, so they could refer to the chapters while answering their questions.
Next, students were provided with a document that revealed the bibliographical information of all textbooks after their first selection/ranking and ratings of the twelve qualities. Bibliographical information included title, author, publication year, publisher, available formats (i.e., print, Ebook), and respective cost/price. Students were given a moment to review this form and were given another questionnaire about whether their selection changed and why. If they changed, they were asked to re-rank the textbooks again with explanations.
Finally, in order to better understand what students value in their textbook selection process, students were asked "What would be your preferences in terms of textbook access?" They were asked to only choose one. Choices included a combination of buying, renting, or borrowing new or used print or electronic copies of books. They were then asked why they made the choice they did.
After students completed the questions and the questionnaires were collected, the instructor then debriefed the students about the purpose of the study-to compare student perceptions of open vs. non-open textbooks and to better understand what students value when choosing a textbook. Participant questions were addressed and the class was adjourned.

Data Analysis
Two quantitative analysis methods were applied. Wilcoxon signed-rank test was used to assess the differences on the book ranking before and after the bibliographical information was revealed to students. The test works with ordinal data (i.e., ranking of book) to compute the difference between pretest and posttest data for each person (Dimitrov, 2008). The confidence interval of the difference was 95% and the standard significance (2-tailed) value level was set at .05.
Simple repeated-measures ANOVA, used to examine a group of subjects with repeated measures (Dimitrov, 2008), was applied to detect the differences in the students' perspective regarding the four books. In the current study, book was the independent variable, including two open textbooks and two non-open textbooks. Each question's rating score (total 12 questions) was the dependent variable.
Additionally, qualitative content analysis was done to analyze student responses from the openended questions, which inquired about factors that influence their ranking at the first time and second time. Content analysis is considered a flexible method to analyze text data from impressionistic, intuitive, interpretive to systematic, strict textual analysis (Cavanagh, 1997;Rosengren, 1981) by summarizing and reporting the content of data and their messages (Cohen, Manion, & Morrison, 2007). The researchers first went over all the responses from the open-ended questions. Next, we further classified responses extracted from each student's response for coding purposes. We rescreened the coded materials again after all the respondents' responses were coded. Then, categories of similar or matched content were merged, and several themes further emerged from the data, thus allowing us to explore the messages beneath students' responses.

Student Perceptions on Textbook Qualities
The review sheet consisted of 12 qualities that students valued when selecting textbooks. Other important criteria, like content accuracy, were not included as students might not have the ability to judge that criteria. For all 12 questions, the only statistically significant difference between the textbooks were on Q6 (helpfulness of image and visual aids) and Q7 (appeal of visual aid), both of which asked about visual images (Table 1). Results from the repeated measures ANOVA indicated there are statistically significant differences in the rating for Question 6, 'Images and visual aids help me understand the concepts' [F (3, 123) = 3.50, p = .018] and Question 7, 'Visual aids are appealing' [F (3, 123) = 3.62, p = .015] between the four books at the .05 level (See Table 2). Introduction to Psychology (open) had lowest average rating on all review items, as it has almost no visual aids in the book.

Student Textbook Ranking
As seen in Table 3, 28 students out of 44 (63.6%) selected the non-open texts as their favourite texts in the first round of ranking. After reviewing the additional bibliographic information, 28 out of 44 (63.6%) students selected the open textbooks as their favourite texts. Regardless the reason for changing book ranking and the direction of changes (upward or downward), 23 out of 44 students changed their ranking.

Factors that Influence Textbook Selection Ranking
Students' responses about the factors that influenced their rankings were coded according to categories, arranged in four groups. Each group represents an emergent theme from the data. The first group referred to the learner-appropriate features. The second group involved the learning supports and application of instructional strategies. The third group addressed the design and layout of the textbooks. The fourth group concerned topics and contents. The final group included responses that did not provide a specific feature or factor relevant to the other categories. As evident in the subsequent discussion, some categories overlap and interact.
Learner-appropriate writing/wording. The most salient theme to emerge from students' responses was learner-appropriate word usages and that the texts were easy to digest, understand, were straightforward, and to the point. Supportive learning tools. Another salient theme that emerged from the responses was the supportive learning tools in the textbooks that help students study, such as overview, glossaries/key terms, summaries, exercises/activities, and study guide. Design of the textbook. The third theme that emerged from the responses was regarding the design of the book that makes reading the book go more 'smoothly' and 'effectively'. Categories under this group include page layout, visual aid, font size, graphics, and text structure "…… has more charts and looked like it was a better set up book." (Student#30) "…… had interesting fronts and caught the readers eye. Throughout the text the [font] families helped assist the reader with specific topics." (Student #17)

'I am more of a visual learner and I like how [title] just didn't contain words but also had pictures and diagrams [has] caught my eye and kept me interested." (Student #29)
Topics and content areas. The fourth theme was that topics or subject-related content were factors that influenced students' decisions on textbook selection. Other: Credibility and general feeling. This group included the responses that (1) did not provide any specific features or factors or (2) did not belong to any of the major groups above. The categories include credibility, ownership of the book, and overall feeling about the books. For Introduction to Psychology (open), the mean of the book ranking before receiving the bibliographic information (0.00) was lower than the mean ranks after receiving the bibliographic information (8.00). Similarly, the mean of Psychology (open) ranking before receiving the bibliographic information (5.50) was lower than the mean ranks after receiving the bibliographic information (8.63). On the other hand, the average ranking for Mastering Psychology (non-open) before obtaining the bibliographic information (9.60) was higher than the average ranking after obtaining the bibliographic information (4.50). We can conclude that in the population, the observed difference/changes in preference ranking regarding the four different books is not likely due to chance.
Furthermore, the extent of the difference in book ranking after bibliographic information revealing was explored. 20.5% (n = 9) of the respondents raised the ranking for Introduction to Psychology (open) by one, 9.1% (n = 4) of students raised the ranking by 2, and 4.5% (n = 2) moved the book ranking up by 3. For Psychology (open), 15.9% of respondents (n = 7) raised the ranking of Psychology (open), around 11.4% percent (n = 5) raised the Psychology (open) ranking by 2, while 6.8% of students (n = 3) dropped the ranking down by 1. In regard of Mastering Psychology (nonopen), around five percent of students moved the ranking for Mastering Psychology (non-open) up by 1. On the other hand, 13.6% of respondents (n = 6) dropped the ranking by 1 and 2, respectively, and 6.8% of students (n = 3) moved the ranking down by 3.
That is, after revealing the bibliographic information, the ranking for Introduction to Psychology

Qualitative Data from Open-Ended Responses
Students were asked to provide open-ended responses for the following questions: (1) Factors that influenced their initial textbook ranking; (2) Factors that influenced their second textbook ranking; and (3) Justification for textbook preference. Each of these questions were analyzed for themes, which are provided in the following sections.
Students who did change their ranking. After reviewing additional bibliographic information, 22 students changed their ranking/preference of textbook selection. When asked what influenced their decision if they change their ranking order, cost/price was mentioned by most of students (19 out of 22), followed by publication date (2 out of 22) and easy access (1 student). Sample responses involving cost/price as a deciding factor include: "Price was def my main reason, along with the content in the more money one is not worth it" (Student #40) "……the introduction to psychology was cheaper and had just as much info as mastering the world of psychology" (Student #41) As mentioned earlier, more than half students were on the Pell grant or student loan. The cost of textbook will potentially contribute to more financial burden to students.
"The price, because as a college student you have budget your money. As a college student you have to buy more than one book" (Student #11) "I am a broke college student" (Student #39) Finally, two students out of 22 indicated that publication date was a factor that influenced their ranking solely. One changed ranking solely because of publication year.
"Year of publication was the major factoring decision. Psychology is a science and therefore always, with that information I would like the newest date option available." (Student #25) Students who did not change their ranking. After reviewing additional bibliographic information, 22 students did not change their ranking/preference of textbook selection. The top reason why their preference remained the same was how books helped them succeed (See Table 4). Comments included "easy to read", "easy to understand", "more interesting", "suit my learning style", and "help me learn" etc. By reviewing their previous response, the helpfulness could be associated with the quality and feature of the books.
Three students expressed that topics of the book covered were the reasons, which were also consistent with the influential factors they pointed out in the first round. Three students' reasons related to cost/price. One said the book s/he likes the most happened to be the cheapest one. One student said the one s/he chose "was within reasonable price." The third student said "I liked the more cheap book." Our interpretation of this response was that the book the student chose was the cheaper one among the four. Under the "other" category, one student said that s/he already had the book as the reason s/he did not change the ranking. It was the same reason why s/he rank the topic choice in the first round. One did not provide any reason; and one said "no reason" for not changing textbook selection preference. As you can see in the responses above, price/cost was also noticed by students who did not change their top choice or ranking order. For some, what helped them most in learning was the primary Open Praxis, vol. 12 issue 1, January-March 2020, pp. 101-114 reason, regardless the price. In some cases, the book they chose happened to be the cheapest one or in a reasonable price range.
"Because the book I ranked at number one is within a reasonable price range" (Student #26) "I did not change my opinion because of the cost of something doesn't change how informational the book is. The book I chose as the best happens to be the cheaper ones." (Student #32) To sum up, based on the student responses to the open-ended questions, we noticed that good textbook features that help students learn do matter to students. They consider factors that contribute to the overall learning effectiveness. These features include message design (i.e., layout, the use of font style and size, organization of text) and instructional support (i.e., study guide, key terms, overview and recap/review, additional resources for further exploring). Students noticed those textbook design and features and mentioned they will choose the book because these features are helpful to their learning. Another thing worth noting was that students also pay attention to the publication date. And the accessibility and the format also play some roles in students' decision process.

Discussion
The purpose of the study was to examine the role various textbook qualities, including print options (electronic versus hard copy), content, quality, design (e.g., images, study aids, organization), and cost have on students' perceptions of textbooks. Specifically, we aimed to draw a direct, blind, nonbiased comparison between two types of textbooks-open versus non-open-to better understand students' perceptions of these textbooks on the textbook qualities outlined above.
Most studies on student perceptions of open textbooks have centered on texts that were currently being used for a course. One study that did draw direct comparison between open and non-open textbooks (Woodward et al., 2017) did not carry out a blind review or focus on textbook qualities. Because teacher (and student) bias may serve as a confounding variable in these studies, we had students blindly review textbooks, not knowing which were open and which were not. Students were therefore able to assess each textbook on key qualities of the text itself, not necessarily price or faculty preference. Other research that has controlled for these factors (Clinton et al., 2019) carried out a between-subjects study, which did not provide the same direct comparison offered in the present study.
Our results showed that students liked the non-open textbooks initially; however, after revealing bibliographic information (print options, publisher, publishing date, cost), students preferred open textbooks. Although cost/price was the primary reason of changing preference, students did not appear to simply change it because of cost/price by itself. Students do weigh the cost with the additional perceived values. In other words, students still care about the quality and features of the textbook, but they do not necessarily feel the price is justifiable. These findings are compatible with previous findings that suggest students value qualities other than cost (e.g., Grissett & Huffman, 2019).
Students also found the textbooks generally comparable, only pointing to differences in visual appeal between an open textbook, Introduction to Psychology and non-open textbooks, Understanding Psychology and Mastering Psychology. The remaining qualities were not statistically significant, illustrating that students did not differentiate between the two forms of text (at least with the texts that were used). These findings suggest that open textbooks-at least those used in the current study-are more similar than different in their qualities. Perhaps if different, more interactive texts were used, students would perceive the two types differently.
Our qualitative data revealed that students found reading ease as a major reason for selecting the textbook they did. Reading comfort is important, as it can motivate students to continue reading. If the wording was challenging or somehow difficult to read, it may turn students away from the textbook, an essential material for their learning.
Still, the present study was not without limitations. First, we examined students' initial perceptions of potential course material. Students had virtually no relationship with the text. The study may have yielded different results had the study examined students' relationships with the material over time, as students would have more time to become familiar with the texts' strengths and limitations. This approach, however, would be difficult to carry out with a within-subjects design (where each student was exposed to each type of text). It is possible, though, that a similar semester-long approach could be carried out, where students would each use a different textbook and their perceptions of each would be compared across groups. Similar research has been done before (e.g., Grissett & Huffman, 2019), but future research may continue to examine the effects of textbook quality and cost on student perceptions over time.
Similarly, the current study occurred outside the context of a classroom experience. Although the study occurred in a class setting with an instructor present, the impact of the class context on students' perceptions of the texts were not considered. In fact, our study intentionally controlled for these contextual factors to isolate and exclusively focus on the textbook. Future research may explore the interrelationships between the textbook and other factors, including the teacher, setting, relationship with other students, level of engagement, and learning activities.
Another potential limitation was that students reviewed chapters from four textbooks, which could potentially lead to evaluation fatigue. If students experienced this, it is possible that students' reviews of later chapters were negatively impacted and perhaps students viewed the first text as most satisfactory. We tried to prevent this through counterbalancing the texts across participants, but future research should try to capture potential order effects, as well.
Finally, it is worth noting the context in which students reviewed the texts. We believe the students were genuinely invested in their textbook reviews because they were helping a faculty member choose a textbook for the next semester. We recommend that future research implement textbook review studies so that students are involved in the process.

Conclusion
Based on the results, we can conclude that textbook quality matters, and not all textbooks meet students' needs in the areas of design and pedagogy, regardless if open or copyright restricted textbooks. Qualitative data shows that students care about the quality of the materials they use. Some students may not be able to articulate how textbook design or quality influence their preference, but they do express the overall feeling on textbook towards their learning effectiveness. Some students in our study were able to point out specific textbook features that help or motivate them to study (i.e., study aid, key points, visual representation of the concepts, wording, visual layout, and aesthetics). While they may not have enough expertise to judge the accuracy or depth of the content, they are able to choose the proper level of difficulty for their learning.
We can also conclude that students value textbook design (i.e., layout, visual representation) and pedagogy, including learner-appropriate wording, scaffolding, and instructional support tools such as overviews, study aids, and key points/summaries. Students may not have the expertise to judge the content accuracy or level of expertise in the area, but they are able to point out what textbook features are helpful to their learning.
Open Praxis, vol. 12 issue 1, January-March 2020, pp. [101][102][103][104][105][106][107][108][109][110][111][112][113][114] Finally, (and most importantly) when students review and select these textbooks, they consider both cost and quality of the textbook as it is relevant to the value of learning. They are not choosing books solely on the cost, although it is an important factor to their academic success. Students choose what they think they can learn most out of the book they choose. However, while the cost was not justifiable to the value of "quality," students choose more affordable options. The study provides insight from students' perspective and helps identify areas to be considered so that students' needs may be better served.