Missing factors for assessing openness: Content volume and “harvestability”

Openness in open courseware (OCW) and open educational resources (OER) requires an open licence, such as Creative Commons licenses, but is affected by several factors both technological and pedagogical. This pilot study examines different factors impacting openness by looking at a very small random sample of 10 relatively recent open courseware offerings from TU Delft and MIT. This paper has two primary objectives: 1) to determine how open the sampled OCW are across eight factors of analysis; and, 2) to determine if the sampled OCW are suitable for educator reuse. The authors evaluated the sampled courses using an existing framework to conceptualize openness. The level of openness was evaluated across eight-factors: copyright/open licensing, accessibility/usability, language, support costs, assessment, digital distribution, file format, and cultural considerations. The framework describes each factor across three dimensions of openness — closed, mixed, and most open — and each author coded the sampled OCW accordingly. This content analysis provided several insights into where sampled OCW succeeded and failed in terms of openness. Courses tended to be relatively open in terms of copyright, assessment, and digital distribution, but closed in terms of language, support costs, and file format. Factors such as accessibility and cultural considerations were more mixed; discipline and course content play a factor in a course’s openness and reuse. This paper also serves a secondary purpose, on the effectiveness of the framework for assessing openness. Openness is a spectrum, with an interplay between factors that determine openness. Greater attention needs to be shown toward pedagogical considerations, rather than technical, when developing open content.


Introduction
The UNESCO Recommendation on Open Educational Resources (OER) (UNESCO, 2019) represents an important development in recognition of the OER movement. While the significance of the UNESCO Recommendation should not be diminished, it clearly defines the "open" element of OER in copyrightcentric terms; OER are either public domain materials or those with an open licence (UNESCO, 2019). UNESCO [1] defines OER as "teaching, learning and research materials ... that reside in the public domain or have been released under an open license." Although an open licence for copyright-protected works is essential for resource distribution, this approach reflects an instrumental and licence-centric approach to openness and masks a more complicated relationship that open licensing has with other elements of openness. Open licensing is essential for granting permissions that allow works to be copied and shared; yet, it alone is an inadequate measure of openness because other factors dictate educators' abilities to adapt existing materials and develop new ones. This inadequacy is made apparent when considering openness in the context of open courseware (OCW), an important, though undervalued, component of open education. The Open Education Consortium (n.d.) defines OCW as "free and open digital publication of high quality college and university-level educational materials. These materials are organized as courses, and often include course planning materials and evaluation tools as well as thematic content." While OCW are often published by universities, the content is freely available to anyone via the Internet. Open licensing is necessary, but not sufficient for educators who might want to adapt and adopt the material. A central tenet of OCW, as the Open Education Consortium's definition suggests, is the inclusion of course planning documents for educator reuse.
McNally and Christiansen (2019)   There are several qualitative approaches one could take to analyze OCW. However, given the lack of clear pedagogical guidelines for evaluating OCW openness, we sought to employ a novel approach as a pilot study. We report not only on how "open" OCW were and whether they would satisfy educator needs; they also reflect on the functionality of the open enough framework as an analytical tool. By employing and suggesting refinements to this framework, this paper outlines practical considerations that would be helpful to both educators interested in adapting existing OCW and administrators of OCW platforms.
The findings suggest that OCW need to be developed with both technical and pedagogical considerations in mind to be truly open. Sampled OCW were less open than expected, particularly in terms of their language availability, file format, and support costs; copyright licences also potentially limited openness. We conclude that the sampled OCW were open from a licensing perspective, but better suited for student consumption rather than educator adaptation. There are relatively low-effort strategies that could be employed to improve OCW openness and make them more educator friendly. These include using less restrictive licensing terms, providing guidance and documents to aid future language translation, supplying documents in multiple file formats, and including more open reading options, including licensed library materials. The sampled OCW were relatively open in terms of their accessibility (stated adherence to commonly observed standards), the availability of assessments, and discoverability through OER search engines.

Literature
The ascendency of open content -from open source software, open access scholarly publications, and OER -have led to a body of scholarship examining what openness means and what unites or distinguishes various forms of open content. Prior to the emergence of OER 20 years ago, education scholarship highlighted the complicated nature of defining openness in education (Noddings and Enright, 1983) with a variety of definitions evolving over the past century (Baker III, 2017). During the relatively short history of OER, there have been attempts to define what is (or is not) OER (Pantò and Comas-Quinn, 2013). Narrow conceptions of openness tend to focus on legal aspects centred on the open licence as the essential element of openness and equating openness with access (Knox, 2013), with many early definitions of openness emphasizing the centrality of open licensing (Wiley, et al., 2014). Wiley's (n.d.) 5Rs (retain, revise, remix, reuse, and redistribute) places legal rights conferred through an open licence as the essential and determining characteristic of openness. In addition to legal conceptions, openness is also connected to the collaborative and participatory nature of the Internet (Maxwell, 2006). Tkacz (2012) argues that openness is primarily a new techno-legal and oppositional approach aimed at countering conceptions of proprietary ownership. Richter and McPherson (2012) posit that, beyond open licensing, OER are only valuable if they fit the learner's context and are fully adaptable. They note several non-technical or licensing barriers including language, context gaps, lack of cultural diversity, and literacy levels among other barriers to the adoption of OER.
While rights-centric/open licence-oriented views of openness are common in OER literature, there exists a broader range of scholarship examining alternative definitions. There is growing recognition of the limited nature of the rights-centric open/closed binary (Rolfe, 2017;Farrow, 2017Farrow, , 2016. Hodgkinson-Williams and Gray's (2009) study of OER usage highlights four elements of openness -social, technical, legal, and financial -and constructs each element along a spectrum of openness. Pomerantz and Peek (2016) provide an extensive review of different types of open content demonstrating a range of meanings of openness. Most importantly their work reveals that there is no single unifying conception of openness across various forms of open content. Similarly, Economides and Perifanou (2018) present a typology of 11 different approaches to openness. Cronin's (2018) doctoral dissertation contains an extensive discussion of various definitions and approaches to openness. She notes that critical approaches go beyond the simple binary of open and closed and focus on issues of risk, participation, and power. Her dissertation builds on her previous work (Cronin, 2017) where she noted four approaches to openness in education. Neylon (2017) argues that the common element of open content in academia is a return to traditional values of open knowledge exchange. However, openness has expanded and contracted in waves throughout centuries of university education (Peter and Deimann, 2013), and certain open practices and concepts (e.g., open society, open stacks/shelves, and even open source beer) pre-date both digital technologies and open licensing systems (Smith and Seward, 2017). In this respect, openness could be thought of as a form of educational transparency. Farrow (2017) underscores the importance of a reflective and strategic value of contemplating what openness means and underscores the importance of linking openness with critical pedagogy. Bayne, et al. (2015) echoes Farrow's sentiment, while Lambert (2018) posits that the definition of open education must explicitly incorporate and emphasize social justice concepts. Finally, it is important to note that the extensive use of the term, a practice coined as "open washing" (Watters, 2014), may result in the term being conceptually nugatory (Weller, 2014).
Frameworks have been developed for assessing many facets of open education and OER. Frameworks exist for examining the quality of OER (Elias, et al., 2020;Achieve, 2011;Moise, et al., 2014;Kuriolvas, et al., 2011), OER adoption (Cox and Trotter, 2017;Abeywardena, 2017), usability facets of open courseware Web sites (Rodríguez, et al., 2017), OER accessibility (Avila, et al., 2016), learning design quality (Stracke, 2019;Avila, et al., 2016), and ethical aspects of OER research (Farrow, 2016). There is even a rubric for evaluating OER quality rubrics (Yuan and Recker, 2015). Although these frameworks provide a means of assessing different aspects of OER, there are a more limited number of frameworks specifically created to examine openness.
One of the earliest frameworks is Hilton, et al.'s (2010) ALMS framework. ALMS consists of four broad dimensions for educators to consider when developing open content. These dimensions include: 1) access to editing tools to edit content; 2) the level of expertise required to edit the content; 3) meaningful editability of the content (i.e., designed with editability in mind); and, 4) if the content is self-sourced. The ALMS framework underscores the importance of considering downstream users and the relationships among licence and filetype. However, ALMS, in its initial expression, lacked a rubric for evaluating openness and instead functions as a general guideline. Gurell (2012) directly addressed this shortcoming. While access to editing tools was the most challenging element to develop a rubric for, Gurell was successful in developing a scoring mechanism for open content and concluded that the ALMS framework underscored an earlier conclusion that educational effectiveness and reusability are inversely related (Wiley, 2001;Wiley, et al., 2004). Abeywardena, et al. (2012) addressed the lack of a rubric for the ALMS framework by developing a desirability measure (or D-Index) for quantifying relevance, the level of access, and the level of openness. At present, no additional attempts have been made to further develop the ALMS framework into a concrete rubric for evaluating the openness of OER and OCW.
While the ALMS framework is the most developed openness framework for OER/OCW, several other approaches exist. Ehlers (2011) presents two matrices about OER and open educational practices. Stagg's (2014) continuum of open practice arranges activities related to open education from least open (basic awareness) to most open (student co-creation of resources). Hodgkinson-Williams and Gray (2009) present four continuums that define openness. The Open Enough framework (McNally and Christiansen, 2019) is the most recent example in this area. Like the original ALMS framework, the Open Enough framework lacks any formal rubric for critically analyzing openness, but it offers more detailed guidance for evaluating openness because it provides eight factors that comprise "openness" (in the broadest sense) and it describes how those factors might be manifested across three dimensions of openness -closed, mixed, and most open (see Figure 2).
Although the literature around openness in education is rich, there is little direct application of these frameworks to OCW specifically. Discussions about how openness might be conceptualized are not always helpful to "on-the-ground" educators who want to use and adapt course content. This paper attempts to fill this literature gap, and builds on McNally and Christiansen's (2019) previous work, by employing the Open Enough framework to conduct an in-depth content analysis of selected OCW, with the goal of demonstrating which aspects of these courses are open, which aspects are closed, and how this openness/closedness affects educators who wish to adapt content. In this paper, we conducted a thorough content analysis of select OCW. Using the Open Enough framework as a coding scheme, we analyzed the content of a random selection of OCW. We pose two research questions: Question 1: Based on the eight factors of openness in the framework, how open are the sampled OCW? Question 2: Are the sampled OCW adequately designed for educator reuse or adaptation?

Study design
To better understand openness, how OCW can be improved, and where instructor efforts to increase openness should be directed, a comparative multi-case study involving a content analysis of 10 OCW courses was designed (Berg, 2004). The overall research design included three sequential steps: identification of institutions offering current OCW, selection of individual open courses (the cases), and content analysis of the cases using the Open Enough framework as a coding scheme.
Purposive sampling of institutions was chosen, as the aim was to find insights that were generalizable to OCW broadly (Creswell, 1998). The purposive sampling was developed through a preliminary review of OCW offerings listed on the Community College Consortium for OER Web site (n.d.). The site lists OCW offerings from several institutions. Delft University of Technology (TU Delft) and MIT were chosen as the sampling databases, as both institutions (despite being science and engineering focused) offer OCW in a variety of disciplines at both the undergraduate and graduate levels. Older courses (courses designed and offered before 2016) were eliminated as newer offerings often had the benefit of modern enhancements such as updated user interfaces and updated literature. We compiled a list of the most recent courses from both databases (late 2019 to early 2020) into two spreadsheets. A total of 37 courses from MIT and 212 courses from TU Delft were compiled. However, the 212 TU Delft courses were reduced to 115 once MOOCs (Massively Open Online Courses) were excluded. MOOCs, by definition, are designed for larger enrollments (in the thousands) and less for educator reuse. To keep comparisons consistent, they were removed, however we assert that MOOCs should be examined in future research. Using a randomization formula, five courses were selected from each database ( Figure 1). The version of the course (date last edited) is indicated in parentheses. The small sample of courses was deliberate; it reflects the time required to conduct an in-depth content analysis of each case, which we contend is a sufficient number of replications to separate unique phenomena from general trends (Yin, 2009).
With the cases selected, we then undertook a content analysis of 10 courses using an a priori coding scheme (Wimmer and Dominik, 2011). The use of an a priori coding approach aligns with Hseih and Shannon's (2005) discussion of directed content analysis. A directed content analysis "is to begin coding immediately with the predetermined codes. Data that cannot be coded are identified and analyzed later to determine if they represent a new category or subcategory of an existing code" [2]. A priori coding using the Open Enough framework ( Figure 2) was chosen for two reasons. First, coding categories should be mutually exclusive, exhaustive, and reliable in that the coding process should yield identical or nearly identical results regardless of the coder (Wimmer and Dominik, 2011;Del Balso and Lewis, 2001).The Open Enough framework aligns with these criteria. Second, using the framework as a coding scheme also allows for the testing of the conceptual framework with real world examples. The framework provides eight factors of analysis to consider when developing OCW. These factors include copyright/open licensing, language, support costs, assessment, digital distribution, file format, and cultural considerations. Each factor's level of openness is described across three dimensions of openness -closed, mixed, and most open. Definitions for each factor along with delineations of each factor as closed, mixed, and most open are detailed in Figure 2. The "closed" category describes the factors when there are no special considerations for openness. The "mixed" category describes the factors as resources or courses that are somewhat open but do not maximize openness. The "most open" category describes what maximizing openness would entail for each factor. We coded each case/course for all eight factors. Each factor was coded based on the framework using the codes "closed," "mixed," and "most open" as defined in the Open Enough paper (McNally and Christiansen, 2019). After coding independently, we discussed and mutually agreed upon final determinations. Because of the dual nature of the study -to examine both OCW openness and comment on the applicability of the Open Enough framework -inter-coder reliability is discussed in qualitative terms as opposed to quantified metrics (e.g., Krippendorff's alpha, Cohen's kappa, etc.). The identification of new factors resulting from the analysis is further discussed in the section entitled "Refining the Open Enough framework". The level of openness across the sampled OCW was inconsistent. From a copyright perspective, the OCW were mixed as they employed an institutional licence across all content -a CC-NC-SA licence. In terms of adherence to accessibility standards, MIT courses were most open while TU Delft were closed. We determined accessibility openness by reviewing each OCW platform's statement regarding which accessibility standards were observed. From a usability perspective, we concluded that TU Delft has the more modern interface, but MIT courses were more navigable. All courses (except one) were categorized as closed since the course content was offered in one language. The majority of support costs (course readings) were proprietary and many of the courses relied heavily on paid books. Two TU Delft courses were the exception to this rule; one course's readings were completely custom (presumably written by the instructor), and another course made its proprietary ebook openly available (albeit with a closed licence). All of the sampled courses were very open in terms of digital distribution. Each course could be located using several federated OER search engines. Surprisingly, the sampled courses were categorized as closed in terms of file format, as all relied heavily on PDF content. We struggled with the cultural considerations factor, given the large volume of content and the factor's subjective nature. Courses were split between closed and mixed/most open. Course content dictated this conclusion, as STEM courses materials were determined to be more culturally applicable, broadly speaking.

Question 2:
Are the sampled OCW adequately designed for educator reuse or adaptation?
Given the limited editability of the course materials, we concluded that the sampled OCW were better suited for learner consumption rather than educator reuse. The lack of editability was surprising and severely limits these courses' utility outside their originating institutions. These findings highlight the importance of prioritizing OCW for editability and adaptability, in addition to making them discoverable to the broader community.  The following subsections provide a deeper analysis of the sampled OCW. The findings are organized into subsections, corresponding to each of the eight factors as outlined in the framework, as opposed to by course, to eliminate redundant discussion as several factors were influenced institutionally rather than at a course level. Examples of openness and closedness are strategically highlighted in each of the subsections.

Copyright/open licensing frameworks
The copyright and open licensing factor is simultaneously paramount and insufficient for achieving openness. Without an open licence, there is no possibility for replication or reuse. Yet, OCW featuring an open licence, but without consideration for the other factors, represent simple open instrumentalism rather than a more pragmatic approach to openness. The Open Enough framework asserts that closed courses would not employ an open licence leaving all rights reserved with the rights holder (which may be the publisher, not the original author). Mixed courses employ an open licence that features restrictive elements such as the Creative Commons NonCommercial or ShareAlike conditions. The most open courses would employ a licence that allows for maximum shareability through the least restrictive licensing terms such as CC-BY or the public domain waiver. All of the courses analyzed fell under the category of a mixed resource. Both TU Delft and MIT enforce a common licence across all courses in their respective databases which is a CC-BY-NC-SA (or Attribution, Non-Commercial, ShareAlike). The NC licence element restricts the use of OCW materials to not-for-profit endeavours. On the surface, this quid pro quo approach to licensing seems reasonable, but it likely holds back the creation of new OER/OCW as combining materials with conflicting licences is prohibited (Creative Commons Wiki, 2015).

Accessibility/usability formatting
The accessibility/usability formatting factor evaluates OCW on their adherence to established accessibility standards and overall usability. A closed course would not adhere to any accessibility standards, except for those built into the learning management platform where the course was hosted. A mixed course would include some basic accessibility features such as closed captions for video or audio transcripts, though this category is not limited to these examples. A completely open course would adhere to established accessibility standards such as the W3C (World Wide Web Consortium) or HHS 508 (U.S. Department of Health and Human Services [HHS] Section 508) in the United States. We chose to evaluate accessibility by each platform's self-reported adherence to a recognized accessibility standard.
Five of the courses were categorized as closed (all TU Delft), and five were categorized as most open (all MIT). TU Delft had several shortcomings across its courses. Learners could search video lecture transcripts, yet there was no option to download a copy of a transcript file. Textual documents were generally available in PDF format and would presumably work with a screen reader, though the FAQ page did not explicitly say if documents or pages were validated for accessibility. TU Delft offered a help page for its OCW and provided more detailed information about screen reader compatibility, but the institution did not explicitly state if it adhered to an accessibility standard. MIT stated that "OpenCourseWare is committed to accessibility for persons with disabilities and strives to meet W3C Web Content Accessibility Guidelines (WCAG) 2.0, Level AA, including validating HTML, captioning the video, and checking the accessibility of course content as part of the authoring process" (MIT, n.d.-a). All video content included closed captions, as well as an ability to download an SRT transcript file. How well each institution's Web site actually adhered to common accessibility standards (Alt tags for images for example) was not verified. Several online tools existed for validating HTML accessibility, but errors could be found on most Web sites. To properly examine HTML accessibility, a more fulsome analysis of the Web site markup and course documents would be required.
Usability was considerably more challenging to evaluate. Unless sticking to a predefined set of evaluation standards, usability was too subjective a measure to reliably label courses as closed, mixed, or most open. Some courses were more or less easy to navigate overall. TU Delft's "Drinking Water Treatment 2" was somewhat confusing to navigate at first, due to changing navigation menus. TU Delft's user interface is more modern and appears to work slightly better on mobile devices. MIT's interface is generally more dated from a design perspective, but MIT courses were more straightforward in terms of their navigation, especially when moving back and forth between course sections to access content.

Language
The language factor is a measure of how many languages course materials had been translated to and/or the level of consideration given to guide translation. Nine of the courses were categorized as closed and one was categorized as mixed. All of the TU Delft and MIT courses were offered in English, with some of the TU Delft materials also being offered in Dutch. Class PowerPoint slides and recorded lectures (including closed captions) were largely provided in English. MIT courses were exclusively offered in English which is not surprising given that it is an American institution.
Language continues to be one of the hardest elements to maximize openness and determine. Creating a bilingual course is a monumental undertaking for an individual so this factor is likely to remain more closed without financial support from educational institutions or government. The challenge occurs when assessing courses that inconsistently offer resources in different languages. For example, TU Delft's "Drinking Water Treatment 2" included English slides and lecture videos, while some of the final assignments were provided in Dutch. The inconsistency is understandable if some of the example assignments were written in Dutch by students. TU Delft's "Public Hygiene" course was primarily in English, but included some supplemental materials in Dutch.

Support costs
The support costs factor examines the openness of a course's required and suggested readings. For example, a closed course would be reliant on a paid textbook. A mixed course might forgo a textbook to save students money but might still rely on licensed library resources which have an institutional cost and are likely not open access. The most open course would rely exclusively on openly licensed textbooks, open access readings, or materials in the public domain.
Eight of the courses were categorized as closed, one was categorized as mixed, and one was categorized as most open. All MIT courses were categorized as closed because they relied heavily on paid books, and course readings lists frequently included links to Amazon for purchase. Presumably, many of these resources could be available through the MIT libraries, or a public library, but there were very few open readings that would be accessible to the public. In contrast, TU Delft's courses offered a better mix of closed and open resources. "Public Hygiene and Epidemiology" (2016) is perhaps the best example of a completely open course as its readings were custom to the course and were openly licensed. In comparison, "The Hydrology of Catchments, Rivers and Deltas" (2016), categorized as mixed, had a proprietary textbook published by Elsevier that was available digitally for free, but a hard copy would have to be purchased. However, like the "Public Hygiene" course, there were many custom course readings as well as citations to several academic journal articles. The journal articles were not openly accessible online, but TU Delft students would presumably have access through a library.
The lack of openness in support costs is somewhat surprising, and the reasons for it may be pedagogical. Many of the courses analyzed did not prioritize open readings. Some variability (two of the 10 courses) arose from situations where a larger variety of educational resources were included with varying degrees of openness. The lack of open readings in OCW highlights an important pedagogical tension in OCW design.

Assessment
Open content for self-directed learning is enhanced when learners can assess their progress. In traditional learning environments, whether face-to-face or online, assessment mechanisms, such as written assignments and tests/exams, are the sine que non for students to demonstrate their understanding of material. Closed represented courses with no assessment available. Mixed represented courses where assessment mechanisms were available but the learner could not effectively self-assess their performance (e.g., short answer, a long essay, or quantitative assessments with no answers provided). Most open represented courses where learners could meaningfully self-assess their understanding of the material (e.g., true and false or multiple choice with answer keys/solutions provided); however, it is important to note that maximizing the openness of assessment does not necessarily produce the most pedagogically appropriate avenues for demonstrating learning.
Assessment was categorized as mixed for six of the 10 courses. Generally, course assignments were made available in the form of qualitative assessment mechanisms (e.g., essays), but self-directed learners, if they elected to undertake an assignment, would have no way of assessing their work. For example, the final assignment in MIT's "Introduction to Art History" course was to write an essay on one of two topics [3]. While most courses that fell under the mixed category relied on essay/paper type forms of assessment, TU Delft's "Structured Electronic Design" was a notable departure because it assigned quantitative exercises but did not provide solutions. There was some variation in the amount of instruction/guidance provided to students for essay assignments. For example, MIT's "Equity and Inclusion" course provided six broad bullet points of guidance for writing papers; this included word count range (500-600 words), leading with a big idea, laying the ground and providing context, making points well, explaining why readers should care about the paper, and providing specific recommendations. In contrast, the MIT "Innovation Systems" course provides an extensive multi-page description of how to write a successful final paper; it included a detailed description of the assignment and topics for students to choose from, in-depth guidelines on paper structure and organization, and additional directions for each paper topic.
Four of the 10 courses were categorized as most open and one was categorized as closed. TU Delft's "System Validation" and "Drinking Water Treatment 2" courses and MIT's "Biological Chemistry II" course had assessments that were categorized as most open. For "Biological Chemistry II", quantitative questions and solutions were provided. Disciplinary differences explain part of this variance as courses in natural sciences and engineering tended toward more quantitative assessment. The two TU Delft courses were notably different in their approach to openness and assessment. Rather than providing quantitative questions with solutions, the assessments were more qualitative; however, examples of student work were provided. In these cases, self-directed learners could judge themselves against student work provided, although this approach was more subjective than having objective answers. TU Delft's "Public Hygiene and Epidemiology" course was categorized as closed because it was the only course that did not include any assessment materials. Beyond a breakdown of how assignments were weighted in the final grade, selfdirected learners would have no means of attempting to assess their learning.

Digital distribution
The digital distribution factor categorizes courses based on their discoverability. Closed courses are completely inaccessible to the public; this would include face-to-face and online courses locked behind learning management systems such as Blackboard or Moodle. Mixed courses are open to the world, but their discoverability is low; for example, OCW could be hosted through a university institutional repository that is not linked to a federated search engine. The most open courses are those with high discoverability. These open courses should be available through one or more of the major OER federated searches, such as Merlot, OER Commons, OASIS, edEX, George Mason University's Metafinder, etc.
All 10 of the courses analyzed were easily discoverable through at least one of the aforementioned search engines. This was a surprise since discoverability is so commonly cited as a barrier to OCW adoption by educators (Cortinovis, et al., 2019). Many platforms and institutions advertised their courses as being available through multiple OER databases. Open Yale advertises their courses as being available through Coursera, and they have their own Coursera home page. TU Delft advertises that its MOOCs are available through edEX and that it is a member of the Open Education Consortium. MIT's Open Learning portal links users to all its various open courses and displays which are available through edEX.

File format
Like licensing, file format is a critical factor that determines the practical openness of any given OCW. Without the ability to revise and remix content, OCW loses all utility assuming you are measuring utility using a rights-centric (or technical) framework such as Wiley's 5Rs (Wiley, n.d.). Closed courses include non-editable materials provided in proprietary file formats. Ideally, OCW would at least fall under mixed, meaning that documents and materials would be editable even if only offered in a widely used proprietary format (such as Microsoft Word/.docx). The most open courses would feature fully editable materials in an open source file type.
All of the courses analyzed were categorized as closed. Text documents (notes, readings, slides, and bibliographies) were almost exclusively provided in non-editable PDF format. While choosing PDF over Microsoft Word (.docx) or Open Document Format (.odf) (either of which would likely have been the formats these materials were created in) does not necessarily undermine the learner's experience, it cripples the course's appeal from the perspective of educators who wish to adapt the course materials. The decision to offer course content in one format was also puzzling because uploading additional file-formats requires minimal effort or skill (McNally and Christiansen, 2019). It is possible that the lack of editable file formats is a result of institutional policy. Unlike many factors that are closed by default (e.g., All Rights Reserved for copyright), closing OCW documents requires an active choice to convert materials into non-editable formats. For example, MIT's "Public Transportation" course has a thorough set of lecture notes, but the use of PDF-only documents makes these resources difficult to revise and adapt.
Video and audio files continue to be a sticky point in file format openness. All multimedia materials were provided to users in MP3 and MP4 format. These are open file types but are less straightforward to edit without expertise using either proprietary or open editing software. What is a more or less editable video/audio file continues to be an area of contention when assessing openness.

Cultural considerations
The cultural consideration factor was arguably the most difficult to address. Of the 10 courses, we categorized five as most open, two as mixed, and three as closed. However, while it was easy to assess how many languages are offered, cultural considerations can be subjective. For example, TU Delft's "Public Hygiene" course (categorized as mixed) made note that Biblical classifications of life on Earth (plants, trees, animals, and finally humans) had influenced early approaches to classification of life. It was unclear whether this constituted some sort of cultural knowledge that would create a barrier to learners and thus whether it should be categorized as closed, mixed, or even most open. Similarly, MIT's "Public Transportation" course (categorized as mixed) made reference to transportation systems in Boston (the metropolis closest to Cambridge), but also other cities such as Vancouver. Certainly, these Western references did not count as "devoid of culturally specific material"; In the case of MIT's "Art History" course (categorized as closed), the course description noted it specifically focused on Euro-American traditions raising the question of whether a course focusing on a specific cultural area is, by definition, closed.
A fundamental problem with the cultural considerations factor was that, upon analysis, it appeared impossible to have a course dealing with a cultural element that at the same time was devoid of culturally specific material. Some of the courses, such as those offered by TU Delft, could be considered most open under the framework because the material is technical and largely devoid of jargon and cultural references. The same could be said for the "Biological Chemistry II", and "Innovation Systems for Science" courses from MIT.

Study limitations
The exploratory nature of the pilot study is not without limitations. Only two institutional OCW catalogues were examined (though several others were purposively eliminated), and only 10 courses in total were analyzed. While the sample is quite small, the intent was not to generalize OCW but examine and improve the framework. Furthermore, the selection of courses, while random, still had a greater number of courses from the natural sciences and engineering than from humanities and social sciences or professional fields such as law and medicine.
In some cases, factors were determined by institutional policy rather than by individual course creators. For both the digital distribution and copyright/open licensing factors, the scores were non-varied, as a result directly from sampling bias. Each institution employed the same Creative Commons license to all OCW. For digital distribution, all courses were determined to be "most open" because both TU Delft and MIT make their courses easily discoverable through several OER portals. A broader set of licences would have made for a more interesting and nuanced analysis. If multiple licences were present within an open course, which category would you associate with the course? In such an event, how would you rank OCW on a continuum of closed, mixed, and most open? This illustrates that the categories could be further divided. A half-way point between "Mixed" and "Most Open" could help remedy this problem. However, if openness is measured on a continuum, the categories could be infinite. It is possible that this limitation could be remedied by "grading" each discrete resource within an open course numerically, though doing so would not necessarily remove educator bias. An inherent limitation of rubrics and evaluation tools is their precision; a number that represents a course's "Open Score" is satisfying, but flawed.

Discussion
This analysis yielded insights into how to conceptualize and practically evaluate openness as well as how openness/closedness affects educators who might wish to adapt such resources. Analyzing OCW is challenging due to the volume and variety of material, hence why this study was limited to 10 OCW cases. Across the sampled OCW, we noticed a certain lack of pragmatic implementation. To make OCW more practical for educator reuse, OCW should be developed with technical and pedagogical considerations in mind. Educators and platform directors need to ask "how will another educator use these materials?" Pedagogical elements can easily be overlooked, but they represent a dimension that -if not adequately addressed -can undermine the educational impact of OCW.

Critically evaluating openness: What does it tell us about OCW?
This analysis underscores that the degree of openness of content must be considered by creators before design (Hilton, et al., 2010). While the ALMS framework (Hilton, et al., 2010) identified four technological considerations in this regard, the Open Enough framework further informs creators on how to consider openness, particularly in terms of the workload associated with the development of OER/OCW. As with Gurell (2012), empirically testing frameworks for evaluating openness is a necessary step in their development and refinement. While Gurell aimed at identifying technological barriers to reuse, the Open Enough framework can help creators pragmatically increase openness during the creation process and better anticipate where to devote their time. This approach should benefit downstream reuse. If creators of OER/OCW are aware of the factors that require the least effort to open at the outset, they can more easily maximize openness in these areas without meaningfully increasing the amount of time spent on creating open content. Overall, it would appear that the variability in outcomes between factors is driven primarily by subject matter. There was a higher variability among openness among the cultural considerations and assessment factors than support costs and file format, for example. Subject matter drives the cultural considerations and assessment differences. More importantly, it would suggest that from an institutional or funder perspective, it is more pragmatic to emphasize openness in STEM where the opportunity costs of increasing openness are lowest. The examination of assessment approaches across the 10 courses raises some considerations when developing OCW. With nine of the 10 courses coded as mixed or most open, assessment appears to be an element of openness that is generally easy to achieve or has been more strictly mandated by MIT and TU Delft. Availability of assessments is unsurprising given that several quality frameworks identify it as an aspect of quality (Moise, et al., 2014;Elias, et al., 2020;Achieve, 2011). It was notable that two of the three most open courses achieved this not through the inclusion of both quantitative questions and answers but through posting sample student work. While this approach increases openness, it could have been further enhanced by providing marked copies of assignments with instructor feedback (particularly any comments that would have identified shortcomings in the assignment). In general, assessment tended toward a greater degree of openness than some of the other factors; this also reflects the uneven number of natural science and engineering courses included in this analysis. While courses in these fields more easily lend themselves to meaningful open assessment, the inclusion of more qualitative open assessments was notable in that it expanded how meaningful assessments may be designed.
The lack of editable file formats across the sampled courses is such an oversight that it begs one to ask if the OCW are more suitable for student consumption than educator reuse. This problem is particularly notable given the heavy recognition in the literature of the limitations of closed file formats (Hilton, et al., 2010;Ovadia, 2019;DeVries, 2013;OECD, 2007). Mandated openness could be to blame. Individual educators who choose to make their courses available through a repository like OER Commons or Merlot might be less likely to pursue openness for reasons of personal reputation and, therefore, upload their course materials in editable formats. From a technical perspective, implementing editable file formats should take first priority for OCW repositories; without it, these courses are not functionally open.
Regardless of how open a course's content is, digital distribution is critical to OCW reuse. A lack of discoverability and awareness of OER databases among faculty has been a consistent barrier in their respective institutions (Cortinovis, et al., 2019). If learners or educators cannot locate OCW, or if courses are buried within institutional Web sites, it is unlikely they will be discovered. Content reuse and digital distribution are inherently linked. It is likely that educators who make the effort to open up their courses do so with the hope that their work will be adopted or adapted. While the discoverability of the analyzed courses is expected given the status of their respective institutions, it is encouraging to see high discoverability of OCW. This bodes well for OCW and speaks to the advancements in, and the success of, federated searches that link various repositories and university materials together.

Refining the Open Enough framework
Overall, the Open Enough framework provides a reasonable means for assessing openness, though additional refinements to the framework are necessary. For reasons of clarity, the "Digital Distribution" factor should be renamed "Discoverability," and "Support Costs" should be renamed "Materials." The "Cultural Considerations" and "Usability" factors should be removed due to their level of subjectivity. In general, science courses involved less culturally specific references; by contrast, the MIT "Art History" course was very culturally specific by nature. While we are confident in these broad observations, removing "Cultural Considerations" would reduce the level of subjectivity. We recommend that "Usability" be removed from the "Accessibility/Usability" factor in favour of just "Accessibility" since usability, like cultural considerations, is somewhat subjective and there exist many metrics for assessing the usability of Web content. Accessibility standards are easier to evaluate objectively as they do not take into account subjective design elements.
The Open Enough framework should not, however, simply be about saving creators' time for OER/OCW construction or attempting to enable greater reuse. It is designed to encourage educators to consider openness more broadly and the consequences of maximizing openness. It should push educators and learners to think about how we learn in addition to what we learn and consider the context of educational resources (Knox, 2013). In this regard, removing "Cultural considerations" and "Usability" as openness factors, but maintaining these as additional considerations is advised. "Localization," which includes both translation into local languages and adjusting for cultural contexts, still presents significant barriers and time intensive undertakings for realizing the supposed openness of OER (Amiel, 2013;Wolfenden and Adinolfi, 2019). Creators of open content must consider not only openly licensing materials produced for a primary audience, but also for a broader, secondary audience of potential adopters and learners (DeVries, 2013). Richter and McPherson (2012) have previously suggested that a contextual description be linked to OER, which would provide reports from educators on successes and failures; while such reports may be time-intensive to create, and for adopters to review, there is a need for supplementary documents to accompany OER/OCW to facilitate reuse. Figure 4 illustrates the revised Open Enough framework.

Missing factors for assessing openness: Content volume and "harvestability"
There is room for other factors when evaluating OCW openness. There is an argument for considering the ease by which users can download an entire open course. This "harvestability" is an advantage that MIT has over TU Delft that would make this analysis lopsided. The ability to download a ZIP file of course content for off-line use is potentially useful for users with limited Internet bandwidth -particularly for educators in rural areas or the nearly three billion people who have never used the Internet (ITU, 2021). Even MIT's implementation is limited, as ZIP files do not contain course video lectures; those need to be downloaded separately. While harvestability is measurable, its nature is binary -either a course is harvestable or not. As such, harvestability is noted as the third "other consideration" and not raised to the level of a full factor (Figure 4).
The volume of content in an open course also introduces important considerations. While it is not a factor of openness, volume is tied to openness; a course with a considerable volume of content is harder to make open. Volume is also tied to quality and the ability for users to pursue self-directed learning. MIT's "Art History" course is somewhat open, but the lack of content effectively renders the course unusable. Of the 10 courses studied, MIT's "Innovation" course was the most educationally compelling due to the depth and quality of content; however, it scored low in openness. This suggests Wiley's (2001) nearly two-decade-old reusability paradox still holds.

Two domains of openness
While this study certainly does not aim to generalize the analysis of this small subset of OCW, it highlights an important delineation between open educational philosophies. The rights-centric approach, as described by Wiley's 5 Rs (Wiley, n.d.) appears to exist in stark contrast to the (often) invisible pedagogical considerations of open education. The rights-centric approach -or "Technical Factors," as outlined by the revised framework (Figure 4) -has dominated much of the early discussion in open education literature. This is not surprising. During the early years of the open education movement, it could be argued that getting educators to "go public" with their work took priority. Sharing materials on open platforms, in whatever form, was prioritized over not sharing. Getting educators comfortable with Internet technologies between 2001 to present, as well as open licensing models, was likely necessary for the open education movement to expand. However, as educators have become more skilled with such technologies, focus has shifted toward pedagogical factors that give OER and OCW context and broader utility, which are also conceptually more complex and contested (Tietjen and Asino, 2021). In this respect, the divide between technical and pedagogical factors comes down to educator challenges. Addressing technical factors requires that educators be willing to share their work publicly and possess the skill/knowledge to do so. When considering OER and OCW holistically, pedagogical factors are likely to require the majority of the development time, in order for educators to develop the necessary ancillary materials and provide guidance to those who wish to adopt or adapt their resources (Figure 4). Open education needs to continue to place an emphasis on these pedagogical considerations.
Technical and pedagogical factors are reconciled in different ways. It is puzzling that OCW do not necessarily achieve maximum openness among technical factors given that these factors are often lowhanging fruit. Any of the courses analyzed could have been made considerably more open by including a less restrictive licence. Files could be available in multiple editable formats. Given the files were likely created in programs such as Microsoft Word or Open Office, the choice of PDF format represents an active choice of closure. Addressing the pedagogical factors is not as straightforward. Aside from being bilingual, how do you meaningfully address language? How do you address cultural considerations? A glossary of all technical terms would be welcomed by any educator hoping to adopt an existing course. Instructions for educators such as colour coding or ranking course modules on their broader applicability, or highlighting where cultural or geographic examples could be swapped out would all go toward making the course more "open." Assessment, though generally well addressed in the sampled courses, could include more example student work with instructor comments. Greater accessibility can be achieved, but it requires knowledge of standards such as image alt tags, properly formatted text for screen readers, transcripts for audio, closed captions for video material, and some understanding of usability and design. In this respect, accessibility requires pedagogical knowledge of course structure.
Finally, since the period of data collection for the study, MIT has introduced a new set of pedagogical resources to support their OCW -"OCW Educator: Instructor Insights" (MIT, n.d.-b). These pages, which complement the curricular materials for open courses, provide instructor perspectives on a range of pedagogical issues such as course design, assessment practices, and even how student time is spent in the in-person offering of the class. These pages allow other open educators to move from the tip of the iceberg to view what lies beneath the surface in the development and delivery of various courses, and thus represent an important improvement, holistically addressing open pedagogy and OCW.

Conclusion
There are several unanswered questions that would be well suited for follow up. Future research could take the Open Enough OCW framework and apply it to MOOCs. Alternatively, comparing a sample of MOOCS and OCW would reveal what, if any, differences exist in the level of openness and design. Also, this study could easily be expanded to include a larger sample of OCW from a more varied subset of institutions. This would provide the open education community with a valuable snapshot of the state of openness in OCW.
We came to two conclusions based on the research questions posed in this study. The level of openness across the sampled OCW was inconsistent, at least across the eight factors outlined in the framework. Courses were open in terms of assessment and digital distribution but closed in terms of language, support costs, and file format. Copyright, accessibility, and cultural consideration were more mixed in terms of openness. There was a lack of general editability that made these OCW more suitable for student consumption than educator reuse. Ultimately, the results highlight the importance of considering educator needs in OCW. A rights-centric approach to openness (i.e., open licensing), while an essential component of openness, is insufficient when standing alone.
1. What was the promise of the new technologies of photography and stereoscopy, and what were some of their problems, according to nineteenth-century commentators Charles Baudelaire and Oliver Wendell Holmes? What, in their view, is the relationship between these new media and art, and what impact do they predict these new technologies will have on art and its audiences? 2. How did women Impressionist artists negotiate the condition of modernity and its spaces of femininity in their art? To answer these questions, students write a critical analysis of Griselda Pollock's account of modernity and the spaces of femininity, and test her arguments by applying them to selected works of art.