A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness

Almost all open educational resources (OER) definitions encompass key concepts such as the 5R activities and open licenses. However, little attention is given to the technical aspects and tools that allow the user to interact with these resources. This study aims to answer five research questions regarding (a) 5R activities, (b) open licenses and intellectual property, (c) technical aspects, (d) tools for developing OER, and (e) the topic of sustainability. To answer these questions, a systematic review of systematic reviews on OER was conducted following the reporting checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). Sixteen studies were eligible and included in this review. The main findings suggest that although most studies did not mention the term 5R exactly, they mentioned related terms, such as share and adaptation . There was also a tendency toward focusing on more legal issues than technical aspects. Besides, most of the studies that mentioned tools discussed them as platforms to access OER, not exactly tools that encourage users to develop or adapt resources in an easy way. In relation to sustainability, several studies highlighted the relevance of developing sustainable OER models, but only a few suggested approaches to sustain an OER project. Therefore, with this article, we hope to raise awareness of the importance of the technical openness and tools that might contribute to fostering users’ engagement with the OER, helping them to act as producers and contributors rather than mere passive receivers.


Introduction
With the advent of the Internet and information technologies, a vast number of digital resources have been created and made available at little or no cost.However, not all these resources available on the Internet are open and can be reused, modified, and re-shared.Most of them are released without clear license terms and are, automatically, protected by copyright, not being allowed to be copied without the author's permission.On the other hand, resources that are made available under open licenses or in the public domain and enable legal use, adaptation, and redistribution are called open educational resources (OER).
In 2002, when the term OER was first coined by UNESCO in the Forum on the Impact of Open Courseware for Higher Education in Developing Countries, the group defined OER as "the open provision of educational resources, enabled by information and communication technologies, for consultation, use, and adaptation by a community of users for non-commercial purposes" (UNESCO, 2002, p. 24), emphasizing that OER is "a universal educational resource available for the whole of humanity" (UNESCO, 2002, p. 28).
Similarly, Wiley (2014) claimed that content is open not only when it is available to be used in other contexts, but also when it gives everyone permission to reuse it in different ways, known as the 5R, which means retain, reuse, revise, remix, and redistribute.Retaining is the right to make, own, and control copies of the content, such as downloading, duplicating, storing, or managing a resource.
Reusing is the right to use the content in a wide range of ways, in a class, in a study group, on a website, in a video, and so forth.Revising is the right to adapt, adjust, modify, or alter the content itself as, for example, translating it into another language.Remixing is the right to combine the original or revised content with other material to create something new.Finally, redistributing is the right to share copies of the original content, its revisions, or its remixes with others.
In 2019, UNESCO updated this definition, saying that OER is "learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been released under an open license, that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others" (p.5).Creative Commons (n.d., 2020) defined OER as "teaching, learning, and research materials that are either in the public domain or licensed in a manner that provides everyone with free and perpetual permission to engage in the 5R activities."Most of the OER definitions highlight the 5R and the open licenses in their concept, but only a few consider the technical infrastructure and tools necessary for the creation, revising, and sharing of content.
If people are given legal permissions with open licenses to interact with OER through the 5R activities, they should also be given technical tools to unlock the material so that they can revise and remix it according to their needs (Wiley, 2014).The ALMS analysis is a framework developed by Wiley (2014) and Hilton et al. (2010) that puts emphasis on the OER's technical aspects.ALMS is an acronym for access to editing tools, level of expertise required to revise or remix, meaningfully editable, and sourcefile access.
Access to editing tools is known as access to software that enables users to edit the resource, not only open and visualize it.There is also the aspect related to the level of expertise required to revise or remix and, thus, the tools to develop the OER must be simple and easy to use.The OER must also be meaningfully editable, meaning that it must be shared in such a format that enables anyone to edit it.
A source file must be accessible which means the file that the web developer edits and works with, for

20
example, is the same one that the web browser displays and the user interacts with (e.g., an HTML file).
Consequently, modifying it must be uncomplicated (Hilton et al., 2010;Wiley, 2014).Applying open licenses which allow users to engage with materials in the 5R activities as well as applying the ALMS analysis framework enable OER creators to maximize the openness of the materials they produce (Hilton et al., 2010).Hylén et al. (2007) defined OER as "digitized materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research" (p.10).This definition includes not only the implementation resources, responsible for the intellectual property licenses which promote the openness of materials, and the content itself, such as the courses, modules or learning objects, and tools, but also the software used to support the development, usage, and sharing of content.
Although this definition encompasses the tools necessary for the creation and distribution of OER, the William and Flora Hewlett Foundation (2018) has recognized that there is a need for a better technical infrastructure to support OER.They have highlighted recurring topics recommended by the OER community as, for instance, the desire for better OER authoring tools, tools to manage the revision and adaptation of OER, and tools to better handle the import and export of OER across different file formats and platforms (Levin, 2017).
The culture of openness must extend far beyond the simple replacement of a textbook with an open textbook and the expansion of access to free or more affordable learning materials.Although cost savings are considered a potential benefit, OER adoption needs to consider other technical aspects and tools that involve students in the 5R activities and, thus, foster student engagement with the OER, helping them to act as producers and contributors rather than passive receivers in their learning process.According to Axe et al. (2020), in contexts where students create resources collaboratively, the platforms used remained traditional or are not mentioned at all.The technical issues were also considered one of the biggest barriers when adopting OER in an educational context by Dichev et al.

Method
This systematic review was conducted according to the reporting checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Liberati et al., 2009) which consists of four phases: identification, screening, eligibility, and inclusion.In the first phase (identification), we began the process of identification of eligible studies by searching pre-selected terms in the Scopus database.
The choice of Scopus lies in the fact that it is one of the largest abstract and citation databases of peerreviewed literature in this field.
The search was conducted based on the title, abstract, and keywords of articles, using the following search terms: open educational resource, open educational resources, OER, OERs, review, and reviews.The reason for searching "review" and not "systematic review" was to avoid missing systematic literature reviews that did not use the term "systematic" in their titles, abstracts, or keywords and, therefore, expanding the results.The data collection was conducted on November 5, 2021, and this round of search yielded 1,023 results.
To maintain the quality and feasibility of the present analysis, a filtering process was performed, using the year of publication (2012-2021), language (English), and publication state (final), and 784 articles were retained.As many of the titles referred to OER as oxygen evolution reaction and not open educational resources, the keywords were filtered, limiting them to those related to education, learning, and teaching, resulting in the selection of 343 articles.The title, abstract, keywords, year, and authors of the identified records were exported to a Microsoft Excel spreadsheet to be screened by an individual reviewer.
In the second phase (screening), the titles of all articles were carefully screened to identify those still related to chemistry and oxygen evolution reaction, and 179 articles were excluded (out of 343).Then, 164 abstracts were read to determine their eligibility, and 56 articles that referred somehow to open educational resources and systematic reviews were selected.
In the third phase (eligibility), 56 full-text articles were assessed to check if they were systematic reviews on open educational resources.Five articles were not available online, one was not written in English, and 19 were excluded because they were not systematic reviews.
In the fourth phase (inclusion), a total of 31 articles had their full text read, and 15 were excluded due Finally, 16 studies were included in this systematic review.The data collection procedures have been summarized in Figure 1.(see Table 1).These items provided information to answer the research questions and to conduct the synthesis that is presented in the next section.
As shown in Figure 2  The systematic reviews that were selected used 71 keywords in total.Figure 3 presents these keywords, and the bigger the font size, the more frequently they appeared in the texts.The most frequently used keyword was undoubtedly open educational resources, which was mentioned 14 times, followed by OER, which appeared seven times.Open educational practices, systematic review, higher education, textbooks, disability, and accessibility appeared twice.The others appeared only once each.Other Open Terms Cited in the Titles, Abstracts, and Keywords Note.Terms used most often appear in larger font size.

How are the 5R activities addressed in the systematic reviews on OER?
Only four texts mentioned Wiley's 5R principles, as displayed in Figure 5. Three texts mentioned it before the presentation of the results, that is, in the introduction and/or theoretical background (Craig, 2020;Moon & Park, 2021;Tlili et al., 2019), and two texts mentioned the 5R in the results (Craig, 2020; Wong & Li, 2019).From these three and two texts that mention the 5R before and in the results, respectively, only one text cited the 5R both before and in the results (Craig, 2020).
A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness Sousa, Pedro, and Santos

26
Moon and Park (2021), for example, said that while most studies have emphasised the 5R, they hardly ever debate approaches to enhance learners' engagement and interactions with the resource.They also said that OER-enabled pedagogy expands learners' interactions with the OER beyond the simple use for information retrieval.Through this pedagogy, learners can create, modify, and share these resources.
In addition to this, Craig (2020) reported that learners could benefit more from the 5R if they had at least a basic understanding of editing tools.The access to editing tools and the level of expertise required to revise or remix materials are two aspects discussed in the ALMS framework (Hilton et al., 2010;Wiley, 2014) and essential to the technical openness requirements.
Only two of the 16 studies selected for this review (Craig, 2020;Yuan & Recker, 2015) did not mention the licenses in their texts, revealing the high importance given to the legal aspects of OER, one of the key elements in the OER definition.

Figure 6
Studies Mentioning the Licenses Note.n = 16.
A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness Sousa, Pedro, and Santos

27
One of the texts that did not mention the licenses is about the implementation of OER in Canadian higher education during the pandemic (Craig, 2020).Although it talks about Wiley's 5R principles, it does not cite any aspect regarding intellectual property and technical issues related to the reuse, sharing, and adaptation of resources.The other text that did not comment on licenses is about the use of rubrics to evaluate the quality of OER (Yuan & Recker, 2015).It used related terms to talk about the 5R activities and did not mention technical aspects (discussed in the next subsection).

How often are the technical aspects discussed?
Only three texts addressed concerns about the technical aspects of OER as, for example, tools that support the development and reuse of learning content, as visualized in Figure 7. Three mentioned the technical aspects in the introduction and/or background theory (Arimoto & Barbosa, 2012;Ivanova et al., 2020;Wong & Li, 2019) and only one mentioned the technical aspects in the results (Arimoto & Barbosa, 2012), being also the only one that mentioned them before and in the results.

Figure 7
Studies Mentioning the Technical Aspects Note.n = 16.
When comparing these numbers from the technical aspects to the numbers from the legal openness discussed in the previous subsection, it can be noted that they are almost opposites.While 14 articles mentioned the legal openness and two did not mention it, 13 articles did not discuss the technical aspects and only three did.
These results supported few studies that have highlighted the importance of technical infrastructure and knowledge that enables technical openness in OER.The William and Flora Hewlett Foundation, for example, has recognized the need for "tools for authoring and automated feedback, tools for metadata management, analytical tools, [and] tools for import/export in different platforms and formats" (Ivanova et al., 2020, p. 67).Arimoto and Barbosa (2012, p. 6) argued that, "The integration of social tools encourages the active participation of developers and users in the construction of OERs, also important in distributed and collaborative development of such resources."These tools can also facilitate the development of OER, besides contributing to the quality of the final resource.

Which open tools are pointed out in the reviews?
Six of the 16 texts pointed out open tools through their texts (Arimoto & Barbosa, 2012;Hilton, 2016;King et al., 2018;Moon & Park, 2021;Tlili et al., 2019;Wong & Li, 2019), as can be seen in Figure 8.According to them, the William and Flora Hewlett Foundation has already recognized the need for a better infrastructure to support these resources.
As was mentioned in the previous subsection, Arimoto and Barbosa (2012) discussed the integration of social tools in the OER to encourage the active participation of users in the collaborative construction of these resources.Arimoto and Barbosa (2012) pointed out that the use of these tools tends to make the development of resources easy, contributing to the quality of the final resource.

How relevant is the topic of sustainability in these systematic reviews?
Only seven texts out of 16 (Craig, 2020;Luo et al., 2020;Meng et al., 2020;Paragarino et al., 2018;Tlili et al., 2019;Wang et al., 2017;Wong & Li, 2019) mentioned the sustainability issue regarding the OER (Figure 9).Sustainability refers to the ability of an OER to continue or be continued for the long term.
According to Wang et al. (2017, p. 303), "Sustainability is a core issue and major challenge faced by not only end-users but also OER developers, foundations and policymakers."UNESCO released a recommendation on OER, and one of the areas of action is "nurturing the creation of sustainability models for OER" (Tlili et al., 2019, p. 11).Wong and Li (2019) also defended the position that developing policies to support OER is essential for the sustainability of their practices.
In Meng et al. (2020), the topic of sustainability is highlighted in the keyword analysis, highly cited publications, and OER practice.Similarly, the study also showed that many projects could not survive beyond their initial funding and argued that self-finance models should be explored to maintain OER projects.Meng et al. (2020) suggested two approaches to sustain an OER project: to advertise or offer other services to obtain extra income, and to extend the traditional mode of donation.
However, although sustainability is frequently considered from an economic perspective, it is not restricted to financial issues.According to Downes (2007), sustainable OER models can be categorized into four aspects: funding, technical, content, and staffing (Wang et al., 2017).Luo et al. (2020) found that sustainability is the most frequently cited barrier that stands in the way of OER, and suggested that partnerships among designers, e-learning staff, academic librarians, and teachers, for example, can contribute to the sustainability of OER.

Discussion and Conclusions
The majority of OER definitions are centered on the 5R activities and legal aspects, such as the open licenses, neglecting the technical aspects and tools that are necessary to fulfill the objectives of the OER movement, giving anyone not only legal permission but also technical support to reuse, adapt, and share materials.
This study performed a systematic review of systematics reviews on OER with the aim of analysing if these reviews mentioned and how often they mentioned the 5R activities, the licenses or intellectual The main findings revealed that all the texts mentioned the 5R or related terms, such as access, share, copy, and adaptation, and almost all the selected texts discussed the legal issues (n = 14), corroborating the definitions presented by UNESCO (2002, 2019), Creative Commons (n.d, 2020), and Hylén et al. (2007), which say that OER is any resource used in the teaching, learning, or research context, that resides in the public domain or is under an open license, giving anyone permission to engage with these materials through the 5R activities.
On the other hand, 13 texts out of 16 did not mention the technical aspects necessary to create, reuse, adapt, and share OER.Besides, the studies that talked about tools did not point out tools that facilitate the creation and adaptation of resources.Most of them were platforms used to access OER.As discussed by the William and Flora Hewlett Foundation (2018) and Levin (2017), there is a lack of technical infrastructure, such as tools, to support the revision and adaptation of OER.This technical challenge was also mentioned by Dichev et al. (2011), who cited that there is a need for environments that facilitate the sharing of content as well as tools for OER modification and redistribution.
There are some limitations to this systematic review.The first one is related to the database used, Scopus.Besides not covering all the publications, we did not search the articles on other databases to complement our research.Furthermore, research outcomes are published in several languages and this systematic review encompassed only those written in English.Because of this, we might have missed some relevant articles due to the database restriction and data collection process.
To conclude, having access to tools and technical knowledge that enable users to engage with resources through the 5R activities and knowing the legal issues that permit users to reuse or adapt these resources should have the same level of importance when talking about OER.If users are given legal permission to engage with OER through the 5R activities, they should also be given technical tools to unlock these resources so that they can interact with them as producers and contributors, and not only as passive consumers, maximizing the openness of content.
Therefore, with this study we hope to raise awareness of the importance of the technical openness and tools that might contribute to fostering users' engagement with OER, helping them to reuse, remix, and redistribute these resources according to their needs.

(
2011).They cited that availability and open access are important factors, but there is a need for supportive environments that foster discoverability and sharing of content, associated with tools for adaptation and redistribution.Therefore, considering the OER definitions, which have focused more on legal openness and open licenses, neglecting the technical aspects, such as the tools and skills necessary to revise, remix, and redistribute the resource, this systematic review of OER systematic reviews aims at analyzing how the 5R activities and the open and technical aspects are addressed in the systematic reviews and identifying tools and practical examples that are beyond the simple use of OER.The research questions that guide this systematic review are: 1.How are the 5R activities addressed in the systematic reviews on OER? 2. How often are the licenses or intellectual property issues mentioned?3. How often are the technical aspects discussed?4. Which open tools are pointed out in the reviews? 5. How relevant is the topic of sustainability in these systematic reviews?A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness Sousa, Pedro, and Santos 21 to not being systematic reviews on open educational resources.As open educational resource is a broad term and is related to other open terms, such as open educational practice, open pedagogy, and open education, many cited OER in the titles and abstracts but did not refer to OER exactly in their text.

Figure 1 PRISMA
Figure 1 , 11 studies out of 16 were published in recent years: 2018 (n = 3), 2019 (n = 3), and 2020 (n = 5), which can indicate an increasing interest in systematic reviews on open educational resources, maybe due to the coronavirus pandemic and the transition to online learning.However, this tendency was inconspicuous in 2021 (n = 1).This may be explained by the fact that the data collection was conducted in November 2021, and there were still some articles to be published that year.

Figure 2 Time
Figure 2

Figure 3 Keywords
Figure 3

A
Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness Sousa, Pedro, and Santos 28 Some open tools mentioned are Moodle, edX, Sina Weibo, XuetangX, CNMOOC, iCourse163, P2PU, Canvas Network, Coursera, EdX, Khan Academy, OpenCourseWare (OCW), Openstax, The Saylor Foundation, Washington State's Open CourseLibrary, and The Minnesota Open Textbook Library.

Figure 8 Studies
Figure 8

Figure 9 Studies
Figure 9

Table 1
Papers Selected for This Review

A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness Sousa, Pedro, and Santos 30 property
issues, technical aspects, open tools, and the topic of sustainability, and comparing if these studies also follow this tendency of emphasizing legal aspects over the technical ones.