Opening Teaching Landscapes: The Importance of Quality Assurance in the
 Delivery of Open Educational Resources

Scholars are increasingly being asked to share teaching materials, publish in open access journals, network in social media, and reuse open educational resources (OER). The theoretical benefits of Open Educational Practices (OEP) have become understood in the academic community but thus far, the use of OER has not been rapidly adopted. We aim to understand the challenges academics face with in attempting to adopt OEP, and identify whether these are related to or stem from the functionalities afforded by current repositories of OER (ROER). By understanding what academics and experts consider good practices, we can develop guidelines for quality in the development of ROER. In this article we present the findings from a study surveying academics using OER and experts who develop and/or work with ROER. We conclude by suggesting a framework to enhance the development and quality of ROER.


Introduction
The learning and teaching landscape is changing dramatically. Academics are requiring new literacies in order to develop digital pedagogies and, more recently, to engage with open educational practices (OEP). Academics today are expected to engage not only with the traditional classroom technologies but also with a kaleidoscope of interconnected digital, open and social practices.
Open Educational Practices (OEP) are defined by the International Council for Open and Distance Education (ICDE) as "practices which support the production, use and reuse of high quality Open Educational Resources (OER) through institutional policies, which promote innovative pedagogical models, and respect and empower learners as co-producers on their lifelong learning path" (n.d.). For Ehlers & Conole (2010) "quality and innovation are inherent characteristics of open educational practices, as education changes to be a social practice, reflective and participative, where learners generate content and validate them in peer-interaction and teachers are facilitating rather than directing learning processes" (p. 9). The critical function of the OER movement is to stimulate academics to share information and knowledge within educational communities, as this supports learning and contributes to bridging demographic, educational, economic and geographic barriers: "OER may ultimately be the genuine equalizer for education and for empowering social inclusion in a pluralistic, multicultural, and imperfect world" (Olcott, 2012, p. 2).
Open Praxis, vol. 6 issue 1, January-March 2014, pp. 29-43 According to Alevizou (2012) while the origin of the OER movement is located on the emphasis of entitlement (of access to, and adaptation of, free pedagogical material), the new wave of policy and advocacy initiatives focus on transparency enabled by the adoption of open educational practices (p. 3).
For Ferguson & Shum (2012) while OERs greatly improve the quality of material available online to learners, this wealth of resources can leave learners adrift in an ocean of information, struggling to solve ill-structured problems, with little clear idea of how to approach them, or how to recognise when they have made progress (p. 316).
Scholars are being asked to share the teaching materials they produce, to publish in open access journals, to network with others in social media, and to reuse OER (Weller, 2011). However, we have noticed that searching and retrieving OER from repositories can be a challenging task as materials are difficult to locate, retrieve and sometimes impossible to download to be adapted, translated or updated. The theoretical benefits of open practices have become understood in the academic community but thus far, the use of open content for teaching and learning has not been rapidly adopted. We aim to understand the challenges academics are presented with in attempting to adopt open practices, and identify whether these challenges are related to or stem from the functionalities afforded by current repositories of OER (henceforth referred to as ROER).
ROER have been cast in a leading role thus far in the OER movement, as they have enabled resources to be collected and "shared"-at least in the sense of being uploaded. In the words of Windle et al. (2010): To date the OER movement has mostly focused on the input or sharing aspect of this equation. A relatively large amount of funding has been made available for the creation of repositories, and the movement has had some success in encouraging individuals to share their resources. Much less is known about the reusability or reuse of the resources that have been accumulated. Who is reusing the resources? How much is being reused? What is being reused? Why are they reusing? What makes it easier or more difficult? (p. 14) For Misra (2013, p. 25), "there are literally millions of open education resources currently available on the Internet. But what differentiates them from one another? How can educators determine whether the resources are high quality?" In this study we report the views of two different samples of experts in OER and academics on the opportunities and challenges of using ROER by understanding what academics and experts consider good practices. We aim to develop guidelines for repository developers, for them to consider the users' needs to support quality assurance and to encourage academics to embrace the use open educational resources.
Our goal is to help improving the adoption of open educational practices by suggesting a model to enhance the development and quality of ROER.

Background
The concept of OER was defined by UNESCO in 2002 as: "the open provision of educational resources, enabled by information and communication technologies, for consultation, use and adaptation by a community of users for non-commercial purposes" (p. 27). The OECD (2007) also define them as "digitised materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research" (p. 10). More recently, UNESCO (2011) refers to OER as learning resources that include curricula, teaching materials, interactive, digital books, videos, multimedia, podcasts and other materials designed for educational purposes and that can be shared on a network, which is available to teachers, academics and students, and can be accessed without having to pay for subscriptions or licenses (p. 5).
OER in order to be fully open should be licensed under Creative Commons; as Jacobi & Woert (2013) mention "Creative Commons licences enable copyright holders to distribute their work on free conditions and in a low threshold manner." Creative Commons licenses are at the heart of growth for the user use of OER as those facilitate the adaptation and modification of the resources (Rolfe, 2012;Bissel, 2009). For Downes (2007) In the system implemented by Creative Commons (widely thought to be representative of an "open" license) authors may stipulate that use requires attribution, that it be non-commercial, or that the product be shared under the same license. So while "open" may on the one hand may mean "without cost," it doesn't follow that it also means "without conditions." McGreal (2004), OER focuses mostly on teaching concepts that facilitate access to open knowledge, and the importance of the creation of ROER is related to the current need to preserve and improve access to knowledge in public education. It is important that teachers understand the social responsibility of sharing teaching resources and collaborate in the construction of learning materials to be able to compete at an equal level with the industrialised model of private education, because the students of private institutions tend to have a better access to learning resources.

For
The value of ROER in the context of the new digital learning landscape is that they facilitate knowledge transfer and the collection and preservation of the information by providing spaces for academics to retrieve and share open resources, by facilitating the access to materials that can be reused, translated and adapted according to their pedagogic needs.
We have identified in the literature a series of indicators for quality assurance, which recommends that in their development, ROER include models to facilitate the search, retrieval, access and download of the OER stored in the platforms as those should allow the users to search, share, reuse and collaborate. Previous research suggests that the ROER should include a set of featured resources to promote the content to the users (Hylén, 2006;Pegler, 2012). Also, it is recommended that user evaluation tools are incorporated into repositories as this is an inclusive and collective method to ensure quality (Downes, 2007;Clements & Pawlowski, 2012;Richter & Ehlers, 2010). Is also recommended that ROER include systems for peer review, to ensure a fair process of revision of the content (Larsen & Vincent-Lancrin, 2005;Schuwer & Wilson, 2010;Windle et al. 2010).
Regarding the question of how to describe the resources, the literature recommends that all the repositories are clear about attributing the authorship of the resources (Browne et al., 2010;Kanwar et al., 2010;Petrides & Nguyen, 2008). It is also suggested that the resources are clearly described by the use of keywords (Davis et al.;2010;Richter & McPherson, 2012). The inclusion of metadata models such as Learning Object Metadata (LOM) or Dublin Core is also suggested to ensure the interoperability of the repositories (Kanwar et al., 2010;OECD, 2007;Barker & Ryan, 2003). However, the question of exactly who can or should catalogue resources remains open.
The use of ROER can create a sense of community of practice. As open platforms, they can allow educators to collaborate and interact by evaluating resources and by sharing content they know other might be looking for. In order for OER to be found, repositories need to be planned and designed to simplify access to resources and to facilitate retrieval. Therefore they need to clearly state the condition in which content licensed and the authorship of these resources, in order to provide enough information about the materials to facilitate it selection, and ensure that the original file can be downloaded to be adapted and reused (Arnold, 2011;Backall, 2007;Tuomi, 2006).
Other authors suggest that the interface of the repositories should allow navigation in multiple languages (Richter & Mcpherson, 2012;Pawlowski & Hoel, 2012;OECD, 2007). Furthermore it is suggested that tools for social media are included to facilitate the sharing of the resources (Jacobi & Van der Woert, 2012;Alevizou, 2012;Kanwar et al., 2010).
Open Praxis, vol. 6 issue 1, January-March 2014, pp. 29-43 It is suggested in the literature that ROER clearly specify the type of Creative Commons License (Hylén, 2006;Pegler, 2012) for each one of the resources to ensure the fair use of the content and to ensure that the resources can be reused or adapted by the users, (Wiley & Gurrell, 2009;Jacobi & Van der Woert, 2012;OECD, 2007;Bissell 2009). Finally, to accomplish the mission of facilitating the reuse, adaptation or translation of the content, the literature remarks the importance of allow the users to be able to download the source code or the original files (Wiley, 2007;Tuomi, 2006;Petrides & Nguyen, 2008;Andrade, Caine & Carneiro, 2011).

Methodology
The study focused on obtaining feedback from academics using OER to teach and from experts who develop and/or work with OER repositories as educational technologists and librarians. In order to be eligible to participate in the survey academics had to be employed part or full time in a higher education institution teaching face to face, remotely or in blended mode. The overriding research question for this study was in which ways OER repositories can be enhanced to increase and improve their adoption for teaching practices in Higher Education.
One sample of academics who teach in Higher Education and one sample of experts in educational technologies were surveyed through two mechanisms, respectively: • A Google online form to survey academics who teach, distributed to 350 participants • Personal interviews with 20 experts, conducted online The Google form was sent out via Twitter and other social networking sites such as LinkedIn and Facebook, obtaining responses from a pool of 350 people from around the world. Respondents who did not meet the selection criteria were excluded-the sample is composed of academics who teach in Higher Education and excludes those who work in Higher Education (HE) but do not teach or who teach in other contexts. We obtained responses from 217 academics from 35 different countries. The survey was piloted once with 20 academics from the UK, Australia, Spain, Italy, Brazil, Mexico, Canada, Chile, Croatia and Germany and was tested four times.
Our sample was composed from academics who teach at many subject fields (see figure 1) and also for academics who teach face to face, at a distance, in blended environments or at a combination of them (see figures 1 and 2).
The general objective of the academic survey was to obtain the following information: a. How many within the surveyed group were using OER repositories b. How they found them and used them c. How they search, find and select content; what challenges they face d. Their opinion on how the repository should work Academics were asked three questions: 1. How would you describe your experience in finding and selecting Open Educational Resources? 2. According to your experience, which are the advantages and benefits of using Open Educational Resources? 3. Which are the challenges and barriers you have encounter when using Open Educational Resources for your lectures?
Personal interviews were conducted online with 20 international experts in order to compare the feedback obtained from the lecturers. The interviews were designed to obtain specialised feedback Opening Teaching Landscapes Open Praxis, vol. 6 issue 1, January-March 2014, pp. 29-43 on how experts think users are employing OER and what can be done to improve the current state of repositories. The general objective of the interviews was to obtain the following information: a. How experts see the current technical development of OER repositories b. How they think users select content within the OER repository c. How they think they could be improved Apart from an initial question requesting personal and professional background information, experts were asked six questions: 1. According to your experience, should the metadata required by OER Repositories include pedagogic objectives? If so which type of pedagogical elements should be included? 2. According to your experience, what are the technical elements that can make an OER repository a successful one? 3. How do you think that educators search for teaching materials? 4. What are the main criteria that educators use to evaluate OER? 5. How do educators select the resources or the repositories they use? 6. Do you think that the current OER repositories are "good enough" or are there room for improvements and standardisation?
The sample of academics was self-selecting and the general aim was to survey academics that could be found online.
We have aimed at presenting the survey and interviews findings in qualitative, narrative form.
The sample of 20 experts in OER was selected from participants in the 2012 World Open Educational Resources Congress (Paris, 20th-22nd June, 2012) and from authors of articles reviewed for this study.

A review of the academic voices in the use of OER and ROER
Participants were asked whether they have used OER. Of the 217 academics who responded to the survey, the majority had some experience of using OER, with text-led resources proving the most commonly used, but showing that video and multimedia resources are also well used across the sample, with some use of animated resources (see figure 3).
When asked about their experiences in finding and selecting OER, respondents' opinions varied quite widely though some issues came up frequently. Generally speaking, the group was divided Figure 3: Use of OER between those who described the experiences in a broadly positive light, those who were more negative, and those who felt they did not yet have sufficient experience to form a judgement. Positive comments included "As more institutions open up their content it is getting easier to find more reliable sources," "A great experience," and "would like my teachers have used more resources like these." The issue of time was mentioned frequently by respondents whether they were positive or negative about their experience overall; one described it as "Easy when dedicate time," while others stated, "it takes some time to identify good resources from the mass, but then it starts to become easier" and "You need to spend time. Researching, analysing, proving, evaluating, choosing, deciding and archiving." Another respondent observed that "it is necessary to spend time researching and evaluating the quality of the resources," and added, "You can find a lot of resources but not all have the same quality. Maybe it's necessary to filter before making them public. Or define a rating system in order to classify previously." The same issue of dedicating time to search and selection of OER was often mentioned in much more negative light: "difficult since it takes me a lot of time" is representative of several similar responses in this more negative category, with one respondent stating "It is difficult so I often make up my own rather than take the time to search." Other responses related to time taken, but focused on other interesting aspects such as pedagogical value of found resources, for example, "In some cases, it is difficult to find appropriate OER that suits with your teaching aims" and "The problem is to find OER for my teaching content." Most reported that they selected the resources using their own personal criteria such as relevance to a specific course or class, although many others answered that they selected resources based on trust in either the author of the materials, the institution where the resources were created, or in a given repository.
Respondents identified numerous benefits flowing from OER use. In contrast to the aforementioned problem of time taken to select OER, some academics perceived OER use as time saving, describing them as "useful free, easy, open reliable tools," pointing out "it's about not (re)inventing the wheel," and also citing the "ability to share and remix sources." The notion of using an OER to increase coverage of a topic was also mentioned. One academic stated OER can "save time and provide different expertise to mine" while another recommended use of OER to complement their own teaching "so that students generate arguments or questions." Another approved of the "Broad range of approaches and positions" available from OER.
Open and free availability was also highlighted as a key advantage: "Cheap, easy access to you and to students, the possibility of sharing on-line and with the digital community through social networks"; "it gives the possibility of free and any-time, any-place access." One academic examined this point in depth, stating: "As an instructor: FREEDOM from the textbook. As a student: reducing the cost of higher education. As an institution: re-imagining higher education for the future. As an advocate: participating in the Openness Movement has been the most profound experience of my career." Others regarded the social element of OER use as a pedagogical benefit, for example, "OER creates a community of learning and changes people's perceptions of learning as a more social activity" (see figure 4).
As well as issues relating to searching for and finding OER already discussed above, the respondents tended to make frequent mention of several other barriers to OER use. Many respondents seemed to feel they lacked relevant experience or training to find and select OER effectively, making comments such as "Don't know where to find materials," "challenging without guidance," "not easy, just beginning." One academic felt that "Teachers' general understanding of the importance of open Open Praxis, vol. 6 issue 1, January-March 2014, pp. 29-43 education" was the main barrier. Another stated "more institutional support to adopt open content and open software" was needed. The skills of academics were highlighted again by a respondent who stated, "From my point of view, the challenges posed by OER have to do with adaptability and knowledge of computer and software that each teacher has. The greater your use of ICT, the smaller your difficulty to adapt and use resources positively," and another mentioned that "repurposing is hard and time consuming." External technological barriers were also mentioned, such as "connectivity at institution level (issues of security, proxies, etc.), and personal resistance of some people to new communication uses and tools" and "IT Connectivity issues and policy (eg. sites blocked for a long time)." Language barriers were another frequently cited area affecting retrieval, use and reuse of relevant resources. In one academic's view, "the language barrier is still an embarrassment." Respondents struggled with both the language of the resources and the language of the repositories. English language resources were specified as being prevalent in repositories, but usually unsuitable for speakers of other languages.
Academics also expressed a general anxiety around the quality of the resources sourced from ROER, mentioning "Evaluating usefulness and quality" and also "pedagogical adequacy." As one respondent stated, "The quality assurance of OER is the challenge and the barrier at the same time." Another problem for several respondents was resource licensing, including the issue of restrictive licenses: "There are some really good materials released but with great restrictions" and also more generally, "The various licenses and formats" represented a barrier for one. Another respondent stated: "Currently, OER are stored in content silos. Open licenses are not a sufficient guarantee of openness. Everyone expects that OER are interoperable, discoverable, remixable, and accessible. I think Semantic Web technologies can be used to achieve these characteristics" (see figure 5).
Overall, respondents were positive about the value of OER, but saw room for improvements. A respondent who felt that OER repositories do not assist in finding resources, stated "Repositories are typically awful, I try to find open information via Google," while another mentioned, "I passively consume via blogs and twitter," indicating that some academics are making use of alternative resource discovery strategies (see figure 6).

A review of the experts voices on the use and improvement of ROER
We aimed at understanding how experts conceived the development of ROER to enable and facilitate access to resources. We needed to understand their views in three key points: • Which type of information ROER should provide to the users • How this information should be included to the platform • Which are the technical characteristics that might lead to the successful development of ROER.
We asked the experts if they believed in the possibility of developing standards to improve the current designs and implementation of ROER. Their opinions were both interesting and passionate, as even when they were pro or contrary to the idea, they proposed models that can enable a better development of ROER. As one of the experts mentioned "the repositories are all over the place and do a bad job of promoting themselves" and for the experts is not also about the platform is about the information those provide as ROER "should have at least: detailed description of content, goals and objectives and teaching methodology." First we asked them if they considered it was necessary to include metadata systems such as Learning Object Metadata (LOM) or Dublin Core in the repositories and their opinions were quite negative. They valued the importance of use of descriptive languages in the repositories, but considered that it can "be quite a complex process as if a resource is too narrowly defined it can have a restrictive effect" also "OER repositories should require the barest minimum of metadata. Every additional field makes it less likely OER will be deposited and released" one of the experts mentioned that "Metadata was the death of learning objects." Regarding pedagogical information, their opinions were quite diverse. One suggested that the resources should be carefully described at their pedagogical level: "quite extensive but I would start along the lines of reflective, directed etc; and then maybe move into approaches such as socialconstructivist, behavioural, emotional etc; and then think of the media, reading, case study, lesson plan, quiz etc.." Another argued that the description needs to kept at a minimum: "content usedinteraction types-skill levels" would be about right as a minimum." Others suggested that adding pedagogical objectives might be useful and "helpful in order to guide users interested in reusing the content" by adding "detailed description of content, goals and objectives and teaching methodology." In general the suggestion from the experts is that the authors themselves need to provide enough information to describe and therefore discover the resources.
Thirdly, we asked the experts about the main criteria they considered educators use to evaluate OER. First, they mentioned that "trust is the key. If trusted colleagues recommend something then it will be used or if a trusted organisation puts its stamp of approval on it." Respondents also agreed that "user friendliness, subject relevance, origin of the producing/ offering institution, number of clicks," are as important as specifying authorship and usage rights by making the resources "useful and applicable," but also their important its "adaptability and perceived quality." One of the experts mentioned that academics "look for how well they meet the goals of the course or the principles of what they want to teach. I think a certain synchronicity with their current approaches would make sense mostly," while others mentioned "professional discernment," "previous experience," "recommendations from peers" and "word of mouth." The experts were asked if they think that the current OER repositories are "good enough" or if there is room for improvements and/or standardisation. One respondent mentioned that "finding and previewing OERs could be easier and more elegant to aid the user experience," and that "quality assurance of OER is a vital area to develop and the key to mainstream uptake. If academics feel they can trust a search service to find quality resources then they will use it (especially if it has the blessing of a national educational body)." Some of the experts were not so positive: "I do not think standardisation is the answer. On the contrary, diversity is what makes it fun and interesting. But definitely there is a lot of room for improvement. We need to be able to index and find resources more easily and to involve users more easily, too." Also, some mentioned that users need more training in using and finding resources as repositories are "good enough, we need more content but crucially it's more about practice, and educators need to think about using OER and feel encouraged to do so." Regarding the technical elements that can make a ROER successful, they highlighted that repositories should be accessible for the users. One of the experts suggested that they need to be "open-searchable (full text) via Google. Approachable-plenty of advice and support available. Well promoted." Another expert mentioned that repositories should have "flexible search parameters. Ability to feedback on the usefulness of an item. Break down the elements of the OER to understand what it involves. Easy to download in multiple formats." Flexibility and openness seems to be the most important concepts for the development of repositories of OER, as they should contain "a good search facility, some pleasant browse options, in fact it is particularly important to offer discipline/subject based browse options." According to our analysis of the answers of the experts, we identified a series of criteria that can lead, according to their opinions, to the development of successful repositories (see figure 7).

Conclusions and recommendations
Experts and academics experience of OER repositories should ideally be in agreement with the theory underpinning OER, with these factors together ideally driving technical development and innovation. Our goal is to improve and increase the adoption of open educational practices by suggesting enhancements to OER repositories sourced from a synthesis of the feedback provided by both academics and open education experts.
The use of OER repositories and OER in Higher Education teaching requires a series of literacies and skills. A digital lecturer is not a "virtual" scholar, but a professional who delivers teaching both face to face and online, who uses technologies for teaching and is also a researcher who uses digital resources to understand the world (Anderson, 2009;Borgman, 2007;Weller, 2011). This study sought to assess the level of engagement with OER practiced by a sample of lecturers, and simultaneously to gather specialised feedback from a sample of OEP experts. The qualitative data we obtained revealed some disconnect between the experiences of those who use them to teach and those who make them available.
As Andrade, Caine & Carneiro (2011) write, "for higher education and adult learning, there is a prevalent notion that there are no specific quality assurance processes in place for OER" (p. 9), and we believe that the importance of quality assurance cannot be underestimated and should begin from the detailed detection of the desirable features in ROER from the point of view of both teaching and OEP specialists. Stella (2010) has indeed argued that "the OER movement would benefit from an exploration of current international quality assurance mechanisms and general guidelines and, potentially, from linking with quality-assurance agencies" (p. 5).
Digital practices in searching for content have changed dramatically. From the very analog library catalogue cards to the online public access catalogue used in almost every library nowadays, the form in which we search and access information has changed (Deegan & Tanner, 2001;Vandendorpe, 2009). The availability of resources has increased, and information about almost any subject can be found online. Moreover, user-interface design has come a long way since its early days. Back in 2000 W. Y. Arms wrote that "during the past few years, the quality of user-interface design has improved dramatically. It is now assumed that new users can begin productive work without any training" (Arms, 2000, p. 160). Thirteen years later, we are witnessing that many repositories leave much to be desired in their functionality and user-interface design, and that many academics find them difficult to navigate without previous training.
Much was written at the beginning of the last decade on the importance for librarians and information professionals to develop digital skills through education and training (Carpenter, 1999;Spink & Cool, 1999;Kenney & Rieger, 2000;Deegan & Tanner, 2001), but as many educational technologists experience in their daily work there is a need for additional training for lecturers in digital skills, including the discovery and use of OER: "Training and didactic and technical support for instructors are also extremely important. Assistance in finding suitable OER, developing the curriculum, adapting OER, and carrying out their new role are crucial for success." (Jacobi & Woert, 2012, p. 19). McGreal (2011) also points out that "in order to take full advantage of Open Educational Resources (OER), instructors, developers and learners should become familiar with the repositories that house them along with some training in how to make optimal use of them" (2011, p. 1).
The abilities that academics are expected to have are, to some extent, the same ones now expected of students; however, in reality there is little support for them to develop these digital skills and much less support in developing open literacies. Engaging with open practices requires expertise, support, time and commitment and universities need to provide both the support for developing the expertise and the time for academics to explore this new world as in general academics are positive and committed to embracing new practices, but they are also scared and worried, as new technologies are not their natural environment.
Collaboration, Searching, Repurposing and Translation were the top four features experts considered could make ROER successful. These signify key abilities of digital scholarship and represent a fairly recent new paradigm, at least in some disciplines, in the way academics think about scholarly outputs and teaching resources. We believe that lecturers cannot merely be expected to "naturally" develop and harness these abilities. Ongoing training and support as well as institutional methods of encouragement (such as policies) might be required. Moreover, as more users develop advanced OEP skills and values, critical assessment and quality assurance of OER are more likely to follow.