Structural Changes in the Landscape of Spanish Distance Universities

The main national distance learning universities were launched before the emergence of the Internet. In order to adapt to digital connectivity, these universities had to modify their organizational and methodological procedures. But in recent years important changes have emerged in the field of educational technology, and this has significantly altered the higher education sector. This work analyzed the recent updates that distance-learning universities in Spain have undertaken in order to adapt to the current higher education landscape. The evolution of enrolment rates in distance universities in Spain in the period 2003–2017 was taken as a reference. Based on the available data, the factors that explain the enrolment gap between types of entities are analyzed and key actions for the evolution of distance learning universities in Spain are suggested. This provides an evidence-based approach that aims to contribute to the reflection of academics, institutional managers and policy makers who are called upon to reorient the strategy of distance universities to make them sustainable in the medium and long term in the new landscape.


Introduction: the evolution of the landscape of distance higher education
The main national distance universities were launched with the purpose of democratizing access to higher education for people who cannot attend face-to-face studies (Tait, 2008). The evolution of the first generation of distance universities has materialized in various types of institutions that currently provide teaching services through multiple methodological models (Orr, Weller & Farrow, 2018). Although these universities have fulfilled the objective of expanding access to higher education, in recent years there have been negative indicators referring to the enrolment rate and the financial balance that suggest that their structures have not been fully adapted to the new social needs.
The current situation of distance universities can be explained by analyzing how they have managed the transition towards a digital model, as well as the institutional adaptation to social changes, especially since the Internet becomes a mainstream technology. In adapting to changes, they have been influenced by internal dynamics, which affect the ability to provide education and research services taking advantage of new digital resources. And there have also been a series of external factors, among which we find the emergence of novel agents competing in the higher education sector as well as a pronounced regulatory advance that sets the conditions for universities to compete in that sector (Guri-Rosenblit, 2009).
Regarding internal factors, distance universities have significantly modified their internal procedures -both organizational and methodological-in order to adapt to digital connectivity. Recently, they also changed the way of providing teaching to give a better response to a broad consideration of lifelong learning. These measures allowed universities to maintain their traditional target of students with a higher average age than face-to-face universities, and also to expand it by adding young people who wish to integrate into a more dynamic and specialized labor market. But in recent years important changes in the field of educational technology have emergedespecially the presence of new stakeholders and companies working on digital tools based on artificial intelligence, with a focus on data-driven learning-and there have been changes in the labor market -mainly the new professions and tasks that require high level skills that are not included in university curricula-, so that all this has significantly altered the structure of the higher education sector. Traditional Distance Universities (TDU) are currently in the process of adapting to the new structure, and are implementing new measures that go beyond the surface changes corresponding to the first phase of digitalization. This is a key process since the viability of many of the pre-Internet distance universities will depend on their adaptation to the current landscape of higher education, and they have done it necessarily within the regulatory framework of each country.
This paper analyses how these generic dynamics have affected the landscape of distance higher education in Spain. Specifically, the analysis will focus on the supply and demand gap that affects distance universities in a twofold perspective: on the one hand, between TDU and New Distance Universities (NDU); and on the other hand, between distance universities as a whole and the new proposals for digitally mediated teaching that come from the very diverse agents that have begun to operate in the higher education sector. Data from Spain confirms the international trend that traditional distance learning universities are not being able to attract the growing demand for online learning (Qayyum & Zawacki-Richter, 2018). The causes that are motivating this phenomenon are analyzed, and alternatives are also proposed with which to change the trend. The proposals are based on the action of agents involved in innovation cycles in emerging markets, and taking into account trends in the higher education sector (Domínguez, Álvarez & Gil-Jaurena, 2016).

Competitiveness and enrolment rates in Spanish distance education universities
The implementation of distance higher education in Spain corresponds to the creation in 1972 of the National Distance Education University (Universidad Nacional de Educación a Distancia, UNED), whose beginnings and subsequent evolution followed a parallel path to first generation distance education universities (Daniel, 1996). The advent of the Internet led to the emergence of a second distance-learning university in 1994, the Open University of Catalonia (Universitat Oberta de Catalunya -Universidad Abierta de Cataluña, UOC), which offers its studies online, but with a smaller scope and structure. These two pioneering universities are considered TDU.
In addition to the TDU, since 2006 NDU have appeared in Spain, which are locally based but offer distance studies throughout the country. They apply various models, ranging from proposals based on quality and high cost to low-cost ones. The main difference between these two types of universities would be the organizational structure: traditional universities have a larger dimension, which corresponds mainly to the needs to produce their own learning resources; whereas the structure of new universities is small, focusing on outsourcing services.
Finally, in addition to this group of institutions, there is a growing tendency for face-to-face universities to offer distance learning. A trend already widespread in Europe, where over 80% of higher education institutions are offering online courses (Gaebel et al., 2014).
In national higher education systems where there are many universities offering their services to the same population, the main indicator of the competitiveness of institutions is the enrolment rate. In the case of Spain, the evolution of student enrolment in the set of non-attendance universities from 2003 to 2017 has been increasing ( Figure 1).

Figure 1: Enrolment rate (official programs) in Spanish distance universities (TDU and NDU), 2003-2017
Source: General Secretariat for University Coordination and Monitoring, Integrated University Information System, Spanish Ministry of Education, Culture and Sport; UNED; UOC Disaggregated analysis of these data shows differentiated behavior for TDU. The positive evolution of student enrolment in Spanish distance universities has not been reflected in the enrolment rates of traditional universities. In the case of TDU, the effect has been the opposite and they have lost students in recent years. In more detail, there are two clearly differentiated cycles: one that corresponds to an increasing stage of enrolment, between 2003 and 2011; and another cycle of decreasing enrolment, where enrolment in traditional universities decreases significantly ( Figure 2). In order to explain the difference between the overall enrolment ratio (TDU and NDU) and those for TDU only, it is important to consider the date on which the new NDU begin to offer their studies. In the period 2002-2007, the universities that monopolized the distance higher education system in Spain were the two TDU. The first distance university to offer its studies outside the TDU was the Distance University of Madrid (Universidad a Distancia de Madrid, UDIMA), in 2008. In 2010, the Distance University of La Rioja (Universidad Internacional de la Rioja, UNIR) and the Valencian International University (Universidad Internacional de Valencia, VIU) were incorporated. And from 2013 onwards, Isabel I of Castilla University (Universidad Isabel I de Castilla) was added. As a result, the current trend is that the number of students enrolled in distance universities is increasing, but the number of students enrolled in TDUs is decreasing. This decoupling leads to a competitiveness gap that affects traditional models (see Figure 3). But university-based courses are not the only possible format for higher education. Distance education has always been offered in more formats than conventional education. And since the advent of the Internet and mobile technologies, there seems to be an increase not only in formats but also in agents providing training. As the world is increasingly connected to the Internet through a variety of devices, online education in multiple formats has attracted the interest of both students and a variety of stakeholders interested in operating in the higher education sector, whether for educational or forprofit purposes. The Massive Open Online Courses (MOOC) is the most outstanding example of a format that is supported by different agents, mainly start-ups that are "accelerated" by Silicon-Valley based innovation processes (Weller, 2014).
Open access courses are certainly not new in the history of distance education, nor is it new whether there is an educational offering with an accreditation option or not. Traditionally, this educational offer was monopolized by distance learning universities. For this reason, the emergence and subsequent popularization of MOOC outside distance universities represents a significant change in the evolution of these institutions, which points to a limit in their capacity for innovation. The classical literature on socio-technical innovation indicates that when this happens there may be disruptions in the previous service delivery system (Christensen, 1997;von Hippel, 1988). And as a consequence, disrupted agents can put at risk the traditional ecosystem dominance of major agents (Kilkkia et al., 2018). With a stable student population and the increase in the number of distance education institutions, the importance of distance universities -especially traditional universities-as the main recipients of students in distance learning methodologies tends to decrease.

Institutional challenges and key factors in the evolution of distance universities
Together with the growing competitiveness among the agents of the higher education system mentioned in the previous point, distance higher education institutions in Spain are facing challenges similar to those identified by Gaskell (2018) in the case of United Kingdom: • Increasing convergence of distance and campus-based education.
• Government funding and regulation of higher distance education. • Retention rates in distance education. • Informal learning.
The evolution of these challenges depends on a number of factors, both internal and external to the universities themselves (see Table 1). In contrast to other milestones in the history of distance higher education, analyzing these developments and understanding the response of institutions to new challenges is now of great importance because of the depth of the changes they can bring about in the structure of the higher education system as a whole. As Gaskell (2018) points out for the case of the Open University in the United Kingdom, the viability of traditional distance universities -"One issue for the future is whether single-mode distance teaching institutions can survive" (p. 94)-is directly related to the evolution of challenges and their management by universities. • Resistance to change in the workforce.
• Emergence of new types of digital educational agents with platform-type models.
• Rigid internal regulation of university governance.
• New ecosystem of companies offering training and digital learning tools.
• Rigid government regulation of the higher education system.
In the case of Spain, the evolution of institutions in recent years has been mainly influenced by resistance to change. This is an internal factor that affects the competitiveness of distance education institutions in Spain, and which is especially noteworthy in the case of more traditional ones such as the UNED. Alan Tait (2018a) explains the way in which the main axes of innovation of distance universities have been eroded since their creation: innovation in institutional vision and mission, teaching methods, learning technologies, educational logistics, scalability and innovation embedded in the organizational system itself. In Spain, none of the universities, both traditional and those born in the last decade, has introduced significant modifications in their methodologies -either in the institutional methodological model or in the specific didactic strategies at the level of the subjects-since their creation. In Spain there are two distinct types of distance learning provision. On the one hand, traditional institutions such as the UNED and the UOC use a mixed method based on self-directed learning (Garrison, 1987) and online interaction with students  through virtual campuses and with tutors in charge of monitoring and evaluating students (UNED, 2019;UOC, 2019). And a second group of newly created distance universities that employ a method based on replicating face-to-face teaching -scaling up groups through growing cohorts of classgroups-and using digital mediation to facilitate the tracking of distance students (UNIR, 2019). None of these methods, which were innovative at the outset, has evolved significantly since their introduction. In fact, no institution has introduced significant variations in its methodology since its creation (Álvarez, Álvarez, Domínguez & Kiczkowski, 2011).
The second (external) factor refers to new types of agents operating on the basis of the Internet and using platform-type models (Kenney & Zysman, 2016) that allow for lower production costs and an alliance-based offering that expands dissemination and maximizes the entry of new students. Kerrison (2016;cited by Gaskell, 2018) mentions the positive economic and image impact for the University of London of his collaboration with Coursera, one of the leading providers of MOOC using a platform model.
And to these agents must be added the new type of companies that are working on particular tools in fields such as tutoring, campus safety, robots in the classroom, social-emotional learning, job training, personalized learning, "critical skills" training, learning analytics, behavior management, grammar and spelling assistance in exchange for your personal data, learning management system, test preparation, language learning, and educational apps (Watters, 2017). This new ecosystem of companies offering training and digital learning tools is mainly aimed at meeting the needs of new professions and the development of tasks requiring high-level skills that are not included in university curricula. So all this has significantly altered the structure of the higher education sector.
A final factor, which acts in a twofold sense, from the outside but with a direct influence within the institutions, refers to government regulation of the higher education system. Spain applies the governance and Quality Assurance model of the Bologna Process (European Higher Education Area, 2018). But there is also national legislation governing access to university for professors and the performance of teaching and research tasks (LOU 6/2001;RD 1313RD /2007. Within this legislative framework, the only universities that have specific regulations for providing distance education services are the UNED and the UOC. The rest of the distance learning universities that have emerged in recent years should be covered by the conventional regulations available to faceto-face universities. For example, depending on the case, new universities wishing to offer distance learning may not offer scalable services, but must respect a teacher-student ratio equivalent to that of face-to-face systems, or may not exceed a certain total number of students. These limitations do not apply to traditional distance learning universities, which allows them to maintain their preferential position within the system as the main providers of distance learning. The barrier effect of this regulatory framework on the implementation of new distance learning universities is not exclusive to the Spanish case (Orr, Weller, & Farrow, 2018), but in the case of Spain it has allowed enrolment rates in traditional universities to remain stable as a result of the limitation of the scope of other emerging competitors. And all this in spite of the greater demand for distance higher education on the part of Spanish citizens, as shown in Figure 1.

Discussion and implications
The combined effect of NDU and the digitization of face-to-face universities increases the overall range of distance higher education offered. A situation that directly affects the sustainability of TDU -which are still slowly restructuring to meet the new needs of students who continue to demand distance learning, but of a new kind-and of NDU. The analysis of enrolment trends in distance learning serves as a basis for proposing the conclusions of this study, followed by a series of considerations on the positioning of distance universities for the future.

Filling the competitiveness gap through innovation
The trend in recent years has been to widen the competitiveness gap between TDU and NDU as shown in Figure 3. A similar gap should also be considered in the case of distance learning universities as a whole and the incipient offer of online education that is beginning to be offered by face-to-face universities. A first condition for reducing these gaps between universities -and from these with the new agents operating in the system-is to orient the changes towards the needs of students who opt for forms of teaching more suited to their interests.
According to the data, in the case of the Spanish TDU, the changes undertaken are not caused an upturn in the enrolment of students, who seem to opt for the offer of other distance universities. TDUs are immersed in the dilemma of the innovator (Christensen, 1997), which arises when organizations must change at a time when they are dominant and need to protect their market. At some point a competitor comes along that threatens your business with a better alternative. And then they face the dilemma of sustaining the market where they are standing out (the traditional model of distance education), but at the cost of losing some great opportunities (focusing on the new demands of students), or focusing on these opportunities that could only bear fruit in the long run (at best).
As a result of the study of the key factors that explain the pattern of behaviour found in enrolment rates, the necessary innovations point to a set of proposals on the configuration of distance learning universities. On the one hand, the scope of the innovations points to a necessary reconversion of traditional institutions, and their repositioning within the higher education system is also discussed. According to the benchmark of the new agents in distance higher education, and the evolution of the digital education sector in the last decade (Palvia et al., 2018), three types of innovative guidelines are proposed: (1) In the methodological field, to deepen the scalable teaching methodologies, which maintain the costs contained while guaranteeing the quality of studies and the social mission of distance universities; (2) at the operational level, innovations aimed at deepening the use of digital technologies based on data, and at focusing the learning process on the student and his/ her particular needs in a distance learning environment; and (3) at the institutional level, to relocate distance education universities to the higher education landscape as agents that facilitate in-depth learning and certify higher-level skills throughout life, which means making the teaching offered at universities compatible with skills acquired in non-formal and informal spaces, whether in digital or face-to-face spaces.

Implications for mission and scalability
It has long been debatable to refer to distance learning as a distinct mode of teaching provision. What distinguishes single-mode distance universities is their mission, not their location and provision on campus or at a distance. And, according to Tait (2018b), the mission remains as it always has been: "to provide post-secondary education at scale with quality at a price that is affordable, and with the flexibility to permit people to come in and out during their lifetimes" (p. 100).
The mission does not change, therefore, but the conditions under which universities provide their higher education services on a massive scale have changed. These universities have always been influenced by technology to develop their mission, but have recently seen how the digital shift has affected certain structural aspects of their model. This forces them to adapt the functioning of organizations to be able to offer their scalable services in the new context. Scalability in distance learning -which allows for low-cost studies to be offered to all interested parties, and which has been the key to distance learning methodology-is being called into question for two reasons. On the one hand, the quality control of the national agency operating under the Bologna scheme requires it to maintain teacher-student ratios similar to those of face-to-face universities and makes it difficult to promote methodologies based on self-learning. Only the UNED applies (residual) methodologies based on self-learning in Spain, and this is at the expense of a high cost in the production of learning materials adequately adapted to the conditions of autonomous learning. The digitalization of the systems for the production of learning resources and also the flexibility of the regulatory frameworks under which quality control operates are necessary in order to continue offering innovative methods of providing distance learning clearly differentiated from face-to-face universities -considering that offering video recordings of lecture class sessions and seminars, as well as tutorials through videoconferencing, is not a differentiating factor characteristic of distance learning in higher education.

Organizational and logistical implications
The socio-technological change generated by the Internet, by mobile services and the interconnectivity between people has radically transformed the very notion of distance. Digital connectivity affects the way in which teaching services are delivered and also has implications for the organizational model of universities.
As has been said, the UNED currently maintains an organizational model that is typical of first generation distance universities. This includes having study centers scattered throughout the country and, in some cases, also abroad. In the beginning, these centers were necessary to support learning with a high practical component. Despite its high cost (the UNED dedicates 40% of the enrolment fee income to its associated centers), its presence gave it a competitive advantage over other agents who did not have a network of study centers. However, the number of people attending face-to-face tutoring at the UNED has dropped significantly in recent years, in parallel with the implementation of digital teaching platforms and e-learning services.
At present, no other Spanish university has face-to-face study centers, and in no case does the teaching method depend on the existence of face-to-face tutors. However, it is common for new distance learning models (especially MOOC) to use centers where the skills acquired in distance learning are validated, particularly in the case of qualifications giving access to regulated professions. Thus, identifying which services require the student's presence as a first step towards advancing physical off shoring in favor of ubiquity based on digital technologies seems to be a suitable way to solve the problem of extra costs affecting institutions that emerged in the first phase of distance higher education.
Sustainability is a challenge that affects not only universities, but also all the agents that operate the higher education system -see Agarwal (2018) for the case of MOOCs on the edX platform. The sustainability of distance learning universities is related to the indexation between expenditure and income, which in turn depends on the methodological model used to provide the teaching service that is derived from the institutional mission. The new Internet-based platform models allow a more fluid relationship between service providers and users, as well as a direct and flexible relationship between agents both inside and outside the organizations involved in the processes. Distance education universities have already explored the platform model in the case of MOOCs -for example, the Open University of the United Kingdom, with the FutureLearn project (Manathunga, Hernández-Leo & Sharples, 2017)-to allow for partnership between the agents who design and deliver courses under the same entity. But it would also be appropriate to explore the extension of the scope of the platform-type models within organizations, so as to favor the work processes between those responsible for the courses, which in the future will necessarily be provided on the basis of the Internet and through digital mediation devices.

Introduction
In e-learning systems, a range of materials are used to facilitate and support students' learning processes. Learners engagement with these materials is essential to provide effective and efficient learning and reach course outcomes. Additionally, as well as learning materials, their engagement on e-learning systems is important to provide learner-content interaction (Moore, 1989) because "student-content interaction can perform some functions of the educational transaction formerly accomplished exclusively through teacher-learner interaction" (Anderson, 2003, p.137). In massive distance education systems, where the engagement with e-learning systems is an indicator for the student-content interaction, it is important to understand it from a broader perspective. In this regard, this study examines the students' achievement on e-learning systems from the perspective of engagement and bounce rate.

Literature Review
Stovall (2003) states that engagement consists of both the time learners spend on tasks and their willingness to take part in activities. Krause and Coates (2008) associate engagement with the high quality in learning outcomes. Engagement requires being active, and having sense making (Harper & Quaye, 2009). The engagement is defined as "the quality of effort students themselves devote to educationally purposeful activities that contribute directly to desired outcomes" (Astin, 1985(Astin, , 1993Pace, 1995;Chickering & Gamson, 1987;Hu & Kuh, 2001).
Bounce rate is the percentage of visitors who enter a site (or a page) and then leave immediately without visiting any other pages. It could also be expressed in terms of time spent on site (e.g., users who spend five seconds or less on the site) irrespective of the number of pages they view. Djamasbi et al. (2014), underline the negative effects of bounce-rate on potential users' engagement in web sites. Krause and Coates (2008) reports regulated measurement of learner engagement from a largescale study of first year undergraduate learners in Australian universities. The analysis of the study presents different type of undergraduate learner engagement, including online, self-managed, peer and student-staff engagement. The findings point out the development of a broader understanding of engagement as a process with several dimensions. The study calls for a more robust theorizing of the engagement concept that encompasses both quantitative and qualitative measures. It takes into account the implications for pedagogy and institutional policy in support of enriching the quality of the learner experience.
In Liaw's study (2008), learner satisfaction, behavioral intentions, and the effectiveness of the Blackboard e-learning system is investigated. The results showed that perceived self-efficacy is a critical factor that affects learner satisfaction with the Blackboard e-learning system. Both perceived usefulness and perceived satisfaction support learner behavioral intention to use the e-learning system. In addition, effectiveness of e-learning can be affected by multimedia instruction, interactive learning activities, and e-learning system quality. This study suggests a conceptual model to comprehend learner satisfaction, behavioral intention, and effectiveness of using the e-learning system. Oye, Iahad, Madar and Rahim's (2012) study examined the application of e-learning model to explain acceptance of the e-learning technology in the academic settings. The study confirmed that in order to foster individuals' intention to use e-learning environments, positive perception on e-learning use is crucial. By using linear regression analysis, the study verified that while attitudes influence intention to use, the actual e-learning use has significant effect on learners' academic performance. In this study, e-learning use is associated with learners' increased academic performance. The study suggested that sessions of training and information on e-learning need to focus primarily on how the e-learning technology improve the efficiency and effectiveness of learners' learning processes.
In their study, Nguyen, Huptych and Rienties (2018) investigated the students' timing of engagement and its relation to learning design and academic performance. The analysis was conducted for about 28 weeks using trace data, on 387 students over two semesters in 2015 and 2016. Students spent less time studying the assigned materials compared to the number of hours recommended by instructors. The timing of engagement also varied from in advance to catching up patterns. Highperforming students spent more time studying in advance, while low-performing students spent a higher proportion of their time on catching-up activities. The importance of pedagogical context to transform analytics into actionable insights emphasized in the study. Research results of Tao, Zhang and Lai (2018) show that there is a positive relationship between perceived online learning environment and university students' learning performance driven by student participation. For this reason, educators should develop online student participation strategies to increase online student participation and improve the learning performance of online students.
McKenna and Kopittke (2018) examined the use of lecture notes, lecture slides, and lecture recording utilized by first-year students through the learning management system. In the study, it was stated that lecture slides were downloaded by more students than other learning resources and 71% of students used at least one type of learning resource. Authors stated that distance learners use learning resources (lecture notes and recording) more often than campus students. The learning resources were mostly downloaded during the 13 th week, revision week and exam week. In the study, there was no relationship between participation and formative quiz scores while there was a positive relationship between participation and final summative exam scores.
In their study, Zhang, Li, Liu, Cao and Liu, (2019) focused on the data-driven online learning engagement detection via facial expression and mouse behavior recognition technology. To improve the accuracy of learning engagement detection, face data and mouse interaction used as two aspects of students' behavior data. Thus, higher recognition rates received.
Studies about engagement, dropout and academic achievement is rich in the related literature. However, there are limited number of studies which compare these variables with large scale participant numbers from the perspective of open and distance learning. Furthermore, transformation of learners' shifting engagement with the educational environments appears to have radically changed from traditional materials to online environments in the last decades. As a new phenomenon, online engagement is related to individuals' online interactions, session duration and navigation (White & Le Cornu, 2011). Therefore, measuring online engagement with psychological test only is almost impossible in massive online learning environments. For this reason, in order to determine students' interactions; new variables such as number of hits, spent time and, bounce rates should be examined. Thus, engagement time in online learning environments became as an important variable that can used in massive education context and learning analytic studies. It is considered that this study will fulfil the gap in the literature and be a model for future studies.

Purpose of the research
The purpose of this study is to compare open and distance learners' e-learning engagement time (the time spent on e-learning portal) and academic achievement according to course, dropout and bounce rate variables. In this regard, the study intends to shed light to following research questions: 1. Is there a significant difference between learners' engagement time levels depending on courses they study? 2. Is there a significant difference between learners' academic achievement depending on their engagement time levels? 3. Do the engagement time levels significantly predict learners' academic achievement?

Methodology
The structure of this study is a post-test only model. The main purpose of these models is to test descriptive causal hypotheses about causes that could be manipulated (Shadish, Cook & Campbell, 2002). Accordingly, this study searches for causal connections between e-learning engagement time and academic achievement. The research design of this research is summarized in Table 1.
In this research, the causal correlation between engagement time, bounce rate, dropout and academic achievement variables are analyzed. Engagement time represents the time learners spend on the web system, bounce rate represents the ratio of learners who stay on the system between 0-99 seconds during a term, dropout represents the learners whose GPAs are 0, academic achievement represents GPA (Grade Point Average) in this study.

Sampling
The participants of this study are chosen randomly from Anadolu University Open Education system. They are the learners who study with e-learning materials (e-books, videos, tests, etc.) of four courses including Ataturk's Principles and Revolution History I, Basic Concepts of Law, Introduction to Economics I, and Basic Information Technologies I. These courses are chosen because learner populations, the number of learning resources and visit times are greater than other courses. Thus, 323,264 learners' e-learning system usage data is used to carry out statistical analysis.
The courses chosen for the purposes of this study are designed in the same way. Each course is designed in a unit-based format. Each unit includes the associated learning resources. The variety and types of learning materials in each unit of the each course is similar. It should be noted here that the primary difference in course design for the chosen courses is the content presented.

Data collection process
When conducting a research in mega systems such as Anadolu University Open Education System, it is required to study with large-scale samples. In this research, the logs of Anadolu University Open Education System's e-learning portal are recorded during the academic term. Learner data like student IDs, pages visited, timestamps are kept in logs when they were active in the e-learning system. These logs, consisting of millions of rows of data, are simplified by using data classification techniques.

Data analysis
The quantitative data is analyzed by using descriptive statistics such as percentage (%), frequency (f), standard deviation (SD), and mean (X -); in addition to parametric tests such as independent two samples t-test, one-way ANOVA, Pearson correlation coefficient, and simple linear regression analysis. While interpreting analyses results, some supportive statistics such as eta square (η 2 ) are utilized due to the big volume of the data. Effect size is a statistical value which shows deviation level of the expectations defined in null hypothesis from the results derived from sample (Cohen, 1988;Vache-Haase & Ness, 1999). As effect size is the quantity of the difference between null hypothesis and alternative hypothesis, it is an indicator of practical significance of the results of the research. While statistical significance is affected by sample size, the use of effect size can help to understand the results in more accurate way (Özsoy & Özsoy, 2013). In this manner, when the difference is significant among the groups, eta square (η 2 ) is used. Calculated eta square values are interpreted according to Cohen d index, which is defined as small, medium and big according to .01, .06 and .14 respectively (Büyüköztürk, 2005).

Findings
In this large-scale research, data is obtained from 323,264 learners who registered in Anadolu University Open Education System in fall term of 2015-2016 academic year. Before parametric tests, descriptive statistics of groups and variables are examined. Distribution of the learners according to the courses given in Table 2.
As can be seen in Table 2, distribution of the learners according to the courses they are registered in is close to each other. Additionally, it can be seen that on the e-learning portal, learners mostly accessed the e-learning materials of Basic Concepts of Law course. Figure 1 shows the distribution of the materials accessed on the e-learning portal according to the courses. In this study, three groups are obtained by grouping the learners' engagement time level. These groups are bounce rate, average and advance groups. Bounce rate represents the ratio of learners who stay on the system between 0-99 seconds during a term. In the study, learners are grouped as average group that remains in the system between 100 and 999 seconds, and advance group that remains in the system 1000 seconds or more. Distribution of the learners according to these groups is given in Table 3. As can be seen in Table 4, dropout rate is rather low. Learners who dropped without accessing are not included. However, when it is considered that the rate of learners who use the e-learning portal is high, dropout rates of the learners are rather low for this group. In this study, the time learners spent on the e-learning portal according to the courses (second) is analyzed. Results of this analysis is shown in Table 5.
When learners' engagement time on the system is investigated in detail, it is seen that the number of the bounce rate learners and advance group learners is close. For this reason, it can be said that the dropout rate is higher than expected. Distribution of the learners in each group is given in Figure 2.   In this study, in addition to distribution of engagement time groups, two groups generated as dropout and non drop-out learners according to their academic achievement. Distribution of the learners according to this classification is given in Table 4.  As can be seen in Table 5, the time spent on the e-learning portal differs according to the courses. To define whether this difference is significant or not, one-way ANOVA test is used. Results of oneway ANOVA test are given in Table 6.
When Table 6 is analyzed, the time spent on the portal according to the courses according to one-way ANOVA findings, it is found that there is difference in .001 significance level [F (3,323263) =130.017, p<.001]. Because of the sample size, to define how effective the significant difference is, eta and eta square values are examined. It is found as η=.035, η 2 =.0012. According to these findings, it can be said that the significant difference of the time spent on the portal according to the courses has low effect size. Bonferroni test was used to define which groups have significant difference. Post hoc test results are given in Table 7.
When Table 7 analyzed, Bonferroni post hoc test findings showed difference in .001 significance level among the time spent on the portal according to the courses. Significant difference is defined for all courses. Accordingly, the learners who studied e-learning materials of Ataturk's Principles and Revolution History I course, significantly spent more time than the learners who studied e-learning materials of Basic Concepts of Law (MD=2151), Introduction to Economics I (MD=4142) and Basic Information Technologies I (MD=5751). Besides, the learners who studied e-learning materials of Basic Concepts of Law course, spent more significant time than the learners who studied e-learning materials of Introduction to Economics I (MD=1991) and Basic Information Technologies I (MD=3600) courses. Lastly, the learners who studied e-learning materials of Introduction to Economics I course, spent significantly more time than the learners who studied e-learning materials of Basic Information Technologies I (MD=1609) course.
In this study, GPA of each student is included with the time learners spent on the e-learning portal. Standard deviation and GPAs of the learners' in the groups obtained at the end of this analysis are given in Table 8.  According to one-way ANOVA findings (Table 9), academic achievement according to engagement time level differs in .001 significance level [F (2,323263) =3244.123, p<.001]. The effect size was calculated as η=.14, η 2 =.02. Despite the fact that obtained effect size value is low, significant difference in the findings has independent impact from sample size. To define among which groups this significant difference comes from, Bonferroni test is used from post hoc tests. Findings of post hoc test are given in Table 10.
According to post hoc test findings, Bonferroni test is used to find among which groups significant difference comes from. These differences are analyzed via one-way ANOVA. According to engagement time groups derived from the time spent on the system, there is a significant difference in terms of learners' academic achievements. It is found that GPAs of advance group learners are significantly higher than both learners in average group (MD=4) and bounce rate group (MD=6). Besides, GPAs of average group learners are significantly higher than the learners in bounce rate group (MD=2). This finding shows that as the engagement time level increased, academic achievement increased significantly. This finding indicates that learners who benefited from e-learning services are more successful.
As it is seen in Table 8, learners' academic achievement increased as they spent more time on the e-learning portal. One-way ANOVA test is used to define whether the determined differences is statistically significant or not. Findings of one-way ANOVA test of the comparison of academic achievement according to engagement time level are shown in Table 9. Simple linear regression is used to define whether there is a significant difference and correlation between engagement time levels and academic achievement. After simple linear regression analysis, with Pearson correlation analysis there is a significant correlation between engagement time and academic achievement (r =.12, p<.001). Both variables' covariance realized to be significant. Engagement time levels in one term predicts academic achievement with R 2 =.015, p<.001 value. Non-standardized regression coefficient is defined as β=39.33, p<.001. Based on these findings, it is possible to assume a causal relation between e-learning engagement time and academic achievement. The regression curve which shows the open and distance learners' engagement time levels and academic achievement covariance is given in Figure 3. When analyzed, it is seen that there are also some other changes besides covariance. Thus, while the open and distance learners' engagement time in one term explains academic achievement until 60 GPA. The highest engagement time is observed in 60 GPA band. After this point, as academic achievement increased, e-learning engagement decreased. It is considered that this remarkable finding needs to be investigated in future studies.

Discussion
There are some remarkable findings to be discussed in this correlation study based on comparisons between two variables. Discussions according to the findings of the data collected from 323,264 open and distance learners during one term are summarized here. Firstly, it is remarkable that the dropout rates of learners are very low and e-learning system bounce rates are high. It is considered that the reason for this is that only the learners who access the e-learning portal are included in this study. Therefore, it is found out that dropout rate is low, bounce rate is high. It is also seen that the learners' engagement time levels according to the courses are differed significantly. It is considered that the main reason of this difference for the benefit of Ataturk's Principles and Revolution History I course is the quality, richness and diversity of the materials presented on the e-learning portal. This finding supports Liaw (2008), who reported that "e-learning effectiveness can be influenced by multimedia instruction, interactive learning activities, and e-learning system quality". Thus, e-learning materials of this course can be benefited from as a measure to keep learners on e-learning environments.
One of the main questions of this correlation research is whether academic achievement differs according to e-learning engagement time levels or not. At the end of the parametric tests conducted, it is found out that the more time learners spend on e-learning environment, the more academic achievement they get. Therefore, bounce rate group has the lowest GPA while the group which spends more time on the e-learning portal has the highest GPA scores. This finding is in line with Sculley, Malkin, Basu and Bayardo (2009) who found that "a high bounce rate can lead to poor experience" statement. In addition, dropout learners' online time on the system is the lowest, which supports Oye et al. (2012) claims that "active usage of e-learning environments increases academic achievement".
After the regression analysis, which supports the first two research questions findings, it is found out that engagement time level significantly predicts learners' academic achievement when it is calculated by the time they spend in e-learning environment. This finding supports the studies about engagement which increases learning outcomes quality and predicts academic achievement like in Krause and Coates's (2008), Oye et al.'s (2012), and McKenna and Kopittke's (2018) studies. Accordingly, it can be said that open and distance learners' engagement time levels directly affect academic achievement in e-learning environments. For this reason, it can be suggested that measures are needed to be taken to increase learner engagement time in open and distance learning practices. The descriptive statistics revealed that distribution of learners, according to the courses they are registered, is close to each other. Learners' amount of access to e-learning materials according to the courses is Basic Concepts of Law, Basic Information Technologies I, Ataturk's Principles and Revolution History I and Introduction to Economics I. The number of the learners who use e-learning materials of Basic Concepts of Law course is more than the number of the learners of the other courses.

Conclusion and suggestions
In this research, learners' engagement time was categorized as bounce rate, average, and advance. Almost half of the learners was in average group, the number of learners in advance and bounce rate groups was close to each other. When we consider this result and the number of learners in the studied data set, it can be expressed that learners' dropout rates are low for this group.
One-way ANOVA test carried out to find whether there is a significant difference between the registered courses and engagement time. Significant differences are found between the time spent for each course to the results of ANOVA and post hoc tests. According to this finding, learners who studied e-learning materials of Ataturk's Principles and Revolution History I, spent statistically more time than the learners who studied e-learning materials of Basic Concepts of Law, Introduction to Economics I and Basic Information Technologies I. Additionally, the learners who studied e-learning materials of Basic Concepts of Law course, spent more time than the learners who studied e-learning materials of Introduction to Economics I and Basic Information Technologies I. Similarly, the learners who studied e-learning materials of Introduction to Economics I course, spent more time than the learners who studied e-learning materials of Basic Information Technologies I course. In conclusion, it is found that the learners spend their time respectively more in Ataturk's Principles and Revolution History I, Basic Concepts of Law, Introduction to Economics I and Basic Information Technologies I courses.
To define whether time spent on the e-learning portal affects academic achievement one-way ANOVA test is applied. It is determined that the spent time on the system significantly differs according to the academic achievements. At the end of the analyses, it is found out that spend increased time on the e-learning portal, academic achievement increased significantly. This finding shows that the learners who benefited from e-learning services for a longer time became more successful. The advance group, the learners who stayed longer on the system, defined as the most successful group. Accordingly, it shows that the usage of the materials on the e-learning portal affects achievement in a positive way. After the regression analysis which supports this finding, it is realized that the time spend on e-learning environment significantly predicts learners' academic achievement.

Limitations of the study
This research has some limitations, listed as follows:

Future implications and suggestions
Based on the findings of this study, the following future implications can be considered: 1. It is considered that organizations who delivers open and distance education may analyze the learners' high bounce rates in e-learning environments to track the engagement of learners and success.
2. Distinguishing characteristics of the e-learning materials of the courses in which the learners spend more time (Ataturk's Principles and Revolution History I in this research) can be examined. 3. Since the time spent in e-learning environments affects learners' academic achievement positively, it is considered that researches can be carry out in the institutions to keep learners in the system. For this purpose, gamification factors could be integrated to the system, personalization and enrichment of the learning environment could be suggested.
It is possible to make some suggestions for future research in line with the findings of this research and in the limitations of the study. First, learning analytics can be actively used when studying with big data in open and distance learning. The reason of learners' bounce rates can be questioned through qualitative researches. In addition, complex qualitative and quantitative researches can be conducted to find out why learners are interested more in some courses' e-learning materials (Ataturk's Principles and Revolution History I in this research).

Introduction
Job satisfaction stands as one among the best and many researched concepts in organizational psychology and work for two reasons at least. One being those researchers who stand interest in subjective evaluative analysis of the working conditions that include communications, task variety and responsibility (Dormann & Zapf, 2001). Gupta and Joshi (2008) agreed that job satisfaction stands very important as people spend most part of their life at work place. Job satisfaction is also termed as psychological response that one's job (Judge, Thoresen, Bono & Patton, 2003). Job satisfaction points towards an employee's judgment regarding her or his job as the result after evaluating all the time a person worked there (Sieger, Bernhard & Frey, 2010). Organizational culture has been defined in a number of ways differing from one author to another one. It can be defined as the difference of values and the expectations of the workers employed in an organization with different culture backgrounds (Tayeb, 1996). Several researches have been carried out to investigate whether the employee's values, corporate attitudes and behavior affect performance and success of the employees working in organization (Tharp, 2009).
Communications taking place in the same or particular organization are found to be related with higher levels service and performance (Hargie, O & Tourish, 2009, pp. 10-15). Organizational communications may prove to be crucial factor for the purpose to understand the worth of organizational assets which are tangible in nature (Ritter, 2003, p. 50). Internal communication may also help to develop relationships among the organization's, supervisors and employees (Sluss, Klimchak, M., & Holmes, 2008).
Supervisor support may play a useful role to establish the work environment and to provide the information as well as feedback to the employees (Griffin, Patternson & West, 2001). It can be defined as the extent to which organizational leaders value their employees' contributions and also care about their wellbeing. Since all the aforementioned variables play important roles in every organizational settings, this research study intended to apply all the selected variables in the IT based distance learning institutions of Pakistan.

Job Satisfaction
This is a complicated and difficult attitudinal variable related to employee jobs in the organization (Poulin, 1995). It has been defined by (Locke, 1969) as positive state of emotions or pleasurable state of emotions that results from an assessment of an individual's job. The variable of job satisfaction further has a wide range of satisfaction determinants (Cranny, Smith & Stone, 1992). Literature produces certain factors and mentions that satisfaction may be because of the pay a worker draws or the promotion one gets or because of relationships established with the supervisors or other colleagues (Cranny et al., 1992). An overall level of job satisfaction of an employee shows his/her affective reaction to work and other related elements or facets (Cranny et al., 1992).

Organizational culture
A study was carried out in Greece by (Bellou, 2010) to investigate the implications of organizational culture as a predictor of employee job satisfaction found that culture of an organization affects the level of job satisfaction working in the organization. Another empirical research study that took place in America by (Lund, 2003) also considered several types of organizational cultures such as that prevail in the organizations of the selected industry and found that culture/cultures of the organization affect the levels of the job satisfaction working in the organizations.
The culture of an organization depicts an active and lively way manner in which the employees of an organization including the supervisors or the executives develop a shared meaning of their surroundings (Morgan, 1997). Members of an organization constantly interpret different features regarding their working environment and through these interpretations along with the ways in which these are developed, create a culture for the organization (Martin, 1992). In another study carried but by (Testa & Mueller, 2003) in a global perspective in services sector for the purpose to analyze the impact of culture fit on the level of satisfaction of employees working in the organization, it was found that cultural fit with the organization would positively affect the level of job satisfaction among the workers in the organization). Louis Gerstner from IBM as well as Heinrich Von Pierer from Siemens are well known for transforming their organizational cultures (Stewart & O'Brien, 2005). Organizations that value advantage seeking and risk taking behavior also further growth opportunities for employees even after temporary setbacks while those organizations valuing innovative culture are found to be growth oriented (Ireland, Hitt & Sirmon, 2003).

Supervisor Support
Cheung and Tjosvold (2007) carried out a study to investigate the mediating role of job satisfaction with its association among the supervisor-subordinate guanxi and other employee outcomes. Findings of the study concluded that guanxi between the supervisor and the subordinates increase the job satisfaction of the employees working in the organizations and also their ability to increase their participation in the decision-making. The guanxi between a supervisor and subordinate may point to the personal relationships between a supervisor and his or her subordinate's relationships that they develop as a result of non-work related social associations during different working hours (Xin & Pearce, 1996;Warren, Dunfee & Li, 2004). Supervisor and subordinate guanxi was found to be associated with subordinate trust with supervisors (Wong, Ngo & Wong, 2003) also with participative leadership (Chen & Tjosvold, 2006) along with communication which is open-minded (Chen & Tjosvold, 2007) and organizational commitment as well as turn over intentions of employees (Farh, Tsui, Xin & Cheng, 1998). It's because of this increased role of supervisor-subordinate guanxi that this concept in Asian settings, the researchers now gives an increased attention to the guanxi between the supervisor and the subordinate. Research also found that this concept is related with administrative decisions of the supervisors (Law, Wong & Wong, 2000); decisions which are joint in nature and also discussions of open minded nature (Chen & Tjosvold, 2006) and trust of the supervisors in their subordinates and also with organizational citizenship behavior (Wong et al., 2003) Chen and Chiu (2008) carried out a research on the topic to investigate supervisor support and organizational citizenship Behavior. The findings of the study proposed certain new dimensions. These also concluded that supervisor support enhances the job satisfaction of employees working in the organization and person-organization fit. Another important variable in the organization setting is supervisor support for employees working in the organization. Griffin et al. (2001) executed a research study on different teams working in the organization with the intention to analyze the impact of supervisor support on the job satisfaction of the employees working the teams in the selected organizations with different hypothesis. Results of the study showed that supervisor support had a significant positive impact on the levels of job satisfaction among the employees studied.

Theoretical framework
The employees working in distance education happen to be separated by distance of more than hundreds of miles from one another. Many differences originate due to this distance between the members of the organization. A supervisor may belong to a different culture while the other employees being remotely supervised may belong to a different culture with different values. But working in the same organization, they need to interact and cooperate with each other in the best interest of the organization. Therefore, the supervisor support, the timely communication of organization policies and other information are very important and have an ultimate impact upon the job satisfaction of employees.
This study is based on Leader Member Exchange theory (LMX Theory) presented by (Graen, 1976). It simply states that when both the leaders as well as the followers have good terms or high quality exchanges (LMX relationships), in such situations, there would be feeling of mutual trust, they would feel better and accomplish more that would ultimately result in the enhanced performance of the unit or the organization as whole (Stringer, 2006). As discussed by other writers, this theory states that when the leaders and followers have good quality exchanges, then there would be trust, respect for each other as well as an obligation (Graen, 1976;Graen & Schliemann, 1978;Graen, Liden & Hoel, 1982;Graen, Novak & Sommerkamp, 1982) positive or good support, mutual bonds, open communication and distributed loyalty (Graen & Uhl Bien, 1995) and affection (Liden, Wayne & Stilwell, 1993). Some other researchers have also associated enhanced LMX relationships to perceived organizational support (Wayne, Shore & Liden, 1997) as well as higher performance (Dunegan, Duchon & Uhl-Bien, 1992;Wayne et al., 1997).

Proposed Model
There are three independent variables used in this particular research study (figure 1). These include 1) Organizational communication, 2) Supervisor support and 3) Organizational culture. The dependent variable of the study is Job satisfaction of the employees working in the selected organization. The researcher in this study intends to execute the selected model with the variables in the online distance education institution of Pakistan. It proposes that all the selected independent variables affect the selected dependent variable directly and positively. This means that all the selected variables would have an impact upon the overall job satisfaction of the employees working in the selected online distance education institution where both the supervisor and the one being supervised are seated mostly miles away from each other.

Methodology
This section deals with the details of all the statistical analysis of the data used in this particular research study to deduce the results. It explains certain items including the data preparation and execution, brief details about the demographics of the sample and also about the hypothesis testing. Hypothesis was tested using regression analysis. Reliability of the constructs was also conducted. Descriptive analysis of the variables in terms of means, standard deviation was also carried out. Data regarding the entire Virtual University academic staff was obtained from the central online system available to the staff. The entire population comprised of 268 academic staff and their contact details including the e-mail address were also available to the researcher. Since the instrument for the data collection was served online through Google forms and the rate of response is also slow in the case of online survey, therefore the researcher adopted to send the link of online survey to the entire population. It was served to all the academic staff due to the small and accessible number of the entire population. For this purpose, the researcher administered and sent the link of the questionnaire (Appendix 1) to all the staff making use of Internet means to share the questionnaire with all the intended staff members. Out of total 268 questionnaires sent to the staff, 215 were returned and used for the data processing and analysis.
In order to enter and process the data in the computer and SPSS, the researcher coded the data. (Zikmund, 2008) mentioned that data is coded for the purpose that each answer may be identified and classified to some numerical score or any other character for the ease in data processing. Therefore demographic section of this study was coded too. The questionnaire designed for this purpose comprised of three questions regarding demographics. These were Age, Gender and Educational qualification. There were 137 males and 78 females and out of all these, 97 had Master's degree, 113 had MS/M Phil degree while 5 had others degree. 80 of the respondents were less than 20 years of age, those under 30 were also 80, those under 40 were 106 and those under 50 were 46 and there were 3 members in the above 50 years of age group.
For this study, the designed questionnaires were circulated to the academic staff in the completely online distance learning institute of Pakistan. A total of 268 questionnaires were sent out of which, 215 were filled and returned and included the study for final data processing. Response rate of the survey was calculated to be 80.22%. The sample size was finalized as per the discussion in the methodology section. The demographic characteristics of the sample were Gender, Age and Education.

Exploratory Factor Analysis
For the confirmation of items used in the study, an EFA (Exploratory Factors Analysis) was carried out by the researcher for Organizational culture, Organizational Communications and Job Satisfaction. Since Supervisor support construct was adopted, hence no EFA for it. The researcher first used Principal component analysis to identify the main factors or the components of the organizational culture comprising a total of ten questions. The analysis revealed two main components. The "Eigen" values in the first phase showed that the first component explained 44.918 % variance while the second factor explained 11.694% variance. They were examined using "Varimax" rotations. After several steps, three questions were eliminated due to low communal value less than 0.4. The remaining ones were used in the final questionnaire.
The initial scale for organizational communications was comprised of ten questions. The principal component analysis revealed one main component. Items with communality value less than 0.50 were eliminated from the study. For the scale of Job satisfaction, Principal component analysis revealed that there was only one component. One of the Items "How likely are you to look for another job outside the company?" was eliminated for having value less than 0.5.

Reliability of questionnaire/Instrument
To check the reliability of the instrument adopted for this study (Appendix 1), Cronbach alpha measures were used to check the internal consistency of the measures (table 1). As table 1 shows, the reliability of all the four variables used in the study is considerable. The reliability of Organizational communication was 0.886 which is reported as good. In the same way, the reliability measure of Organizational culture was 0.808 which is also a good one. The reliability measure of the third variable Supervisor support was 0.885 which is also considered a good one like the others. The reliability measure of Job satisfaction was 0.889 which is also considered reliable. Correlations (table 2) reveal positive linear associations between the variables under study. It also depicts that there is significant positive correlation among all the variables of the study including the organizational culture, supervisor support, organizational communication and the job satisfaction of employees working in the organization.  Table 3 shows the regression beta values for the proposed model. It shows that beta values for all the variables are positive with 0.244 for organizational culture, 0.267 for supervisor support and 0.322 for organizational communications. The t values are also high mentioning significant relationship between the organizational culture, organizational communication, supervisor support and job satisfaction of employees (p<0.05). Hence all hypotheses H1, H2 and H3 are accepted. The higher values of F=96.448 at (p<0.001) show that the proposed model is significant and therefore its regression values can be used for further analysis (table 4). Results obtained through correlations among the variables were further processed for regression through SPSS. Beta values for all the variables of the study were found to be positive and therefore significant as well. The value (r = 0.760) indicates that the selected independent variables have a positive relationship with the dependent variable of the study. To address the issue of multicollinearity, VIF and tolerance values in SPSS were used (table 5). When a variable carries a VIF value larger than 10, it may call for more investigation. In this study there was no issue of multicollinearity as all the VIF values were less than 10 and tolerance level one. Also there was no issue of autocorrelation as "Darbin Watson" value in SPSS was 1.972 (Neter et al., 1996).

Results
• Organizational culture positively affects the job satisfaction of employees working in the IT based distance learning institutions of Pakistan. • Organizational communications positively affect the job satisfaction of employees working in the IT based distance learning institutions of Pakistan. • Supervisor support positively affects the job satisfaction of employees working in the IT based distance learning institutions of Pakistan. Lund (2003) studied the organizational culture and job satisfaction of employees and found a positive relationship between the two. In another research study by (Testa & Mueller, 2003) conducted to analyze the impact of cultural fit of and job satisfaction of the employees and showed that cultural fit positively affects the levels of job satisfaction of employees working in the services sector. There is a positive relationship between the organizational culture and job satisfaction of employees. Findings of the study in light of the data collected and processed support this claim that there is a positive relationship between the organizational culture and job satisfaction of employees working in the online distance learning Institutes. Empowerment and Teamwork is a very important part of this culture. There are multiple reasons for this, including that there are teams working within these organizations which are empowered to design activities from their own, the ideas that they present to the high ups in academic domain gets considered and followed by feedback as well. In virtual organizations, the load of work is many fold due to the mega enrollment of students in every course, this work load is divided and carried out by the teams established for this purpose. This accomplishment of job through teams adds to the job satisfaction of employees. Another objective of the study was to analyze the impact of the organizational communication upon the job satisfaction of the employees working in the online distance learning institutes. (Giri & Kumar, 2009) carried out a study in India to analyze the impact of organizational communication on job satisfaction of employees and concluded that organizational communication has a positive impact on the job satisfaction of employees. The hypothesis tested for this purpose in the light of the data collected was supported in regression analysis. This is because of the reason that employees who are working away from each other when timely get the organizational policies or the working procedures well on time find it easy to get the assigned work done at time. Since employees remain connected all the time, the achievement of tasks gets easier as they discuss the procedures and later frequently discuss issues that may arise with other colleagues. It can also be because of the reason that there is an effective mechanism through which the employees can put their concerns to the management and also the policies that affect the employees in either case are communicated to the employees in a clear language.

Discussion
The last objective of this study was to analyze the impact of the supervisor support on the job satisfaction of the employees working in the online distance learning institution. The hypothesis developed for this purpose was also supported by the regression analysis. This is in line with many researches in the past including Stringer (2006) and Yukl (1989). Superior subordinate relations and quality of information are very important components of organizational communication. The reasons responsible for this factor include that since the supervisor and the employees work away from each other, in such cases a supervisor extending cooperation and support to the employees working in remote locations enhances their satisfaction as it becomes and establishes an element of trust among them. Employees having support of the supervisor would get timely intimation of the organizational policies and also other areas of their domain such as applying for further education.

Conclusion
This research study was carried out with the intention to analyze the impact of organizational culture, organizational communication and the supervisor support on the levels of job satisfaction of employees working in the online distance education institutes of Pakistan. In doing so, it really made substantial contribution to the present literature on the selected proposed model with variables to be associated with job satisfaction not only in the context of distance learning education but also in the specific context of distance education in Pakistan where this concept is flourishing. Due to the flourishing or new concept of such type of organizations, there is really a need to carry out research on many issues related with these types of organizations including the job satisfaction of employees working in this sector. This study is the first of its kind here in Pakistan and elsewhere as well to have worked on the selected three independent variables, as mentioned above, and their impact on the job satisfaction as all the three really matter a lot in online sector where the supervisor and the one being supervised is seated miles away from each other. Virtual University of Pakistan is the only and purely distance learning institution of Pakistan and its members could be accessed easily. Therefore only one university and access to the members was the limitation of the study.

Introduction
A central role in teacher training is given to teacher-educators, a particular group of professionals with specific responsibilities, expertise and commitments in their respective educational systems (Kelchtermans, Smith & Vanderlinde, 2018). The development of their expertise constitutes an important issue, considering their leading and modelling role in assuring quality and the introduction of technological and pedagogical innovation. However, in a hyper-connected world, the initial and continuing teacher training proposed by higher institutions remains somewhat limited to national borders, with a need to meet the emerging challenge of increased openness and cross-institutional collaboration, both formal and informal, among higher education institutions and practitioners (Inamorato dos Santos, Punie & Castaño-Muñoz, 2016; Nerantzi, 2018). Consequently, teachereducators often deal with the national perspective and are not incentivized to confront international issues. Furthermore, research on the professional preparation of teacher-educators is less widely researched than that of teachers (Kelchtermans, Smith & Vanderlinde, 2018), as is their international professional learning. The deep introduction of technology in the so-called Global South starts to introduce some change in the quality of teacher-educators' professionalism. More research is needed to analyze the effects and impacts of international training on local teaching practices of teacher-educators.
Indeed, fostering international training and sharing of practices between teacher-educator could be a strategy to tackle common issues in a wider and more complex vision of teacher professionalism for both developed and developing countries. Considering this, the aim of the paper is the analysis of the impact of an international blended course on Pakistani teacher-educators' professional learning. A theoretical discussion is presented about the international teacher-educators' professional learning, especially supported by blended learning modality. Afterward, the study about Pakistan teachereducators professional learning is presented and discussed.

Why does the professional learning of international teacher-educators matter?
The word 'teacher-educator' covers a mixed and diverse group of professionals considering the great differences across countries (Berry, 2007;Darling-Hammond & Lieberman, 2012;Kelchtermans, Smith & Vanderlinde, 2018). This paper adopted the general definition of all those in teaching or research who actively facilitate the learning of student-teachers and teachers (European Commission, 2013).
Following the considerations of Snoek, Swennen and van der Klink (2011), the intensive international exchange of learning by teacher-educators will contribute to their increased professionalism. Engaging in an international training course gives new knowledge to teacher-educators but also gives them an opportunity to challenge themselves with the management of resources, building of new relationships and growing a professional network that could play a role in modelling their professional career. Indeed, professionalism is built through a long process based on information, knowledge exchange and studying one's own daily classroom practices, as well as learning with and from peers, enhancing the experience of others (Ritchie, 2018). Active participation in a learning community brings the sharing of experiences and meaning, facilitating a new form of collaboration and an active knowledge building process, with open discussion, sharing, negotiation, and integration of ideas (Gruber, 2018). For this, it is important that teacher-educators be exposed to innovative training, with an international and intercultural perspective too, as supported, for example, by the European Union with the Erasmus programme's mobility or the Capacity Building programme.

The role of Blended Learning to foster International Professional Learning
In recent years, many teacher learning initiatives proposed by higher education for initial or continuing training have been developed through the Massive Open Online Course (MOOC) movement (for a discussion in a European perspective, see Schuwer et al., 2015), including blended learning (BL) to meet the challenge of keeping teachers professionally up-to-date and providing them with professional development opportunities on a continuing basis (Philipsen, Tondeur & Zhu, 2016).
Considering that the definition of BL encompasses different educational experiences, modes of communication, types of technologies, methods of learning, time management and skills involved, it is here understood as the optimal combination of learning and online presence, enhancing the potential benefits of both approaches (Graham, Allen & Ure, 2003), thus becoming a suitable approach to introduce innovation and international added value for international teacher-educators (Carlsen, Holmberg, Neghina & Owusu-Boampong, 2016;Naylor & Gibbs, 2018). It emphasizes a participant-centered perspective, facilitates access to continuing education for new audiences and the better use of educational resources, therefore rethinking the teaching model to adapt them to the specific context. Moreover, a recent review of the literature evidence that BL appears to facilitate learner empowerment more than either face-to-face or fully online courses, with a greater chance of succeeding (Owston, 2018). This condition is valuable if we consider a thoughtful duration (Consuegra & Engels, 2016), the need for professional and peer support (Desimone & Garet, 2015), active learning (Consuegra & Engels, 2016), link and relevance between theory and practice (Wilson, 2012) and the implications for professional identity (Baran, Correia & Thompson, 2011).
Despite the variety of potential described above, the adoption of BL for cross-international teachereducator training is still in its starting phase. In their review about global BL, Spring and Graham (2017) demonstrated that in educational literature there exists a lack of connection between countries and regions. However, BL could be an interesting choice in developing countries looking for flexible and effective solutions to adapt to a challenging context with limited technological solutions. Indeed, BL is based on a non-expensive technical infrastructure, such as a free online learning environment like Moodle, that is already rich in online tutorials in different languages and is well adapted for mobile. Indeed, in BL what really matters is to have a vision of educational technologies, which should be fully integrated into the psycho-pedagogical models.

Blended Learning in the Global South: Focus on Pakistan
In Pakistan, the quality of teacher education still needs improvement through the introduction of innovative strategies and practices, together with economic and social recognition (Dilshad, Rehman, Ahmad & Iqbal, 2010) and new accreditation procedures (Mirza, 2005). The government's budget for education is still very low, especially when compared with the 20% recommended by UNICEF and the 6% agreed upon in the Dakar Framework of Action at the 2000 World Educational Forum.
Pakistan's National Education Policy (1998-2010) observed "the qualitative dimension of the teacher training program has received marginal attention resulting in mass production of teachers with a shallow understanding of both the content and methodology of education" (Government of Pakistan, 1998). Numerous factors affect the low performance and quality of teachers in Pakistan, which include, absence of academic qualification, of continuous professional development and of recruitment on merit. New Teacher Induction Policy 2017 is the most recent education reforms. Right after this reform, the government advertised recruitment of more than 17000 teachers of various cadres and scales with no requisite condition of professional qualification for any post. Indeed, as Halai, Begum, Niaz, Hussain and Baig (2018) stress, education in general and teachers in particular, are mostly influenced by governing political parties.
About teacher-educators' professional developing some practices are supported by the specific national association, in particular about reflexive practices (Khamis & Sammons, 2004). The success of some professional learning initiative stressed the importance of field-based rather than universitybased programmes, with still a limited impact on whole school development. From the results of the research analysis done by Khamis and Sammons (2004), factors involved in hindering the development of the role of teacher educators in the context of schools in Pakistan include: role ambiguity and competition for the available human resource; efforts of the teacher educator not viewed as a priority by school management; self-perception of the teacher educator as an expert and not 'just a teacher'; initial intensity and demands placed upon teacher educators to initiate, plan, and support improvement efforts without the requisite assistance or support; lack of a school vision leading to lack of support from management and system inertia. Considering all these factors, the educational system disregards the teacher educator's professional skills.
In the so-called Global South, the training is increasingly dematerialized in online courses recognized as a viable alternative instructional delivery method in higher education (for a review of BL in Asia, see Tham & Tham, 2013). Also in Pakistan, growing attention has recently been paid to the opportunities created by technology and their implications for economic and social growth. In the rural and urban areas of Pakistan, social network platforms are frequently used, as there is easy access to high-speed Internet and mobile broadband -like 3G and 4G Long-Term Evolution LTE. For teachers and teacher-educators, the integrating technology or e-learning facilities are still in their beginning stages (Bashiruddin, 2011). Social media has influenced the performance of teachers in developing countries like Pakistan. However, teachers face challenges regarding the use of technology in the teaching process; these barriers are mostly related to the training of teachers, support and lack of awareness about the technical skills needed to fully utilize these technologies (Hassan & Sajid, 2012).

Research question
This study sought to answer the following question: What are the main competences developed by Pakistani teacher-educators involved in an international blended course? For this reason, we analyze the professional learning of Pakistani teacher-educators that attended an international blended course between Europe and Asia. In the next section, the formative project is briefly presented.

The programme: Teacher-educators between Europe and Asia
This section aims to present the formative project on which the blended curse was conceived and supported, to give an overview of the international dimension of the training proposed to the Pakistani teacher-educators.
The "Blended Learning Training for Teachers educators" project (hereafter "BLTeae") -for more information, see http://blteae.eu/ -led by Aix-Marseille University (France) is supported by the European Capacity Building Programme. This European programme has the aim of fostering and increasing trans-national cooperation projects between higher education institutions, addressing challenges and cross-cultural awareness. The BLTeae project is oriented to European countries (France, Belgium, Denmark, and Estonia) and Asian countries (Malaysia, Bangladesh, Bhutan and Pakistan) in a collaborative community capable of reflecting on teaching trainers' practices. It responds to the common European and Asiatic issue of improving and revising teacher training programmes. Indeed, improving the quality of teacher training effectiveness is one important schoolrelated factor affecting student achievement in different regions (see OECD, 2014). The BLTeae project was delivered across three years (2017-2019): 1) In the first year, an initial questionnaire was conceived and submitted to the community to understand the needs and skills of teacher educators. At the same time, all the institutions were involved in co-developing content for a training course organized with 20 online modular courses (of which three developed by Pakistan partner) and two face-to-face training sessions (one host in Bangladesh and the second in Pakistan); 2) In the second year, all the members were involved in video sharing their teaching practices (related to the modules) and an online community discussion about the teachers' practices; 3) In the third year, the main activity was the sharing of best practices of teacher-educators to build a common reference curriculum for teacher-educators.
BL is proposed considering the three macro levels: 'personal', 'peer' and 'group'. The personal dimension of teacher-educators in the project is supported by the use of ePortfolios for individual reflection. The peers share in the community what they have learned by using the online resources and later re-evaluate these experiences to see them in new ways that might suggest new practices. In particular, the role of the peers is being active in the sharing of digital video about their teaching practices (Santagata & Angelici, 2010). Indirectly, this online activity could elicit an impact on their local teaching class and, more widely, in the educational community. Learning innovative practices, the embedded use of educational technology in the training course and continue discussing knowledge and personal experiences on a common space could strengthen effective teaching Additionally, the international dimension in the project between the partners makes possible the sharing and critical discussion of common topics and visualization of the links to the local educational context. Teachers are sensitized to the use of adequate and clear communication for the international community using the common language of English; moreover, they critically consider the perspective (for example, the pedagogies of the flipped classroom) of local situations. The international cooperation enriches teachers' practices, facilitating a critical comparison and the sharing of reflections.
Finally, the BLTeae project has a special focus on the design of shared and concrete objectives through the mediation of technologies. Indeed, the BLTeae project proposes the teacher-educators improve teacher practices by joining in common discussions finalized to a shared and joint curriculum. This way, each participant can discover and explore the potentiality of ITC (Moodle, social networks, etc.) for the growing of the collective reflective community.

Participation and Data
In this paper, we focus on Pakistani Teacher-educators group. In Pakistan, Teacher-educators are considered Faculty member involved in the teacher training. They can have a doctoral level and more usually a master degree. Also, doctoral students are involved in teacher training too, working as teacher assisting. To collect data, we adopt the blended ethnography, as a form of ethnography that integrates techniques of traditional face-to-face research with those of virtual ethnography, such as participant observation and face-to-face interviews (Hammersley, 2006). This gives participants a direct voice (Hine, 2000) to obtain better completeness of data (Miller & Slater, 2000). The analysis was carried out starting from three main sources: -traces are written in the online platform by Pakistani teacher-educators; -47 questionnaires at the end of the face-to-face training sessions in Pakistan (35 by a woman and 12 men). The questionnaire was composed of closed and open questions. The closed questions were about the agreement of some items, where 80% strongly agreed with the engagement, the relevance of the topics and the quality and structure of the content. 20% of participants complained about the logistics needed (there was no computer available in the room for the participants) and said that the session time was too short (20 minutes on average for each session); -10 deep-interviews with Pakistani educators carried out in Pakistan (8 women and 2 men).
The written traces, the open questions, and the deep-interviews were qualitatively analyzed. The three datasets were analyzed in a triangulation perspective. All the data were read by two researchers, and a thematic and content analysis was done (Charmaz, 2006;Neuendorf, 2016) in a grounded approach.

Developing Technological Skills
Generally, teacher-educators appreciated the practical competencies developed, in particular, the skills related to how to use social media in teaching, how to do video-clips and how to use their ePortfolio. These three aspects are considered to be fairly new topics in their repertoire. Teacher-educators were interested in understanding the functionality and utility of recent technology: "I have used YouTube video clips, video lectures, feature films, and WhatsApp during class. I have just started using Google Classroom to incorporate eLearning. I plan to incorporate Google Forms in [the] classroom for formative assessment[s]". Also, teachers appreciated best practices that can be quickly adapted to their context.
Pakistani teacher-educators involved in the project had positive attitudes toward the use of technology in teaching ("I [have integrated] technology in teaching since 2008. I have observed that students are fully involved in learning when we integrate technology"), using a variety of tools ("Generally I use video clips from YouTube, and TeacherTube. I share video clips through [my] WhatsApp group and assign students' tasks for future [classes]"). Laptops, desktops, mobile phones, and tablets have been used for Internet surfing & searching, as well as for preparing presentations and watching videos, with growing attention to social networks.

Developing Pedagogical and Instructional Design skills
Furthermore, teachers' attention is centered on the use of ePortfolios ("I will develop the ePortfolio of my students and their teaching practices as they [have] already [developed] it manually"), videos and social media to teach ("To use Facebook as a platform").
Teacher-educators are interested in BL, which they discovered through their participation in the project, which proposes an online platform and face-to-face meetings with an international audience or at national events. Consequently, they propose using BL in classrooms ("I will try to follow [the] BLT methodology in my training"), adopted in their practices: "I have tried blending mediums with [lectures] and discussion methods. The students are better able to attempt analysis questions after watching a video clip and discussion. They are motivated and explore other sources on the topic on their own. They also share their learning with [their classmates] afterward".
BL is interpreted as a suitable and equilibrate introduction of technology in the classroom, with a good integration of activities with both technology and paper: "If the students watch a movie of [a] video lecture, they are given a small questionnaire [in which] to express their opinion about the content, as well as the concepts discussed. This activity can be for individuals or in [the] think-pair-share methodology". This broad interpretation of BL is linked to the contextual resources and the novelty of the concept ("Careful planning is needed in blending as it is still a new concept in Pakistan"), stressing the role of preparing the lesson with the use of technology: "Blending needs careful planning and monitoring. Some students, although users of social media, are more into leisure activities and have to be guided in [their] use of technology. Some students take it lightly so they have to be motivated".
The introduction of technology in learning has some rapid advantages, such as leading to more active student participation ("Student classroom participation increased. Absenteeism [lessened] and they started to work enthusiastically on the given online assignments. They became more open in giving their opinions and participating in discussions"), motivating the teacher-educators to continue to use it.

Developing Interpersonal and Intercultural skills
Thanks to these experiences, teacher-educators became more aware of the introduction of more cultural and social aspects in the instructional design of their lesson, valorizing the interactive side. Indeed, all teachers appreciated the sharing with international trainers and participants ("[It is] good to share and exchange ideas, good practices, challenges and situations", "Exposure to international perspective[s] [and] learning from each other [facilitates] future collaboration"). Indeed, from face-to-face training, they have had the opportunity to spend time with international participants, sharing formal and informal time together ("Yes, get to know their culture, personal life, hobbies and personal experiences in different subject areas"; "Yes, I enjoy [the] exchange as they [clarify] those concepts which I could not ask during session"; "Increases cultural awareness [and] build[s] good relationship[s]"; "Because different cultures have different taste of customs"). The motivation to appreciate the international exchange allows them to appreciate experiences across the globe ("To know … the new technique[s] in teaching internationally, we learn many things directly related to their culture"; "It is important. It encourages the authority to provide and arrange more training sessions for us"). Generally, teachers express the value of the international exchange, stressing the need for more sessions, including informal ones ("Provide social trip[s] to know the participants better"). The international confrontation also helps to increase motivation in the challenge to renew their teaching practices, as expressed by the teacher-educators: "The training (4 days) has not only provided [us] with knowledge and skills in blended learning but also [brings] together with the teaching community from many countries. It has allowed [us to] share common goal[s], common difficulties and common reflection[s] which will help each institution to move forward.

Discussion
The results of this study show the engagement and the willing of the teacher-educators about their professional learning, despite the difficult working conditions as of educational system in Pakistan. This international engagement has given them the opportunity to develop and consoled technological skills; pedagogical and instructional design skills; interpersonal and intercultural skills. All these skills could directly improve and increase the quality of teacher training. Teacher educators in Pakistan, therefore, need to seize the opportunities of international formations in order to make up for the limits of the national system. In the same time, they need to be supported in how to implement innovation and new practices in the local context. Indeed, as shown by Ali (2018) pedagogical changes through teachers cannot be effective without addressing critical aspects of the classroom and school structure and culture, improving working conditions like basic facilities and structural resources.
At the same time, it is also important to focus on the quality of the design of appropriate international training, which take into account the possible limits and challenges of the context. The design of innovative learning courses, like BL or MOOC, is strategic for today's international globalization. BL is based on international collaborative learning and can be particularly useful in designing interventions, with a scenario that includes social interaction, open and complex issues requiring investigation and active engagement and shared knowledge objects and collective efforts to advance knowledge. It becomes interesting to reflect on its benefits and challenges (King, Luan & Lopes, 2018) and how the students will transfer the skills of reflection tested during the online course in their future professional life. Furthermore, the online learning environment could be a boundary between the formative and the professional dimensions. Certainly, the adoption of BL is not easy to implement nor is it able to guarantee satisfactory results.
In an international BL design, collaboration is particularly important for the learning community. Indeed, the adoption of technology for collaborative learning situations include mainly knowledge creation; public sharing of practices and opportunities for continue and guided reflections about local practices and international innovations and trends, which need to be adapted and changed. Also, the collaborative design of BL for teacher educators have to stress the planning of activities around shared objectives on which the teacher educator have to feel responsible, with a strong local impact. In this way, creativity and reflections are engaged in a long term process of professional developing, open across local and international communities and institutions, supported by flexible tools adapted to, often, limited technological resources.
Finally, if well designed, BL from an international perspective could be a space in which to improve professional competence and transversal skills, subsequently improving reflective and intercultural skills, considering that learners have to deal with complex, ill-defined working contexts (like periods of uncertainty and transition) in their personal and professional lives (Lakkala, Toom, Ilomäki & Muukkkonen, 2015). Thus, the design of a BL course could help to support a new generation of international teachers who are connected beyond national borders.

Conclusion
In this paper, we considered the development of complex professionalism such as that of the teachereducators and how this can allow participants to take advantage of international networking, which today technology makes possible but is not yet fully proposed by higher institutions. Indeed, as stressed by Kelchtermans, Smith and Vanderlinde (2018) "there is a need to raise awareness of the different and distinctive national and institutional contexts teacher educators are working in and how they affect their practices as well as their opportunities to develop professionally" (p.10). Quality of teacher-educators training in different parts of the world -and in this case Pakistan -could take advantages by a continuous connection between people, knowledge, and communities, opening new space of discussions and valorising contextual educational experiences outside of the occidental mainstreaming in teaching and learning.

Introduction: open education policy for contemporary education ecosystems
In education, we are witnessing a shift from analogue to digital across both content and practice -a shift which enables, though certainly does not guarantee, a parallel move from closed to open. The philosophy of the Open Education (OE) movement is based on the idea that knowledge is a public good which should reside in the public domain, for everyone to share, use, and reuse (UNESCO, 2002;Gourley & Lane, 2009;Andrade, Ehlers, Caine, Carneiro & Conole, 2011;Rolfe, 2012;McAndrew, Farrow, Law & Elliot-Cirigottis, 2012;Abeywardena, Tham & Raviraja, 2012;Willems & Bossu, 2012;Jacobi & van der Woert, 2012).
Digitally-driven forms of openness such as Open Educational Resources (OER) and Massive Open Online Courses (MOOCs) have been heralded as greatly beneficial to learners and educators (Lane, 2009;Ehlers & Conole, 2010). The concept of OER as an enabler of 'universal' education was introduced in 2002 by UNESCO, defining them as educational resources that are openly available and modifiable by anyone, without the need to pay royalties or licence fees; similar perspectives have been echoed by the OECD (2007), and by UNESCO (2012). However, this definition is evolving and adapting to the changing technological and social landscapes, and to legal and policy frameworks regulating educational systems and markets. The rise of OER has further enabled a wider discussion about Open Educational Practices (OEP) which recognises a diversity and history of initiatives opening-up access to education, embracing openness as a pedagogical approach that involves networked and collaborative open approaches to learning and teaching, as well as the creation and use of OER (Havemann, 2016;Cronin, 2017).
International OE declarations (e.g. Cape Town, 2007, Paris, 2012 have called for an OE ethos to be embedded within wider education policy, as it is understood as a key spur to democratise access to quality education. Nowadays, a series of international initiatives are attempting to stimulate action on educational and scientific openness, including guidelines to foster sustainable policies (Zajda, 2005;UNESCO, 2012;Swan, 2012;UNESCO & UNICEF, 2015;Deepwell, Weller, Campbell & Wilson, 2017 ;Amiel, da Cruz Duran & da Costa, 2017;Hecker et al., 2018;OpenMed, 2018). Yet, the question of precisely what sustainability represents within OE policy continues to be debated (MacKinnon, Pasfield-Neofitou, Manns & Grant, 2016;Oliver & Cairney, 2019).
Most European Member States now include opening-up education among their education policy objectives (Inamorato dos Santos et al., 2017); in addition, OE national commitments have been made via the Open Government Partnership (OGP). However, most initiatives still tend to focus on the provision of OER rather than fostering the development of OEP (Conole, 2012). Furthermore, in our view, OE supranational and national policy guidelines, tend to focus on fostering the production and dissemination OER or MOOCs, and thereby treat these somewhat in isolation from the wider education sectors, and the current social and economic contexts in which these sectors operate. Consequently, our aim in this paper is to present a panorama of the current OE policy-landscape which includes some key contiguous issues, which are transforming educational ecosystems and have the potential to derail or pose challenges to the implementation of OE initiatives.
These issues are, in the first instance, datafication of education, as nowadays, as data is used to monitor almost every educational and research activity, affecting policy-making in education and science (Mandinach, Honey & Light, 2006;OECD, 2015). Secondly, we review a recent suite of copyright reforms, as changes in intellectual property (IP) legal frameworks can drastically change and limit public access to knowledge and information (Nobre, 2017). Finally, we discuss the challenges of unbundling and open-accreditation systems, as the roles of these emerging practices need to be considered if they are to be harnessed in the service of universal access to not only knowledge, but, recognised credentials (Swinnerton et al., 2018).
The paper therefore presents a landscape review of OE policy, combined with our analysis of the impact of these key contiguous issues, forming a framework for sustainable policy development. In addition, we discuss our use of this framework as the basis for a series of policy co-creation workshops that aimed at ensuring that co-created policies have a real impact in the target community, following the recommendations for successful policy implementation given by Macintosh and Whyte (2008) and Oxman et al. (2010).
The workshop was piloted in October 2018 during two international events, the OpenMed project final conference 1 (Rome) and the OE Policy Forum 2 (Warsaw). We contrast the feedback and data obtained from the pilot workshop participants with our findings from the reviewed literature to identify the key elements needed to foster sustainable OE policies.

Context: where does openness stand in educational policy?
Several high-profile OE policy developments have occurred in recent years. UNESCO has fostered OE policy discussions from the 1st Global OER Forum 3 to the 2nd World OER Congress 4 . Also, the EC's OpenEdu Policies project 5 has analysed OE policies across the 28 EU Member States, identifying different typologies of policies aiming at opening-up education. Also, the OpenMed project 6 has published guidelines for OE policymaking in South-Mediterranean Countries. Furthermore, in the last years some countries have included OE-related commitments within OGP National Action Plans (NAPs), showcasing how OE can interconnect within wider policy actions and priorities. In the following pages we analyse these policy developments, underlining whether the three issues introduced above (datafication of education, copyright reform and OE accreditation) are somehow considered within the current OE policy-landscape.

From Paris to Ljubljana: supranational initiatives
Recent years may have given the impression of a gradual saturation of international policy recommendations on OE, without much take-up by public bodies, perhaps exemplified by the 2012 Paris World Declaration on OER, which aimed to raise the OE awareness of governments and institutions (UNESCO, 2012;Pawlowski & Hoel, 2012). To foster direct interaction between stakeholders, there was a need to move beyond advocacy, and the 2nd World OER Congress (Ljubljana, 2017) was therefore designed to be the culmination of five phases, each of which produced tangible results.
The first phase mapped the state of OER globally by surveying government and stakeholders. Responses were received from 102 countries and over 600 stakeholders responded, and these results are summarised in OER: Global Report 2017 (COL, 2017). This led to six regional consultations with 105 countries, building towards the 2nd World OER Congress. The second phase of the Congress produced the Ljubljana OER Action Plan 2017 (UNESCO, 2017), and operationalised it with 30 ministries committing to it through a ministerial statement 7 , fostering, as a third phase, the As expected, the state of the art across the EU is a composite of alternative approaches and levels of engagement. For example: the Estonian Lifelong Learning Strategy supports platforms for sharing OER amongst teachers; Germany's Mainstreaming OER programme aims at fostering OER and MOOCs; Ireland has a dedicated funding mechanism with emphasis on Open Access that includes OER; in Italy a number of OER and MOOC-related activities have been started without any direct public policy support; the Netherlands' policy aims at modernising HE with a strong component of OER; Poland's Programme for Knowledge Education Development explicitly states that all EU funded resources should be openly-licensed; and, in Scotland, their OEP plan works to build capacity of educators, based on the experience of their leading national universities.
Despite the diverse local approaches, the aforementioned study has identified some common enablers for OE to thrive, including prioritising the development of OE policies alongside with raising awareness of OE among leaders and educators by building and developing capacities for educators to empower and incentivise them in adopting OEP, while supporting grassroots communities and coalitions to foster OE initiatives.

Open education in the Open Government Partnership
OGP is key arena in which drives toward greater openness are interacting with education and related policy concerns. To understand how states pledge to foster OE, 216 education-related national commitments were retrieved from the OGP database, and those focusing in OE were reviewed.
As OE-related commitments are co-created by the civil society, OE advocates and education ministries, this collaborative method is a good practice to encourage governments to support OE initiatives, as can be seen below.
In Brazil, thanks to the work of the Iniciativa Educação Aberta 10 and Educar Digital 11 , the country has pledged to provide a platform for the continuous use and adaptation of OER, valuing the plurality and diversity of Brazilian education 12 . Chile, through the work of the Library of Congress and the Open Government Academic Network 13 is developing an OER-based competency framework for citizenship education at school level. The US, and because of the advocacy of SPARC 14 , has committed to expand access to educational resources through open-licensing and technology, with a view to thereby increasing access to high quality education and reducing the cost of educational opportunities in the US and around the world. In the case of Slovakia, thanks to the efforts of the Alliance for OE in Slovakia 15 , the country has committed to identify existing teaching and learning materials that can be openly-licensed towards promoting the reuse of educational resources.
In the case of Greece 16 , its Ministry of Education, Research and Religious Affairs has proposed the development a platform to make OER available to the public, educators and students. In Romania 17 , a joint effort by the Ministry of National Education and the Romanian OER Coalition aim at providing a Virtual School Library including OER especially for secondary education. In Spain 18 , the Ministry of Education, Culture and Sports aimed at promoting the use of OER, guaranteeing that educational resources produced with public funding are accessible to all.
As noted by Gondol and Allen (2015), "Support from national governments can help accelerate the open education movement both directly through supportive policies and projects, and indirectly by promoting awareness and support within civil society" (p. 275). However, and despite that improving access to education is widely discussed in the OGP commitments, concomitance between stated intentions to foster OE, and the commitments themselves is lacking, as these are mostly focused on the provision of platforms, rather than on much-needed capacity-building and OEP.

Contemporary issues for OE policy
OE initiatives form part of a larger education ecosystem (Bindé & Matsuura, 2005;Jacobi & van der Woert, 2013;Thorne, 2016;Alevizou, 2017), which in turn exists in an interdependent relationship with society, culture, economy, and governance. Consequently, OE is not only driven or impacted by OE or even education policy, and OE policy-makers must take a wide-angle view of the landscape 10 Iniciativa Educação Aberta: https://aberta.org.br within such practices occur. Therefore, aside from explicitly OE-focused considerations, our literature review and subsequent workshop discussions facilitated the identification of three policy-sensitive areas where participants expressed the need for support: 1. Use of data in education and educational policy; 2. IP licensing, copyright and copyright reform; 3. Unbundling and Open Learning accreditation.

Datafication of education
We live in a datafied society (van Es & Schäfer, 2017), consequently, data has pervaded different domains of education, including policy-making. Learning analytics and educational data-mining are often perceived as a panacea for optimising learning and increasing efficiency through personalisation and data-driven interventions (Baker & Inventado, 2014). This tendency, widely known as 'datafication' of education (Selwyn, 2015;Lupton & Williamson, 2017), turns students and learning activities into data-producers for statistical and algorithmic analysis, used to validate arguments, rearticulate educational discourses and, construct policy (Williamson, 2016).
Education-related data is neither transparent nor innocuous. Data has become the key element to assess learning-performance in education, and the need to produce data to justify educational activities is jostling against traditional imperatives such as curricular design (Atenas & Havemann, 2019). When developing policy to support OE, we need to consider the relationship between datafication of education and the broader rise of surveillance capitalism (Zuboff, 2015), as performance-data can be used as currency when sold on to third parties, while offering an apparently free service.
Acquiring services from for-profit ed-tech providers and publishers carries the risk of tracking and monetising data generated as a result of learner interaction with platforms and content, which may well include OER, MOOCs and Open-Textbooks (Anderson, 2013;Jones, Ryberg & de Laat, 2015;Rienties et al., 2016). As van Dijck and Poell (2015) have noted, the main MOOCs corporate providers "are built on the same mechanisms underpinning the ecosystem of connective platforms: datafication, (algorithmic) selection, and commodification" (p. 2675).
Learners' data, when crossed with socio-economic data released by governments can provide tools, opportunities and a landscape perspective to aid understanding of a society's key educational challenges, and supporting policymakers to develop strategies to improve education; however, it can open up windows for surveillance, discrimination and unethical uses of data by tech corporations and governments (Kupchik et al., 2009, Fuchs, 2013Srnicek 2016;Sadowski, 2018). Therefore, at policy level, Mapstone, Buitendijk and Wiberg (2014) recommend, "public and philanthropic opportunities for supporting online learning" (p. 14), instead of fostering opportunities for venture capitalist investments operating under a freemium model.
Critical perspectives questioning the ethical implications and possible ramifications of tracking students' behaviour are needed. In this scenario, students need to become data literate, to understand how society operates and how their data is used. Consequently, we argue that open data should be reframed as OER (Atenas & Havemann, 2015;, presenting educators with opportunities develop the literacies people need to participate in the datafied society, as for Eynon (2013), "access and use of open data is unlikely to be equally available to everyone due to existing structural inequalities" (p. 239).
Therefore, the use of data to foster problem and research-based learning activities allow students to learn and experiment using the same raw data researchers, governments, civil society, international organisations, and policy-makers generate and use, to foster information, statistical, scientific, media, political, critical thinking, collaborative and citizenship skills, narrowing the participation and knowledge gap (Johnson, 2014;Atenas & Havemann, 2015;Manca, Atenas, Ciociola, & Nascimbeni, 2017).

Copyright reforms
The OE community has placed a high priority on communicating the value of openly-licensing educational materials to facilitate their reuse and adaptation (Kapitzke, Dezuanni & Iyer, 2011). OER advocates have especially promoted the use of the Creative Commons licensing framework, launched by Lawrence Lessig in 2002 19 , the same year in which OER was defined by UNESCO, and the Budapest Open Access declaration was issued. While open-licenses are a logical mechanism to support the democratisation of knowledge, it is worth examining the role of copyright, and the trends towards increasing restrictiveness.
Education is adversely impacted by the limitations placed on access and dissemination of knowledge and information, which are imposed by copyright and intellectual property legal reforms in recent years, as publisher interests have been prioritised. Copyright infringement thereby becomes the norm rather than the exception, threatening to criminalise anyone who uses copyright material for educational purposes.
The current Copyright Reform 20 in Europe is imposing barriers to the fair use of digital content. According to Nobre (2017), copyright must empower teaching and learning, but this reform can have a severe impact in the current EU education and science landscapes, affecting national and institutional policies and commitments, by intervening in three main areas: cross-border uses of digital content in education; text and data mining for scientific research; and preservation of heritage in the cultural sector 21 .
The EU directive on Copyright in the digital single market 22 aims at harmonising rights across the EU, however, it remains uncertain whether these exceptions will achieve a fair balance between the interests of rights holders, and the users of the copyrighted material. Furthermore, the debate has been asymmetric, as the voice of the publishers' lobby has seemingly been better heard, leading towards corruption and obstruction 23 of the negotiations between educators, scientists, universities and the EU Parliament 24 . Therefore, according to Communia, 25 exceptions and limitations to copyright for education should allow access and reuse of copyrighted content in different formats across borders as currently, the EC focuses only on digitally-supported education leaving unharmonised a large spectrum of non-digital educational activities such as music or arts teaching. The US is meanwhile attempting to postpone and modify the rules for the eventual transfer of copyrighted material into Public Domain, through international agreements (Lessig, 2013), creating a series of debates amongst governments, educators and scholars in the EU 27 , Uruguay 28 , US 29 , New Zealand 30 31 , Colombia 32 and many others, as an instrument used to strengthen copyright in the international arena is the Trans-Pacific Partnership (TPP) 33 , which has aimed at increasing the copyright length retroactively, based on the Sonny Bono Copyright Term Extension Act, which includes strong legal enforcement for copyright infringement (Fergusson, Mcminimy & Williams, 2015;Travis, 2016;EFF, 2017).

Unbundling and accreditation of open learning
In the last decades, Higher Education (HE), has been subject to the forces of increasing fees and marketisation, massification and internationalisation, and prevalence of technology in delivery, both in the transfer of distance mode provision to fully online, increasing blendedness of the campusbased experience. At the confluence of these forces has been the growth of 'unbundling', the separation of constituent parts of educational services such that they might be recombined in novel ways, which may bring various potential benefits. Of particular interest in the OE community has been the question of how to facilitate learners to gain credentials, as well as knowledge, via open learning routes (Tuomi, 2013).
A key role in this unbundling has been played by MOOCs, which had started as an exploratory form of social, online, peer-to-peer driven learning, but in recent years became the focus of commercial platforms and therefore, somewhat notorious as OE's most celebrated and critiqued initiative. Platforms, their university partners and the press widely touted MOOCs as a revolution in learning, owing to their free and open online enrolment and close association with elite US universities. As the hype of the MOOC phenomenon cooled, course formats, platforms and audiences have diversified, creating low or no cost opportunities to develop skills, encounter specialist and cuttingedge knowledge from researchers, and potentially earn micro-credentials. Consequently, UNESCO (2018) have argued that accreditation mechanisms should be adopted in order to formally recognise the learning acquired through open courses.
For Swinnerton et al. (2018), however, the unbundling of HE has "followed the neo-liberal economic logic which has shaped priorities and relationships across all areas of public policy since the 1980s; the influence of internationalisation and a variety of concurrent business models has been particularly visible in HE -especially in the UK as in other English-speaking countries" (p. 3  32 Colombian biologist cleared of criminal charges for posting another scientist's thesis online https://www.nature.com/ news/colombian-biologist-cleared-of-criminal-charges-for-posting-another-scientist-s-thesis-online-1.22057 33 Trans-Pacific Partnership https://ustr.gov/trade-agreements/free-trade-agreements/trans-pacific-partnership/tpp-full-text While Connor (2014) argues that "unbundling the teaching component of faculty work to assign tasks such as facilitation, assessment and academic advice to specialised staff improves both the quality and the cost effectiveness of teaching" (p. 3), for Holmwood (2013) on the other hand, "the playing field is not so much levelled as tilted in favour of for-profits", who "can enter relieved of university functions other than those of teaching at the lowest cost".
Educators and learners are at risk in a market-driven model. According to Lynch (2015), the social idea of HE as a common good has been supplanted by the idea of training institutions serving a corporate labour market. For Robertson and Komljenovic (2016), as part of this commodification process, universities are partnering with the private sector, developing new business models by providing those excluded from provision with MOOCs, which then go unaccredited and unrecognised.
The current tendency towards a policy vacuum in terms of credentialing of open learning risks this space being filled by hype of for-profit ventures that are normalising transformations in HE, which affect the most disadvantaged groups, as well as promoting precariousness in academic jobs. If we look at these arguments, we can see that MOOCs are seen as a business opportunity instead of a means to provide universal access to knowledge, and that currently MOOCs are acting to widen social inequalities by catering largely for the already educated (Bass & Eynon, 2017;Czerniewicz, 2018;UNESCO, 2018).

Open education policy co-creation workshops
In addition to conducting the review of policies, OE literature and contemporary issues discussed above, we identified a need for a policy co-creation method which would enable a range of stakeholders to build capacity in OE policy development and consider policy elements in relation to their local contexts.

Workshop design and method
The aim of the workshop is to foster peer-learning amongst policy-makers and bench-learning from global successful OE policies in line with the recommendations from the literature review, the Ljubljana Action Plan (UNESCO, 2017), and the JRC study results (Inamorato dos Santos et al., 2017). Therefore, we provided policy stakeholders with a common knowledge base, a canvas for OE policy design based on the business canvas methodology 34 and a set of change cards adapted from those created by the UK Policy Lab 35 , grounded on the participation and co-creation standard developed by OGP (2017OGP ( , 2018. The canvas and cards are used to foster the development of sustainable OE policies at institutional and national level. To design the workshop, we followed the guidance given by Klein, Lankhuizen and Gilsing (2005) that describe the most common errors that lead to failure of policy implementations, and by Sanderson (2002) who showcases a series of recommendations for policy evaluation and guidelines on use of evidence to build policy, to enable successful policy implementations.
The workshops aimed to assess whether or not OE stakeholders considered the elements discussed in the literature as crucial, relevant or unimportant to foster OE policies, to identify elements that are 34 Business Model Canvas: A Simple Tool For Designing Innovative Business Models https://www.forbes.com/sites/ tedgreenwald/2012/01/31/business-model-canvas-a-simple-tool-for-designing-innovative-business-models/#a63e30b16a73 not mentioned in the literature to ensure the impact and success of OE policies, as according to Marsh and Connell (2010), "What constitutes success can differ according to the perspective and/or interests of the participant in, or observer of, the policy process" (p. 581).

Workshop elements
The OE policy elements considered within the workshop are drawn from a diverse range of literature.
A key influence was Haddad and Demsky (1995), who recommend to assess the sector drawing on data, research, experience and international knowledge, through a contextual analysis the sociopolitical, economic, demographic, and cultural conditions, evaluating the interest groups, their rationalities and their roles in education change to foster what Thompson and Cook (2014) call global policy convergence.
During the workshop participants are asked to consider key elements drawn from our review of literature, such as the need to involve a wide range of processes and partners in the co-design policy process (Spillane, Reiser & Reimer, 2002;Zajda, 2005, Pawlowski & Hoel, 2012Oliver & Cairney, 2019), and addresses the need to consider the context, for example, any socio-cultural issues at play, and international policy bench-learning (Ball, 1998;Phillips & Ochs, 2004;Start & Hovland, 2004;UNESCO, 2013). Furthermore, participants should identify the key stakeholders needed to develop and implement the policy (Bell & Stevenson, 2006;UNESCO, 2015;Inamorato dos Santos, Punie, & Castaño Muñoz, 2016).

Outcomes of pilot workshops
The workshop methodology was piloted twice at the OpenMed conference (Rome), and the participants were a varied group of stakeholders from Egypt, England, Italy, Jordan, Morocco, Palestine and Spain, comprising 20% females and 80% males. The third pilot took place at the OE Policy Forum (Warsaw), with participants from Germany, Malta, Poland, Romania, Spain, Slovenia, Sweden and The Netherlands, with a group of 40% females and 60% males. Across the pilots, each workshop lasted for two hours, the participants' age ranged from 28 to 55 years, and their roles included HE senior management, government advisors, educators, OE advocates, policy-makers and civil society leaders. At the workshop, participants were asked to discuss a range of possible policy elements. Below we provide a synthesis of their reflections.
The participants agree that the key process is collaboration, bench-learning and co-creation with a community. This requires involving as partners educators, researchers, librarians and copyright experts, institutional senior management, government advisors, and local and international OE and policy experts, as they can provide a landscape perspective on the context, because considering local needs and cultural approaches in education are key to ensure the successful implementation of a policy. Therefore, stakeholders need to support senior management, students, educators, unions and quality assurance agencies in understanding the value of the policy.
In relation to solutions and approaches, participants favoured reviewing Open Science, Open Access and OE policies and declarations, copyright regulations and accreditation and credit transfer systems at national international level, to ensure coherence amongst policies while identifying potential policy opportunities at local level. They also discussed the need for regional platforms to open resources, the importance of fostering capacity building in copyright, open-licensing, Open Science and OEP. Additionally, they highlighted the need to promote certification for learning recognition and to foster regional collaboration for cross-country accreditation.
These elements were regarded as essential to overcome some challenges and barriers, examples of which were lack of ICT skills, lack of copyright and IP literacy, and lack of awareness of open practices, amongst educators, senior management, and policymakers. These challenges were highlighted as preventing the recognition of OEP for career progression, and furthermore, preventing funding being unlocked to support implementation.
For the participants, the key elements to support and enable a policy are recognition and accreditation of open learning, managerial support and funding to support OE activities. Therefore, as evidence, the participants agreed that evaluations from learners and educators on OE-based programmes, and examples from international good practices, data on cost-benefits of OER and attainment data, national educational data and performance data, can help persuading senior management and the government, to understand the value of OE.
Policy beneficiaries were primarily understood to be learners and educators, while some mentioned families, as OER may lower the cost of buying resources (e.g. textbooks). Moreover, the participants mentioned that governments and public universities should benefit, as OEP may widen participation in education. However, participants highlighted the importance of assessing the risks that OE policies may face, including, management's increasing demand for data collection from learning activities to perform analytics, modifications of copyright regulations, lobbying and collusion by publishers and ed-tech vendors, and furthermore, change of management and governments, as these elements can affect or derail and OE policy.
The participants were enthusiastic and their feedback positive, as the workshop provided OE advocates and stakeholders a space to share ideas and design and draft a policy by exchanging expertise and experiences, fostering participative and inclusive dynamics, as the cards enabled discussions about the different elements of the canvas from an international perspective, allowing the participants to learn each other and to acknowledge their own knowledge gaps. After each workshop, the participants gave us ideas for improvement, such as simplifying the language of the toolkit, giving some people specific roles in the tables, and printing out the instructions so these can be revised during the sessions.

Conclusions and considerations for open education policy-making
Our central theme in this paper is that OE does not occur in a vacuum; ergo, policies aimed at fostering sustainable growth of OEP must acknowledge that such practices sit within a wider landscape of social, economic and educational ecosystems. This landscape includes widespread efforts to democratise access to research and knowledge through Open Access and Open Science policies; however, the development of national and institutional-level policy focused on opening educational content and practices appears to lag behind. Supranational calls to action have been met with a range of specific national initiatives and commitments, and some degree of take-up from institutions and organisations, but in the main, there remains much further to go to embed OE as a core element of education policy and sector activity. Consequently, in preparing this paper, we sought to examine relevant literature and policies, based upon this review to design a policy co-creation workshop, and further to contrast the results from the workshops against our review of policy and issues.
The results of the pilot workshop discussions tend to confirm the findings of our literature and policy review, indicating that the workshop methodology operated as intended, as a facilitation and capacitybuilding exercise to structure and deepen discussion around policy issues. Most of the challenges identified during the workshops are connected to copyright and technological developments and to social and behavioural developments, showing a certain dichotomy between the technical and the social understanding of education policy (Bell & Stevenson, 2006).
The workshop results confirmed that OE policy needs to take greater cognisance of three key issues for education: copyright, open accreditation, and datafication. Copyright reform poses challenges that must be considered and addressed at policy design level but also at implementation level, to provide a wide range of actors with the requisite level of copyright literacy. Accreditation of learning through OE, participants agreed this should be addressed by policy beyond experimentation, as learning recognition in fact can be a key OE enabler.
It is necessary to raise awareness of the risks of over quantifying educational activity through analytics and metrics. The relationship between technological developments (including commodification of data) and policy, while less addressed within OE literature, was discussed deeply during the workshops, leading to a consensus view that OE policy should both adapt to, and try to influence, technological developments. Participants tended to agree that policy can hardly anticipate the outcomes of technological innovation, as these are likely to continue to grow at a rapid pace while technology corporations are currently operating in a semi-regulated way, pushing the limits of the possible and ethical at times. Therefore, a key challenge for sustainable OE policy is to remain fit for purpose and as "technology neutral" as possible, while ensuring it addresses not only educational activities, but also, the social and ethical effects and impacts of technologies in learners' and educators' lives.

Introduction
Successful adoption of Open Educational Resources (OER) in higher education arguably depends on the effectiveness of the materials. If an argument is made that OER is at least as effective as traditional non-free closed (proprietary) educational resources (CER), then why not choose the free, more adaptable and accessible OER? There may be other factors to consider, such as development of the materials and potential administrative roadblocks. But regardless of these factors, effective OER should be considered a viable substitute to CER.
In this paper, we report on the results of a study designed to compare free OER to the proprietary CER used in Calculus at our university. Four sections of first-semester Calculus were taught face-toface during the Fall 2017 semester, two using an open textbook and web-based homework system, and two using a proprietary textbook and homework system. Performance measures were used to compare student achievement between the two groups.
Using statistical analyses, we develop multilinear models for predicting student success, and we demonstrate various significant effects measured to within a probability of 5%. First, the OER students accessed their web-based homework earlier than the CER students, which contributed to higher scores on their first exam, on average. However, this effect was not sustaining, as the CER students performed better later in the semester. Second, student preparation prior to class was an indication of better performance, and this effect is sustaining, affecting all performance measures throughout the course. Third, students in the CER group performed better on their homework, which directly correlated with final exam scores. To understand the reasons for this difference, we consider student comments concerning the quality of experience using the OER and CER homework systems.

Literature Review
In his 2016 survey paper, J. Hilton (2016) investigated the state of OER by summarizing the literature on effectiveness and perceptions. At that time, the landscape included a total of just 16 published articles, of which nine focused on the efficacy of OER textbooks as a replacement for CER textbooks. In his address at the 2018 open education conference, Hilton (2018) emphasized that many more studies are needed to fill the gap.
Development of OER courses typically involves replacing traditional textbooks with open textbooks and online resources. Several studies have shown that such a transformation does not significantly affect student performance and or progression, such as Allen, Guzman-Alvarez, Smith, Gamage, Molinaro and Larsen (2015), Weller, de los Arcos, Farrow, Pitt and McAndrew (2015), Hilton, Gaudet, Clark, Robinson and Wiley (2013) and the papers reviewed in Hilton (2016). Some studies indicated an improvement in performance and retention using OER, such as the open psychology implementation in Hilton and Laman (2012), and the secondary level science study in Robinson, Fischer, Wiley and Hilton (2014) based on end-of-the-year state standardized tests. In Hilton and Laman (2012), their study showed a withdrawal rate of 7.1% for the OER group compared to 14% for the traditional group.
In determining the effectiveness of OER, one approach is to define and attempt to validate statistical hypotheses. In Weller et al. (2015), eleven such hypotheses are considered, concerning: Performance, Openness compared to other online resources, Access (equitability), Retention, Reflection (among faculty teaching), Finance (cost-savings) and a few others. However, their investigation specifically concerns perceptions, not efficacy. Note also that their hypothesis on "access" is different from the "access hypothesis" stated in Waters, Mallick, Grimaldi and Baraniuk (2018) which hypothesizes that barriers in adopting the traditional materials led to poorer performance than with OER materials. Among the more analytical studies on efficacy is . In it, they conduct a statistical approach to hypothesis testing by demonstrating non-inferiority of their OER. In general, all these studies indicate OER is at least as good as CER.
The above studies consider the textbook as the sole resource. However, in mathematics, web-based homework is often the dominant resource. Nguyen, Hsieh and Allen (2006) and Williams (2012), as well as many others cited in these papers, investigated the effectiveness of online homework versus traditional homework in mathematics and statistics. These studies indicate that instant grading and feedback is an advantage of online homework. However, we note that Williams (2012) showed that the traditional homework group scored higher on homework and final grades than the online group. They attribute this in part to the lack of partial credit and feedback with their online homework system.
A widely used open online homework system is WeBWorK. It is open source software that can be downloaded and installed for free and contains an open problem library of over 40,000 math and science problems. Good references on using WeBWorK include the reports (Carpenter & Camp, 2008;Denny & Yackel, 2005;and Gage & Pizer, 1999), and papers on effectiveness WeBWorK include Hauk, Powers, Safer and Segalla (2014) and Swanbom, Moller, Evans and Reeves (2016). This homework system plays a primary role in our study that follows.

Filling the Gap
The goal of this study is to provide new insights on the effectiveness of OER compared to CER. The first aspect of our work that is different than the existing literature is that the textbook is not the main, or only, resource. In our work, web-based homework plays a much more prominent role. Indeed, students indicated that the textbook was the least important resource for them in learning Calculus. The second aspect of our work is the tightly controlled side-by-side comparison between the traditional proprietary CER materials and replacement OER materials. The same instructor taught all classes. A third aspect of our work is the quantitative nature of our study. We provide a statistical analysis (in particular multilinear regression models) that predicts student outcomes as well as marginal gains. Finally, we provide validation to certain hypotheses conjectured in the literature concerning benefits of early access on performance using OER.

Methodology
This study was conducted during the Fall 2017 semester, involving four classes of Calculus taught in a face-to-face format, each meeting four hours a week in lecture and one hour in recitation. Two of the classes used the textbook and web-based homework normally used in our department: Thomas' Calculus and MyMathLab. These two classes comprise the CER group. The two other classes, the OER group, used an open source textbook and web-based homework system installed at our university: OpenStax Calculus and WeBWorK. Table 1 compares features of the two homework systems used in this study. The biggest differences between the two homework systems are cost, problem selection, and help resources. The following procedures were followed to reduce variability between the two groups: 1 • Same instructor for all classes (the author of this paper) • Same number of students (70 in each group at the beginning of the semester) • Students had no prior knowledge to opt-in or not • Same lecture notes • Same exam reviews • Similar exams • Similar homework • Similar student demographics To determine the effectiveness of the course materials, student data was collected throughout the semester, including a pretest, homework login (first date web-based homework was accessed), homework scores, and exam scores. The data was analyzed using statistical tests, leading to the results and conclusions of our study.

Cost Savings
One advantage for students using OER is the money they save. The traditional course materials used at our institution include a textbook and the web-based homework system MyMathLab. These come as a bundle for $288.40 including a hard copy of the book, or $106.90 with an electronic book (based on 2017 prices). Hence, the total cost for 70 students in the CER group ranges from $7483 to $20,188 total. Based on information provided by our bookstore, the vast majority of the students opt for the online textbook. Hence, the cost would be closer to $10,000 than $20,000 for these 70 students using CER, which equates to the amount saved by those students using free OER.

Statistical Analysis
In this section we analyze our data to determine significant effects and differences between the CER and OER groups, and to create multilinear models that can be used to predict outcomes on exams based on the covariates in our study. To carry out the analysis we use the statistical libraries in the programming languages R, Matlab and LibreOffice Calc.
In our first comparison, the performance means for the CER and OER groups are analyzed. The results are in Table 2. The last column lists p-values, which are the probability that the Null hypothesis "means are the same" is valid based on the T-test (we also used a rank-sum test on the medians with similar results). A p-value of less than 0.05 (5%) indicates significant differences. The first item of the table compares the average scores of the pretest students took on the first day of class. The mean scores of 8.40 and 8.43 out of 20 are nearly identical, with p-value of 96.5%, indicating the ability of the two student populations at the beginning of the semester are nearly identical. Henceforth, we can assume that differences in other performance measures are due to differences in instruction methods, not prior experiences. The items in the table that indicate significant differences are the "login day" (which is the first date students accessed the webbased homework), "exam 1" and "homework". Of these, the difference in first login day is highly significant, with p-value 3e-5.
Next, a multilinear regression analysis was conducted to determine the significance of the covariates on student performance in our study. The covariates are variables that may affect performance measures but not directly used in calculating performance. For example, the four exams and homework averages are variables but not covariates of the course average. However, other factors, such as students' pretest scores may indirectly affect their course average, hence are covariates. We aim to determine which effects are significant and to derive linear models for performance measures based on these covariates.
The covariates considered in our work include the Pre-Test score (P), whether a student is in the OER group (G=1) or the CER group (G=0), the first day accessing the online homework system (A ranges from -3 to 28 in our study), and the final homework average (H). We look at the significance of these effects on the performance measures (exam scores and final average) according to equations. The results of this analysis are in Table 3. In Table 3, the significant effects include only those covariates with probabilities less than 5% (i.e., .05). Those with probabilities higher than 5% are crossed off. After removing those covariates, a linear regression is run again with only those significant covariates. From this, we arrive at the multilinear models in Table 4.
For example, the equation Another way to understand this formula is by marginal gains. For each gain of 1 point on the pretest, the student is expected to have a gain of 1.58 points on their final, and for each gain of 5 points on homework, they would have about two more points on their final. In this way, one can compute marginal increases. For our linear models, these marginal gains correspond to partial derivatives.

Early Access Effects
Students using OER typically have access to course materials on the first day of class at no cost and with little effort, while students using CER may delay purchasing materials due to costs or other considerations. In Waters et al. (2018) it was hypothesized that this early access to open textbooks would enhance student performance. To our knowledge there is no research to validate their hypothesis, we can demonstrate the access hypothesis with regard to web-based homework. In a typical semester at our university, students using the proprietary homework system delay purchasing the needed access code, or have difficulty logging on using the access code. This delays when they can start using the homework. The access day is illustrated in Figure 1. At the end of the first week, 91% of the OER students had logged in to their online homework system, compared to 65% of the CER students. This difference was shown to be significant in Table 2. Moreover, from the multilinear regression analysis in Table 3, we have the fit: Exam 1 = 26.109 + 3.526*P -0.840*A This shows that for every day a student delays logging onto their homework system would lower their score on the first exam by .84 points.
Alternatively, we can express this result in terms of whether students are in the OER or CER group: Exam 1 = 17.197 + 3.665*P + 8.989*G.
Here, the marginal gain in exam performance due to student preparation (as demonstrated on their pretest) is 3.665 points higher, while the students in the OER group (G=1) gained 8.989 points more than in the CER group (G=0) on their first exam due to the early access. The gains in first exam scores for the OER over the CER group is illustrated in the linear regression fits in Figure 2.

Preparation Effects
Student preparation was measured by a pretest given on the first day of class, before instruction. The linear models in Table 4 show that student performance on all exams depends significantly on their pretest. Hence, we consider the effect of preparation on student performance as sustaining, lasting throughout the semester.

Homework Effects
In our study, the web-based homework was assigned weekly, and the homework average was recorded before the Exam 4 (the final exam). By Table 4, the final exam depends significantly on the pre-test and homework: Exam 4 = 32.434 + 1.580*P + 0.397*H Every point (out of 20) on the pre-test correlates to a gain of 1.58 points on the final exam, and every homework point (out of 100) corresponds to a gain of .397 in the final exam. By this model, there are two paths to success: by pre-class preparation and by homework performance.
Another model for the final exam from Table 4 is: Exam 4 = 64.255 + 1.923*P -8.199*G Students in the CER group (G=0) do better on the final exam on average than students in the OER group (G=1), which is consistent with the higher homework average in the CER group. In understanding this difference in performance using these two homework systems, we consider student comments. Students using the proprietary homework system appreciated the online help, similar problems generated, and reference to the textbook, while students felt the open system did not provide enough online support. As a result, some students had a better overall experience using the proprietary system.

Conclusion
In this paper, we investigated the effectiveness of OER in teaching College Calculus using faceto-face instruction. We demonstrated significant differences in performance between the OER and CER groups, and we developed multilinear models to predict performance based on the covariates: pre-test, homework access data, group (CER or OER), and homework average. Based on the pretest, we showed that the preparation prior to instruction was similar between the two groups, and had a significant effect on performance throughout the semester. We also showed that the students in the OER group likely scored better on the first exam due to the earlier access to the web-based homework, verifying a kind of "Access Hypothesis." On the other hand, the overall homework average and final exam scores were higher in the CER group, which, based on student comments, may be attributed to the level of help resources and overall experience with the homework systems.
There were a few limitations to our study. While four sections with 140 total students were enough to determine significant effects, a large-scale implementation involving more students and faculty may provide further insights. A second limitation to the study was in effectively selecting homework problems in the open system consistent with the proprietary system, which possibly affected student performance. Finally, while this is a quantitative study measuring student success based on exam scores, we do not claim our study directly addresses student learning.
There are several directions to consider in follow-up studies. In particular, in a forthcoming paper, we are investigating student perceptions and attitudes of OER. Another area of investigation is the impact of OER on remediation. We believe that reaching out to students prior before class begins would help improve student performance. We also suggest that careful choice of the homework problems and consideration of help resources available with the open homework system can affect the overall experience, and close the gap between student performance using the OER and CER homework systems.

Introduction
The open educational resources (OER) movement is an established trend in the broader 'open' movement, based on the idea that knowledge is freely available on the internet and open for use with few or no restrictions. Wiley (n.d.) introduced the 5Rs (Retain, Reuse, Revise, Remix, Redistribute) framework of permissions for using OER: (a) retain refers to permission to preserve the authorship of the original work and control copies of the content; (b) reuse refers to permission to reuse the materials exactly as they are; (c) revise refers to permission to adapt, modify, improve, and change the content, including translating into different languages; (d) remix refers to permission to mix and incorporate the original content with other material to produce new materials or content; and (e) redistribute refers to permission to distribute revised and mixed original copies among educators or friends. The William and Flora Hewlett Foundation (2013) produced a set of goals stating that by 2017, OER would be significantly integrated into all educational systems at different levels. However, the current status of OER adoption in education is variable, and it has not expanded in higher education due to several obstacles that hinder the spread of OER at the local and national levels (Allen & Seaman, 2016).
In addition, several scholars (e.g., Pitt, 2015;Wiley, Webb, Weston & Tonks, 2017) believe that OER might have other potential capabilities alongside the sharing of knowledge and providing cost-saving alternatives to expensive textbooks. These capabilities, according to the William and Flora Hewlett Foundation (2013), are: (a) to offer access to knowledge for everyone, (b) to reduce the cost of education, (c) to deliver greater learning efficiency, (d) to promote continuous improvement of instruction and personalized learning, and (e) to encourage translation and localization of content. However, these perceived benefits of OER cannot be robustly demonstrated without empirical evidence.
To widen the adoption of OER across higher education institutions, OER must prove its effectiveness in teaching and learning. That is, faculty need empirical evidence to substantiate the potential benefits of OER in delivering greater learning efficiency and innovative teaching practices. Accordingly, several experts (e.g., Wiley et al., 2017) and associations (such as the Open Education Group) 1 have endeavored to demonstrate the impact of OER in teaching and learning contexts. For example, in June 2017, the William and Flora Hewlett Foundation funded the Designing with OER (DOER) Fellows Program, administered by the Association for Educational Communications and Technology (AECT), Open Education SIG, and the Open Education Group. The main goal of the DOER Fellows Program is to encourage instructional designers, in partnership with a subject matter expert, to design and implement effective OER practices in teaching and learning contexts. Specifically, the DOER grant focuses on eliminating the disposable assignments that have no further value beyond the limits of classrooms and developing renewable assignments that add value to the world of knowledge. Thus, the purpose of this study was to examine this emerging trends in OER movement that might contribute to widen the use of these open resources across higher education and provide evidence that OER is effective in teaching and learning practices.

Conceptual Framework
Openness OER is a phenomenon under the broader umbrella of openness, based on providing knowledge as a "public good" and the idea that "openness" can have a significant impact on education (Murphy, 2013;The William and Flora Hewlett Foundation, 2013). With this spirit, in recent years, the intention of the OER movement is to promote equalized access to knowledge and educational opportunities over the world, as well as to bring about a change in educational practices through improving pedagogical approaches, promoting collaboration, and receiving support from several communities in different aspects.
OER can be considered as a new technology innovation that individuals will use and repurpose to fulfill their needs. The concept of openness associated with OER features several traits: transparency, flexibility, credibility, and creativity (Biswas-Diener & Jhangiani, 2017). Transparency refers to the development of courses on web-based platforms that offer an opportunity for reviewing OER materials such as open textbooks by educators online. Flexibility means that users can adopt OER in different contexts, through flexible learning pathways, and through an international network. Credibility represents the support of OER by professional agencies through leading research on how OER (e.g., open textbooks) affects learning and teaching practices. Finally, creativity pertains to students' contribution to the creation of OER and sharing these resources online with no or few restrictions.

Students' Contribution to OER Creation
Recently, there has been renewed interest in making students' assignments matter by engaging students in OER creation. Renewable assignments, also called student-created OER, involve students' engagement in creating artifacts that have personal meaning to them and sharing them publicly under the Creative Commons CC-BY license (Ross, 2018;Wiley, 2017). These artifacts with CC licenses are students' OER assignments that will be undertaken through a peer-reviewed process to ensure the high-quality of OER content (Ross, 2018). DeRosa and Robison (2017) defined students-created OER approach as "an approach in which students are not just consumers of content but active and visible participants in the construction of knowledge" (p.115).
Several lines of evidence suggest that learners play a vital role in co-production of open content (Ross, 2018;Pitt, 2016). For example, the University of Edinburgh launched a "Digital Futures for Learning" initiative that aimed to create OER in one course by utilizing the core students' assignments (Ross, 2018). The results of this initiative were high-quality OER assignments that received feedback from three peers, the instructor of the course, and an author to ensure their quality before they were released in public. The initiative raised the question of how students can be engaged in the OER developmental process in order to evolve these published resources over time.
In addition, Pitt (2016) reflected on a case study in which students were co-creators of OER at Medical School in Dundee University in Scotland. This initiative drew from the Higher Education Authority (HEA) project that advocates engaging students as partners and collaborators in curriculum production. In this spirit, students were engaged as co-producers of learning resources with medics using WordPress alongside Blackboard for creating OER, sharing, and reviewing it online. Although students were enthusiastic about their accomplishment, some of the produced OER materials could not be published due to copyright, accessibility, and pedagogical issues. Regarding the impact of engaging students in the creation of OER, students favored this new approach over traditional ways for a wide audience to access and read their materials.
With this in mind, Ehlers (2011) articulated that the level of OER usage and creation is based on the level of learning architecture or pedagogical practices. A high level of OER usage and creation would be approached by designing a course at a high level of open learning and teaching practices, in which both objectives and methods are open. At this level, the learner is self-regulated and active in open content production, and the instructor acts as a facilitator in the learning process. Geser (2012) insisted that the use of OER should be integrated in a course designed based on constructivist approaches in order to promote user-generated OER.
The substitution of the traditional textbooks with OER content and transformation to the studentcreated OER approach have emphasized the student-centered design of curriculum. OER proponents advocate the use of open pedagogy approach or OER-enabled pedagogy approach to encourage user-generated OER. Wiley (2017) defined OER-enabled pedagogy as "the set of teaching and learning practices only possible or practical when [users] have permission to engage in the 5R activities." (para. 7). Utilizing open pedagogy practices promotes instructors an opportunity to create a new way of teaching and learning. Instead of allowing students to merely download the knowledge and use it as it is in their learning, they could view it as content that can be continuously produced, revised, and improved over time (DeRosa & Robison, 2017;Ehlers, 2011;Geser, 2012).
According to Schuwer (2017), open pedagogy adds value to teaching and learning practices in two ways: (a) offering instructors the flexibility to engage in multiple pedagogy approaches to make the learning process more active, and (b) challenging learners to use the available learning technologies to connect with peers and practitioner from different cultures online, and thereby, to develop communication and collaboration skills. To date, innovating pedagogical models with the use of OER has focused on effective employment of the 5R practices to reuse and produce OER content.
Hodgkinson -Williams, Arinto, Cartmill and King (2017) attempted to understand the hindrances that hamper the long-term effective use of OER in education. A meta-synthesis of 15 empirical studies that examined OER use, creation, and adaptation in Asia, South America, and sub-Saharan Africa was undertaken. The study showed that the major obstacle to OER use in these three regions was educators lacking legal permissions to share their materials online under open licenses. On the other hand, several factors optimize OER usage with respect to using and repurposing open content: ensuring the quality of OER content; integrating OER use and creation as essential elements in institutional initiatives; and creating a culture of sharing among students and educators. In addition, the provision of a cohesive catalog of OER content in a course has a major influence on the use, creation OER as well as practicing the 5R activities.
In summary, since there is an emphasis on the role of students in this digital age to be co-producers of OER content and contributors to the world of knowledge beyond the boundaries of classrooms, there is a need to explore the value of publishing the core assignments that students themselves create in college courses online under CC license. This study provides indicators about students and instructors perceptions of the emerging trend in the use and creation of OER.

Purpose of study
The purpose of this pilot study was to examine the intervention of student-created OER renewable assignments in the Advanced Instructional Design class at George Mason University through one semester, from February to May 2018. The concept of OER renewable assignments was examined from both students and instructor perspectives. The goals of this study were: • To gain a better understanding regarding the potential impact of renewable assignments on students and instructor perceptions of the attributes of OER. • To identify the factors that play a critical role in motivating students to adopt the concept of renewable assignments. • To develop more targeted research questions for the next phase of testing the intervention of student-created OER.
The study focused on the following research questions: Q1. To what extent are graduate students in the Advanced Instructional Design course aware of the core concepts of OER?
Q2. How do the students and their instructor perceive the concept of integrating renewable assignments into the Advanced Instructional Design course?

Mixed methods research
A sequential mixed method (MM) design was used in this study. This design combines quantitative and qualitative research methods in a single study for the purpose of complementarity, seeking enhancement and clarification of the results from one method with the results from the other method, and triangulation by integrating multiple methods in both data collection and data analysis leading to an integrated explanation of results from both methods in order to construct the conclusion of the study (Greene, Caracelli, & Graham,1989;Maxwell, 2013;Teddlie & Tashakkori, 2009).

Instruments/measures
A survey (Appendix 1) was administered to students to collect quantitative data about their perceptions of OER and the concept of renewable assignments, and to determine the factors that would motivate them to share their assignments publicly under an open license. The quantitative survey instrument was developed based on the benefits of OER and the concept behind renewable assignments. One question used was based on an existing and validated survey conducted by Allen and Seaman (2016), and another question was based on a survey developed by Hilton and Wiley (2018) with the Open Education Group on behalf of the University System of New Hampshire Open Education Initiative, adapted by permission (Hilton, personal communication, January 31, 2018). The survey consisted of 14 questions, including two unstructured questions, 10 structured questions, one rating question, and one contact information question to collect participants' emails for participating in future studies. It was divided into three parts: (1) demographic information, including gender, length of work experience, and job title; (2) awareness of OER, aiming to measure and describe participants' thoughts on the value and quality of OER used as supplementary resources in the course; and (3) participants' satisfaction with the concept of renewable assignments, their willingness to adopt renewable assignments in future courses, and the factors and obstacles they faced.
In addition, an open-ended face-to-face interview (Appendix 1) was used to gather qualitative data from the instructor of the course. It was a non-directive and general approach to gain an in-depth understanding of the instructor's thoughts about the concept of renewable assignments (Teddlie & Tashakkori, 2009). Questions relevant to the current paper sought the instructor's perspectives on the value of implementing renewable assignments in the class for both students and instructor, and the types of pedagogical practices that might support student-generated OER renewable assignments. Other questions addressed related topics such as motivating factors that might encourage students to turn their assignments into renewable assignments as a form of OER, as well as areas where the intervention could be improved for the next iteration of this research.

Setting
This study took place in Spring 2018 through a course at George Mason University entitled Advanced Instructional Design. The course was 16 weeks long, running from January 23 to May 8, 2018. This course was selected for implementing the Designing with Open Educational Resources (DOER) Grant administered by the Association for Educational Communications & Technology (AECT), SIG Open Education, and the Open Education Group, focusing on the development of renewable assignments based on the principles of OER-enabled pedagogy. Fundamentally, the course provides students with the knowledge and skills for designing highly contextualized and engaging problemsolving learning environments (PSLEs) based on the principles of constructivism, situated cognition, and distributed learning.
The criteria for selecting this course were as follows: (a) it offered easy access to participants through the instructor; (b) the objectives of the course included promoting individual knowledge construction; (c) the nature of the course assignments aligned with the essence of the renewable assignment concept and could be published under an open license; and (d) publishing the OER renewable assignments of this course would contribute to increasing the quality of OER in this discipline through sustaining, producing, and repurposing the current OER renewable assignments over time. The intervention in this study encompassed two course assignments: a group presentation of the theoretical principles and instructional characteristics of constructivist learning environments, and a research brief on constructivist pedagogical models or instructional strategies.

Sample overview
The pilot study used a convenience/purposeful sample because it was easy to reach these participants through the instructor of the course. The participants in this study were graduate students in the Instructional Design and Technology (IDT) program enrolled in the course (Advanced Instructional Design) during the period of this research. Participants were 11 graduate students (25% male (n=3); 75% female (n=9)) and the instructor of the course. The students were asked to complete an online survey, and the instructor was interviewed. All participants were novices in the area of OER adoption with respect to using, creating and repurposing OER (Hodgkinson- Williams et al., 2017).

Procedures
At the beginning of the Spring 2018 semester, the instructor of the course (first author of this paper) introduced to the class the aims of the study, which were to explore the concepts of OER and renewable assignments. Then, in week 2, a presentation was given to the class about OER, Creative Commons licenses, and the concept behind the benefits of OER to learning communities as well as the benefits students could accrue from making their assignments OER. An account on the Multimedia Educational Resources for Learning and Online Teaching (MERLOT) resources platform was created. MERLOT is an online repository for the submission of OER in a range of academic disciplines for use by higher education faculty and students. After the instructor evaluated students' assignments based on rubrics that developed as a component of the course, an email was sent to each group and each student seeking their permissions to publish their assignments as OER in MERLOT. Once the permissions were received from the students, the CC-BY license was added to their assignments and uploaded to MERLOT under the students' names. Then, the citations of the works hosted in MERLOT were sent to the students so they could add them to their CVs and share them with friends and colleagues. At the end of the class, after students had been exposed to the intervention of renewable assignments, they were asked to complete an online survey. In addition, the instructor of the course was interviewed to explore her points of view regarding the concept of student-generated OER in the form of OER renewable assignments.

Data Collection
The data collection for this study included a survey containing open-and closed-ended questions that took 10-15 minutes to complete, and a semi-structured in-depth interview that lasted approximately 30 minutes. An informed consent page was presented prior to the survey and interview; participants were asked to agree with the consent statement before answering the survey questions and interview questions.
The online survey was administered to students at the end of the course through the Survey Monkey website. First, the participants responded to the closed-ended questions, followed by a set of open-ended questions designed to illuminate some aspects related to OER adoption and renewable assignments as well as to avoid missing data that might remain undiscovered through closed-ended questions. The open-ended questions were particularly aimed to gather data about students' perceptions of the concept of renewable assignments and their thoughts about the main barriers that might hinder them from sharing their renewable assignments in public with no or few restrictions. These two methods (interview and survey) were used because both are self-reporting methods that sought to determine participants' beliefs about the adoption of OER, specifically about the concept of renewable assignments.
A semi-structured in-depth interview was conducted to "construct a shared understanding of 'what is going on'" in an authentic relationship with the subject (Brown & Durrheim, 2009;Kvale, 2006). The instructor of the course in which the intervention was implemented was interviewed in an attempt to understand her perceptions and to give her the opportunity to address the advantages and disadvantages of employing OER renewable assignments in the course and to provide input on prospective improvements. Kvale (2006) described interviewing a participant as "an interchange of views between two persons conversing about a theme of mutual interest" (p. 2). The interactions with the instructor in this study can be described as "warm personal dialogues" due to the curiosity of constructing a real understanding of the impact of renewable assignments on teaching and learning (Kvale, 2006). Through the interview, the desired information from the instructor was extracted by probing and asking questions other than those existing on the agenda (Kvale, 2006). The interview was a face-to-face conversation with the instructor to formulate ideas about the study themes and to enhance understanding of the topic of interest. The interview was used as an instrument to build a descriptive narrative and, in turn, the results were interpreted in a report.

Data analysis
Descriptive statistics (Teddlie & Tashakkori, 2009) were used to analyze the survey responses and the data from both the survey and interview were presented thematically. The data from the closedand open-ended survey questions were analyzed simultaneously, and interview data were analyzed after the interview with the instructor. The data were analyzed employing the constant comparison analysis technique according to Onwuegbuzie, Dickinson, Leech, and Zoran (2009). The constant comparison analysis involves grouping the data into units for open coding. Then, these generating codes were classified under broader themes. Coding data, as well as the changeover process between data collection and the iterative data analysis (Saldana, 2016) to search for meaning (Hatch, 2002), is a design decision that must be systematically planned (Maxwell, 2011). Saldana (2016) stated that the code is a "researcher-generated construct" and "an interpretive act" (p. 4). The process of coding was iterative including listening to the interview-tape recording before transcribing and reading the interview transcript several times. While listening and reading, notes were taken and looked for patterns to explain and interpret the instructor's perspective regarding the topic as well as to develop tentative categories or patterns. After generating the codes, the qualitative interpretation obtained from the responses to the open-ended questions was merged with the qualitative narrative data obtained from the interview in order to identify emerging themes and patterns. Finally, the entire set of findings from the interview and both the closed-and open-ended questions was merged to generate the final report on the conclusion of the study.

Survey findings
Of the 11 students approached for the study, the number of respondents to the survey was 8 participants. Of these eight, two (25%) were male and six (75%) were female. However, the responses were varied between 6-8 responses to the closed-ended questions. That is, two respondents skipped some of these questions. The majority of participants had professional work experience of more than 10 years. The participants were full-time and part-time students, and among the participants were a professor, a language instructor, a training program manager, an instructional technology coordinator, a product developer, an elementary school teacher, a principal, and a PhD student.
Students' awareness of OER. The first goal was to measure and explore the awareness of OER among students in Advanced Instructional Design after they were introduced to it. The results indicate that among the eight participants, three (42.86%) were knowledgeable about the term OER, two (28.57%) were very knowledgeable about it, and two (28.57%) were somewhat knowledgeable about it. The majority of participants, six (85.71%) students, chose promoting shareability as the greatest benefit of OER, followed by four (57.14%) who chose equalizing access to information for all, followed by three (42.86%) who chose personalized learning and enhancing learning performance as a benefit. When the participants were asked to rate the quality of OER used in the course, five (71.43%) respondents rated it as about the same quality as the traditional (non-OER) instructional materials. The databases participants used for their own searches included Google (100%) and Wikipedia (42.86%). No students used open digital repositories such as MERLOT and OER Knowledge Hub. Only one participant reported using the Directory of Open Access Journal.

Students' perceptions of renewable assignments.
The second goal of this study was to explore participants' satisfaction with the concept of renewable assignments, their willingness to adopt renewable assignments in future courses, and the factors and obstacles they faced. The results, based on six participants who responded, indicated that three participants (50%) were very satisfied, two (33.33%) were satisfied, and one (16.67%) was somewhat satisfied with the concept of renewable assignments. One of the participants who was very satisfied valued the idea of sharing assignments with future students in this class and found it helpful to see other student work samples for the same projects the participants were working on. The results show the influencing factors that contributed to adopting renewable assignments. Generally speaking, students tended to endorse sharing works with others and receiving publication credits more than factors such as intrinsic motivation, the pleasure of being involved in peer production, and stimulating innovation. One participant was influenced by understanding OER from the content creator perspective. When participants were asked if they would be willing to publish their future assignments for other courses in open digital repositories with others under an open license, five (71.43%) were quite willing to share future assignments in public, and two (28.57%) were uncertain about publishing their works because they were still on the fence about OER. Participants' responses regarding what they liked best about the renewable assignments approach in Advanced Instructional Design included the availability of OER online, helping others in immediate work or community learning, sharing knowledge and personal credit, removing financial barriers to knowledge, and finding it helpful to look at work samples of students going through the same program or course. The main barriers that kept participants from publishing their assignments under the Creative Commons license included deficiencies of quality or professionalism in the work, lack of peer review, and worries about the ability of others to change the work without consulting them.

Interview findings
The data from the interview with the instructor suggest that the concept of renewable assignments is exciting because it engages both students and instructor in the development of OER materials. The instructor commented: "The renewable assignments concept is probably something that is more visible, more pragmatic, and more implementable than the actual textbook idea of OER." However, the instructor pointed out that there are lots of difficulties in the implementation process, such as finding good-quality OER and a cohesive collective list of OER that can be used over time, as well as determining how faculty can balance resources that they are going to use as commercial or traditional course materials versus OER reading. One issue related to the quality of OER is the difficulty of continually assessing its quality as these resources keep changing over time through the 5Rs practices.
The instructor believed that the primary value of renewable assignments for students was letting students feel they owned their assignments and could do something with them such as putting them up for public consumption. These virtues of renewable assignments encourage students to become more proactive and value their assignments beyond the course limits. Another source of value is that OER like renewable assignments might become citable; people will cite them when they use them and the students can add them to their CVs.
The instructor argued that the value of renewable assignments for faculty is the chance to compile activities done in a course into a publication online under a Creative Commons license and add it to their CV in a section such as "non-refereed journal articles." Maybe when more higher education institutions adopt OER, they will encourage faculty to show that their publications are being reused by others under a specific section called "Creative Commons or Open Educational Resources." The instructor stated that students understood renewable assignments as a primary driver that helped them reflect about their learning not only by giving their permissions, but by understanding that they can publish their assignments in MERLOT and put them on their CVs. In other words, students' awareness of the attributes of reusing and repurposing behind renewable assignments motivated them to be more engaged in making their assignments open and publishing them publicly. Another motivating factor that encourages renewable assignments creation is students' willingness to go back and spend the time to polish their assignments to make them publishable. Students can receive good feedback from the public over time, not only from the instructor and peers, for continuous improvement purposes.
The interview data suggest that current pedagogical practices may or may not need to change to support student-created renewable assignments. If there is a need for a change, it may lie in including feedback given to previous classes on their assignments for current students before they use them as best practices. The instructor mentioned that students were mimicking the highly graded assignments without having more information about whether these models represented good implementation. Thus, instructions can be included in the assignments to have students look for new examples themselves. This may engage them more to find best practices for implementation and critique them for further improvement.
It was also suggested that one area of improvement for redesigning the intervention of renewable assignments for the next class would be to integrate the instructions for the assignments as part of the class "instead of being voluntary options." This feedback of integrating instructions for creating OER renewable assignments suggests developing a guide for students on how to do renewable assignments in the syllabus of the course. Instructions for the assignments should be added to the guidelines, such as listing OER resources for students to use and instructions to include the Creative Commons link in the assignments and upload them to MERLOT. In addition, the instructor suggested adding official processes of peer review among students (e.g., commenting on assignments through WordPress). The instructor also believed that creating a rubric to assess the quality of these assignments could help to ensure their quality before posting them publicly.

Limitations of Study
The limitations of this study were the small sample size and deficiencies in the adoption of OER. First, the study was limited by the 11 participants enrolled in the Advanced Instructional Design course of which only eight responded to the survey questions. Second, there was no real exposure to OER throughout the course other than the presentation on OER in the beginning of the course and seeking students' permissions to turn their assignments into OER materials in the form of renewable assignments. In the future study, an intervention will be developed to allow students to use OER in their assignments extensively. Moreover, using additional instruments to collect data besides the survey, such as focus groups and think aloud, which is a scaffolding instructional strategy used for teaching high-level cognitive strategies (Abu Raihan, 2011), would help provide an in-depth understanding of participants' experiences with the concept of renewable assignments.

Conclusion
In this study, the intention was to explore students and instructor perceptions of OER in general and renewable assignments in particular. The study had only a small number of student participants who were knowledgeable about OER. Notwithstanding the relatively limited sample, this work offers valuable insights into participants' perceptions regarding the concept of renewable assignments. Overall, both students and instructor viewed the idea of renewable assignments as interesting and believed it could add value to their teaching and learning outcomes. For example, almost 50% of the student participants indicated that they were very satisfied with the concept and production of renewable assignments and the instructor endorsed renewable assignments as more pragmatic than disposable assignments. Both the instructor and students can collaborate to produce open textbooks based on OER renewable assignments. In addition, both students and instructor reported that the factor that most influenced them to produce renewable assignments was making them publishable and citable. The potential to share work with others and make the assignments available for future students also encouraged students to adopt the idea of OER renewable assignments in their future courses.
One of the more significant findings to emerge from this study is that the majority of students, 71.43%, would be willing to share their future works in public under a CC-BY license to make knowledge available publicly, and to help others by sharing their works as examples for future students. The findings from this study underline that the main concern regarding sharing students' assignments in public is the level of quality of these works. As a result, it was suggested that a change in current pedagogical practices could be made to provide students with feedback on previous class assignments to support current students to enhance the quality of their assignments.
The research reported in this paper suggests that the idea of renewable assignments can be accommodated by both students and instructors. Areas of improvement when redesigning the intervention for the next class were suggested, such as making the instructions for assignments an integral part of the course syllabus. This can be approached by developing a guide for students on how to develop the renewable assignments in more detail, letting students reflect about their learning and value their works beyond the boundaries of the classroom. Finally, this study lays the groundwork for future research into immersing students in the use of OER and taking them through the whole process of turning their assignments into OER. These findings provide insights for future research to investigate the pedagogical models that tend to support student-generated OER. Wiley, D. (2017, May 2). OER-enabled pedagogy [Blog post]. Retrieved from https://opencontent.org/ blog/archives/5009 Wiley, D., Webb, A., Weston, S., & Tonks, D. (2017). A preliminary exploration of the relationships between student-created OER, sustainability, and students success.

AWARENESS OF OER
The remaining questions are related specifically to the open educational resources (OER) and the concept of renewable assignments that your instructor used in this course for your class readings and class activities.
Q4. "Open Educational Resources are teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and repurposing by others. OER include full courses, course materials, modules, textbooks, streaming videos, tests, software, and any other tools, materials, or techniques used to support access to knowledge" (William Flora and Hewlett Foundation, 2002 The following questions are related to your Satisfaction with the concept of renewable assignments, willingness to adopt renewable assignments in future courses, and the factors and obstacles to adopt this new idea Q8. The renewable assignment is defined as "an artifact that has personal meaning to students and is shared publicly under the open license of creative commons CC-BY." Please rate your satisfaction regarding the concept of renewable assignments in the course:

Introduction
With the rapid growth of technology, cost and rising demand for educational materials in developing nations (James, 2014;Stork, Calandro & Gamage, 2014) like India, OER-creating organizations are required to keep up with the constantly changing environment. For this reason, such organizations are continuously attempting to identify pathways through which sustainability issues can be overcome and organizational quality maintained, with consideration of the ethical and social responsibilities (Hertz, 2011). Sustainability is one of the key concerns for OER providers since it is indispensible for the prolonged stability and viability of OER initiatives, particularly during periods where financial resources are scarce (Farisi, 2013). A major limitation of OER business models (Olcott, 2012a;De Langen & Bitter-Rijkema, 2012) is the fact that most of the present day OER initiatives heavily rely on constant sources of funding (Johansen & Wiley, 2011). With increasing online presence, active interactions on social platforms (Pitt, Watson, Berthon, Wynn & Zinkhan, 2006;Di Benedetto, 2014), and a growing rate of user-generated content (Mallapragada, Grewal & Lilien, 2012), the possibility of creating OERs is gaining attention and interest. At the same time, the idea of content being created by public entities has also given rise to significant reliability and quality questions (Ingawale, Dutta, Roy & Seetharaman, 2013).
The objective of this paper is to engage specifically with the case study of Pratham Books and its open access creation and distribution model. Its technology and creation costs are significant and funding is limited or temporary, with persisting quality concerns that require innovative solutions. The study found that barriers are being circumvented in two key ways: by using volunteers and platform users to create and maintain materials online; and by constantly engaging in fund-raising measures. This paper reflects on the complex interplay between the need to increase access to educational materials and the need for quality assurance and sustainability measures.

Methodology
This discussion is derived from a doctoral research, which sought to explore the role and specific ways in which OERs could impact primary education in developing nations, through the qualitative case study analysis of three OER-providing organizations. This paper solely discusses the case of one of the three organizations, Pratham Books, a children's books publisher, established in 2004 with three key aims: produce good quality multilingual books that are low-cost and easily accessible (S. Singh, personal communication, May 18, 2016). Being a small organization, it did not have the time or the manpower to cater to the growing demand for their books. This problem resulted in the creation of StoryWeaver, an open platform which has been encouraging openly licensed usergenerated literacy content (see Table 1). This approach is unprecedented in the education field, although this is not uncommon to social media and gaming platforms. Six key members from Pratham Books, who were heavily involved in the StoryWeaver platform's content creation and distribution were approached and interviewed for this research (see Table 2). The research was conducted at the Pratham Books headquarters, in Bangalore, India. While research participants were very cooperative and informative, one of the key challenges was the inability to interview more than six members, due the size of the organization. Another difficulty was the researcher's inability to undertake multiple school visits as planned earlier to overview how Pratham Books' resources are used, due to the election and school holiday period in India.
The semi-structured interviews covered a range of issues, comprising of a series of questions on the ability of OERs to narrow literacy gaps, the existing barriers to OERs, intellectual property challenges, funding models, and the potential of OERs in transforming the primary education space. Interviews were conducted either in participants' offices, lasting between 1 to 2 hours. Naturalistic observation also formed a part of the data gathering phase, with notes being made of the work environment. Following data collection and data coding, three key themes emerged: literacy, language and culture; access, distribution and pedagogy; and user-generation, quality and sustainability. The third theme forms the discussion of the present research paper.
This research suggests that OERs are favorable to the educational theory, referring to the alternate pathways through which education could be propounded, reflected by many eminent early theorists such as Vygotsky (1978), Dewey (1916), Piaget and Inhelder (1967) and Illich (1971). Based on their theories, this paper proposes that OERs have the potential to overcome many of the challenges associated with distribution and access, however also recognizes that the creation and assurance of quality materials is still necessary. Having said that, the original research from which this paper is derived from, was an observational type of study without a high emphasis on theory. It was a pragmatic study, in which intepretations and conclusions were developed following the data gathering phase, as validated by the grounded theory research approach (Dillon, 2012;Strass & Corbin, 1990;Glaser & Strauss, 2017;Corbin & Strauss, 2008). For this reason, this paper will not have an in-depth discussion of theories.

Results
The results of this study suggest that while OERs can be a potential instrument in helping OER providers achieve increased creation of works, rapid dissemination and the circumvention of costs, they also present a two-fold problem: this type of a creation method carries quality implications combined with the constant need for funding to sustain the business model and afford quality assurance undertakings. This section details these implications, in parallel to the measures undertaken by Pratham Books to overcome the concerns caused by user-generation, which relies on the public for content, in order to determine the system's efficiency and suitability for literacy and primary-level education. The findings retrieved are discussed under key themes that emerged during the study.

The need for continued fundraising to sustain the business model
Funding plays a substantial role in determining the sustainability of an organization. The root problem that led Pratham Books to opt for the utilization of users and volunteers to create and disseminate content is the cost involved in the physical publishing and distribution of books and the lack of sufficient physical distribution channels. However, having found an alternative to these issues, the cost of maintaining their business model and their OER-providing technology remains: the StoryWeaver platform's long-term sustenance would not be feasible without sufficient funding. Consequently, ensuring the acquisition of grants, continued donations and support are of utmost importance to such organizations. Funders tend to opt for the more "rewarding" projects and this limits Pratham Books' capacity in managing the direction of its enterprise. Therefore, acquiring funding and dealing with the expectations of funders are challenges that the organization continually faces. Even though funding creates the material environment that enhances the creation of quality organizational outcomes, the organization's freedom in using the funding as it sees fit can be heavily constricted. For this reason, the organization resorts to multiple and varied methods of fund-raising. The following extract gives the example of how crowdsourcing and partnerships were used for this purpose. In order to remain a sustainable operation, Pratham Books acknowledged that being a full-fledged philanthropic model is not sufficient. Often philanthropic income is used to cover the costs of employing the key staff and technology. This implies that volunteers and an online public user community then form the bulk of the resource creators. This business model has created a two-tiered model of online educational content: those funded and created by experts and those unfunded and created by volunteers or the public. This again raises the specter of transparency and quality assurance. However, it is only in this manner that OER providers like Pratham Books can maintain their social organization and business model.

Using public opinion and attribution as cost-free quality assurance tools to establish end-user trust
Pratham Books' example reveals that public opinion and feedback can be cost-free alternatives to traditional content reviewing options, which are often time-consuming and require committed expert reviewers. The respondents of this study suggested that this has been aiding them in quality assurance, enabling a transparent relationship between the organization and its stakeholders. Encouraging the platform's users to partake in the quality review process is helping increase their engagement and ownership of the OER platform, whilst enabling quality to be augmented without any additional cost to the organization. Similarly, Pratham Books also uses attribution to build the trust of end-users on the quality and information reliability of materials. At the same time, this also creates trust within the contributor community, which leads to a rise in the number of volunteers.
Because we are so mission-focused and so transparent in the way we function, authors are willing to work for us. (Singh) Pratham Books has thus witnessed a significant growth in user contributions and involvement in their platform through their willingness to credit all contributors at the end of each created resource, fortifying the organization's image, dedication to their mission and transparency in the eyes of the public. This has also largely added to their efforts in their outreach goals, which is necessary to ensure the ongoing and increase of users joining the platform as well as content creation on the platform.

The potential of user-generated OERs in eliminating costs whilst achieving greater outreach
While the sustainability of technology is key to the sustainability of Pratham Books' open platform, ensuring the continued production and dissemination of resources is also equally important for its prolonged sustenance. For any small organization or publisher with limited funding, this would pose a problem due to the significant physical production and distribution costs involved. To remain sustainable whilst achieving growth in terms of increasing content and the user base, Pratham Books relies on user-generated OERs.
In the digital market, user-creation is allowing greater distribution of works, enabling widespread interactions between consumers and creators. As exemplified by the below extract, it is forming an entire value chain of content, which involves production, distribution and consumption (Blackman, 2016 Pratham Books sees increasing possibilities for OERs in multilingual developing nations, considering that their resources are being translated and modified by users from different cultural backgrounds within and beyond India, capturing new audiences and markets. Entirely created and disseminated by users, the organization is saved from time, labor and costs that may otherwise be required to produce and distribute books on their own. Therefore, the possibilities and flexibility to tailor resources to suit specific needs, languages and cultural contexts have certainly encouraged minority groups in India such as indigenous tribal communities to utilize OERs for literacy education and become OER creators in their on own right on the StoryWeaver platform. Pratham Books is achieving mass impact through OERs, reaching even the most remote locations, yet, it is still looking to find an efficient system to measure, control, and guarantee the caliber of every resource being created on its platform.

While user-generated content increases dissemination pathways, it also raises quality concerns
Encouraging a user-run OER platform has not only helped StoryWeaver generate large number of online storybooks but it has also allowed the generation of individual elements such as images, which are systematically stored in an image bank for future re-use by users for free.
We had a set of 60 words and we told illustrators to pick any 6 words out of our collection of words and create 6 frames, meaning 6 illustrations and tell a visual story with no words. More than 70 illustrators drew 6 illustrations and sent it to us as a visual story. So, we were able to get about 450 new illustrations from the campaign. (Vijayan) User-generated OERs come as a solution to the problem where professional quality authoring and illustrations are expensive, causing the employing or commissioning of professionals to be an unsustainable option for Pratham Books. At the same time, expecting professional quality content creation out of volunteers or online users is also unreasonable and unlikely.
So, these are the ongoing challenges: to find the best people in the field [translators, writers, illustrations, etc.]. So, we don't select writers unless we need to commission a story. Most of the stories come from people who want to write for us. So, they submit stories and we, on an average, get at least one story everyday. (Kumar) Thus, a reliance on users or volunteers to create, modify and translate OERs does mean that quality of the content is less certain. Pratham Books acknowledges this problem but struggles to balance the sustainable creation of novel quality resources with the use of unaccredited contributors.
At several instances, the research participants stated that partnerships and collaborations with external entities were few of the main factors that led to the increased dissemination of OERs. Pratham Books views partnering with cause-driven organizations as a pathway through which a different and much larger audience could be captivated (Y. Vijayan, personal communication, May 18, 2016). The involvement and support of another organization also helps in adding value and a quality mark to every resource being shared through the collaborative initiatives, in the eyes of the end-users. That being said, not all consumers seek quality in the resources they find.

In a context where people lack access to knowledge, resource quality is only secondary
Although the case study organization engages in a filtering and reviewing process to help endusers, quality is often left for users to discern. One of the factors that often influences end-users in evaluating resource quality is its price.
Anything that is free, may not have a value. (Kumar) The problem lies with people judging OERs' quality based on other products in the market, where quality is determined by the price paid. That being said, where children have poor levels of literacy and limited or no access to education, the mere availability of any resource can be beneficial to their development (S. Singh, personal communication, May 18, 2016). Respondents acknowledged the difficulties involved in guaranteeing quality whilst pursuing the mission of enhancing access to materials simultaneously. Creating access hence is a priority and prime motive for OER providers like Pratham Books, pushing quality to the second place. The following extracts illustrate the growing demand for the StoryWeaver platform's OERs, with consumers considering access before quality.
We launched with 24 languages, we are now at 38 in just 8 months and every single language that we added is on user request, which is the power of open. It is increasing. (Shah) Aiming to be a catalyzing agent for creating more literacy content for children, Pratham Books engages in multiple measures to address the quality issue. Particularly, recognizing the prevailing quality questions surrounding user-generated OERs, the respondents of this research elaborated the methods they are currently employing to operationalize and ensure quality. Since StoryWeaver's inception, the organization has been attempting to find ways of distinguishing quality content from the rest. For instance, it has created filters to differentiate between materials created by different categories of users, as the below extract details.
There are processes to find certain kinds of quality content [on StoryWeaver]. For instance, you can search by publisher. There are different categories. One is Pratham Books content, which means that (…) this is quality content. Then there is community-created content. There are so many users, so we cannot say that this is quality content. It is for the user to decide. Hopefully, we are looking at ways in which we can do a quality check even for this so that the user sees more relevant and appropriate content. (Vijayan) to determine quality or spend time in curation. The platform also allows users to rate the resources and provide an opinion in the comment section of each resource; adding another layer of quality filtering for other users to benefit from. Therefore, although quality is not an imminent concern for several consumers, it is a concern for the organization, to which quality reflects its brand image and is therefore pressurized to find an effective remedy.

Discussion
Three key discussion points emerge from the above results. The first one outlines the cost implications for OER organizations like Pratham Books, which are divided between the will to invest in quality and the necessity to remain a sustainable operation. The second discussion point explains how the cost and sustainability issues are circumvented through the driving force of users, which also serves as an outreach and impact mechanism. The third point discusses the consquences of utilizing a public community of practice for the generation and dissemination of children's resources, giving rise to serious quality concerns, in ways that it does not in other contexts, such as social media (Mallapragada et al., 2012;Noguti, 2016;Douglas, 2014) or gaming (Hayes, 2008).

Cost and OER Sustainability
The cost of printing and distributing educational resources is prohibitive for developing nations such as India where a large proportion of the population lives in rural settings. For Pratham Books, the use of OERs through an open and collaborative platform is the only sustainable option with the ability to extend outreach and augmented access to child literacy materials. Nevertheless, this method too involves a certain cost in terms of technology maintenance (Aroyo & Dicheva, 2004;Kinshuk, Huang, Sampson & Chen, 2013). In addition, it also has an impact on the organization's decision-making about the creation and quality assurance of resources.
Philanthropic fundraising is an approach that is widely used in the higher education sector, which constantly endures cost-related barriers and limitations in government aid received (Rohayati, Najdi & Williamson, 2016). As a result, constantly seeking funding from multiple sources is a strategy employed to ensure sustainability and competitiveness (Rohayati, Najdi & Williamson, 2016), as in the case of Pratham Books. Thus, financial sustainability is highly critical to OER organizations. Yet, scholars in the area identify a severe neglect of this in the existing literature on OERs (Larson & Murray, 2008). However, in this regard, this research draws attention to two important points. Firstly, it is clear from this study that the case study organization is aware and focused on the long-term sustenance of its model. Secondly, it is also apparent in the literature that OER initiatives that do focus on their self-sustenance and on being business models that support an open online presence are often times the most successful in attracting long-term financial aid from funders like foundations, government bodies and other external agencies (Olcott, 2012b;Leahy, 2009;Johnstone, 2016;Jacobs, 2007). Pratham Books reflects this attitude very clearly; consistently sourcing funds through a variety of pathways like crowdsourcing, grants, donations and other forms of sponsorships to maintain its model.
Corporate Social Responsibility (CSR) has risen into becoming a major strategic contributor to the value of ventures on a global scale, from being viewed as a mere philanthropic movement. Business models too have developed genuine measures to combine their corporate governance and ethical behavior, to be able to stimulate the performance of their business and public image (Xuemel & Martin, 2012). In this context, Pratham Books is in an optimal space to access such funding, given its social mission and open access nature. However, reliance on such funding sources also implies that OER-providing organizations are obligated to tailor their objectives and works around the expectations of external stakeholders and irrespective of the success of the partnerships or collaboration, there is no guarantee that funding will be continued.

Accessibility and OER Outreach
Ensuring the equitable access to knowledge is a challenge for many developing nations due to cost, infrastructure (Harttgen, Klasen & Misselhorn, 2010;Ally & Samaka, 2013) and the insufficiency of distribution channels (Baral, 2012). OERs have the potential to address these problems and achieve large-scale distribution due to their flexibility: for instance, they offer the possibility for offline use of works (Oyedemi, 2015;Darries, 2004), as low-resolution freely downloadable and printable resources (Mitra & Rana, 2001;Mehta & Shree, 2015). In developing nations like India, OER provision for the purpose of increasing access to quality educational resources is uncontroversial. The complexity lies in sustaining a business model, which is open access and thereby available to the public for free use and re-use. Such a situation demands OER organizations to depend on volunteers, whether offline or online, as well as on funders to support the creation of good quality resources and achieve increased outreach.
With the rapid advancements in technology, book-publishing organizations like Pratham Books are shifting from print to electronic provision of materials; with it is an increasing recognition and need for innovation, in order to ensure corporate sustainability. However, this development has been uneven, varying depending on the location and the efficiency of technological infrastructure of a nation. The results of this study illustrated that Pratham Books has had to align its online presence with a capacity to offer users an offline consumption option. These include online and offline user-friendly resolutions that are viewable online, downloadable, shareable and printable by anyone. This is verified in the literature, wherein it is argued that a "web-centric environment will vary in both shape and substance depending on the country and region involved" (Xuemel & Martin, 2012, p. 104). The literature also defines this approach as an "online-offline strategy"; a method where both online and offline creation and consumption are combined to create increased accessibility, leading to high outreach rates.
As the results of this study also indicated, being an OER provider, Pratham Books uses Creative Commons (CC) licenses to increase the possibilities for end-users to access, modify and share OER content (Garcelon, 2009;Bissell, 2009), to achieve two interrelated goals: to maintain its viability as a business model and achieve accelerated and increased outreach. This has also allowed the organization the flexibility to make use of copyright expired materials, as a means of expanding and populating their content base, which in turn serves to reach more consumers. Additionally, open licensing is also used to avoid the chances for copyright claims by creators over their works as well as to minimize the chances of copyright infringement by users (Koščík & Šavelka, 2013;Bannister, 2011;Ferullo & Soules, 2012). Simultaneously, Pratham Books also sees the benefit of persuading publishing houses and other literacy organizations to use the StoryWeaver platform to upload and share their works for increased accessibility to content and dissemination.

Quality Assurance and OER Success
This study seems to align with scholarly works that argue that although user-generation has the power to multiply the creation and distribution of content and eliminate the costs of physical production, it still raises quality and information reliability questions (Ingawale et al., 2013). Consequently, userdependent OER-providing organizations like Pratham Books are found to have considerable hurdles in ensuring the quality of their resources (Pawlowski & Zimmermann, 2007;Pérez-Mateo, Maina, Guitert & Romero, 2011).
Literature in the area strongly associates the success of open content repositories with the quality of OER content provided (Clements & Pawlowski, 2012;Cechinel, Sánchez-Alonso & García-Barriocanal, 2011;Tate & Hoshek, 2009). Quality sustenance starts from the content creation process and works in two parts: measuring the resource quality is one part of the creation process, while the other is the ability to sustain the generated resource's quality. Different forms of collaboratively created online content is published everyday in several thousands on the Internet. In such a case, it is only natural that evaluation, quality control and assurance are greatly expected by consumers. (Palavitsinis, Manouselis & Sánchez-Alonso, 2013;Downes, 2007).
The practice of transparency could definitely act as a key quality indicator with the potential to build and sustain the multi-directional relationships between organizations and their various stakeholders, whether internal or external (Schnackenberg & Tomlinson, 2016). In the literature, transparency is positioned as the cornerstone of trust for organizations (Masson & Udas, 2009;OECD, 2007;Liesegang, Albert & Schachat, 2008) and holds differing definitions for differing contexts. For instance, in the context of strategic alliances with external entities, the term is defined as the openness exhibited towards and between partners (Larsson, Bengtsson, Henriksson & Sparks, 1998). In the case of electronic markets, it is defined as the extent to which information is accessible and visible (Zhu, 2004). This research found that Pratham Books acknowledged the same, illustrated by its commitment to the incorporation of public opinion and contributor attribution.
Whilst the study of public opinion is not uncommon in academic spaces such as political science, which have abundant discussion on public trust and perceptions on public institutions and politics, literature on the relationship between public opinion and the quality of educational resources has been largely neglected (Fladmoe, 2012). However, attribution and its effect on success of OERs are discussed in prior works. One of the works in the area (Wicherts, 2016) outlines the main difference between traditional resources and OERs, with regard to the potential of attribution in denoting quality. For instance, traditional educational materials like peer-reviewed academic journals do not leave room for deliberation: end-users trust the quality of the published works on the basis of the journal's ranking and review process. Whereas, on OER content, it is the attribution that permits end-users to judge the quality of a resource based on the listed names of contributors and the number of people who have contributed (Wicherts, 2016).

Conclusion
This paper discussed some of the main challenges inherent in using user-generated content for literacy and education proliferation (Mallapragada et al., 2012), which despite being a solution to prevailing accessibility and dissemination issues, poses significant quality questions (Ingawale et al., 2013). Prior research suggests that quality holds a key role in the sustainability of OER projects (Clements & Pawlowski, 2012;Cechinel et al., Tate & Hoshek, 2009). Understanding that consumers greatly regard quality assurance, evaluation and quality control (Palavitsinis et al., 2013;Downes, 2007), Pratham Books has displayed evidence of its efforts through its reviewing and filtering processes, which help determine quality.
Aside quality, another main barrier that this study discussed is associated with cost and sustainability. To cover certain material development costs such as special commissioning charges and technology upgrade costs, such organizations are often pressurized to continuously raise philanthropic funds. (Annand, 2015;Hannon, Huggard, Orchard & Stone, 2014;Olcott, 2012a). However, to circumvent these costs, whilst still achieving the generation of novel content and wide distribution, a reliance on volunteers and online users is a necessity; both of which are required to allow Pratham Books to create and disseminate content both online and offline, as well as cover a range of topics in multiple languages and divergent audiences, whilst also ensuring that their notfor-profit business model survives.
Moreover, this research acknowledges that the adopters of OERs and especially those using usercreated content have a number of challenges to address. This understanding is in line with prior works, which insist the implementation of strong initiatives advocating the adoption of OER systems into the educational space, to suit the rapidly evolving learning needs and environments (Toledo, Botero & Guzman, 2014) could make a difference. At the same time, scholars also warn that the OER system is currently at an infancy stage in developing countries (Canbek & Hargis, 2015;Aydin & Ulutak, 2010), where the traditional educational system is more trusted for quality because of its history, hinting that competing with that established standard would be a challenge to be overcome.
Nevertheless, a few best practices can be drawn from the Pratham Books' case. First and foremost, since the StoryWeaver platform's launch in 2016, it has grown multifold due to its outreach measures. This has not only enabled StoryWeaver to increase its user base and multiply its online content in the form of derivatives but most importantly, it has been able to receive continued financial support from major donors and grants till date. This indicates that advocacy and marketing played a major role in not just promoting their work but also in establishing trust amongst end-users and funders. In this manner, Pratham Books has so far been able to sustain the StoryWeaver platform.
Secondly, despite quality concerns, Pratham Books has been utilizing user-generation to circumvent physical books creation and distribution costs. This means that the StoryWeaver model is a lowcost model, which only needs funding to sustain its technology, while sustaining and multiplying its content occurs automatically through user-generation. This could be a valuable best practice model for countries in need of a high impact-low cost solution to create access to educational resources.
Lastly, Pratham Books has established a number of partnerships with literacy organizations, publishers, schools, as well as with individual groups such as teachers and communities. Although partnerships and collaborations may not guarantee funding or the sustainability of the platform, they can help strengthen the brand, its trustworthiness and the possibility of being offered support. Therefore, the three key lessons drawn from Pratham Books' case could be valuable for both researchers and literacy content providers in trying to explore alternate sustainability models and pathways in augmenting access to literacy resources In resource-constrained countries.
In conclusion, while OER providers such as Pratham Books draw immense inspiration from higher education OER models such as MIT MOOCs on their creation and utilization of OERs, they do not have the capacity or resources to adopt a similar model due to funding contraints and lack of qualified or dedicated content creators. Moreover, the higher education system in developing countries like India only serves a very small proportion of the population, most of which is illiterate and living under the line of poverty (Ilie & Rose, 2016;Carnoy & Dossani, 2013). As a result, utilizing systems such as user-generation to minimize expense and maximum impact is working well for them. On that note, a number of researchers cite the lack of funding, inaccessibility and inadequate infrastructure to be common contraints for higher education institutions too, which hinder the promotion and sustainability of OERs (Anderson & McGreal 2012;Olcott, 2012a;Olcott 2012b;De Langen & Bitter-Rijkema, 2012;Cohen & Soffer, 2015;Sener, 2010;Joseph & Nath, 2013). Another significant problem associated with OER content creation at the tertiary level is the unavailability and resistance of academics in creating OERs due to time limitations, pedagogical concerns and peer influence (Cox, 2013). In such cases, higher education institutions investing in OERs could potentially benefit from researching and experimenting with user-generated OERs, with Pratham Books as a best practice model to overcome these problems.