A design-based approach to support and nurture open educational practices

Purpose – A critical attribute of open educational practices (OEP) is the pursuit of open scholarship which comprises the release of educational resources under an open licence scheme that permits no-cost access, use, reuse, adaptation, retention and redistribution to others. The degree of openness in relation to this attribute will depend on the context and culture of the place and the people in it. When left to chance, the adoption and practice of open scholarship by educators is at best sketchy. For optimum impact, a design-based approach is essential. A central focus of such an approach will need to target educators’ belief systems and practices about their scholarship. Any such work will involve researchers collaborating with practitioners in real-life settings to improve educational practices through iterative analysis, design, development and implementation. The purpose of this paper is to report on how the development and use of such a designbased approach, implemented by the Open University of Sri Lanka, impacted the adoption and uptake of open scholarship among teachers in the Sri Lankan school system in terms of changes in their use of instructional resources, pedagogical thinking and pedagogical practices. Design/methodology/approach – The study adopted a design-based research (DBR) approach (Reeves, 2006), which involved researchers collaboratively working with practitioners in real-life settings to improve their educational practices along three aspects – instructional resource use, pedagogical perspectives and pedagogical practices. Based on the four stages of the DBR approach – analysis, solution, testing and refinement, and reflection, a professional development intervention programme was designed and implemented to support teachers on the integration of open educational resources (OER) and adoption of OEP in their teaching-learning process. Data collected throughout the process using multiple strategies such as questionnaire surveys, concept mapping, lesson plans, focus group interviews, self-reflections and “stories”, were analyzed using both qualitative and quantitative methods. Findings – By the end of the intervention, significant changes were observed in teachers’ use of instructional resources, their pedagogical thinking and pedagogical practices. While resource usage has shifted from no or low usage of OER to reuse, revise, remix and creation of OER, the pedagogical thinking and practices of teachers moved from a content-centric and individualized patterns to more constructivist, context centric and collaborative ways. The diffusion of OEP was prominent along two dimensions – enhancements in the individual practices in innovative OER use as well as collaborative practices of sharing of resources, knowledge and good practices. Practical implications – The systematic and flexible methodology adopted based on the DBR approach via a framework designed as a contextualized, process oriented and a self-reflective enquiry has been very useful to support changes in OEP among practitioners over time. Originality/value – This iterative process allowed the researchers to function as “designers”, while investigating real-life issues in collaboration with the practitioners through reflective enquiry to further refine Asian Association of Open Universities Journal Vol. 12 No. 1, 2017 pp. 1-20 Emerald Publishing Limited 1858-3431 DOI 10.1108/AAOUJ-01-2017-0010 Received 9 January 2017 Accepted 6 April 2017 The current issue and full text archive of this journal is available on Emerald Insight at: www.emeraldinsight.com/1858-3431.htm © Shironica P. Karunanayaka and Som Naidu. Published in the Asian Association of Open Universities Journal. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons. org/licences/by/4.0/legalcode This work was carried out with the aid of a grant from the International Development Research Centre, Ottawa, Canada, through the Wawasan Open University of Penang, Malaysia, as part of the Research on Open Educational Resources for Development (ROER4D) programme. 1 A designed based approach to OEP D ow nl oa de d by T he U ni ve rs ity o f T he S ou th P ac if ic A t 2 1: 53 0 2 A pr il 20 18 ( PT ) innovative practices towards OEP. This provides valuable insights for improved design solutions for future interventions in similar contexts.


Introduction
The imperative for open educational practices (OEP) has been on the rise for some time now. Foremost, this comprises; open access to educational opportunities; open learning, at the heart of which is time, place and pace flexibility; and open scholarship which means free and open access to educational resources for use, reuse, adaptation, retention and redistribution to others. The concept of openness is not new. Open access to educational opportunities and flexible approaches to learning and teaching have been always a core defining characteristic of open, flexible and distance learning which sees openness and flexibility in relation to time, place and pace of learning as a value principle, much like we see diversity, equity or equality in education and society more broadly. Openness and flexibility are at the heart of what distance educators do and promote. And in the contemporary world, both in the developed and developing contexts, technological infrastructure is a key component of it.
The notion of open scholarship is a more recent phenomenon and an addition to the broader concept of openness. Its value principle is that education in general, and knowledge more specifically, is a public good which should be available to all. A key component of this is the release of educational resources under an open licensing scheme (e.g. Creative Commons Licensing Framework) as open educational resources (OER) for it to be able to be used, reused and shared freely and openly at no cost to either party. The imperatives for adopting this path to open scholarship are enormous. For teachers and learners, both in developed and developed educational contexts, this means access to valuable and adequate educational resources for better teaching and learning. And it includes the opportunity to adopt and adapt these essential educational resources in ways to suit local contexts and its requirements. These affordances are critical and crucial for the achievement of our sustainable development goals towards education for all and promoting equity, and equality of educational opportunity more broadly but especially in developing social and economic contexts.
While the educational imperatives for open scholarship are clear and convincing, its adoption and implementation more widely are till fraught with challenges. Some of these challenges have to do with the economics of open scholarship. Someone, and at some point, must pay for the production of educational resources so that these can be distributed widely and openly among the community, and at no cost to its users. Issues around this kind affordability are still being worked out by authors, developers and publishers of any such content.
The more serious challenge has to do with the adoption and integration of OER in teaching. Foremost this requires an understanding of the concept of open scholarship and what it involves. This includes an understanding of what is an OER, where is it found, and how to find it? How is it distinguishable from any other kind of educational resource? And also, how to develop an OER? What, if anything, is qualitatively different about it? And once found or developed, how best to make use of it in supporting teaching and learning? What are its pedagogical affordances? And how these affordances differ from those that are possible for proprietary educational resources?
A lot of this has to do with developing new perceptions, and perspectives about pedagogy, about what it means to teach and to learn. And this involves, not just developing new and technical skills but it is about shifting mindsets. It requires practitioners to move beyond a focus on access to OER, to a more comprehensive view about creation and integration of OER in order to make a difference in the teachinglearning process. This requires engagement of teachers and learners in open practices 2 AAOUJ 12,1 two focal areas of openness in education. The release of teaching, learning and research materials under an open licensing framework (Creative Commons, n.d.) as OER has increased no-cost access to and sharing of knowledge in an efficient and affordable manner (UNESCO-COL, 2012;Wiley and Green, 2012). This has enhanced emergence and adoption of various scholarly practices of openness, promoting open scholarship, which is a critical attribute of OEP.
The "spectrum of rights" provided by the open licensing framework to users giving access to resources through "most open" to "least open" licence types, allows an opportunity to adopt varying degrees of openness in the use of educational resources via the 5R framework of OERretain, reuse, revise, remix and redistribution (Wiley, 2014). The ability to adapt and use educational resources in this manner, which they were unable to do with the proprietary materials, has empowered educators to become more creative and innovative in their educational practices. In other words, the OER movement has stimulated innovations in teaching and learning practices. This shift of focus from access to resources, to innovative practices in the use of resources promotes the concept of OEP Open Educational Quality Initiative Report, 2009).
The various definitions of OEP fundamentally focus on how "openness" can be practiced in teaching and learning. For instance, OEP are explained as, a set of activities and support around the creation, use and re-purposing of OER (Conole, 2010); a combination of open resource use and open learning architectures to transform learning (Camilleri and Ehlers, 2011); use and production of OER in such a manner to improve the quality of education and innovate educational environments (Ehlers and Conole, 2010); and practices that support the (re)use and production of OER through institutional policies, promote innovative pedagogical models, and respect and empower learners as co-producers on their lifelong learning path . The emphasis on practices instead of resources and on enhancing innovations and quality in education are common features stressed in all these definitions. These practices are further described in relation to the use of educational content through "open pedagogies" and "open technologies" (Beetham et al., 2012). While the adoption of pedagogical models based on more constructivist and participatory approaches will allow educators to become more open in their teaching practices, increased availability of free and open technological applications and services let educators select appropriate technology to support innovative practices. OEP will thus require engagement of teachers and learners in innovative practices which are participatory and collaborative, harnessing the potential of both technology and pedagogy (Cape Town Declaration, 2008). However, such changes in practices would essentially require changes in practitioners' belief systems and mindsets. Hence, promoting adoption of open scholarship and OEP becomes very challenging.

Facing challenges in implementing OEP
The degree of openness in educational practices, and hence open scholarship will vary, depending on the context and culture of places and people. While the degree of openness in actual content will differ based on their licence type, the degree of openness in practices will change depending on the pedagogical models practitioners adopt in the use of the educational resources. In order to make an impact, implementation of OEP would essentially require practitioners to change from a simple focus on access to a broader view about integration of OER in the teaching-learning process and creation of OER, which is a challenging process (Ehlers and Conole, 2010). Even though adoption of OER enhances access to resources, OER by themselves would not have any impact unless those are used in particular ways for a change in practices to occur (Smith, 2016). This will require practitioners using innovative methods in the integration of OER. Research studies conducted on OER integration in teaching and learning reveal that adoption of OEP demands a culture of sharing and use of novel pedagogical models (Beetham et al., 2012; 4 AAOUJ 12,1 Carey et al., 2015;Coughlan and Perryman, 2015;Karunanayaka and Naidu, 2013;Karunanayaka et al., 2014Karunanayaka et al., , 2015Lane and van Dorp, 2011;Petrides and Jimes, 2008).
The realization of successful OEP will thus depend on having opportunities for pedagogical innovation, yet innovative thinking and practices may occur if and when more flexibility and choices are available for practitioners (Educause, 2010;Weller, 2014). Five principles of openness necessary for implementing OEP have been identified, comprising open tools and processes that will promote: collaboration and sharing of information; connected communication about learning and teaching; collectivity to grow knowledge and resources; critique for the promotion of scholarship; and serendipitous innovation (Conole, 2013). Digital technologies will play a key role in achieving such requirements effectively and efficiently and hence digital competencies also becomes a crucial requirement in supporting OEP (Beetham et al., 2012;European Commission, 2013).
Various initiatives on OER integrations have presented frameworks for implementing OEP in different contexts. For instance, the "OPAL framework" (Andrade et al., 2011), provides strategies to consider when designing, developing, implementing and evaluating OER initiatives. Capturing the link between resources and practices, OEP is explained in two dimensions in this framework: openness in resource usage and creation versus openness in pedagogical models, suggesting different degrees of openness in both aspects from low to high levels . Similarly, the 7C learning design framework (Conole, 2014) presents seven elements: conceptualize; capture; communicate; consider; combine and consolidate, integrating the trend for openness by using OER and collaborative practices. A model of "open pedagogy" discusses eight interconnected attributesparticipatory technologies; people, openness, trust; innovation and creativity; sharing ideas and resources, connected community; learner generated; reflective practice and peer review (Hegarty, 2015). Such models exemplify how OEP can be facilitated by creating structured enabling environments.
It is apparent that adoption of OER and OEP by practitioners can be truly effective only if it reflects a change in their thinking and actions. Educational change is a complex process, and to deal with such complexity, the best way is not to control change but to guide it, by individuals taking action as "change agents" (Fullan, 1993). Enacting change towards OEP is best achievable through a strategic approach via systematic design of appropriate learning experiences (Naidu and Karunanayaka, 2014). OER integration endeavours in the Sri Lankan context, have revealed significant influences in supporting changes in thinking and practices of educators towards OEP through design and implementation of effective, efficient and engaging learning experiences based on innovative pedagogical models (Karunanayaka andNaidu, 2014, 2015;Karunanayaka et al., 2015). Experiences which are designed to create more intense and close interactions with the practitioners in context, while engaging them as co-participants in the process would be more desirable in promoting changes towards OEP.
Affordances of a DBR approach to support adoption of OEP A DBR approach that involves researchers collaborating with practitioners in real-life settings to improve educational practices (Reeves, 2006), offers a feasible solution to ensure a sustainable and lasting impact on the adoption of OEP. DBR is a systematic but flexible methodology aimed at improving educational practices through iterative analysis, design, development and implementation in real-world settings (Wang and Hannafin, 2005). It comprises four phases: analysis of existing levels of expertise and practices by researchers and practitioners; designing, developing and implementing solutions as appropriate; testing and refinement of solutions in practice; and reflection by researchers and practitioners on authentic problems produce design principles and enhance solution implementation (Reeves, 2006). Being grounded in real-life context and conducted in collaboration with the practitioners, a DBR approach would have more potential in enacting desired changes of authentic educational practices. The project implemented by OUSL involved the integration of OER and adoption of OEP by student teachers of the Faculty of Education, and ascertaining its impact on the use of instructional materials by teachers, and changes in pedagogical perspectives and practices of teachers. The focus of the current study was to explore how a DBR approach impacted on supporting OEP among the teachers.

Research design
The DBR (Reeves, 2006;Wang and Hannafin, 2005) approach adopted in this study involved researchers collaboratively working with practitioners in real-life settings to improve their practices along three aspectsinstructional resource use; pedagogical perspectives; pedagogical practices. The full programme of activities comprised the design and implementation of a professional development intervention programme for teachers on the integration of OER in teaching and learning, and ascertaining the impact of integrating OER and adoption of OEP in their teaching-learning process. This intervention programme was implemented in several stages, at nine OUSL centres representing nine provinces of the country.

Participants
Participants in the study comprised 230 student teachers during Stage 1, and 85 student teachers in 21 teams in Stage 2, representing the nine provinces of the country. A summary of the participant profile at Stage 1 is presented in Table I.
The participants constituted more females (66 per cent) than males (34 per cent), and all (100 per cent) were graduate teachers with more than half (57.8 per cent) being science graduates, and some (17.4 per cent) were with post graduate qualifications. A majority of the participants (75.7 per cent) were newcomers to the teaching profession with less than five years of teaching experience, and only very small percentage (1.7 per cent) with experience above 15 years. The distribution of participants at Stage 1 in the nine OUSL centres -Anuradhapura (A), Badulla (B), Batticaloa (Ba), Colombo (C), Jaffna ( J), Kandy (Ka), Kurunegala (Ku), Matara (M) and Ratnapura (R) is presented in Figure 2.

The intervention framework
The intervention framework was designed as a contextualized, process oriented and a selfreflective enquiry following the four stages of the DBR approachanalysis, solution, testing and refinement and reflection (Reeves, 2006). This allowed examining the impact on the three aspects in focusinstructional resource use, pedagogical perspectives and pedagogical practices, through the provision of a carefully structured intervention process with specific strategies at each stage. A diagrammatic representation of the framework developed to design and implement the intervention using the DBR approach is presented in Figure 3. The design of the intervention involved designing and developing a set of strategies based on existing theories and frameworks, tested design principles and prior research.
Strategies were developed to monitor and support teachers to gradually move from "no" or "low" usage of OER to "high" degrees of usage and creation of OER enhancing openness   in their use of instructional resources, based on the 5R framework of OERretain, reuse, revise, remix, and re-distribute (Wiley, 2014), and the OPAL framework to support OEP in two dimensions: openness in resource usage and creation vs openness in pedagogical models . To enhance changes in teachers' pedagogical perspectives and practices towards OEP, the experiences in the intervention were designed based on situated learning principles underlying a constructivist approach to learning (Brown et al., 1989;Duffy and Jonassen, 1991). These included specific strategies designed to support teachers moving from low towards high degrees of context centric, challenging, critical-thinking, creative, and collaborative thinking and practices (termed "5Cs").
Specific strategies of the intervention in line with the four phases of DBR are presented in Table II.
As indicated in Table II, the four phases in the DBR approach were implemented in iterative cycles during the intervention, using different strategies. These strategies also served as a variety of data gathering methods, both qualitative and quantitative, systematically used at different stages of the intervention.
Initially, the current situation in relation to the three aspectsinstructional resource use, pedagogical perspectives and pedagogical practices were analyzed, in collaboration with the practitioners. Next, as a solution, the intervention consisting of two key componentsa series of interactive workshops and an enabling online environment in Moodle learning management system (LMS) was designed and implemented. Both these components included specific activities for capacity building, guiding, monitoring and supporting, as well as reviewing and evaluating, in relation to integration of OER by teachers. A variety of hands-on individual and group experiences were provided to engage teachers in the integration of OER in their teaching-learning process, while encouraging collaborative practices and promoting reflective practice.
Iterative cycles of testing and refinement of solutions in practice occurred during the series of pre, mid and post intervention workshops conducted at the nine OUSL centres, as well as constant interactions through the online environment in the LMS. Data gathering via multiple sources continued throughout, and these helped researchers to work together with practitioners in refining design strategies, based on contextual needs and complexities.
Throughout the intervention process, both teachers and researchers were encouraged to maintain reflective journals, writing their self-reflections on the experiences gained at different stages. Based on these reflections, teams of participant teachers and researchers compiled narratives in the form of "stories" at the end of the process. This helped in discussing and finding solutions to authentic problems, and to recommend effective design principles.

Methods of data collection and analysis
In keeping with the DBR process which involves integrative use of multiple data gathering methods over time, data were collected throughout the intervention process using questionnaire surveys, concept mapping, analyzing lesson plans, focus group interviews, records in the LMS, self-reflections and narratives ("stories") of teachers. These strategies were employed at different stagespre, mid and post intervention, and analyzed using qualitative and quantitative methods to capture changes in relation to adoption of OER and OEP in terms of changes in instructional resource usage, pedagogical practices and pedagogical perceptions of teachers.
Such methodological triangulation allowed providing a more comprehensive view of the whole, since each source of data gave a different views of the issues being studied (Morse, 1991). This process helped gaining a broader understanding of the effects of the intervention.
Interpretative phenomenological analysis (IPA) was the methodological construct used in the analysis of qualitative data that comprised content analysis and interpretation of 9 A designed based approach to OEP Teacher reflections researcher reflections Use teachers and researcher's reflections to find and implement solutions to authentic problems Compilation of "Stories" -by teachers and researchers, based on their reflections; Creation of a weblog to share the stories of their experiences Table II. Strategies adopted during the intervention to promote adoption of OER and OEP 10 AAOUJ 12,1 concept maps, open-ended questions in surveys, self-reflections, interview transcripts and narratives. The IPA strategy was used to explore in detail how individuals were perceiving the particular situations they faced and making sense of their personal and social world (Smith and Osborn, 2003), that helped discovering the meaning of experiences of participants through their and researchers' interpretations by examining their "lived experiences" (Reid et al., 2005). Further, such a "realist, process-oriented approach" that relied on understanding the processes by which a situation occurs via direct physical connection in the real world, offered a feasible approach for ascertaining causation, rather than a comparison of situations depending on the existence of the presumed cause (Maxwell, 2004;Mohr, 1999).

Results and discussion
Changes in instructional resources usage Data obtained from questionnaire surveys revealed that at the pre-intervention stage, teachers' use of OER was minimal, as indicated in Table III. Only 10 per cent have even heard the term "OER" and 3.9 per cent have used some OER materials, while use of online resources was only 20.4 per cent.
However, by mid-intervention this has significantly changed as shown by Figure 4, where a majority have claimed developing competencies in searching and identifying OER, identifying CC licence, adopting 5Rs, creating OER and integrating OER in their teaching practices.
Even though the number of participants decreased by the end of the intervention, many participants who remained were actively engaged not only in reusing OER, but also re-purposing them by translating into local languages, adapting to suit their contexts, and even creating OER on their own, as evident by the following excerpts: When I use OER I modify it to local language. Some OER's are advanced than I expect. Thus I edit it according to my lesson.
We were able to find interesting presentations on photosynthesis. We translated one presentation to Sinhala and used it to teach students. Sometimes we downloaded exercises and tests and made copies. Then we distributed among students.
The provisions of hands-on experience during the workshops and the Moodle LMS have vastly supported their use and adoption of OER: Workshop activities helped us to identify relevant OER and identify the nature of their licenses […] it helped us to gain some knowledge and practice of the 4R concept through practical activities organized during the workshop.
We could also access the OER site created for us […] in the Moodle LMS and search for OER materials relevant to our subject areas. We could identify appropriate OER to integrate in lesson plans.
Hence, it was evident that starting from an initial state of "no or low-usage" of OER, the participants gradually moved towards adopting 5Rs and creating OER. This move from

11
A designed based approach to OEP "low" to "high" degrees of openness in two aspectsuse, sharing and creation of OER, and innovative use of instructional resources, signifies enhanced diffusion of OEP (Ehlers, 2011). The specific strategies of the interventionavailability and accessibility of a variety of OER through the LMS that allowed 5Rs, and provision of hands-on activities to search, select and integrate OER in lesson plans have facilitated enhancing creativity and innovation in their use of resources.

Changes in pedagogical perspectives
The structural and content analysis of the concept maps created by participant teachers at different stages of the intervention revealed the developments in teachers' understandings and changing perspectives over time. The comparison of different versions of concept maps illustrated incremental changes in teachers' perspectives, moving towards more "open" thinking, as depicted by Figures 5(a) and (b). At the end of the process group concept maps were created by teams of participant teachers, elaborating their collective understandings and perceptions, as illustrated by Figure 6.   The final group concept maps constructed by teams of teachers demonstrated their enriched perspectives towards adoption of OER and OEP. The concept mapping exercise has also supported teachers to reflect on changes in their own pedagogical thinking during the intervention process as claimed below: The concept maps was a new experience […] which provided an easy way to summarize our findings throughout our activities and that new knowledge had made us special.
Even though teachers' awareness on OER was minimal at the beginning, their perceptions on sharing resources was quite positive. However they were not very much concerned about copyrights when using resources. This view has changed during the process and teachers have started thinking more about using online resources legally and sharing resources, as indicated by the their reflections: I have already used teaching materials which are available in the internet without checking the license agreement. But today I could get a real idea about the license agreement of the resources.
I can understand the meaning OER […]. Now I can use useful data and information legally. Before that I mostly use copyright data and information without permission […] now I use free usable data source with permission.
Further, it was evident that the participants' pedagogical thinking has become more context centric, challenging, critical, and creative as well as collaborative, as following excerpts specify: Through the integration of OER […] we have got the opportunity to "think out of the box" and release ourselves from the traditional teachers' role. The creation of our own OER enhanced our thinking capabilities and contributed to the development of our personalities as well.
OERs helped us to plan and implement very attractive lessons. Here the teachers as well as students were encouraged to look for new knowledge. This created an opportunity for us to share subject related resources such as activities, assessments, video clips etc.
As evident by these quotes, engagement with OER has stimulated critical reflection among the teachers about their current practices and offered inspiration to attempt new practices with OER, and leading towards OEP (Beetham et al., 2012). The provision of specific "reflective" experiences in the intervention such as concept mapping, reflective journal writing and peer discussions, together with the individual and group activities on OER integration, and the enabling environment of the LMS to share OER, have significantly supported this. The compilation of "stories" based on their experiences with OER adoption process also promoted critical reflection and collaborative thinking leading to further enhancements in teachers' pedagogical perspectives towards OEP.

Changes in pedagogical practices
Analysis of the lesson plans of teachers at the pre-intervention stage revealed that even though a majority (W60-70 per cent) demonstrated use of a learner-centred pedagogic approach with learning activities in their overall approach to teaching, only a small percentage (W10-20 per cent) demonstrated innovative learning designs, use of a variety of media types as learning resources, use of technology, providing opportunities for learner creativity, promoting self-regulated learning, linking with real-life situations and creating an enjoyable learning experience.
Yet, during the intervention, their pedagogical practices have notably changed, as revealed by the OER-integrated lesson plans, and also supported by their responses and reflections: OER supported us to prepare a quality lesson plans for our lessons in all subjects. Then we were able to incorporate new techniques in our teaching-learning process.

15
A designed based approach to OEP Our lesson evaluators and colleagues highly appreciated OER incorporated lessons […] It was very easy to get students' attention and it supported to create a better learning environment.
Further, it was encouraging to observe many examples of sharing OER and disseminating the OER concept among others by the teachers via creation of OER and sharing them, conducting awareness raising workshops, publishing booklets etc: I shared my OER knowledge with my school teachers by organizing a workshop. I created more than 30 OERs and uploaded to LMS and also searched more than 50 lessons to different subjects.
One member wrote a booklet in Sinhala about OER for teachers and any others who are interested in this concept.
We felt proud to publish a magazine on OER titled "Integrating OER in Learning Teaching Process.
Strategies such as motivating teachers to engage in lesson planning with OER integration and share them in the LMS, organizing a competition to identify the most active centre and the most active student teacher in each centre, have certainly impacted on promoting OER adoption among teachers. The design of such challenging activities related to their own contexts has stimulated teachers adopting critical, creative and collaborative approaches in their teaching practices. The shift from low to a high degree of OER use, and change of practices in the creative use of OER clearly suggests a move towards OEP. The diffusion of OEP was prominent along two dimensionsenhancements in the individual practices in innovative OER use as well as collaborative practices of sharing of resources, knowledge and practices , that has resulted due to the intervention.

Impact of the DBR approach in supporting OEP
The basic features of the DBR approach has been very supportive in enacting changes in educational practices of teachers towards OEP. In order to address a meaningful current problem faced by practitioners in relation to effective implementation of OEP, an educational intervention grounded in a robust theoretical framework was designed, developed and implemented in real-world settings, in collaboration with the practitioners (Reeves, 2006). Situating the research work in naturalistic contexts enabled close interactions among researchers and practitioners, in their pursuit to find solutions to face the challenges in the adoption of OEP. The co-partnership link with the practitioners (teachers) was useful for researchers to test and refine design strategies in collaboration (Wang and Hannafin, 2005), according to the contextual and emerging needs during the process.
The iterative and flexible nature of the process allowed recognizing the complexities and dynamics of the real-world interactions and contextual limitations in the designs (Collins et al., 2004), and refining them as appropriate during the series of testing cycles, in the form of workshops and online activities via LMS.
The integrative use of multiple methods and a variety of data gathering strategiesboth qualitative and quantitative, as required by the DBR approach enhanced the credibility of findings (Wang and Hannafin, 2005). It also generated a large amount of "thick" descriptive data sets (DBR Collective, 2003) which required systematic analysis and consensus building around interpretation of data to provide a comprehensive view of the process.
The armoury of tools and strategies created during the iterative process generates evidence-based claims about new design solutions, guidelines and frameworks to address the contemporary issues in integration of OER and OEP by practitioners, based on theoretical relationships. Thus, DBR approach, through its coherent methodology has enabled bridging theoretical research and educational practice (DBR Collective, 2003), and refining both theory and practice (Collins et al., 2004) through their synergy.

Conclusions and implications
It was evident that OEP among practitioners can be supported and "promoted" through a DBR approach using iterative analysis, design, development and implementation of carefully planned intervention strategies at different stages. Significant changes were observed in teachers' use of instructional resources, their pedagogical thinking and pedagogical practices, due to this DBR intervention. The specific strategies designed and implemented in an intensive sequence of activities in the intervention during a series of workshops and provision of an enabling online environment supported both practitioners and researchers to gradually move through the four phases of DBR, analyzing issues, developing solutions, testing and refining solutions and reflecting on the experiences in their collaborative journey towards OEP.
The careful selection and design of activities to develop teachers' competencies in identifying, using and creating OER has facilitated a shift in resource use from no or low usage of OER to reuse, revise and creation of OER. The opportunities provided to engage in sharing resources and challenging and innovative use of OER have enacted a change in the pedagogical perspectives and practices of teachers shifting from a content-centric and individualized pattern to more constructivist, context centric and collaborative ways. The process of capturing teachers' and researchers' real-life experiences through a collection of "stories" around their experiences, enabled sharing as well as transfer of "good practices" in relation to adoption of OEP by practitioners.
The systematic and flexible methodology adopted via DBR by designing a framework aiming at improving educational practices was very useful to support changes in OEP among practitioners over time. This iterative process allowed the researchers to function as "designers", while investigating real-life issues in collaboration with the practitioners through reflective enquiry to further refine innovative practices towards OEP. This provides valuable insights for improved design solutions for future interventions in similar contexts.

Introduction
Learning analytics (LA) refers to the process of collecting, evaluating, analysing, and reporting organizational data for decision making (Campbell and Oblinger, 2007). It involves the use of big data analysis for understanding and improving the performance of educational institutions in educational delivery. Open and distance learning (ODL) institutions present an ideal context for the use of LA as, with their large student numbers and the increasing use of the internet and mobile technologies, they already have a very substantial amount of data available for analysis with analytics.
Despite LA being increasingly applied in a wide range of educational organizations, the literature in this area has usually focussed on conventional face-to-face institutions. In the ODL setting, there is yet to be a systematic review summarizing existing work on the potential benefits of LA to open and distance institutions (Firat and Yuzer, 2016;Prinsloo and Slade, 2014), and relevant research findings potentially applicable to these institutions (Rienties et al., 2016). This paper gives a systematic review of the mounting research work on LA that has been published in recent years to provide an overview of this emerging field and serves as a foundation for further exploration. It addresses the potential problems of ODL institutions that could be solved by using LA, and the benefits that could be obtained according to the existing case studies. It also presents a meta-analysis of relevant empirical studies which shows the effect of intervention for at-risk students based on the use of LA.
Related studies LA involves the use of a broad range of data and techniques for analysiscovering, for example, statistical tests, explanatory and predictive models, and data visualization (Arroway et al., 2016). Various stakeholders, such as administrators, teaching staff, and students, can then act on the data-driven analysis. Without a standardized methodology, LA has been implemented using diverse approaches for various objectives. Gašević et al. (2016) summarized three major themes in LA implementation, namely, the development of predicators and indicators for various factors (e.g. academic performance, student engagement, and self-regulated learning skills); the use of visualizations to explore and interpret data and to prompt remedial actions; and the derivation of interventions to shape the learning environment. The diversity in LA implementation poses a challenge for education institutions which plan to be involved in it, leading to a commonly voiced question -"How do we start the process for the adoption of institutional learning analytics?" (Gašević et al., 2016, p. 4).
As an emerging field of study, an increasing number of case studies relevant to the implementation of LA in higher education have been published. However, only a small number of reviews summarize these individual case studies. Among them, Dyckhoff (2011) reviewed the research questions and methods of these studies. The findings showed that existing studies have focussed on six types of research questions: qualitative evaluation; quantitative measures of use and attendance; differentiation between groups of students; differentiation between learning offerings; data consolidation; and effectiveness. The research methods used include online surveys, log files, observations, group interviews, students' class attendance, eye tracking, and the analysis of examination grades. Based on the results, suggestions were given on LA indicators for improving teaching.
Papamitsiou and Economides (2014) focussed on the impacts of LA and educational data mining on adaptive learning. They reviewed the experimental case studies between 2008 and 2013, and identified four distinct categories, namely, pedagogy-oriented issues, contextualization of learning, networked learning, and the handling of educational resources.
Also, Nunn et al. (2016) discussed LA's methods, benefits, and challenges. It was found that the methods used included visual data analysis, social network analysis, semantic analysis, and educational data mining. The benefits of LA were seen to revolve around targeted course offerings; curriculum development; student learning outcomes; behaviours and processes; personalized learning; improvements in instructor performance; post-educational employment opportunities; and enhancement of educational research. The challenges included the tracking, collection, evaluation and analysis of data, as well as a lack of connection to learning science, the need for learning environment optimization, and issues concerning ethics and privacy.
Focussing on computer science courses, Ihantola et al. (2015) surveyed LA case studies in terms of their goals, approaches, contexts, subjects, tasks, data and collection, and methods of analysis. The goals were related to students, programming, and the learning environment. The approaches included case studies, constructive research, experimental studies, and survey research. They also found that most of the research work was undertaken in a course context, with the number of subjects ranging from 10 to 265,000, with 64 per cent of the studies having 500 or fewer subjects. In most of the studies, students 22 AAOUJ 12,1 were required to complete multiple programming tasks. Over 60 per cent of the studies used automated data collection that logged students' actions, and a variety of data analysis methods such as descriptive and inferential statistics.
The existing reviews of LA case studies provide a basic descriptive summary. However, as a new area in education, there remain many uncertainties for ODL institutions about involving themselves in it. To make an informed decision on whether or not to implement LA, a key question is: "What are the expected benefits for the institution?" This paper addresses this issue by surveying the outcomes of LA implementation for institutions.

Methodology
This study aims to investigate how LA has been used in higher education institutions and the outcomes obtained. Relevant case studies were collected from Scopus, using the key terms "academic analytics" and "learning analytics" for the period from 2007 to 2016. The studies were selected based on the following criteria: (1) the study reported one or more empirical cases of the use of LA in a higher education institution; (2) the institution in question was accredited by the government or government-related bodies; (3) the institution had 1,000 or more students; and (4) the source information contained the aims of using LA, a description of the analytics, its implementation and the outcomes.
An initial search returned 1,492 results. After screening, a total of 43 cases which fulfilled the criteria for inclusion were selected for further analysis. They were analysed in terms of their objectives, approaches, and major outcomes. A meta-analysis was also conducted to synthesize the empirical findings reported in the case studies. Studies which included relevant quantitative data analysis were chosen, resulting in six studies on student support and analysis of learning behaviours, with the effect of LA intervention validated and reported.

Results
Benefits for institutions, staff, and students A summary of the objectives and approaches of the use of LA in the institutions chosen is presented in Table AI. The benefits of LA for the institutions, staff and students revolve around the following aspects.
Improving student retention. Table I presents the use of LA which improved student retention. By closely monitoring students' learning and persistence, undesirable learning behaviours and emotional states can be detected, and students who are at risk can be identified early. Factors leading to student dropout or retention can be identified and prediction models developed. Staff can take prompt follow-up action and provide proper assistance to students who need extra support, such as counselling, suggesting learning resources, and formulating individual learning plans. Students' level of achievement, as well as their retention, can be enhanced.
Supporting informed decision making. Table II shows the use of LA which supported informed decision making. Institutions are provided with information and analyses generated from a massive amount of data for informed decision making. For example, planning can be carried out on course development and resources allocation on the basis of information about the popularity of courses, and types and frequency of materials reviewed by students.

Learning analytics in higher education
Increasing cost-effectiveness. Table III presents cases of LA use which increased costeffectiveness. LA can be integrated with other platforms such as the learning management system. Instructors can then access various kinds of information online for providing feedback and support to students. Analyses and feedback on students' study progress can be delivered to staff, students, or parents in an automatic and cost-effective manner.
Understanding students' learning behaviours.
The Technical University of Madrid Information was provided by the LA system which helped to prevent problems, carry out corrective measures and make informed decisions to improve students' learning  The student retention rate for those who got support was higher than the university's average rate Atif et al. (2013) Harvard University The results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning Robinson et al. (2016) New York Institute of Technology An at-risk model of high predictive power was developed Sclater et al. (e.g. learning management systems and social networks), institutions and academic staff can understand the relationships among students' utilization of resources, learning behaviours and characteristics, and learning outcomes, which helps them to evaluate the effectiveness of pedagogies and instructional designs for improvement. For instance, the use of LA helps to capture the students' behaviours in watching course videos by highlighting the patterns of their preferences and behaviours as well as showing the parts of videos which were watched most and least frequently. Curriculum and learning materials can thus be better designed to address students' preferences and needs.
Providing personalized assistance for students. Table V illustrates the use of LA for providing students with insightful data about their learning characteristics and patterns, which can make their learning experiences more personal and engaging, and facilitate their reflections and improvements while a course is still in progress. Early alerts can be automatically generated and sent to students if their academic performance is below a There was greater interaction between teachers and students Wong and Li (2016) University of Adelaide Lecturers were allowed to assess and monitor students' collaboration in an online environment, without having to traverse a large discussion forum Tarmazdi et al.
University of Michigan The system demonstrated high scalability and extensibility Mattingly et al. The utilization of open source resources could be modified and adapted by anyone to meet specific user needs Prasad et al. (2016) University of Sydney LA features such as instant feedback and auto-grading are especially useful for instructors teaching subjects in computer science education Gramoli et al. (2016)  Students who displayed more higher-order thinking behaviours learnt more through deeper engagement with course materials displayed by their discussion behaviours These students in turn also learnt more than students who were constantly off topic in the forums Social-oriented topics triggered richer discussion compared with biopsychology oriented topics, and higher-order thinking behaviours tended to appear together within threads in the forums Wang et al. (2016) McGill University It provides an unprecedented opportunity to use data from real learners in authentic learning situations to better understand learning processes The study demonstrated how to detect learner misconceptions Prediction precision and weighted relative accuracy were significantly increased The Hong Kong Institute of Education Potential indicators were found for predicting student performance, such as the contribution of in-depth contents in online discussion Wong and Li (2016) The Open University (UK) Common pedagogical patterns were identified from learning designs, showing the relationship between learning activities and students' learning outcomes  performing poorly or are isolated from the main discussion, and then provide intervention during discussion in real time. This is especially important for ODL institutions, where students may be using different study modes and social media is a major communication channel.

Meta-analysis of the effect of interventions on student success
An important function of LA is to predict at-risk students and deliver early alerts and interventions to them, in order to improve their academic attainment, and their retention and graduation rate. This section provides a meta-analysis of the various prediction models utilized in LA systems, and the effect of the intervention solutions on enhancing students' success. Among the case studies examined, only six which provided quantitative analysis results were selected and the results are synthesized in this section. The effect sizes for each analysis were calculated where the data required for the calculation were available, and a

Institution
Major outcomes Source

Albany Technical College
Based on analysis of students' study results, demographics and social data, at-risk students were identified for providing individual counselling Karkhanis and Dumbre (2015) Bridgewater College Tutors were provided with detailed information to discuss with students on their progress against targets and suggested actions Sclater et al. (2016) Open Universities Australia Students obtained from the system recommended content and activities and a personalized learning environment Atif et al. (2013) The Technical University of Madrid The LA system provided information for preventing problems, carrying out corrective measures and improving students' learning   (2010) Purdue University Interventions were provided to at-risk students, and a higher student retention rate was achieved

Arnold and Pistilli (2012) San Diego State University
Interventions through e-mails were shown to be the best treatment within constraints, while having an impact on student achievement Dodge et al. (2015) University of Adelaide The LA system allowed instructors to be aware when particular students are behaving differently from the others for making appropriate and timely interventions

27
Learning analytics in higher education descriptive comparison of the effect sizes across the studies was made. Table VII presents a summary of the predictive models and intervention solutions employed in the six case studies; and Table VIII summarizes the results of quantitative analyses for the intervention solutions and the effect sizes for each study.
To summarize, a common approach utilized in the cases of intervention for student success was to collect and analyse data from students' learning activities and employ a specific computational model to predict and prioritize those students who were at-risk of dropping out or getting poor academic results. Based on the findings of the predictive modelling, subsequent measures can be taken for intervention. A common practice was to get academic staff to contact the at-risk students and provide personalized learning support to them. Such an approach to prediction and intervention was found to effectively enhance students' success, as measured by various indicators such as GPA, study progress, the retention rate, and the graduation rate.
According to the meta-analysis of the quantitative results, all the institutions found improvement in the students' success in the intervention group compared to the control group, although the effect size varied across different types of indicators for success and different institutions. For instance, the intervention groups in the case of Marist College showed a 6 per cent improvement in the students' final grades compared to the nonintervention control groups (Sclater et al., 2016), while the effect size was in the range of small to medium based on Cohen's (1988) convention. For the retention rate examined in Mattingly et al. (2012) for the Course Signal System of Purdue University, the intervention groups showed a nearly 50 per cent performance improvement compared to the control groups. In spite of the small sample size, the meta-analysis showed an encouraging result for the benefits of LA in aiding institutions to make effective informed decisions to improve students' learning performance and success.

Discussion and conclusion
This study shows that positive outcomes have been widely reported in relevant case studies. The results suggest great potential for ODL institutions to utilize LA for analysing existing data, which is expected to benefit their operations in areas such as quality assurance and student support. This study also reviewed various predictive models for student success which were developed and validated to identify and prioritize students who may be in need of support. The quantitative analyses confirmed that the learning performance of these students improved after they had been approached for LA-based interventions. The findings of this study thus provide various stakeholdersinstitutions, staff, and studentswith the benefits they may gain from LA.
In particular, the results related to student learning suggest that, to change students' behaviours, it may suffice to simply make them aware of their learning engagement through LA tools in relation to other students or indicate that they are at risk ( Jayaprakash et al., 2014;Sclater and Mullan, 2017). Complex data visualizations or dashboards may not be necessary. What is more important, as recommended in Gašević et al. (2016), is to help students to interpret correctly the information from visualizations or dashboards.
The meta-analysis revealed that only a few case studies related to LA implementation provided quantitative analyses dataa limitation which may be caused by the relatively new development of LA. Therefore, empirical investigations and validation of many new models and new theories in this area remain to be carried out. While an increase in the quantity of empirical and quantitative research can be expected in future, it is also important to develop and test innovative solutions supported by LA. Present LA-based interventions, as reviewed in this paper, were mostly based on the interaction and discussion between students and instructors. Although such interventions were shown to be effective in general, their effectiveness may vary among different groups of students in different contexts. Interactive-Constructive-Active-Passive (ICAP) framework It was predicted that engaging in higher-order thinking behaviours results in better learning outcomes than paying general or focussed attention to course materials Students' online discussion behaviours were categorized into three types: Higher-order -the student has contributed at least one constructive or interactive post during a course Paying-attention -the student has contributed at least one active post during the course but has not displayed any constructive or interactive posts No contribution to any on-topic discussion during the course Together with the students' other persistent characteristics, treatment and control groups were formed to investigate differences in their learning outcomes Hong Kong Institute of Education (Wong and Li, 2016) KeyGraph algorithm and Polaris (a software tool) A test-mining analytical tool was used to predict students' academic performance. The tool visualizes the hidden patterns and linkages among students' learning activities. The findings of the study showed that this approach can provide insights into predicting students' performance, and students with a higher grade tended to contribute more in-depth contents in an online learning environment Students' posts in an online learning forum were extracted and analysed -how the students presented concepts, specifically whether they can make linkage among various concepts. Such a pattern was correlated with the grades they obtained. The findings can be used to guide interventions on students' learning process, and inform ways to give feedback to improve teaching and learning Marist College ( Jayaprakash et al.,

2014)
Open Academic Analytics Initiative A machine learning algorithm and logistic regression were used to predict whether students are at risk based on their demographic details, aptitude data, and various aspects of their usage of the virtual learning environment obtained from the LA system An online academic support environment was developed containing study skills materials and community support for specialists and student mentors. At-risk students identified by the predictive model were directed to the support environment (continued ) Notes: a The results presented in the case studies of these two institutions did not involve any statistical tests and complete information for the data -that is, sample size for each category was not provided. Therefore, no effect size could be calculated from the available data; b the effect size was computed by combining the data for the second-year retention rate for three cohorts (2007,2008,2009)

31
Learning analytics in higher education A challenge in measuring the effectiveness of LA implementation lies in the difficulty of identifying the extent to which any change after the LA implementation is attributed to the LA itself. As discussed in Sclater and Mullan (2017), it may not be feasible to isolate the influence of LA when it is part of a wider initiative to develop data-informed approaches in an institution. The case studies published and reviewed in this paper would thus be biased to the institutions which only deployed LA without other measures in their data-informed approaches.
In the ODL context, work on LA remains at an initial stage. Features of ODL, such as open admission which allows a broad range of students to study the same course with very limited face-to-face interaction, are yet to be studied in relation to LA implementation. It is therefore suggested that future research can involve more fine-grained validation studies to identify the effect of the various factors involved the implementation of LA. In particular, investigation on those factors related to ODL institutions, staff and students, as well as the plausible constraints on their use of LA, would shed light on how they can benefit more from involvement in LA.
Mat, U.B., Buniyamin, N., Arsad, P.M. and Kassim, R. (2013), "An overview of using academic analytics to predict and improve students' achievement: a proposed proactive intelligent intervention", IEEE 5th Conference on Engineering Education (  Monitoring, analysis Better scaffolded online discussion to improve learning in a MOOC context Explore effects of higher-order thinking behaviours in learning Identify kinds of discussion behaviours associated with learning Investigate types of learning materials which trigger richer discussion Wang et al. (2016) 9. Harvard University Monitoring, prediction Analyse the extent to which students' responses about motivation and utility value can predict persistence and completion of study Robinson et al. (2016) (continued )

Open Universities Australia Intervention
Identify at-risk students Suggest alternative modules to students which are more appropriate for their needs Atif et al. (2013) 19. Open University of Catalonia Information collection and management Identify automatically pieces of knowledge taught in each subject Gather students' information Keep information updated Guitart et al. (2015) 20. Oxford Brookes University Monitoring Improve student experience Support progress evaluation of modules and programmes, and the identification of priorities at an institutional level  Approximately 50 percent of respondents perceived OUSL as an institute which facilitates working people by conducting part time-based or distance mode education with self-learning features. In total, 56.9 percent students perceived the role of an ODL teacher correctly as a facilitator or a guide. The educational process was perceived correctly as an ODL system by 52 percent, while the remainder also identified the system to be a more self-study and student centered flexible learning system. However, the role of a BSc student at OUSL was recognized as self-independent learners by only 36.7 percent and the majority had no clear perception of the role they have to play as an ODL student. Hence, more attention should be paid to make students recognize the role they have to play in an ODL system in order to succeed at OUSL. Originality/value -Although research has been carried out periodically on the process of ODL education system at OUSL, on the graduate (output) and dropouts, etc., not many have focused on the nature of input such as characteristic features of first registrant and their prior knowledge on ODL. As the output invariably depends on the input and the process, this type of survey is timely and novel. Keywords Open and distance education (ODE), BSc undergraduate, First registrant, ODL concepts, OUSL, Teaching methodology Paper type Case study

Introduction
Teaching at a distance is characterized by separation of teacher and learner and learner from the learning group, by replacing interpersonal face-to-face communication of conventional education by an impersonal mode of communication mediated by technology (Keegan, 1996). At present, distance education systems and open education have become a feasible and viable alternative to those who were denied education due to one reason or the other (Nigam and Joshi, 2007). Although subjects dealing with humanities and social sciences have been offered through open distance learning (ODL) in the absence of major constraints, teaching science and technology courses through ODL is more challenging, mainly due to intensive practical work (Nigam and Joshi, 2007). Student dropout rates in ODL higher education systems are typically higher than in face-to-face conventional systems of higher education (Barefoot, 2004), and a study focusing dropouts of the BSc degree program at the Indira Gandhi National Open University, India (Fozdar et al., 2006) has come out with several significant issues faced by students. The lack of time (due to employment, family commitments), distance to study centers, absence of interaction with students, insufficient academic support, expectations of the program not being met and financial constraints were some of the key issues identified in this study. Investigations by Aluwihare and De Silva (2016) demonstrate that similar reasons contribute toward the prolonged time duration taken toward completing the Engineering degrees at the Open University of Sri Lanka (OUSL).
In the Sri Lankan context, OUSL is the premier and only state ODL institution where students can pursue their studies through ODL methodology. Established in 1980 following the Open University UK concept, the institute has the same legal and academic status as any other national university in Sri Lanka. Due to the nature of its teaching methodology and infrastructure, the OUSL is able to serve a large student population spread throughout the country. Data for 2015 indicate that it caters to 42,911 students, in either the Central campus in Nawala (Colombo Regional center) or in any one of the other seven regional centers in Kandy, Matara, Jaffna, Anuradhapura, Badulla, Kurunegala or Batticaloa. The University has five faculties: Natural Sciences, Health Sciences, Engineering Technology, Humanities and Social Sciences and Education.
The Faculty of Natural Sciences conducts several study programs such as short courses, diploma programs, certificate programs, etc. However, the Faculty's primary concern is the degree program leading to the BSc degree. As at 2015, 4,261 students were registered for and following the BSc degree program at OUSL and approximately 1,300 students enroll annually.
Unlike in the other state universities, the students enrolling to follow degree programs at OUSL are comparatively diverse ( Jayathilake et al., 1997). Tantrigoda et al. (2014) state that as the majority of students enrolling for the BSc degree are adolescents who have undergone formal education, shifting to distance education with self-learning maybe a totally new experience for them. Students sometimes opt for ODL expecting it to be easy (Carnevale, 2000) and often face huge difficulties when they realize ODL requires equal if not more effort than conventional education (Fozdar and Kumar, 2006). Further, Kuruppuarachchi et al. (2016) ascertain that, the components of the education processing system (PS) of the OUSL play a comparatively different role to that of a conventional university and that the input (students enrolling) and process are comparatively different in ODL, but the quality of output (the graduate) should be equal or higher than that of a conventional setup. Evaluation of the nature of input of PS is very important for future educational strategic changes and for efficient quality products Kuruppuarachchi et al. (2016). Due to these reasons the present study was carried out with 456 (35 percent) prospective BSc undergraduates of OUSL at first registration at the Colombo Regional Centre (CRC) in the 2013/2014 academic year. The total number of new registrants for the BSc degree program at OUSL was 2,373 in the 2013/2014 academic year. Of this 55 percent 42 AAOUJ 12,1 registered at CRC. The survey explores the existing knowledge on ODL concepts of these prospective students while also attempting to get an insight into the socio-economic background characteristics of these students, in the view that the findings of the study will help ODL managers/administrators to make future direction of the BSc degree program.

Methodology
The surveying technique was adapted with a piloted structured questionnaire. The questionnaire consisted of two components. The first (structured) component: with 14 sections was used to evaluate personal, socio-economic and demographic data (Kuruppuarachchi and Gunerathne, 2014), while the open ended component facilitated the recognition of student's existing perceptions on varying ODL concepts. The questionnaire was randomly distributed at the CRC (Central campus) and collected from 456 (35 percent of students registered at CRC) prospective BSc undergraduates of OUSL at first registration for the academic year 2013/2014. The statistics for BSc student enrollment clearly shows that all other six regional centers and 18 study centers belonging to OUSL collectively represent only 45 percent of the new registrant population. As 55 percent of new registrants are at CRC, it seems justifiable to collect data from CRC and generalize results of the study for BSc undergraduates of OUSL. The open ended part of the questionnaire was adapted to explore the prior knowledge of BSc undergraduate of OUSL on "different ODL concepts." This consisted of four parts. These are: teaching-learning process of OUSL; main instructional media used in OUSL; the role of the teacher of OUSL; the future role as an OUSL student. The responses provided were scrutinized with an open mind, without any restrictions, discrimination or preformed perceptions. The responses of students for each part were categorized, then quantified and are presented as described under the result and discussion section.

Structured component
The results of the structured component of the questionnaire describe the socio-economic characteristics of BSc first registrants of OUSL.

Student characteristics and ODL knowledge
The highest educational qualification of parents or any other family member of most registrants was found to be (ca. 84 percent) below G.C.E. (A/L). This implies that the majority of parents are middle class employees. Further, the permanent residences of the new registrant to the BSc degree are mainly rural or semi-urban (74.4 percent). The monthly mean household income in year 2012 of Sri Lankan people was recognized as Urban (USD471), rural (USD291), estate sector (USD220), respectively (Department of Census and Statistics, 2013). Result of the study show that the majority of students registered for the BSc degree program in 2013/2014 academic year belong to the low income category. That is, the total monthly family income is less than SLR30,000/-(USD208) for 68.8 percent (Figure 1). The government policy of Sri Lanka after independence in 1948 was to provide free school and university education. However, after 2010 this policy was changed somewhat to allow recognized private sector and international universities to established and award degrees at a fee for those who wished to obtain them. Charges of such institutes are comparatively high. Thus, attraction of lower income categories to OUSL would be due to the affordable and easy payment scheme provided by OUSL (total Ca. SLR150,000.00 for 108 credits for the BSc degree). Similar results were seen in the Bangladesh Open University, where 58.3 percent of the learners belonged to the lower-middle-income group population (Anwarul et al., 2010).
Most of the students that registered for the BSc degree are unmarried (92 percent), employed (53 percent) ( Figure 2) and belong to the late adolescent (48 percent age 18-22, 92 percent below 27 years) age category. The age at graduation of students of OUSL in 1996 showed that the majority of BSc graduates were less than 30 years (76 percent Open University of UK, the majority of undergraduates were aged between 31 and 60 and the average age was 43 years (Feinstein et al., 2007). However, a study carried out in 1997 in Sri Lanka ( Jayathilake et al., 1997), to recognize characteristic features of OUSL students in comparison with conventional university students proved different. Our study in 2013 results in similar findings for students' characteristics but attraction of employed students to OUSL has increased compared to 1997. This is a good indication from the ODL concept point of view. However, one has to bear in mind that now there are more private universities. Some of these provide part time-based education; hence the competition faced by OUSL is higher. If the BSc program of OUSL continues to attract a majority of lower middle class or low income groups into the system, it will affect the students' intra-social interactions as well as the sustainability of the institution in the long run. Thus, the administration of the university should attempt to attract clients of other social strata as well. Most employment opportunities in Sri Lanka belong to the private sector and usually they consider the social strata, school, etc. of the candidate. If the OUSL is recognized as a university which caters mostly to low income social strata, it will affect the employability of OUSL, BSc graduates in the long run. In addition, retaining learners has always been an ongoing challenge for ODL institutions worldwide (Fozdar et al., 2006;Bandarage, 2016). According to OUSL statistics, 15 percent ±7 of new registrants of BSc program dropout without starting the academic activities as "Non-starters" (Bandarage, 2016). The student dropout rate has been high (ca.50 percent) in first and second years of BTech. study programs at OUSL (Dadigamuwa and Senanayake, 2012). It appears that a considerable percentage of undergraduates of OUSL give up their degree program without completion. One reason for this could be the "poor socio-economical background" of BSc undergraduates.

Open ended component
The primary purpose of this component of the questionnaire was to evaluate the existing knowledge on "ODL concepts" of BSc undergraduates of OUSL at initial registration. The open ended part consisted of four questions as: what is the teaching-learning process of OUSL; the main instructional media adapted by OUSL; the role of a teacher of OUSL; the role of a student of OUSL. The responses provided by students for each of these questions are tabulated from Tables I-IV, respectively. Response number Response category (example of answers given) Response frequency % 1.
Open distance system (does not strictly follow the basic qualification for entry as a state university, any person can enter, not a full time program, most activities are conducted during weekends); does not have a fixed age limit; can follow the program while working 51.97% (237) 2.
Not answered for the question 17.76% (81) 3. Self-study system (students have to study and learn concepts themselves, students are self-motivated, academic achievements depend on self-ability and efficiency) 12.06% (55) 4.
Flexible learning system (a student can select the amount of workload favorable to him) 5.04% (23) 5.
Conducting ODL system of OUSL is more student centered. Different media used to facilitate students. Selection of subject units and amount is a student's own choice 4.16% (19) 6.
Teaching-learning process is strictly part time 1.75% (08) The responses for the "nature of teaching-learning process of OUSL" clearly show that students have sufficient knowledge on this concept. For example, they have recognized that the teaching-learning process as an open distance system (51.97 percent), self-study system (12.06 percent), flexible learning (5.04 percent), and students centered (4.16 percent), respectively (Table I). Only, a very low amount of response frequencies (1.53 percent) were allocated as "OUSL is conducting formal lectures" similar to a conventional setup.
Manly educational video and electronic media 0.52% (02) The OUSL student should be a good time manager: Ability to balance time and manage arise conflicts in different roles (university, working place and home) 6.38% (24) 5.
Exam-oriented mentality: somehow OUSL students should be able to get through exams within a minimum period for future development 5.85% (22) 6.
Not answer the question 7.
Should be a hard worker and a target-oriented person 3.19% (12) 8.
Should have developed a balanced personality to solve and tolerate difficulties in the ODL endeavor, should be a different role player (as an employer, as an undergraduate, as a husband/wife/family member) 1.32% (05) OUSL teacher recognized as "a helper for studies," "a good counselor," "a facilitator "or "a good advisor" 56.9% (185) 2.
Formal lecturer role: teaching subject matter like in our school class room, formal teacher role, should be a lecturer like in other government universities 20.3% (66) 3.
No Idea on the role of the OUSL teacher 2.76% (09) 5.
Teacher should be a very friendly, kind hearted person, should able to make interactions as well as emotional attachments with students 2.46% (08) 6.
OUSL teacher should play multiple roles as a good teacher, guide, helper, etc. 0.3% (01)  (Table I). Efficient counseling and orientations programs can be recommended for those who show poor perception of the ODL teaching-learning process. Generally, the OUSL uses printed material as instruction media and to a lesser extent electronic aids as supplementary material. More response frequencies (45.7 percent) clearly highlight the existing condition in relation to the instructional material. The latest statistics issued by the Telecommunication Regulatory Commission of Sri Lanka show that active internet users have increased up to more than 5 million and the number of mobile broadband connections has doubled each year from 2009 to 2015 (Internet usage of Sri Lanka, 2016). The computer literacy of the younger generation (aged 15-29) in Sri Lanka is quite high. Of the 18-24 age group 65 percent could manage basic functions of the computer as opposed to the 25-29 year olds where only 43 percent demonstrated knowledge in computer usage (De Silva et al., 2013). We found that more than 92 percent of the BSc undergraduates registered for OUSL are below 27 years age, and as such should be computer literate. In addition, responses for the open ended part of the questionnaire recognized the main instruction media of OUSL as "purely internet" (9.73 percent) or a "mixture of internet and other electronic or printed material" (35.26 percent) ( Table II). The answer may indicate their subconscious preference to use electronic media including the internet as instructional media. In this context OUSL is still not on par with the use of technology by the younger generation. Hence, this study recommends the increase of online teachinglearning processes for BSc undergraduate of OUSL.
Responses to the "role of the teacher at OUSL" can be summarized into six categories (Table III). The majority of response frequencies correctly recognized the role of an ODL teacher as "a helper"; a "good counselor," a "facilitator" or "an adviser" (56.9 percent).But quite a large frequency percentage (37.5 percent) of responses recognized the role of an ODL teacher to be similar to the role played by a conventional "teacher" or did not respond to the question.
Most of BSc undergraduates of OUSL are comparatively young and below 27 years of age. After facing the highly competitive G.C.E. (A/L) examination, most do not gain the opportunity of entering a conventional state university. From among those who are eligible for university education, it is actually only 28.87 percent that qualify to enroll at conventional state universities in biological science discipline and only 38.12 percent in physical science in the year 2012 (UGC-Sri Lanka, 2016). The majority of students, who perform relatively well at G.C.E. (A/L) do not get the opportunity to enter state universities, due to the limitation of resources and state higher education institutes in the country. Further, 38.05 percent of students, did not qualifying for university/tertiary education based on G.C.E. (A/L) 2012 results (Department of Examination, 2016). This clearly shows that the larger percentage of students either do not qualify G.C.E. (A/L) exam or even if qualified do not get the opportunity to gain (state) university entrance. These students either select suitable middle class employment or directly enter for further studies at private institutes or OUSL. Even those who are employed may select higher education at OUSL.
Our study results shows that, 48 percent of new registrants are between the ages 18 and 22; and therefore, fresh school leavers. The role played by secondary school teachers would still be fresh in the minds of new registrants. Hence they prefer close contact with teachers. Basically, ODL students are physically isolated. However, psychologically, adolescent students prefer and need emotional contact with teachers or their peers. Situations such as these make students distressed or in the long run even give up the study program. It has been identified that 64 percent of BSc undergraduate of OUSL are psychologically distressed (Kuruppuarachchi et al., 2012a, b). This could be due to physical isolation or problems faced during adjustment to the ODL system of OUSL or both. Therefore it is recommended that appropriate orientation and counseling programs be conducted during 47 Student characteristics and ODL knowledge initial registration time (of BSc undergraduates) are strengthened to overcome these issues. The major components of the teaching-learning process of the ODL system are course development and delivery. During course development, more attention is given to develop knowledge or for the cognitive development of the students. The course delivery part of the ODL system therefore, should mainly consider the development of skills, attitudes and values. In the long run the teaching-learning process of OUSL should be reformed to fulfill student's psychological needs, in addition to the knowledge-based domains. Hence, course delivery components such as practical, day schools, and other extracurricular work should be reformed to provide more interaction between teachers and students as well as among students. For example, peer interaction sessions, active learning, cooperative learning or group learning techniques can be strengthened to develop soft skills and values and enhance social interaction to eliminate the social isolation of adolescent BSc undergraduates of OUSL.
According to the Sri Lanka Qualification Framework (2015), 12 learning outcomes should be achieved by any undergraduate of a Sri Lankan University. This includes only 25 percent for knowledge-based domains and the remaining 75 percent are mainly targeted toward attitude and skills development. Hence, OUSL too has to reform its teaching-learning process to include more attitude and skills development while maintaining its identity as an ODL institute.
The responses on "The role of OUSL students" could be summarized into eight categories. The relevant response frequencies are displayed in Table IV. Results showed that mainly response category number 1 (36.7 percent), 4 (6.38 percent), 7 (3.19 percent) and 8 (1.32 percent) (total 47.59 percent) represent clear ODL concepts on the "role of an ODL student." However, the responses of students indicate that the majority (52.1 percent) did not have a clear idea of this concept.
Central to Carl Rogers' personality theory (Rogers, 1959) is the notion of self or self-concept. This is defined as "the organized, consistent set of perceptions and beliefs about oneself." The self is the humanistic term for who we really are as a person. According to Carl Rogers, the main duty even of a counselor is to help the client understand "himself." As such the students should be helped to get a more clear perception of their responsibility and role in the ODL system. Students should be made to realize that their success in the ODL setup is clearly dependant on their own efforts, and even more so that in a conventional University setup.
The Sri Lankan school education system is mainly knowledge oriented. The G.C.E. (A/L) science discipline is a highly competitive and exam-oriented teaching-learning process. A "spoon feeding system" has been developed by A/L teachers as well as private tutors. This maybe reflected in the BSc undergraduates' inability to clearly recognize ones' role and responsibility as an ODL student. The new teaching-learning process, the physically isolated nature of the ODL system along with other problems faced during adjustment to the new system would seriously affected the new registrant to the OUSL.
ODL students are more likely to have insecurities about learning. New registrant are particularly vulnerable, as they struggle to establish study habits, time management and maintain motivation. The adjustment difficulties, financial difficulties, limited access to resources, and conflict between education and family life or work place, etc., could lead to further insecurities (Owens et al., 2009).
Due to these reasons, the academic and administrative structure of OUSL, especially in the Faculty of Natural Sciences should not be restricted to the "conventional ODL concepts and methodologies." It should be flexible, to promote students' interactive sessions, peer learning, group learning, active learning or collaborative learning systems which can be include into course delivery. In addition, the BSc undergraduate should be trained to recognize "self-concepts" and "self-learning concepts" during the time of initial registration by strengthening fruitful orientation and counseling programs. In addition to the limited existing 48 AAOUJ 12,1 on-campus orientation and academic counseling programs for new registrants in the Faculty of Natural Sciences, it has recently (2014) introduced "Peer-Assisted Study Sessions". These are conducted with the help of existing and newly graduated OUSL students as young mentors. This will help students to developed social and academic interactions and share experiences gained by past students (Bandarage et al., 2015). Unfortunately, students' participation in such academic support work has not been encouraging. Hence, new methodologies need to be developed to improve attitudes and social skills of OUSL, BSc undergraduates by way of introducing group assignments, presentations and case studies, etc. which are considered as a component of the continuous assessments.

Conclusions/recommendations
In conclusion, the majority of BSc undergraduates of OUSL represent the employed, late adolescent lower middle class. Prior knowledge on ODL concepts were developed by the majority of BSc undergraduates while not properly recognizing the "role of an ODL student." Hence, more attention should be paid to make students aware of their role in the ODL system, in order to succeed at OUSL. In addition, the following recommendation can be highlighted from the findings of the study: • Increasing online-based teaching-learning processes for BSc undergraduate of OUSL.
• To fulfill student's psychological needs, the ODL delivery components should be reformed to provide more interaction between teachers and students as well as among students, i.e. active learning, cooperate learning, group learning, case studies, etc. in limited contact sessions.
• A fruitful orientation and counseling program during the initial registration period to adjust to "self-learning," "ODL methodologies," "self-management" to work efficiently in physically isolation.

•
The teaching-learning system of OUSL should not be restricted to the "Conventional ODL concept and methodologies" and should be flexible to promote possible students interactions.

•
The academic and administrative structure of the OUSL especially in the Faculty of Natural Sciences should try to diversify its clientele. While commending the institution for providing opportunities for the less privileged, the university should make an effort to attract clients of diverse social classes for the long-term sustainability of the institute.
The majority of studies on ODL focus on the nature or improvement of the educational process or the output (the graduate). However, as the output depends on both the process and the input (students that enroll) it is equally important to pay adequate attention to this aspect and conduct such surveys on its cliental periodically as well. This will enable the institute to formulate/modify its policies on educational administration, program development, delivery, student welfare, etc. to better cater to its cliental and in the process lead to the betterment of the institution.

Introduction
Many higher educational institutions in the Philippines have started to implement web-based learning environments capable of delivering online education in a blended learning academic setting. Blended learning, also called hybrid learning or mixed method learning involves both face-to-face classroom style instruction as well as the use of online methods (Prasad, 2015). Researchers are unanimous in stating that the blended learning strategy enables educational institutions to implement a more learner-centered approach to teaching where learners are given space and flexibility to indulge with effective learning activities (Alonso et al., 2005;Hughes, 2007;Roby et al., 2013). To implement blended learning, a web-enabled tool or learning management system (LMS) is often utilized to design a particular course in asynchronous mode. Moodle, a free open-source software package used by educators to create online courses (Borromeo, 2013;Maila et al., 2014;López et al., 2016). It provides a modular design that makes it easy to add contents that will engage learners and supports a social constructionist pedagogy style of teaching (Romero et al., 2008).
The use of Moodle has been cited by several literatures as an effective tool for teacher course administrative tasks (Perkins and Pfaffman, 2006); improving student inquiry and critical analysis skills (Regueras et al., 2011); inducing self-directed learning (Woltering et al., 2009); as well as promoting collaborative activities (McLuckie et al., 2009). However, even with many cited benefits of using Moodle particularly in higher education institutions, there are still factors that need to be looked upon to ensure its effective implementation. One of the most difficult factors has to do with assessing how the utilization of the various features of Moodle within the online environment affects the overall course performance of students. Are there patterns of utilization that can lead to better success in learning and higher course grade? This aspect can be analyzed by looking into the sort of activities that students often engage with. Due to the nature of the design of Moodle, it is able to routinely collect detailed activity data on students through its log files. Unfortunately, because of the inherent difficulties in handling these enormous log data files generated online by students; teachers would not agree to analyze them manually. Traditional assessment techniques, on the other hand, do not provide appropriate measures on the kind of skills that students develop while interacting with the features of the Moodle environment (Macfadyen and Dawson, 2010).
Fortunately, in the last few years, data mining technologies have been making a lot of headway in capturing and analyzing massive amounts of data (Romero et al., 2008).
These technologies utilized techniques adopted from machine learning and text mining which have enabled researcher to gain unique insights from huge amount of data with minimal effort (Blikstein, 2011). This paper presents part of an on-going research focused on analyzing students' behavior in a blended learning environment. Hundreds of activity logs for each student were collected, filtered and analyzed using a machine learning technique known as vector space model (VSM). The paper also describes some prototypical coding trajectories generated using these logs, look on probable relationship to student's overall course performance and finally on effects teaching and learning in blended environments.

Previous work
VSM has traditionally been used to search and process important information in large collections of unstructured texts (Raghavan and Wong, 1986;Mikolov et al., 2013;Farid et al., 2016). Recently, however, there has been some progress on utilizing VSM for purposes outside the domain of information retrieval. Sreeja and Mahalakshmi (2016), for example, explored the use of VSM to automatically detect emotions in English poems. They compared the performance of VSM with a probabilistic corpus-based method and found that VSM performs better in recognizing emotions in poems mined from public websites. Fraser and Hirst (2016) investigated using VSM to detect language impairments among people with Alzheimer's disease and compare it with those from healthy controls. Initial findings showed changes in word usage in Alzheimer patients after analyzing their words when mapped in the VSM semantic space. Younge and Kuhn (2016) used VSM as a measure to detect patent similarity and concluded that VSM is a better measure to use for this purpose. Li and Zeng (2016) also used VSM as a foundational technology to develop a system that can be used to filter spams in mobile text messages.
The dynamic explosion of information in web-based educational system in recent years has required additional efforts in finding appropriate learning materials suitable for learners. Salehi et al. (2013) developed a hybrid recommender system that can overcome this problem by finding appropriate learning materials based on some specific attributes of each learner.

53
Analyzing students online learning behavior In the same manner, in this study, we attempt to apply VSM to data generated within web-based environments in the context of blended learning courses to enable instructors to overcome the voluminous amount of activity data generated by students as they interact with resources and with each other within the Moodle system. Student activity logs are a key resource for gaining insight into student behavior in online environments. Behavior patterns observation, in turn, is a necessary step in detecting students' learning style. Govaerts et al. (2011), for example, developed a tool called student activity meter (SAM) which can visualize the amount of time spent by students on learning activities and resources used in online learning environments. They found that visualizations generated through SAM contribute to creating awareness for teachers and that this awareness enables them to develop various teaching strategies. Ateia and Hamtini (2016), on the other hand, connected students' behavioral patterns to specific features of an online environment then used this to define the effect each visual, auditory, and kinesthetic (VAK) learning style will have on each pattern using the VAK. They claim that web-enabled learning systems "that are supported by a dynamic approach to detect the learners learning styles are better and more effective than traditional ones that extract learner learning style using traditional questionnaires." Similarly, the work by  mines the web usage data from Moodle to predict the student performance. They use features such as assignment, quizzes, and forum activity to predict students final ratings based on four categoriesfail, pass, good, and excellent. The paper also presents a mining tool to extract data from Moodle. The results of the paper compare multiple algorithms and show that the fuzzy rule learning algorithms and decision trees perform well with an accuracy of 65 percent. Agnihotri et al. (2015) focused on studying login data extensively and used it to cluster the students using the data they generate while interacting with a tool called Connect. The work used machine learning-based clustering techniques to group students based on their attempts, scores, and logins. Their results identified three distinct student clusters: "high-achieving students," "low-achieving students," and "persistent students" (Luik and Mikk, 2008). They also found a non-linear relationship between logins and performance based on the cluster results.
Wen and Rosé (2014) also attempted to look at the varying patterns in the behavior of students relative to their grades. Utilizing clickstream data from MOOC courses to characterize the sessions, they were able to mine student behavior within individual sessions. Results of their experiments show distinctive behaviors among students who pass, fail, and receive a distinction. This provides an indication on how different students with varying levels of course performances distribute their activities differently in online environments.
While many works in the literature describe methods to identify patterns in the student behavior, Champaign et al. (2014) posited that there is a strong negative correlation between student's skill and the time they spend doing online tasks. They likewise observed a negative correlation between the improvement in skill and the time on task. This finding provided added motivation for the current work in terms of verifying whether similar correlations also exist in and among students exposed in blended courses.

Research questions
The main goal of this research is to find effective ways to sift through the vast quantity of data generated by web-based learning environments. In particular, it aims to look into the action log data maintained by Moodle to determine whether processing models can be developed that can extract useful information that instructors can use in monitoring class activity. This involved the extraction of sample log data for selected blended learning 54 AAOUJ 12,1 courses offered at Jose Rizal University ( JRU). Specifically, the study addresses the following research questions: RQ1. How can students' learning behavior be extracted and visualized from activity logs recorded by Moodle?
RQ2. Can Moodle's action log of student's online activity offer meaningful insight into students' course performance?
RQ3. Does the demographic profile of students have any effect on their level of activity in an online learning environment such as Moodle?

Data set
The data analyzed and processed in this exploratory research was extracted from various blended learning courses offered at JRU during the second semester of SY 2015-2016. "A blended learning course is defined as a formal education program in which a student learns at least in part through online learning, with some element of student control over time, place, path, and/or pace" (Blended Learning Definitions, 2017;Horn, 2013). The courses under study include Elementary Statistics (MAT22), Human Behavior Organization (MGT26), Engineering Management (EGR36), and Ethics in Information Technology (ITC56). These courses are being offered to undergraduate students taking up BSA, BSCpE, and BSIT, respectively. These blended courses were chosen because they all served as pilot implementations of the Course Redesign Program (CRP) of JRU where extensive use of the Moodle environment was introduced to enable instructors to deliver course contentslearning materials, supplemental links; promote student engagement thru online forums and chats; assessment tasksquizzes and assignments more effectively. As such, the online structure of each course consists of coursesreadings, assignments, exercises, lecture quizzes, and a final exam, which the students are required to complete with a minimum grade of 3.5 to pass the course.
In this setup, instructors are still required to spend an one hour lecture time with the students each week, afterward, their tasks consist mainly of monitoring students' online activities while students, aside from attending class lecture will need to spend a minimum of two hours laboratory time with Moodle every week at their own discretion.
To support this setup, a pre-designed course content template is already placed in the Moodle course resources before the start of each class; however, individual teachers can customize this template by uploading additional learning materials like PowerPoint presentations, video clips, and web resources. They can also require forum participations and add additional exercises, assignments, and quizzes as they deem fit. Students can browse the contents independently through individual accounts. Some student accounts can only be used locally in the laboratories but there are also experimental accounts which are cloud-based and can be used online and accessed conveniently anywhere. Many students even access the online courses using mobile devices.
The predefined course template divides each course into several modules. For each module, students were asked to complete topic-related readings and perform the prescribed exercises, on scheduled dates, and take the online quizzes. Students can also download content materials and exercises and work on them offline.
Relative to this setup, Moodle has built-in features that can produce several types of reports that can be used to track student activity. One of these reports, called action logs, enable instructors to keep track of which resources and activities in a course have been accessed, when, and by which student. For the purposes of this study, logs of students' action for the entire semester for each of the courses were collected then cleaned up. This resulted in a data set with a total of n ¼ 199 students.

55
Analyzing students online learning behavior

Action logs
Each event record in the raw action log has six attributes (see Table I): course name, time of the event, IP address, username, action, and information. In this study, we only focused on the username and action attributes, the other attributes from the raw data were reserved for future use. The action attribute represents actions initiated by students on various items that can be accessed from Moodle such as assignment, quiz or assessment, course content, forum discussion, resource, and URLs. The actions that can be performed on these items include: • view individual and view allopening the items on Moodle; • view forumopening the forums; • forum add discussionadd or post a forum topic; • submitupload completed assignments or quiz; and • submit for gradingsubmit the uploaded assignments or quiz for grading. Table II provides the total number of action log records extracted for each course as well as the average number of actions per student. These logs constitute actions initiated for Moodle tools identified previously.

Student demographics
Finally, student demographics were taken by means of a structured survey using purposive sampling, participants in the survey conducted were the same students whose activity logs were extracted and processed. These data as shown in Table III are essential in identifying possible focal determinants of students' online behavior exhibited by Moodle recorded action logs. These attributes would be checked later to determine whether students online activity is affected by gender, year level, enrollment status or the number of CRP (online) course they are taking for second semester.
The two latter attributes (device ownership and access mode) would help describe how students took advantage of the mobility factor of an online course relative to the blended

Data dimension Description
Course Identification string of the course in which the action is related Time Date and time stamp of when the action was executed IP address Unique numerical label assigned to the device used by the user User full name The user who initiated the action Action Type of action initiated Information General information on learning activities Respondents with no PC or have a computer but no internet access relied on using JRU Open Lab (33.33,19.05,25.00,and 18.92 percent). This implies a positive observation because it implies that students who lack personal devices can still access course contents and perform online tasks through the university infrastructure as provided in the open lab.
Use of mobile device was differentiated from use mobile device home/mobile by verifying IP address stamped in each action log (the IP address is a set of numeric values that specifically identifies the device being used by the student). Another notable item in the survey shows that MAT22 students have taken more CRP (Moodle online) courses during the semester compared to the other three courses.
5. Analysis of activity data 5.1 Extracting and visualizing learning behavior After data collection, the first question that was addressed is how to process the data set and extract patterns of activity that can be used to visualize students learning behavior. Following the mining process described by Romero et al. (2008), as shown in Figure 1, a two-phase process was used which include initial preprocessing of data then afterwards applying data mining algorithms that transform the data into a form suitable for interpretation and evaluation. In the context of this study, Moodle log data were collected from JRU LMS for a particular CRP course as depicted in Figure 1.  (Mohamed, 2014). In this phase, the raw log files were first processed to clean and prepare it for further processing. This is critical because many of the data sets extracted in Moodle can have missing values, noisy data, and/or irrelevant and redundant information. For this purpose, the raw log files were first imported into an Excel worksheet. Here, the actions logged by instructors and course administrators were selectively removed and the data set was anonymized by removing each student's name and replacing it with a unique identification number. Processing then started by filtering the data set by course, user identification, and action. Then, two-dimensional tables for each course were built containing the list of student identifiers as row headers and specific types of actions as column headers. Table IV presents the set of action types used in this study for analyzing the students' online behavior. The key aspect of these actions is that collectively, they can be used to represent the different types of activities that students can engage with inside Moodle, that is: accessing course content, engaging with peers, and taking assessment tests. The key assumption here is that student's actions indicate intentionality which in turn provide clues, as to their learning preferences. Thus, when categorized based on class activities, the actions helps to infer whether the student prefers to study by accessing learning materials, by engaging with peers and/or the instructor or simply by taking assessment tests. Each cell in the two-dimensional table was filled with values representing the total number of times each action type was initiated by each student. Figure 2 shows a sample table generated after pre-processing the raw data files. The process of counting this value

59
Analyzing students online learning behavior was automatically done using a customized Excel macro. The total counts extracted for each action type are shown in Table V wherein course views, view forum discussions, view forum, and assignment views have the highest occurrence while quiz view, quiz attempt, add forum discussion, and URL view are relatively low and assignment submit, resource view, and assignment view actions are the fewest actions initiated by the students. 5.1.2 Data mining algorithm -VSM. Data mining algorithms enable extraction and visualization of patterns of activity that can be used to infer students' behavior.
VSM, a statistical model representation often used in processing documents in information retrieval (Raghavan and Wong, 1986). The main idea behind VSM is to construct vector representation for documents and use these vectors to analyze and compare the contents of each document. A vector is simply a labeled set of values arranged in a specific order. In the case of VSM, the labels are the unique words that occur in the document and the values refer to the number of times each unique word occurred in that document. So for example, if there is k number of documents to be represented and these documents contain n number of unique words, a k × n matrix can be built as shown in Figure 3. In this matrix, D 1 to D k represent the set of documents while W 1 to W n represent the set of unique words. The values in each cell represent the number of times a specific word W occurred in a particular document D. Each row in this matrix is considered a vector representation for its corresponding document.
The analogy used in VSM is that the vector representation acts as a sort of coordinate that can be used to plot the position of the document in an n-dimensional semantic space where n corresponds to the number of values in the vector. Figure 4 depicts what a threedimensional semantic space looks like along with the documents plotted in this space using vector representation.
Using this analogy, to compare the contents of documents VSM simply determines how far the location of their vector representations is within the semantic space. For instance, to determine how closely related the topic of document D 1 is to the topic of D 2 , VSM simply  measures the distance of D 2 relative to the position of D 1 . The most common method for measuring this distance is by calculating the cosine of the angle formed between the two locations (represented by the symbol ⊖). The formula for computing the cosine angle is as follows: Basically, for two vectors with n values, this formula simply computes the scalar product of the two vectors for the numerator; computes the product of the length or norm of the two vectors for the denominator. So for example, if vector x is represented by the values (1, 1, 1, 3, 0) and vector y is represented by the values (0, 0, 0, 1, 1) the computation for the resulting cosine angle is as follows: The cosine formula returns a value between 0 and 1. The rule in VSM is that the more similar the contents of two documents are, the higher their cosine value will be. So a cosine value of 1 for two documents means that the documents are completely identical and a value of 0 means they are totally unrelated. Any value in between reflects the degree of comparison between documents, the higher value means documents are highly related. 5.1.3 Representing student activity using VSM. Given the previous discussion, representing student activity using VSM requires the construction of activity vectors for each student. An activity vector can be defined as simply a list of action types with their corresponding values depicting how many times each action was initiated by the student. Here, a value of 0 means that the action type was not initiated at all. For instance, in Figure  There are two ways by which this vector representation can be used. First, it can be used to compare students' activity to each other in order to group them based on how similar their level of activity is. Second, it can be used to assign students to a predefined set of categories based on how close their activity level vis-à-vis defined activity level for a specific category. Both cases will enable the identification of similar characteristics that occur within each group of students. In this paper, the latter approach is explored. The color-coded header in Figure 3 indicates the type of class activity to which the action type belongs, such as content access, engagement related, and assessment activity. These sets of activities can be used to classify students to determine which type of activity they implicitly prefer. To do this, an archetypal activity vector for each activity class needs to be constructed. This can be done by setting the corresponding action types for each activity to a non-zero value while the rest of the action type values are set to 0 as shown in Table VI. Thus, the archetypal vector for each activity class would be as follows.
To classify, each student's activity vector will be compared to the archetypal vector of each activity. The student will be grouped accordingly as per archetypal vector which generated the highest cosine or similarity value. Student activity vector can also be analyzed and grouped and compared on a per course/class basis.

Analysis of correlation between activity level and course grades
Prior to delving further in analyzing activity logs, it is necessary to first determine whether there is a relationship that exists between the action types initiated by the students and the students' course achievements, what the direction of the relationship is and its strength of magnitude. For this purpose, the student's final course grade is treated as an indicator of course achievement; which can reflect both student knowledge and level of engagement. The goal is to gain insight into how students' actions in the online environment correlate with their course grades. Pearson coefficient correlation (r) was used to investigate the significance and computations were done by importing the excel data worksheet to SPSS.

Analysis of the effect of students demographic profile with level of activity
Correlation and descriptive statistics were conducted to examine whether student demographic attributes, namely, gender, year level, enrollment status, and device ownership could affect the level of LMS utilization as exhibited by total activity logs. Descriptive statistics used to determine mean activity logs of students while Pearson coefficient (r) was also used to establish possible relationships.

Results and discussions
The development of easily interpretable graphic that can depict trends in student activity based on action logs is a useful tool for instructors to constantly visualize and monitor course progress with minimal effort. Each line point in the graph ( Figure 5) shows representative visualizations of the cosine values generated by each student in their respective course. Although students are anonymously depicted, the graph depicts the degree of activity among the participants, and can possibly be even be refined to drill down to each individual student's level of activity. The visualizations clearly depict some patterns of online behavior relative to three different activities: content access, engagement, and assessment. It indicates that different classes vary widely in how they utilize the tools

63
Analyzing students online learning behavior provided within Moodle. And that, even within a certain class, students undertake complex behaviors in allotting time between different tools and activities. Students from MAT22, for example, generally login into Moodle mainly to do assessment tasks with little access to educational resources and engagement. Whereas EGR36 students seem to prioritize content access and engagement over access to assessment tasks; students from MGT26 and ITC56, on the other hand, while more or less showing equal interest on content access and assessment show very little interest in engagement. These visualizations can help course administrators to determine the type of strategic interventions that each course would need to ensure that student's activities are kept within intended learning outcomes. Unfortunately, while some of these online activities aim to provide effective teaching strategies, the visualizations (along with the cosine ratings) does not seem to correlate with the students' course accomplishments. This suggestion comes from observing that some course with a low level of online activity (MGT26 e.g. average cosine: 0.33) have higher average grades (e.g. 2.7) than a course with higher levels of activity (e.g. EGR36, average grade: 3.46, average cosine 0.47). This seems to suggest that longer time spent on Moodle may not result in higher course achievement. An implication of these observations is a need to redesign the online component more effectively in order to achieve quality instruction. Another issue that the visualizations reveal is the lack of a standard teaching approach. Since students' activities are often governed in part by the teacher requirements, what the visualizations indicate is that teaching methods among different classes seem to vary. Some instructors mainly focus on uploading lectures, while others focus more on assessment tasks or activities; some require a certain level of engagement among their students. More studies are needed to determine which pattern of teaching approach would be most beneficial to the students.
A simple correlation analysis was conducted, in order to determine whether there are relationships between the action types initiated by students and their course accomplishment represented by the final grade given to them by their instructors. As shown in Table VII, the results seem to suggest that there is some variability in terms of positive and significant correlation in the final grade between courses. In Mat22, for instance, the only action which shows correlation with the final grade is the URLView (r ¼ 0.266, p ¼ 0.01). Whereas, in the EGR36 course, all the actions correlated positively except AssignmentSubmit and AssignmentView ( p ¼ 0.01). For MGT26, it is the QuizView and QuizAttempt actions that correlated positively ( p ¼ 0.01) and for ITC56 it is ResourceView and QuizView shows medium to high correlation at p ¼ 0.01 and small correlation for AssignView at p ¼ 0.05. This variability in terms of correlation, to some degree, seems to agree with our previous observation regarding the lack of correlation between the students' online activity level and final grades. In other words, in some cases, it may correlate but in others, it may not. The magnitude of correlation also varies from activity type which reflects how students have prioritized tool or activity performed in the online environment. The best explanation for this observation is that instructors are considering other factors in assigning grades to the students which may not be present in the online environment. The analysis of correlation coefficients for student demographics and total action logs (TAL) obtained as depicted in Table VIII can be observed per course wherein MAT22 has relatively negative, positive small correlation on gender, enrollment status, and device ownership while statistically significant relationship between TAL and CRP courses taken (0.356, p o 0.01, 58). This was contrasted by results for MGT26, with negative/ positive medium to high correlation was significantly established for gender and enrollment status (−0.564, p o 0.01, 0.506 at p o 0.05, 19) while EGR36 got a negatively small coefficient for gender (−0.305**, p o 0.01, 78). ITC56 results did not show any significant relationship among any factors considered. Closer analysis of the IP address stamped on the log hits indicated that students still displayed a high level of reliance on computer units provided in the open laboratories. In effect, this nullifies the mobile access advantages of the online course. It was also observed that majority of the task performed online are assessment related. This inference could be attributed to the environment of blended learning implementation of the university where students despite mobility and availability of learning resources online still rely on classroom discussions performed during the face-face session which is one primary structure of a blended mode. Likewise, prior experience in LMS tool indicated by the number of CRP course taken is not a factor on students' online LMS activity level nor students' achievement.

Limitations and future work
The paper showed how students of various courses utilized the LMS system, it may not be indicative of the overall effectiveness of the system but a structured analytical study of actual online activity thru a data-driven approach gave important highlights. In summary, this work has shown that VSM can be used to aggregate the action logs of students and quantify it into a single numeric value that can be used to generate visualizations of students' level of activity. While the visualizations do not seem to depict course performance, the value it presents is in terms of helping instructors monitor course progression and enable them to rapidly identify which students are not performing well and adjust their pedagogical strategies accordingly. Since the VSM visualizations generated in this study, for the most part, relied on the structure and contents of the activity reports generated by Moodle, adopting the same methodology to other LMS will necessarily involve minimal adjustments, specifically, in the construction and generation of the activity vectors. Nonetheless, once the necessary adjustments have been made, VSM can be applied to other courses even with varying designs and structure.
A plan to continue the work by developing a complete dashboard style interface that instructors can use is already underway. The study also looked into whether or not various action types can be used as indicators of student's class performance. The current investigation indicates that there is a lot of variability in terms of the correlation between these two variables. It is highly likely that the design and nature of the course as well as the individual teaching strategy of instructors are introducing other factors that are not present Analyzing students online learning behavior in the online environment. A comparative inquiry on a per subject basis can also be done to explore the effectiveness of a particular course module taken by students in the different disciplinal area. More data need to be collected, and more advanced processing tools are needed in order to obtain a better perspective on this issue.

Introduction
The success and effectiveness of the distance education system "depends heavily on the study material, particularly in the form of printed course material" ( Jayaram and Dorababu, 2015). Quality in the Open and Distance Learning (ODL) institutions and their programs are often judged in relation to study material because "student learning is at the center of the ODL experience" (Kirkpatrick, 2005;Hashim, 1999). These are the pivots on which the whole learning pedagogy revolves. ODL learners in Malaysia are also very concerned about the quality of learning material. This was borne out in a recent survey conducted among post-graduate learners in the Open University Malaysia (OUM) in which "good quality modules or study guides" was listed as one of the five items that were important to the learners (Abdol Latif et al., 2015). The survey concluded that if the course contents, among others, are positively addressed, the learners are likely to continue studying at the OUM and attain their educational goals. At the end of the day, the ODL learners aspire to achieve three main outputs, namely, "gaining new knowledge, acquiring new skills and most importantly obtaining higher qualifications" which they regard as necessary for their social and economic progress (Idrus, 2006).
Historically, quality assurance (QA) first gained prominence in the fields of engineering and manufacturing, as a means of ensuring that quality products are made. The quality gurus have provided a number of definitions of quality, such as conformance to customers' requirements (Crosby, 1979) and fitness for purpose or use ( Juran and Godfrey, 1999). Deming, on the other hand, pointed out that customer's definition of quality is "the only one that matters" (Deming, 1982). Hence, QA and quality control (QC) are part and parcel of attaining quality. Moreover, QA is "a broad concept that focuses on the entire quality system" and includes quality objectives, whereas QC looks at the micro level and focuses on the process of producing the products (ASQ Statistics Division, 1983). In short, QC can also be viewed as a subset of QA (Eriksson, 2016).
From the perspective of ODL, QA should cover a wide range or areas including curriculum design, content and delivery, and the teaching and learning process (Pitsoe and Maila, 2014). In this paper, we will explain and illustrate how the Centre for Instructional Design and Technology (CiDT) at the OUM implements QA in the print module development at various stages. We will also touch on the quality management system (QMS) established in CiDT since it was first awarded the MS ISO 9002:2008 certification in 2005. Finally, we will look into the satisfaction level of learners with OUM's learning material.

QA in ODL
Around the world, the burgeoning demand for ODL institutions to provide opportunities to learners to pursue higher education has shifted the attention of stakeholders and interested parties to the importance of quality. In light of the increasing competition with conventional universities, the ODL institutions need to offer quality courses and modules that satisfy learners and make them feel there is value for money and the education is a good investment; this in turn ensures that the ODL institutions can continuously "sell and survive" (Koul, 2010) and, at the same time, overall improvements in quality in addition to openness and transparency become the key drivers of "competition and collaboration in education and research" (Anderson, 2008;Ossiannilsson and Creelman, 2012).
With respect to higher education, QA has been defined as the "systematic management and assessment procedures adopted by higher education institutions and systems in order to monitor performance against objectives and to ensure achievement of quality outputs and quality improvements" (Harman, 2000). Alternatively, QA has been defined as "a set of activities or procedures that an organization undertakes to ensure that standards are specified and reached consistently for a product or service" (Robinson, 1995) as well as a set of activities that provide "value for money and transformation capabilities" to ODL students (Gandhe, 2010). In a nutshell, QA "helps disseminate best practices, with the prime objective of leading to overall improvement of higher education systems" (Belawati and Zuhairi, 2007).
In addition, an ODL institution can build its good name by having "a systematic and consistent QA system" resulting in a positive impact on its stakeholders, e.g. more satisfied students, more competent personnel and better public image; meanwhile, students too will benefit as they achieve "enhanced learning outcomes from quality instruction, learning materials and interactions with the institution and its staff" (Kirkpatrick, 2005). Hence, providers need to emphasize quality in various aspects such as products, processes, production and delivery systems (Commonwealth of Learning, 1997). For educational institutions, including the ODL, ISO 9000 assures quality in terms of programs and courses, meeting the intended goals and needs of the users, in particular, the learners (Koul, 2010).

QA related to tertiary education in Malaysia
In general, there are two best-practice QA and accreditation agencies in Malaysia, which are Malaysian Qualifications Agency (MQA) and Scientific and Industrial Research Institute of Malaysia (SIRIM).

70
AAOUJ 12,1 MQA was "established under the Malaysian Qualifications Agency Act 2007 by the Ministry of Higher Education" (Malaysian Qualifications Agency, 2009) and is responsible for QA of higher education for both the public and the private sectors in Malaysia. One of the initiatives of MQA is carrying out a rating exercise of Malaysian Higher Education Institutions at the undergraduate level, known as SETARA. In 2011, the ODL category was introduced for the first time to rate the quality of teaching and learning at the ODL institutions in the country, "with 23 criteria and 52 indicators for ODL institutions which include process dimensions such as quality delivery and pedagogy" (Mahsood and Chenicheri, 2016).
MS ISO 9001 is one of the Malaysian Standards developed by SIRIM, whereas ISO 9001 is an international standard that many companies use to ensure that their QA system is established and effective. ISO 9001 demonstrates "an organization's ability to maintain quality in products and services, meeting the requirements of customers and regulations" (Scientific and Industrial Research Institute of Malaysia, 2016).
QA for the ODL programs and institutions is one of the main concerns among institutions and stakeholders today. This is in line with the Malaysian Education Blueprint (2015-2025) which aspires to see lifelong learning as a way of life for all Malaysians. The blueprint further stipulates that "there will be high quality formal, non-formal, and informal programs in a wide range of disciplines and topics to support both professional and personal development" (Ministry of Education, 2013).
A brief background of the OUM and the CiDT OUM has had a long history in developing study materials for its distance learners. OUM was the first ODL institution in Malaysia and commenced operations in 2001, beginning with the face-to-face mode of learning aided with printed modules and a learning management system platform. Since its inception, the University has been at the forefront championing an accessible, affordable and flexible approach to learning via the ODL mode. OUM has also been highly ranked in the SETARA ODL category with an excellent Tier 5 ratinga testament to the overall quality of the university (Malaysian Qualifications Agency, 2013).
The university has given working adults in Malaysia the opportunity to pursue higher education anytime and anywhere via the ODL mode. Over the years, the university has grown from strength to strength and this is evident from the fact that presently the OUM has 154,000 cumulative learners, 1,000 international learners and 70,000 graduates. The university also employs about 8,000 part-time tutors, 100 full-time faculty members and 500 support staff. Unlike the other ODL providers in the world, the OUM is unique in that it is owned by a consortium of 11 public universities. These public universities have been generous in lending their prestige and expertise in the various fields; this collaboration has added value to the wide range of programs provided by the OUM. Presently, the University practices a blended pedagogy which comprises self-managed learning, face-to-face tutorials and online learning. This pedagogy is dependent on print-ready self-instructional learning materials, commonly known as "modules." At the OUM, the CiDT is primarily responsible for the development of learner-friendly and specially constructed modules (Open University Malaysia, 2008). As of May 2016, the CiDT has developed 1,392 print-ready modules, 366 study guides, 865 hypertext markup language (HTML) modules and 181 video lectures. These HTML-based modules help to enrich the learning experience with colorful graphics, interactive activities and exercises, embedded audio and video files as well as links to other online learning resources. Since the HTML modules and video lectures are sourced from printed modules, it is vital for the module development process to undergo a stringent QA process.

QA in learning material development
On October 2005, the CiDT was among the four learner support service centers in the OUM to be awarded the prestigious MS ISO 9001:2000 certification (Ali and Fadzil, 2013). The growing number of learning materials made it vital for the center to embrace a QA culture. Moreover, the center is the largest unit in the university comprising more than 60 staff with expertise from various disciplines.
The importance of quality print modules in the OUM When the CiDT opted to pursue the MS ISO 9001:2000 certification, the scope was limited to printed learning materials for programs offered by the OUM. Currently, there are five core activity procedures for print modules: (1) acceptance, selection and appointment of subject matter experts (SMEs); (2) instructional and graphic design; (3) desktop publishing; (4) management of module writing and moderating workshop; and (5) measurement of customer satisfaction.
Ideally, quality learning materials in an ODL setting must be "simple to interact with and should be understood by learners," so much so that the materials should be self-explanatory, requiring minimal or even no assistance from tutors (Adegoke and Oni, 2015). It has been shown that learners who scored poorly on reading tests were more likely to drop out before completing their courses (Macdonald-Ross and Scott, 1996). If learners are unable to understand the module, they may underperform in their exams, lag behind in their studies, become demoralized and eventually contribute to the high attrition rate of learners at the ODL institution. In fact, the factors for the high attrition rate among ODL learners include "study material in tough or difficult language and delay or non-receipt of study material" (Kumari, 2012). Meeting deadlines is also very important, as failure to achieve the set deadlines can affect other subsequent processes and lead to "devastating consequences" (Belawati and Zuhairi, 2007).
In addition to this, QA is very much output-outcome driven. Initially, the OUM identified three outputs of QA in the ODL for the University, namely, "graduates with new knowledge, graduates with new skills and graduates with new qualifications" (Idrus, 2006). New skills here include language skills in the English language and relevant skills, such as skills in Finance that will allow learners to perform finance-related calculations. As the learners are mostly working adults, the outcomes among others include job promotion, respect in the community and the pursuit of new interest. Additional relevant skills acquired after graduating from the OUM will be part of the additional list of accomplishments to show value that will be useful for "moving up the corporate ladder" (Zhang, 2014).
Hence, the accuracy of the content in learning material is crucial. Information must be up-to-date and new changes must be taken into consideration as these will "ensure the appropriateness and quality of the learning materials" (De Fazio et al., 2012). Therefore, there should be zero-tolerance for errors such as grammatical error, factual error or mistakes in formula or calculations in the learning material. Learning material which is difficult to understand will contribute to high attrition (Kumari, 2012), whilst "inaccurate and outdated learning material" will not make the learners achieve the desired outcomes (Idrus, 2006).
All five of the SOPs were designed to ensure that QCs are in place so that the print-ready modules are of benchmark standards that satisfy the learners' expectations. Since the CiDT was first awarded the MS ISO 9001:2000 certification in 2005, the center has undergone a number of quality audits which saw corrective and preventive actions (CAPAs) implemented by way of PDCA or Plan, Do, Check and Act. This requires mastering each 72 AAOUJ 12,1 process, knowing how to govern it in a way that enhances communication, transparency and accountability. Understanding and analyzing the root cause of problems is imperative to prevent the same problems from recurring (Olzak, 2008). This calls for stringent QC measures which will be discussed in the next section.
The QA process for print module development At the CiDT, there are three stages involved in the QA of print module development. These include the pre-development, actual development and post-development stages. Figure 1 shows all the stages and the respective QC measures that will be explained in the following subsections.
QA at pre-development stage In this stage, there are four QC measures in place to ensure that the OUM modules are of benchmark quality.
The first QA check at this stage is the selection of module writers. This has always been stringently done. The modules are written by writers who are (SMEs) nominated by faculty and vetted by the CiDT to ensure that they have the right academic credentials and relevant work experience in their respective fields. Following ISO requirements, the records of all writers are well-maintained and documented.
The second QA check is the module writing and moderating workshop. Upon appointment by the CiDT, the writers and moderators are required to attend a module writing and moderating workshop where they are briefed on general principles of instructional design, house style and editorial format, timelines and expectations, graphics, referencing and citation, as well as the rubrics for evaluation and payment. Attendance at this workshop is a pre-requisite for module development at the OUM, as everything begins with the writers and moderators who must be clear what the requirements and expectations are. Furthermore, writing modules for ODL learners is "an art in itself" because its interactive styles and techniques are distinct from conventional textbook writing (Rahman, 2006;Hashim, 1999).
At this workshop, the writers and moderators are also introduced to the module development team from both faculty and the CiDT so that they feel a sense of collegiality and esprit de corps for the journey that lies ahead. The development of learning materials by a team of SMEs, academic moderators, module writers, tutors and consultants is part of the

73
QA in learning material development OUM strategy of giving priority to learners and providing them with quality services and products (Kirkpatrick, 2005).
The third quality check is enforced once the module content comes in. This comes in the form of a plagiarism/similarity check via Turnitin © software to ascertain whether the content submitted by the writer is original and not taken from other sources. Turnitin © is a popular web-based plagiarism detection software used in Malaysia and abroad that is "able to match the work submitted by a module writer against works done by other writers from its huge database repository" (Rogers et al., 2009). If the content is found to be highly plagiarized, it will then be returned to the writer for amendment.
Besides plagiarism, the raw content will undergo a screening process conducted by the instructional designers (IDs). Detecting plagiarism is only one aspect of screening. Furthermore, the IDs will ensure that learning outcomes incorporate the Blooms Taxonomy (a progression from lower order to higher order thinking skills) and the desired outcomes. This allows the learning outcomes to be stated in "measurable behavioral or operational verbs" for the learner to monitor his or her own progress (Koul, 2010). Behavioral verbs have two distinct advantages. First, we can use them to "devise activities that make the student an active participant in his or her learning" and second, "the verbs may be used to help to translate the course goals and objectives into situations that are more concrete to the learners and others" (University of North Carolina at Charlotte, 2016). At this stage, contents are checked against the table of contents (TOC) to ensure that they are in accordance with the TOC given by the faculties. References are standardized according to the APA style used in the OUM. A ten-item checklist identifying the elements that need to be present in the content is used as a tool by the IDs throughout the screening procedure. If there are any issues such as plagiarism, improper learning outcomes or improper in-text citation and references, the content will be immediately sent back to the writer for amendments.
Only after these three QC checks have been carried out is the content forwarded to a moderator, an SME in the same field who is nominated by faculty and appointed by the CiDT. The role of the moderator is to ensure that what has been written follows the specifications spelled out by faculty. So, in this sense, the content is vetted for breadth and depth in terms of scope and coverage by a second expert in the field. Moderators are selected based on specific criteria, such as being of similar, or higher, academic standing as the writer. Professionalism is a necessary ingredient in the production of quality learning material (UT, 2002); hence, the proper selection of competent writers and moderators is taken seriously at the CiDT. If the moderator feels the content is lacking or inappropriate in any way, the writer is obliged to make the necessary amendments based on the moderator's feedback.

QA at the actual development stage
This stage sees the involvement of IDs, editors, graphic designers (GDs) and desktop publishers (DTPs). At the CiDT, there are currently 11 IDs, 5 editors, 12 GDs and 10 DTPs who have expertise in different disciplines. In addition to these different units, there are four dedicated Project Coordinators or PCs for each faculty at the CiDT, and they are responsible for liaising with both internal and external parties in module development for their respective faculties and in coordinating the movement of modules to completion. The PCs play a critical role in the overall coordination of module development as they track the progress of each and every module as early as the pre-development stage, to ensure the modules are delivered within the stipulated timelines given to the writers in the appointment letter. The whole module development process can be likened to "the procedures in an assembly line" and any delay at the preliminary stage can have a devastating outcome on subsequent downstream stages (Belawati and Zuhairi, 2007).

74
AAOUJ 12,1 At this stage, the module content is first assigned to an ID who will develop the module based on the quad alignment model shown in Figure 2. All the four components of the model, namely, body of content, learning outcomes, assessment tasks and learning activities are explained and illustrated during the module writing and moderating workshop. Besides understanding each component, it is vital for all the components to be aligned with each other, that is "support and be directly relevant to each other," thereby helping the learners to easily understand the modules (Fink, 2007).
Hence, the IDs and the writer have to work closely to ensure that every topic is "quad-aligned" so that the learners will find the module easy to comprehend. Apart from writers, the relevant faculty members who are known as program coordinators also provide academic input to beef up the content and ensure that the learning outcomes in each topic are elaborated adequately.
The content in a module must be designed in a systematic way to facilitate learning without regular supervision of a tutor. The content must also be interactive with "an element of guided didactic conversation that can create warmth, closeness and two-way communication between learner and materials or between learner and the module writer" (Holmberg, 1977). Toward this end, IDs have specific roles to play in ensuring that "the learning outcomes are clear to the learner, content is arranged in the proper sequence, there are adequate activities, diagrams and illustrations to aid learners and also assignments to regulate learners' progress" (Koul, 2010). In some cases, the ID may suggest to the writer parts of the content that should be revised or rewritten to better meet the learning needs of learners. For instance, a well-written introduction to a topic can grab a learner's attention and arouse his/her interest to continue reading the module.
Among the activities that can be found in the OUM modules are self-checks and self-tests, i.e. self-assessments; for post-graduate learners, the modules include several case studies as well. These are designed to make learning interesting, interactive, meaningful and collaborative. Some of the activities also encourage learners to communicate with their peers through the e-forum and post their answers on myINSPIRE, the learning management system of the OUM.
The content is then edited to improve the language, flow and readability of the content without compromising on the original meaning. The upshot of this is that the content becomes more readable, self-explanatory and easier for the learner to understand (i.e. written in a user-friendly language), requiring minimal tutor guidance. The editor also ensures that the content meets the in-house style standards and verifies headings, statistics and data in graphs and figures. After the editing process, the IDs then work closely with GDs to visualize and incorporate graphic elements to enhance the overall visual presentation of the content. A picture is said to be worth a thousand words; thus, the GDs play an important role to ensure the graphics and diagrams can make the module content more coherent to the learners. Finally, the content is sent to the DTPs for layout in the OUM in-house style to maintain the OUM identity. The DTPs also combine text, Throughout this stage, there will be an open door communication and mutual consultation among all relevant parties, both within and outside the CiDT. Every staff must also be competent in his or her respective field. A high level of professionalism (UT, 2002) is required in a quality work culture to ensure the process is done correctly and within the set deadline.
QA at the post-development stage Once the actual development stage has been completed, the ninth QC check kicks in as a hard copy of the module is printed out and sent to the ID for a process called internal review (IR), to ensure that everything has been put together correctly. Here, the IDs will perform a final review of the content to verify that the learning outcomes, content, learning activities and assessments are in sync with each other and to make sure that the module is error free.
After amendments based on the IR have been done, the module is sent to faculty for review. At this stage, the program coordinator or the vice-dean of learning materials will vet the module prior to sign-off for print or upload. Amendments may still be requested at this stage.
Once a module has been signed off, a dedicated QC team at the CiDT vets through the final copy once more before it is printed or uploaded. This QC team comprises the director, academics and senior staff at the CiDT who have strong language skills and are well-versed in module development. To standardize the QC process, the CiDT has come up with a checklist to be used by the team. Some of the elements listed in the checklist include checking and correcting the following: (1) the credit page for the latest version; (2) language errors; (3) figures, tables, activities and self-check questions in terms of suitability and relevance; and (4) alignment and indentation.
After this stage, a soft copy of the final module is sent back to the writer for feedback, on the understanding that if there are no amendments from the writer within a week, the module is deemed approved and ready for print or upload.
According to the quality objectives of the CiDT, the process development must be completed within four months of receiving the content from the moderator. As mentioned earlier, deadlines must be met in order to prevent delay of subsequent activities such as timely delivery of modules to learners.

From print to E
With effect from May 2016, the OUM no longer produces print modules. This is in line with global trends and initiatives to completely migrate the learning content toward E. Instead, e-modules are made available on the learning management system of the OUM known as myINSPIRE for learners who wish to download them in PDF format. Learners can either download the entire module or download each topic seperately, which has been made possible after the process of chunking. Meanwhile, bookmarking allows learners to navigate to a particular topic or subtopic within the module by clicking on the TOC.
There is also a module feedback mechanism in place on myINSPIRE, whereby learners can convey suggestions to the CiDTthis feedback is then channeled back to the faculty, who may request for module enhancement, in line with the PDCA model.

76
AAOUJ 12,1 A module which is deemed stable is categorized as Red Spine. As a general rule of thumb, a Red Spine module is used for about three years before it is again looked into, in keeping with the philosophy of continuous improvement. Knowledge is the "currency a university deals in," and this knowledge is constantly upgraded to remain current and abreast of changes (De Fazio et al., 2012). The role of the CiDT is to assist the university in this task.
The entire module development process is presented in Figure 3.

Customer feedback
The commitment from the OUM top management throughout the years has been a critical factor in driving the quality culture at the CiDT and the organization as a whole.
In line with this commitment to quality, the CiDT conducts a customer satisfaction survey at the end of every academic cycle to gauge feedback regarding the learning material developed and used in that cycle. The benchmark was set at 80 percent by the CiDT management. Consequently, as shown in Table I, customer satisfaction surveys have shown the satisfaction results to consistently hover above the benchmark of 80 percent for the past four years. Meeting or exceeding 80 percent of satisfaction for each production cycle is another quality objective of the CiDT.
Quality learning can be said to be achieved when the feedback indicates that the outputs and outcomes of the modules are satisfactory (Idrus, 2006). Even if the benchmark set is not at 100 percent, it does not mean that there is a lack of quality as "quality assurance does not expect perfection"; rather there is an expectation from the CiDT to commit toward improvement (Pitsoe and Maila, 2014). The feedback gained from the survey as well as other feedback mechanisms will provide useful input toward PDCA and further enhance the print module development process.   Table I demonstrates the positive impact that the QA culture has on the print module development process at the OUM. CiDT has been adopting a stringent auditing process yearly, since 2005, and the continuous improvement efforts have produced a desirable result. These audits include self-audit, internal audit and, finally, quality audit by SIRIM. The QA process for print module development is very clear in defining what the standards of achievement are, including documented procedures for each process.
The QMS in the CiDT is crucial to ensure the continuity of the quality work culture. The various procedures and work instructions that are in place set out a number of tools to facilitate the module development process. Checklists are available in most processes such as screening, conducting module writing and moderating workshops and also during the actual development stage. If followed correctly, the staff will not leave out any important work instruction and the processes can be executed smoothly and effectively.
Besides checklists, the CiDT has a number of feedback processes in place. Clause 8 of the MS ISO 9001:2015 requires the QMS to establish proper measurement, analysis and improvement (Department of Standards Malaysia, 2016). Implementing quality programs such as ISO certification encourage higher education institutions in Malaysia to collect data, which will "enable them to measure progress in key areas and establish benchmarks" (Sohail et al., 2003). The screening checklist for instance also doubles up as a feedback mechanism as it allows IDs to evaluate whether the writers fulfill the requirement set by the CiDT. After the module writing and moderating workshop, the participants consisting of module writers and moderators are given a questionnaire to gauge their level of satisfaction and also to gain input from the participants regarding the workshop. The customer satisfaction survey shown earlier in Table I is also important to gain vital feedback from the ultimate customerthe learners. In addition to these, there are frequent meetings with the faculties to discuss and thrash out issues pertaining to progress in developing the learning materials. Apart from learners being satisfied with the modules, the respective faculty or in this case program coordinator from the faculty, should also be satisfied with the outcome of the modules (Idrus, 2006).
As a result, many potentially serious issues that can lead to non-conformance have been detected and successfully resolved through CAPA. The initiatives introduced after the ISO certification include the screening process, the use of Turnitin software and the setting up of a QC team. Several tools were added to the processes from the pre-ISO stage, such as providing a step-by-step guide to participants for the module writing and moderating workshop and detailed work instructions for all staff. These have led to continuous improvement by reducing wastage in terms of redundancy and having a clearer workflow. The road ahead is full of challenges as the OUM migrates to more modules that are fully online because the modules must be of good quality to meet learners' needs and expectations. Migration also indicates that new QC measures must be established to ensure that the fully online modules meet or exceed the learners' expectations. We further plan to study the impact that the migration to fully online modules has had on the customer's satisfaction index, and data are currently being gathered for this purpose. The risks are high as the global education platform becomes increasingly competitive and learners more discerning. But at the same time, opportunities abound for the CiDT to assist the University in its mission to provide quality learning materials and make education accessible to all. Sohail, M.S., Rajadurai, J. and Abdul Rahman, N.A. (2003), "Managing quality in higher education:

Introduction
Online examination is a relatively more open and flexible learning platform, and is more convenient than a pencil-and-paper examination. As such, open and distance learning (ODL) institutions are urged to lead the endeavor to provide quality and excellence in the online examination services. Khare and Lam (2008) argued that "the implementation of online examination for distant learners can no longer be ignored or indefinitely postponed" (para. 1).
Conforming to such stipulation, Universitas Terbuka (UT) conducted the online examination in 2010. The number of participants saw a considerable rise from 332 in the first semester of 2010 to a staggering 2,293 in the second semester of 2012 (Sapriati and Pardede, 2015). This paper reflects on the profileration of online examination as a newly introduced education platform and analyzes how and why it broadened the extent to which the aforesaid students' demographics were interrelated, to bring those demographics into a unified diffusion framework (Iftakhar, 2016). Lane and van Dorp (2011) defined the diffusion of innovation as a process of adoption involving four elements, i.e. an object of innovation, a group of people, communication channels, and a period of time. Accordingly, the diffusion of innovation is an adoption process of an innovation object in a group of people through communication channels over a period of time. This paper weighs in the concept of innovation-decision process that the students of UT depend upon to make decisions about rejecting, suspending, or using and adopting the online examination as an innovation in UT. Rogers (2003) has broken the innovation-decision process into five stages: (1) knowledgethe individual (or other decision-making unit) is exposed to the innovation existence and gains some understanding of how it functions; (2) persuasionthe individual (or other decision-making unit) forms a favorable or unfavorable attitude toward the innovation; (3) decisionthe individual (or other decision-making unit) engages in activities that lead to a choice to adopt or reject the innovation; (4) implementationthe individual (or other decision-making unit) puts an innovation into use; and (5) confirmationthe individual (or other decision-making unit) seeks reinforcement for an innovation-decision already made, but he or she may reverse this decision if exposed to conflicting messages about the innovation (p. 206).
The innovation-decision period is referred to as adoption rate, which in this paper attributes to the students' time elapsing from awareness-knowledge of the online examination to the point at which it is put into use.
ODL has gained tremendous growth with an ample amount of innovations, most notably, in the strict sense, for students' learning support. UT as an ODL institution should accordingly keep providing sustainable innovations of learning support to its 300,000 students currently studying in nearly 13,000 islands in Indonesia. There has therefore been an urgent need for ODL institutions to observe and integrate the online examination as a modeling effort of diverse types of interpersonal characteristics, which are the students' demographics in this case, into a valid single framework of diffusion. Research in the diffusion modeling will have to expand the horizon. This paper proposes potential guidances for that expansion.
The purpose of the study was to elucidate how the adoption rate of online examination as an innovation at Bengkulu Regional Office of UT was linked to a number of students' demographics. The length of time between the beginning of the second semester of 2013 and their first participation in the online examination was taken into account to extend the measure of the students' adoption. The diffusion of innovation theory was then used to analyze the students' classification based on the adoption rates, i.e. innovators, early adopters, early majority, late majority, and laggards.
On the basis of the aforementioned purpose of study, the researcher therefore outlines research question as follows: RQ1. How is the students' adoption rate of online examination related to their academic factors (program of study, GPA, credit earned), socioeconomic status (age, sex, marital status, employment status), and residential factor?
Research context UT has provided two forms of students' learning outcome evaluation, known as the final evaluation, i.e. paper-and-pencil test and online examination. The paper-and-pencil test was organized simultaneously in a fixed schedule and place determined by UT as written in the 83 Online examinations at Universitas Terbuka academic calendar. The online examination was intended for the students who wanted to take the final evaluation beyond the schedule of the academic calendar of UT. In UT's regulation, the online examination is an option for the students to partake the final evaluation. Bengkulu Regional Office of UT began administering the online examination since the second semester of 2013. In the first semester of 2015, the number of students participating in the online examination was 1,239 or about 30 percent of the total number of students in the educational program for non-teachers. Since the first semester of 2016, re-examination has been introduced as the online examination in the event of students failing paper-and-pencil examination. In such capacity, the number of the participants is most likely to continue to have a growing number in the future. The data showed that while the online examination was still carried out at Bengkulu Regional Office of UT in Bengkulu City, the capital city of Bengkulu Province, 66 percent of the participants came from outside Bengkulu City. They consequently had to make a travel arrangements, as they stayed hours away from the exam venues, so as to manage to snag a spot. Students from Mukomuko Regency, as an example, had to go on a seven-hour trip to get to Bengkulu City. A research result found that the cost factor of commuting to the exam venues was a determinant factor, though it might depend on the commuting distance and proximity to the exam venues (Sugilar, 2016). Despite the travel and financial burden, most of the students had high academic aspiration and a strong commitment to the university.
Prior to sitting for an online examination, students have to meet a number of procedures. Using their logins, students have to register for the desired examinations. It is their responsibility to ensure that they are aware of the examination details, including the date, the starting time, and the venue, set out by Bengkulu Regional Office of UT. Hence, it is highly recommended that they contact the regional office to get notified at all times about the online examination arrangements.

Related studies
Socioeconomic status has effects on the adoption rate. A study by Ganiyu et al. (2013) described that socioeconomic characteristics of adopters were closely associated with the adoption of innovation, which in turn heavily modulated it. Zhang et al. (2015) studied the adoption process of the e-appointment service focused on some factors influencing patient acceptance and the usage of this process, and found that "the patients' social and demographic characteristics, including age, education level, and work status, appeared to have influenced their choice of use or non-use of the e-appointment service" (p. 13). The research also found that the variable of sex play-acted as a moderating variable on the correlation between employment status and usage of the e-appointment service. Accordingly, a recent study by Penjor (2016) used age and gender to predict the adoption rate for the virtual learning environment among academic staffs. To conclude, in some cases of adoption process of an innovation, the socioeconomic status variable was related to the level of innovativeness of the adoption. This paper assumed that the socioeconomic status had an effect on the adoption of the online examination and conceptualized socioeconomic status which included the variables of age, sex, marital status, and working status.
Several personal characteristics have explained the individual adoption rate of an innovation. Rogers (2003) generalized that "earlier adopters had greater intelligence, […] more favorable attitude toward education and science, […] had higher levels of achievement motivation than later adopters, [and] […] had higher aspirations (for education, occupations, and so on)" (p. 258). A study by Campbell (2015) about the diffusion of innovation of videoconference technology in distance education pointed out that the interaction of the students with instructors and material were key elements affecting adoption decisions of students regarding video-conference. In an academic setting, as discussed in this paper, students' pursuance of a program of study, GPA, and credits earned may attribute to such characters.

AAOUJ 12,1
The students' program of study, among other variables in this paper, was included due to an assumption that the characteristics of students of teacher-training programs were different from those of the regular programs. The students of teacher-training programs are typically professional teachers who are required by the law to earn a bachelor degree, while those of regular programs are non-teachers, be it employees or fresh graduates from high school, who attend UT with considerably diverse motivations and goals which they might want to achieve.
Another characteristic equally relevant to this study is the degree of cosmopoliteness, i.e. "the degree to which an individual is oriented outside the social system" (Rogers, 2003, p. 259). Rogers specified that early adopters are more cosmopolitan than later adopters, since the innovators' networks are more likely to be outside, rather than within, their social system and "they travel widely and are involved in matters beyond the boundaries of their local system" (Rogers, 2003, p. 258). On the basis of this theory, this study broadened in scope from focusing on socioeconomic status to encompass the so-called cosmopoliteness with respect to the variable of residence, which denoted whether the respondents currently resided in or outside Bengkulu City.
Despite the growing trend of the innovation, online examinations at UT have yet to garner much research attention to date. One of the researches was regarding a number of determinants of students' participation in the online examination by Sugilar (2016), who documented group comparison analysis between the groups of participating and non-participating students. The results showed that the following factors influenced the students to participate in the online examination: self-efficacy in using computers, perceived easiness in operating an online examination, perceived importance of online examination, intrinsic value of online examination, and cost of online examination. In addition, the following personal factors of students were of equally crucial importance in relation to their participation in the online examination: age, grade point average, gender, and marital status. Based on the rank of the amount of effect size for each factor, the order of the five most important of those nine factors was as follows: first, perceived easiness, second, intrinsic value, third, self-confidence, fourth, marital status, and fifth, grade point average.
Another research was conducted by Iriani (2010) who evaluated the online examination practices at Surabaya Regional Offices of UT. The results concluded that 64.22 percent of the students had not known the online examination, and only 2 percent of the students thus 2 percent participated in the online examination. However, the students who participated in the online examination showed exceptionally favorable attitudes toward the arrangement of the online examination. Sapriati and Zuhairi (2010) observed that in a regional office of UT about 75 percent of students who took a computer-based testing were also participating in a paper-and-pencil test. For the students taking both examination modes, Sapriati and Zuhairi found that the mean score of computer-based testing (equal to 54.00) was higher than the mean score of paper-and-pencil testing (equal to 46.00). Since, as the students know certainly, the examination results do not rely on the mode of examination, rather they depend on the learning process before the examination, these findings expressed that the students had a good preparation before they took the computer-based testing; this was probably because the paper-and-pencil testing was held prior to the computer-based testing.

Research method
This case analysis of 1,540 first-year students at the beginning of the second semester of 2013 at Bengkulu Regional Office of UT employed ordinal regression (Norusis, n.d.) to examine the adoption rate among students of the online examination based on explanatory factors and variables, i.e. academic factors (program of study, GPA, credits earned), personal factors (age, sex, marital status, working status), and residential factor (within or outside the city). The adoption rate of the online examination as an innovation was measured for each student.

85
Online examinations at Universitas Terbuka The adoption rate was measured by the number of semesters between the beginning of the second semester of 2013 and the first time they participated in the online examination until the first semester of 2016. Because the number of semesters from the first semester of 2013 to the first semester of 2016 was seven, the adoption rate was represented by 7 − d, where d was the number of semesters between the beginning of the second semester and their first participation in the online examination. On the basis of this formula, it can be identified that students who were first participating in the online examination in the second semester of 2014 had the adoption rate equal to 7 -2 ¼ 5, since the students needed two semesters before they took the online examination. The students who participated at the time of their first enrollment (the second semester of 2013) would have the adoption rate equal to 7 -0 ¼ 7. In this case, the students who never participated in the online examination, until the first semester of 2016, were considered to have the adoption rate equal to 7 -6 ¼ 1. Therefore, there were seven adoption rate categories, from 1 to 7. Whenever the students showed higher adoption rate, they were more likely to adopt the online examination faster.
The adoption rate categories in this research were presented in the ordinal quantities from 1 to 7 and are related to adopter categories based on Rogers (2003), who classified the categories as laggards, late majority, early majority, early adopters, and innovators. The relationship between the adoption rate and the adopter categories was not a one-to-one mapping, for two reasons. First, the number of adoption rate categories in this research was seven, while the number of adopter categories in the diffusion of innovation theory was five. Second, the adopter categories in the diffusion of innovation theory described adopter categorization with a certain standard. For example, the laggards category lies at X þ2sd interval and the innovators category lies at X À2sd interval in a normal distribution, while the adoption rate in this research is merely indicated by the number of semesters. Therefore, the adopter categories were simply a direction to define the adoption rate categories in a process of innovation diffusion. In fact, there was no clear-cut boundary for adoption rate categories in relation to adopter categories. Table I describes the relationship between adoption rates and adopter categories.
This study used an ordinal regression model to analyze a relationship between adopter categories and academic factors (program of study, GPA, credits earned), personal factors (age, sex, marital status, working status), and residential factor (within or outside the city). The ordinal regression required only the dependent variable as an ordinal variable. The independent variables could be an interval, ordinal, or categorical. However, to avoid too many empty cells and for convenient interpretations, the independent variables are all converted to categorical numbers, as shown in Table II.
The parameters of ordinal regression need different interpretation compared to parameters in simple linear regression. The basic concepts for interpreting parameters of ordinal regression are the concepts of odds, odds ratio, and logit (National Centre for First time enrolled in Bengkulu Regional Office of Universitas Terbuka  Research Methods, n.d). The odds express the likelihood of an event occurring relative to the likelihood of an event not occurring. For a dichotomous explanatory variable, the odds ratio is simply the difference between the odds for the base category (x ¼ 0) and the other category (x ¼ 1). Logit is the log of the odds ratio. The parameter estimate in ordinal regression as shown in Table IV is in the form of logit numbers. For instance, the parameter estimation for teachers' category is −0.853. Therefore, the odds ratio for teachers and nonteachers is exp(−0.853) ¼ 0.426. This means that the odds of teachers achieving the higher adopter category are 0.426 times compared to non-teachers. Similarly, the odds ratio for non-teachers compared to teachers is 1/exp(−0.853) ¼ 2.345, which means the odds of non-teachers reaching the higher adopter category is 2.345 times that of teachers. In some contexts, the odds ratio could be also expressed in percentage terms, e.g. the odds ratio of adoption rate for teachers was 42 percent for non-teachers.

Findings and discussion
Description of adopter category by explanatory variables This study dealt with 1,540 students who first enrolled in the second semester of 2013 at Bengkulu Regional Office of UT. In Table III, teachers make up 53.4 percent of the students, most notably elementary school teachers. None of the students in the teacher program was included in the innovators category. More than 50 percent of the students had a GPA of 2.9 or less in the first semester of 2016, and made up the bulk of the innovators category. Meanwhile, the students who earned 62 credits or less stood out in the laggards category with 64.0 percent. In terms of the age group, the number of the students aged 25 or less surpassed the number of students aged over 25 in the lower adoption categories 1, 2, and 3. This led to a domination of student groups aged over 25 in the higher adoption categories 4-7, where in fact none of the students aged 25 or less were found in category 7 or classified as innovators. Classifying on gender basis, the number of female students surpassed the number of male students in almost all adoption rate categories, except for category 7. However, no such systematic difference was found for variables of marital status and working status, since the percentage between the levels in each variable converged to a percentage of the total.
As for the students' residences, the low adoption categories 1-5 were dominated by the students residing outside Bengkulu City. In contrast, those residing in Bengkulu City

Factors
Variables Variable values Academic Program of study ¼ 1, non-teacher program (regular program) ¼ 0, teacher-training program GPA ¼ 1, more than 2.9 ¼ 0, less than or equal to 2.9 Credits ¼ 1, more than 62 ¼ 0, less than or equal to 62 Personal Age ¼ 1, more than 25 ¼ 0, less than or equal to 25 Sex ¼ 1, male students ¼ 0, female students Marital status ¼ 1, married ¼ 0, unmarried Working status ¼ 1, working ¼ 0, not working Cosmopoliteness Residential ¼ 1, residing in city ¼ 0, residing outside city 87 Online examinations at Universitas Terbuka outnumbered those residing outside Bengkulu City in categories 6 and 7. This signifies that the students residing in Bengkulu City were more inclined to adopt the online examination faster than those residing outside Bengkulu City.

Model of adopter category on online examination
The parameters of the ordinal regression model for the effects of academic (program, GPA, credits earned), personal (age, gender, marital status, working status), and residential variables (within city or outside city) on the adopter categories are presented in Table IV. The model of ordinal regression proposed a significant χ 2 statistic ( po0.0005) to suggest that the final model led to a significant improvement upon the baseline or intercept-only model. The Nagelkerke of the pseudo R 2 statistics value was 16.6 percent, stipulating that the explanatory variables defined 16.6 percent of the variation among adopter categories. The model also corroborated the assumption of goodness of fit, indicated, respectively, by the χ 2 test for the null hypothesis of deviance statistics that was not rejected by χ 2 test at po0.997 and the χ 2 test for the assumption of parallel lines that was not rejected at po0.278. The threshold in the Table IV represents the intercept in the model of ordinal regression. All intercept coefficients were significant at p o0.05 by the Wald test, which in turn would be included in the model. The location in the table represented coefficients of the explanatory variables in the model. If the coefficient was significant at p o0.05, the associated variable would be included in the model. Table IV demonstrated that all variables were significant at p o0.01, denoting that there was a strong association between each of these variables and adopter categories. However, no such significant association was found for marital and working status. As described earlier, the parameter estimate in ordinal regression as shown in Table IV is in the form of logit numbers. The logit is log of odds ratio between two categories. For a dichotomous explanatory variable, the odds ratio is simply the difference between the odds for the base category (x ¼ 0) and the other category (x ¼ 1). For easy interpretation of the parameters in Table IV, the last two columns of Table IV provide the exponential values of the parameters and their inverses.

Academic factors
In Table IV, the educational program consists of teachers and non-teachers (or regular program). Adopter category of the students of teacher program was indicated by the parameter estimation equal to −0.853. Therefore, the odds of a student of teacher program breaking into a higher adopter category (from laggards to innovators category) was exp (−853) ¼ 0.46, or 46 percent of the odds of a student of the regular program. Conversely, the odds of breaking into a higher adopter category increased by 1/0.46 ¼ 2.35 times for a regular student compared to those for a teacher student. The regular program students, simply put, were more likely to adopt the online examination much faster than the teacher program students. Referring to the previous argument, the students majoring in teacher education and training program are typically teachers required by the law to earn at least a bachelor degree The students of regular program, on the other hand, are of tremendously diverse kinds, from high school fresh graduates to retiring people. The differences in characteristics of such two groups brought about differences in the odds of falling into a higher adopter category to 46 percent for those in the teacher-training program. The parameter estimate for the variable of GPA was 0.703, which was GPAW2.9 as a baseline, suggesting that the odds of a student with a GPA of 2.9 or less to fall into a higher adopter category were 2.02 times (or 202 percent) as much as those with GPA above 2.9. The GPA data were taken in the first semester of 2016, before the inception of the online examination, which let the students decide whether to participate or not in the elapsed timefrom the second semester of 2013 to the first semester of 2016. This research yet stumbled upon unexpected finding that a student with a higher GPA turned out a slower adopter. On the other hand, this finding was consistent with that of earlier research that a student with a lower GPA had a high score in expectancy and value toward the online examination (Sugilar, 2016). The study also discovered that a student with a lower GPA, in fact, needed the online examination much more, as a re-examination, to improve his/her GPA. Students with lower GPA, simply put, were more likely to adopt the online examination faster. Some students, who could be categorized as laggards or late adopters, explained that they did not yet need the online examination because they were obtaining good grades for their courses. This should have supported the fact that the students who were fast adopters of the online examination were largely the students who acquired lower GPA.
It is important to note, however, that the online examination is just been another tool for measuring students' competence in one course and assessing their failure in improving their learning so that they can learn from their mistakes. The finding that the students with lower GPA belonged to the category that adopted the online examinations faster indicates that the online examinations provided the students with the benefits of re-examination but did not lead to significantly higher scores. Therefore, there is a need for further research regarding the examination results and the students' preparations before taking the online examinations.
The students that earned credits above 62 in the first semester of 2016 had a 3.12 times the chance to be in higher levels of adopter category as compared to those that earned 62 credits or less. In accordance with the earlier findings, this finding suggests that while the online examination did not add to the GPA significantly, the students acquired sufficient grade for credit accumulation. To conclude, although the online examination did not pull off the GPA above 2.9, it helped the students in the second semester of 2013 to gain a credit greater than 62.

Personal factors
Students aged 25 or less were 0.59 times slower adopters than students aged over 25. In other words, those aged above 25 were 1.7 times faster adopters than those aged 25 or less. This was contrary to the assumption that younger people, as digital natives, tended to be better adopters of communication technology. On the other hand, this finding was in line with a research result that older students demonstrate higher scores in expectancy and values toward the online examination (Sugilar, 2016). This indicated that the students considered the online examination less related to technological issues, but more associated with tools that enabled them to frequently take the re-examination. The participation in online examination was, thus, closely linked to a sense of learning seriousness. This was underlined by a research conducted by Darmayanti (1993), among others, in UT's context, who posited that older students had seriousness and self-directed learning compared to younger students.
The odds for the female students to be part of higher adopter categories were 1.7 times that of male students; the male students had only 0.59 times the females' possibility.

AAOUJ 12,1
This finding extended the result of a research in online examination by Sugilar (2016) that the male students tend to participate more frequently in online examinations. The male students are consequently slower adopters than the female students, but once the male students participate in online examinations, they will participate several times more than the female students.

Residential factor
Residential factor refers to the place where the students resided during research period. Bengkulu Province has nine regencies and one city, which is its capital, Bengkulu City. Bengkulu Province has an estimated population of 1.828 million, of which about 310,000 are concentrated in Bengkulu City. As the capital city, Bengkulu City has better infrastructure and facilities for communication and a more heterogeneous mix of people coming from other places in Indonesia, with several foreign-born people. The people of Bengkulu City are assumed to be more cosmopolitan than those belonging to other places in Bengkulu Province.
Cosmopoliteness is a characteristic strictly associated with innovation, and people who are more cosmopolitan are earlier adopters of innovations (Rogers, 2003). Table IV reveals that students' residential factor gives different odds for higher levels of adopter category. Those residing in Bengkulu City were 2.50 times more likely to hit the higher levels of adopter category, compared to those residing outside Bengkulu City. The city-dwelling students were thus faster adopters of the online examination.

Conclusions and recommendations Conclusions
The results of this study showed that while program of study, grade point average, credits earned, sex, and residence had systematic effects on adoption rate categories, no such effects were found in employee status and marital status. The conclusions of this study are as follows: • The students majoring in teacher education and training program were faster adopters of the online examination than those majoring in a non-teacher program of study. The adoption rate of the online examination among the students in the teacher program was 46 percent of the adoption rate of the students in the regular program.
• There was greater likelihood of the online examination adoption among students with lower GPA (2.9 or less) than among students with higher GPA (above 2.9), implying that students with lower GPA benefited from the online examination as a re-examination effort to improve their GPAs.

•
The students with credits greater than 62 were 3.12 times more likely to achieve higher levels of adopter category compared to those with 62 credits or less.
• Students aged 25 or less had 59 percent likelihood of those aged above 25 to gain higher levels of adopter categories.
• Female students were 1.7 times more likely to pull off the higher adopter categories. The male students were thus slower adopters of the online examination as compared to the female students.
• The students that resided in Bengkulu City were 2.50 times more likely to reach higher levels of adopter categories than those residing outside Bengkulu City. These city-dwelling students, in other words, pushed on the adoption of the online examination 2.50 times faster than the non-city-dwelling students.

Recommendations
This study implies that the institution of ODL should accelerate the introduction of new learning resources based on diffusion of innovation modeling. Upcoming technological changes in the nature of ODL innovations are expected to extend this scope further and are expected to be integrated with students' demographic profiles. Diffusion modeling has attempted to reflect the increasing complexity of newly introduced innovations in the adoption process. As a technological innovation, the online examination should be diffused by applying the technology acceptance model. To maximize students' acceptance of the technological innovations, it becomes pertinent to emphasize the usefulness and easiness in terms of students' perception. Mkhize et al. (2016) has concluded that "Technological innovation has been the pillar of success in many organisations in the knowledge age, irrespective of the organisation type" (p. 298).
In addition, it is recommended, to the developer of learning resources for the students of ODL, that all the learning resources should be developed and evaluated with students' expectancy-value in mind (Sugilar, 2016). The following factors should be noticed: easiness, students' ability, importance in the students' view, possible intrinsic values of students, and cost. It is also recommended to encourage the students of ODL to have the feeling of easiness, self-confidence, and importance of using online examination, especially for the new students joining ODL. To achieve this objective, regional office staffs can guide the students to try out the online examination in the new student orientation study activities.
As this research is a case study in Bengkulu Regional Office of UT, further study is needed to compare adoption pattern of the online examination among regional offices of UT, since Indonesia consists of more than 10,000 inhabitant islands with cultural varieties. The analysis should include some variables that represent the uniqueness of each region in Indonesia. Further study is also necessary for comparing the adoption of new learning technologies among ODL students of various nations.

Introduction
The revolution in information technology dramatically impacted the lives of billions of human beings especially through its influential tool of internet (Nie and Erbring, 2000). This storm of wireless technology affected numbers of domains including education. Education is considered very important not only for the personal development of an individual but also for the development of any society. Schools, colleges and universities have been spearheading this endeavor for hundreds of years.
However, it is not feasible for everyone to get education from universities, especially top universities, not even for those who live in developed parts of the world because of many reasons (Tobin, 2015). Therefore, some educators conceived the brilliant idea of massive open online courses (MOOCs) to provide meaningful learning experience to anyone around the globe for free. The term MOOC was first coined in 2008 by Dave Cormier of the University of Prince Edward Island (Miguel et al., 2013).
MOOCs are increasingly considered as an opportunity for millions of people who want access to the higher education for free or at low cost. Although academic institutions are jubilantly jumping on to the digital bandwagon of MOOCs, the majestic success of MOOCs in the recent years has also been creating panic among the university officials who believe that this free and low cost model of education would cannibalize their share (Christensen et al., 2014).
Distance learning education is not new to Pakistan. Allama Iqbal Open University was the first in Asia to start distance education in 1974 which has currently 1.3 million enrolled students (AIOU, 2016). Pakistan has got huge MOOC potential as a developing country where most of its students cannot afford to study at world's premier academic institutions.
EdX, a joint venture of Harvard and MIT, and a premier MOOC provider have got nearly 90,000 students from Pakistan. EdX has also established international regional office in Lahore. In addition, Anand Agarwal, CEO of edX, has expressed his interest in an interview to collaborate with Pakistani universities (Ahmed, 2016a).
This research idea emanated from the personal experience and interest of researcher 1 when I enrolled in MOOC offering of the University of Edinburgh in 2012; this was the same year when The New York Times labeled it as the year of MOOCs because of the sharp rise in the MOOC enrollments worldwide (Pappano, 2012).
Despite of the considerable boost of MOOCs and its published literature in recent years, there is dearth of research and insight from developing countries. The objective of this research paper is to review the development of MOOCs, its characteristics in general and to explore its potential and challenges in Pakistan particularly. This research study would help us to understand that what students and teachers in Pakistan think of this massive rise of this online education genie (MOOCs).
For this exploratory qualitative study, the data collection was done once from multiple groups, hence it was a cross-sectional study. Focus group was conducted with students and interviews were used to collect data from faculty members. The collected data were recorded and transcribed. We used thematic and content analysis to draw conclusions.
Finally, we concluded this paper with some useful insight that facilitate us to create a better comprehension of MOOC potential and challenges in general and particularly in Pakistan.

Literature review
With the economic progress of mankind and development of new technologies, new medium of imparting knowledge evolved. The emergence of MOOCs exploded in the year 2012 primarily through the US universities' platforms such as www.udacity.com, www.coursera. org and www.edx.org (Davis et al., 2014). Many of the premier British universities joined futurelearn.com to emulate their Americans counterparts. FutureLearn was the first British MOOC platform. However, some British institutions such as the University of London and Edinburgh were already offering courses through Coursera (Parry, 2012).
These platforms are result of genuine endeavors of IT experts, scholars and professors of academic institutions (Liyanagunawardena et al., 2013). It started not long ago but its success has been phenomenal in terms of participation. Now millions of people have been learning through these platforms. In addition, Wong et al. (2016) analyzed research development in the area of distance education. They conducted content analysis of 288 research articles published in online and distance learning peer-reviewed journals. They noted that MOOCs were the most frequent keyword emerging in 18 percent of their selected articles.
Some universities are using MOOCs along with the face-to-face classes as blending teaching is increasingly evolving in this digital era (Bo Tso, 2015). It gives excellent opportunity to the students if their university is offering. However, most of the MOOCs are

95
The potential and challenges of MOOCs in Pakistan based upon a freemium business model where the basic product or service is provided for free but money is charged for additional services (Dellarocas and Van Alstyne, 2013). In addition, Ming  explored the effect of the pedagogic orientation of MOOC platform on their courses. They examined 32 courses on the Coursera, edX, FutureLearn and openlearning. This study suggests that the platform offering MOOCs invariably influences the delivery of the MOOCs. As edX and Coursera comprehensively focused videos while FutureLearn and openlearning focused more on the participant interaction. Moreover, Kalman (2014) analyze that the distance learning education is neither new nor the pedagogic and technological approaches applied are unique. However, the concept of free education from top universities has sparked the general interest. Altbach (2014) see the advent of the MOOCs as some form of neocolonialism where English-speaking academic cultures are dominating the MOOCs. But the millions of learners around the world are not much concerned with pedagogical philosophy and nature of knowledge they are acquiring. On the other hand, they are appreciating the MOOCs capacity to provide opportunities to everyone. Regardless of national or financial background, one can enroll in the courses through the internet access. It also opens the doors for the learners of developing countries who want to refine their skills or even start up their own ventures. According to Christensen et al. (2014), it is great opportunity for the business schools to expand in to new markets as nearly half of the MOOC offerings of Wharton School of the University of Pennsylvania have international enrollments. Furthermore, Ming  suggested that MOOCs help instructor to experiment his/her online teaching to large number of diverse learners. Some instructors have found MOOC experience useful for refining and enhancing their teaching skills.
MOOCs adeptly question traditional teaching method where instructor asks students whether they understand the topic and clever ones nods their heads and he/she moves on with the lecture. All five fingers are not the same, in almost every class, there are some students who hesitate to ask questions. This platform smartly addresses this crucial issue by putting video in the recorded lecture form, enabling students to access it numerous times. According to Zapalska and Brozik (2006), the instructional design in online educations must oversee different individual learning styles. Student learning is more effective if it is consistent with his/her learning styles.
Discussion forums are another exciting and informative feature of MOOCs. Students put up their questions in threaded discussions and receive numbers of responses from their fellow students dispersed around the globe (Welsh and Dragusin, 2013). While late joiners might experience problems in understanding the course requirements, and in such cases, the help from older participants was recorded very useful (Waite et al., 2013). In many courses, students from the same town and city form groups for the in-depth discussions. Li et al. (2014) also reported on the longitudinal study that the study group especially the one co-located had a pleasing experience in learning difficult topics. Moreover, self-management is sine qua non for digital learners in this information age. It will help learners to enhance learning efficiency and effectiveness in communities by developing better comprehension of valuable strategies and related concepts (Zhixian and Jianhua, 2011).
The participation certificates from top universities breed sense of achievement and encourage students to continue education. In some accredited courses, you automatically get the credits for the university education. For example, students can get undergraduate credits for some of the Coursera courses offered by Duke University, University of California, Irvine and University of Pennsylvania (Coursera, 2013). In UK, FutureLearn introduced its first MOOC having university credits, an offering by the University of Leeds that will allow student to have ten credits for its respected degree (Havergal, 2016).
Business schools need to watch closely how the genie of MOOCs is going to behave in the future. Clayton Christensen, a prominent Harvard Business School Professor envisaged that 96 AAOUJ 12,1 the growth of MOOCs would shake up the foundations of business schools in the next decade and lead many of them to the bankruptcy (Kaplan and Haenlein, 2016).
The virtual platform no-pay MBA claims that it provides opportunity for those motivated individuals who ever considered MBA but fees or flexibility hampered their admission. It uses MOOCs to give a learning experience of top business schools, i.e. MIT, Harvard and Yale with the cost less than US$1,000. Their flexible program allows you to learn at your own pace. It has 14 core courses, three electives and four concentration electives. The members work together for group assignments and projects, and also meet for video discussion on a regular basis with professionals, entrepreneur and professors (Pickard, 2015).
Despite of MOOC appreciation, there has been some serious consideration, such as the assessment methodology of enrolled students has been raising questions for the MOOC credibility. For many of the courses, peer-reviewed assignments are mandatory as it is impossible for instructors to evaluate individual work. In this regard, Krause (2013) found out that some learners got commendable feedback despite of submitting substandard work. Another big challenge for the MOOCs is to ensure that the student taking the exam is the same one who is enrolled. Some experts believe that online examination with webcam could be suitable option. If their computer is locked down to use any other program or documents, this can enhance effectiveness of the invigilation (Coughlan, 2016). Knox (2014), a Coursera Instructor interpreted the response of many participants as anxiety and burdensome because of the mass participation. Zhenghao et al. (2015) in a Harvard Business Review article claimed that over quarter of billion people had enrolled in MOOCs but expectation went down as research showed that only small proportion of enrolled students went on to complete the course (Ho et al., 2014). On contrary, Frick (2016) argues that we should not use dropout rate to claim that MOOC does not work. According to him, the other way to use MOOCs is to scan all the materials and lecture but not necessarily taking quizzes and submitting assignments. The disadvantage is evident, one would not get certificate. The advantages are that one has not got the pressure to complete the assignments and can learn at one's own pace. Zemsky (2014) foresees that MOOCs have been losing ground as it has nothing much to offer from technological and pedagogical perspective. In addition, many business leaders think that the university graduates lack essential skills to perform the job. In this vein, MOOCs are not impacting the business world significantly because the courses are designed by the same academics who cater their own interests rather than focusing on the skills sought by industries (Weise, 2014).
In case of Pakistan, many students attempt to go to English-speaking countries particularly Australia, England and New Zealand for the higher education where securing good bands in International English Language Testing System (IELTS) is fundamental requisite for the admission. British Council which is the awarding body of IELTS certificates has launched its free course through FutureLearn, i.e. Understanding IELTS: Techniques for English Language Tests (Coughlan, 2015). This course is blessing for students who want to prepare for IELTS, as there is lack of established and standardized platform for learning English linguistic skills for higher education. Furthermore, it also prevents students to pay huge fees to substandard English language institutions.
Moreover, Agha Khan University was the first in Pakistan to launch three-week long MOOCs in 2014 on Bioinformatics of Drug Design (Abidi et al., 2016). They launched another course from the same platform in the year 2016. The organizers highlighted many factors in the context of Pakistani society for launching successful MOOC programs. First, faculty members need to leave comfort zone and review their approach in the light of global context. Second, the course design must be astutely done as course must be learner friendly with technological factor considerations.

97
The potential and challenges of MOOCs in Pakistan Furthermore, the current Vice Chancellor of Information Technology University (ITU) and former Massachusetts Institute Technology Instructor Dr Umar Saif announced that ITU is working out with edX to launch online courses (Saif, 2016).

Methodology of research
This exploratory research was conducted for developing better understanding of the researchers (Babbie, 2008). Interpretivist paradigm with cross-sectional research design was assumed to look at the potential of MOOCs in Pakistan from people who are already aware of MOOCs. For this purpose, two set of respondents were selected: one included seven students for focus group and another comprised four faculty members for the interviews. All the individuals selected had completed at least one MOOC. Almost all of the students have done one MOOC while a student has done three MOOCs. Among faculty, all have done either eight or more MOOCs. In addition, the methodology and philosophy of qualitative study is different than quantitative. According to Marshall (1996), if the research question is adequately answered then the sample size is not an issue in qualitative study. Even for very detailed studies it could be in single figures.
The participants were requested to fill in a self-administered questionnaire requesting for personal information and brief details about MOOC already attended, before the start of the focus group or interview. The link for this questionnaire was sent to the respondents through digital means.
Following an interpretive strategy, respondents were asked to comment freely on the potential of MOOCs in Pakistan. Following were the main research questions: RQ1. How do you foresee the potential of MOOCs in Pakistan?
RQ2. What will be the challenges with regard to MOOCs implementation in national and regional languages in Pakistan by Pakistani universities/professors?

RQ3.
What merits with regard to benefits, skills, advantages and demerits with regard to challenges, difficulties, practical and logistic problems the MOOCs will have?

RQ4. What do you like/dislike about MOOCs?
These open-ended questions were asked through focus group discussion among students and through interviews from faculty. The focus group discussion last for around 1 hour 23 minutes while each interview ran from around 12-18 minutes. We had reached saturation after conducting focus group and four interviews, and according to one of the most prominent qualitative scholars, Creswell (1998), there are no rules to determine the sample size what we consider is a saturation point in the qualitative study where the new themes stop emerging. Furthermore, all discussion and interviews were recorded, transcribed and read several times to identify emerging themes or discussion falling in already identified themes. It was seen that respondents' answers had both similarities and differences. Both thematic (Hycner, 1985) and content (Silverman, 2014) analyses were conducted as themes were already derived from the literature review and few were finalized during the analysis process. The triangulations of methods and sources were conductedtriangulation of methods, i.e. the focus group and interviews; and triangulation of sources, i.e. students and faculty.

Qualitative findings
Findings are presented based on both thematic (Hycner, 1985) and content (Silverman, 2014) analyses which were conducted along with the triangulation of methods, i.e. the focus group and interviews; and triangulation of sources, i.e. students and faculty.

MOOCs potential in Pakistan
Many respondents foresee MOOCs enjoying good future in Pakistan because it makes education affordable for masses. It can play positive role in the human development without huge investment in the shape of state-of-the-art buildings and it also saves considerable amount of resources. In addition, respondents also highlighted that most of the courses offered by MOOCs are from top academic institutions of the world for free. It enables us to acquire knowledge from the best professors and scholars of contemporaneous era. Apart from subject knowledge, it also boosts their general confidence level. Couple of respondents mentioned that distance education is not new to Pakistan as Allama Iqbal Open University has been doing it for more than four decades through national television channels. As one of the respondents, Mr Khurram Adeel Sheikh said, "It is the same old candy in the new wrapping." On the contrary, many of the respondents regarded teaching methodology of MOOCs are nontraditional and learner oriented. The lectures are recorded and uploaded in bite size videos which are easy for students to digest. They can download it and play it as many times as they want; a deed which is hard to do during real-time lectures. Moreover, it also offers opportunity to discuss topics and ideas through discussion forums. Furthermore, many of these programs are self-paced which helps to integrate every type of learners in MOOC. As one of the students in discussion forum responded, in his words, "I enrolled in MOOCs because I wanted pristine knowledge from best professors not certificates. I only downloaded the lectures and read books but never appeared in quizzes and assignments. Of course, not many people want to do quizzes in summers break." According to one of the university professor we interviewed, universities have to follow certain guidelines and timelines for the courses and academic programs. At times, it makes it hard for universities to incorporate all the topics in which a student could be interested to learn. Second, commencement of courses needs to have certain number of students at Pakistani universities which make students to withdraw if they do not meet quantitative requirement of the class. However, MOOCs provide learners the platform to choose the course which could be useful for the individual. Importantly, it enables learners to enjoy wide range of courses including the ones not taught in Pakistan but could see development in future.
MOOCs could be beneficial for graduates as well, as they cannot always go back to university to resume the studies. Similarly, there are some other courses as well whose teachers are not available or maybe that professor from which you are trying to get your lessons is not available. As one of the respondents uttered he wants to learn the course for artificial intelligence and if he wants to learn under the supervision of the best MIT professors so he is left with few choices. He can either join MIT but for which he might not have the IQ level or join MOOC offering where he can do it for free and also meet with hundreds and thousands of students of similar interest. This is a huge benefit which is provided by these learning platforms.
Many respondents regarded two factors financial resources and location as important factors when it comes to knowledge acquisition in the context of Pakistani society. In Pakistan, it is common to see families moving from villages and smaller towns to cities for the education of their children and most of them come from prosperous background because not everyone can afford living in the city. However, MOOCs have provided the opportunity to learn everyone at their doorsteps.

Challenges
Some respondents from the focus group concerned that MOOCs' ability to attract large number of students becomes biggest challenge for teacher. As in many of these courses, more than hundred thousands of students are enrolled. Regardless of the competency of teacher, it is

99
The potential and challenges of MOOCs in Pakistan humanly very difficult for the teacher to answer the query of hundred thousand students from around the world. So, without the real-time interaction and the questions, curiosity might be diminished particularly if concepts are not clarified. Second, students' social interaction at the university with fellow students and teacher is very important and it refines their interpersonal and communications skills. Group assignments help them to develop team skills which are one of the most cherished set of skills at work. Moreover, one of the faculty members said in our interview that by looking at the student physically, teachers gauge their level of confidence. They closely monitor the effort of the performance. With MOOCs, they might not be able to do that because student are going to be connected via the powerpoint slides and online quizzes, consequently, evaluation of the students' performance becomes harder. Few respondents were skeptical regarding MOOCs success in rural area because of power failures and fluctuation of voltage. Moreover, one of respondent raised another set of concerns in the context of Pakistan that online courses unfortunately do not enjoy deserved eminence and if someone has done an online course, it is considered of lower quality without any reason. The biasness against distance learning and online learning exist in some form which we need to work on as a society. Second, some respondents were skeptical of MOOCs as they do not think employers would consider these courses because anyone can do these courses and crucially, there is no credibility of the evaluation method. However, they considered it as supplemental to one's current knowledge and resume.
Couple of respondents mentioned that the role of Higher Education Commission (HECa government body responsible for facilitation and development of higher education in Pakistan) is very important in the development of MOOCs in Pakistan. If HEC promotes MOOCs at universities, it can gain recognition in a very short span of time, especially if it awards credits for the courses. On the contrary, some respondents envisaged that HEC would not do it because it might put future of the universities to questions.

4.3
The potential of local MOOC in national and regional languages Many respondents think that MOOCs initiated in Pakistan by domestic teachers will be more useful and beneficial for the Pakistani students, as international teachers do not understand the social, religious and cultural aspects of the country.
MOOCs by Pakistani universities will have more recognition amongst their student as there would be domestic examples. As in current MOOC from western universities, most of the examples are from the institutions and organizations of developed countries which lack the social context of developing country. Doing business in USA and Pakistan are two very different cups of tea.
There are number of good-quality tuition centers which charge very high fee from O and A level students and most of them are based at cities. If some Pakistani teachers make videos of their lectures and offer it to the students, it would be a lot more convenient and beneficial for people living particularly in countryside. It may also provide opportunity to those students who cannot afford higher fee of the prestigious tuition centers.
There is great potential if MOOCs are developed in our national language Urdu and other regional languages such as Punjabi, Sindhi, Balochi and Pashto. This will have a prospect of covering most of the underdeveloped areas of Pakistan, particularly those remote areas where women are not encouraged to go out and study. However, internet availability and connectivity is one of the greater challenges in Pakistan along with the frequent power failures. We have good internet connection only in major cities and town whereas most of the population lives in the rural areas.

Discussion
This research reflects that distance education programs have been firmly ingrained in our academic system for a long time. However, MOOCs are a product of information age which 100 AAOUJ 12,1 uses internet to impart education and importantly, rather than operating inside national boundaries, MOOCs are global in nature, influencing people all over the world. In addition, this study notices that MOOCs got boost in Pakistan because it is free and from top academic institutions as Kalman also claimed that MOOCs did not bring revolution in the teaching methodology. Nevertheless, free courses from premier universities have enabled MOOCs to earn distinguished status.
The surge of MOOCs could see customized degree in near future where learner is free to choose from the range of courses as there are already platforms offering similar courses. The design of the MOOCs has been particularly the size of the videos and discussion forums. Similarly, Waite et al. (2013) and Welsh and Dragusin (2013) also consider it very useful as it connects all the class fellows spread around the world. Massive participation and communication makes these discussion forums like online coffee shops where you can discuss, chat and argue with your class fellows. This interaction and connection among co-located group members enables them to reap maximum learning from MOOCs.
This research shows that the enviable ability of MOOCs to attract millions of learners has also been limiting its potential as very small numbers of participants complete the courses. There could be many reasons in this regard, such as for many students, interaction with the teacher is vital for their learning style if they do not comprehend they lose interest, some students just want access to knowledge. So, they download learning material and do not participate in quizzes and assignments. Likewise, Frick (2016) believes that some students avoid evaluation because they do not like to study under pressure. They enjoy learning at their own pace.
Internet access and power failures are considered as some of the challenges for MOOCs. Pakistan is currently struggling to manage power failures as shortfall heightens up to 5,000 megawatts (Hasnain, 2017). According to the World Bank (2016) report, only 18 percent of Pakistanis are using internet, whereas Pakistan population is around 200 million with more than half of the population is under 30. Hence, there is great potential for online education as drastic changes in power and internet connectivity are imminent by dint of Chinese investment of $46 billion investment in Pakistan's infrastructure (Ahmed, 2016b). Pakistan recorded IT export with over $2 billion last year with the growth rate of 41 percent (Baloch, 2015), this is reasonable performance considering not even quarter of the population has internet access. This research shows that students are using Coursera and Udacity courses to refine their software development knowledge and skills. Hence, MOOCs are impacting and transforming the lives of thousands of learners in Pakistan.
The support of HEC could witness the sharp rise of MOOCs culture in Pakistan but it could also endanger the future of the universities particularly business schools as predicted by prominent HBS Professor Clayton Christensen (Kaplan and Haenlein, 2016).
Hence, it could be argued that MOOCs are fine to develop basic knowledge of the concepts and theories. However, the significance of the traditional universities will remain intact in some form, seeing that some disciplines require rigorous use of expensive tools and equipment at laboratories such as disciplines of engineering and medicines which cannot be afforded by the individuals. In this vein, the augmented reality might be a game changer in the future (Castellanos, 2016a), which could help students to guide, envision nature of structures and attain deeper comprehension of the processes. This change could be expected in couple of decades. Before people used to dissect animals as part of their experiment but now there are holographic projections which can be used to explain anatomy and even pilots are being trained through it. Microsoft has also launched its holographic lenses (Castellanos, 2016b). This could be very beneficial for MOOCs and other online courses. Furthermore, idea of online laboratory is growing but this could be not deemed alternative to real one as students will not be able to develop and refine hands-on skills which are indispensable for engineers (Rasika Nandana et al., 2015).

101
The potential and challenges of MOOCs in Pakistan Western MOOCs platforms are undoubtedly valuable and beneficial. However, they lack our domestic context, and believe that MOOCs by Pakistani universities could greatly help local students. On the other hand, Agha Khan University has already launched couple of MOOC programs in English language. Abidi et al. (2016) mentioned many factors universities need to consider if they are going to launch MOOCs in low-and middle-income countries. In addition, the idea of MOOCs in regional languages emphatically put forward in focus group by most of the respondents as Pakistan is much diversified on linguistic grounds and nearly 26 different languages are spoken. Nevertheless, it will require big effort and kind intentions from government as the medium of imparting education in all the premier Pakistani universities is English and most of the books and academic literature especially for higher education are available only in English language. So, this endeavor will require the translation of all the books to Urdu and regional languages. Introduction of institutional reforms to galvanize learning in the rural areas could improve the education level of the country. On the other hand, it might have some limitation as these MOOCs will be only limited to the audience who understand those regional languages.

Conclusion and recommendations
MOOCs intervention influenced the academic world through many conceptual and technological changes. Many academicians and scholars considered growth of MOOCs as a big threat for the traditional universities, but it has not endangered the traditional ways to imparting knowledge yet. As for many subjects, student needs close supervision, expensive equipment and state-of-the-art laboratory. However, major technological advancement in augmented and virtual reality could revolutionize MOOCs future.
MOOCs are admired in Pakistan as hundred thousands of people are already learning in the country primarily through US and British MOOCs platform. It is the best way to acquire quality knowledge at the time of heightening education cost. As discussed above, there are many sustainable benefits which MOOCs are bringing to the Pakistan society. Although limited internet access and power failures are hindering the streamlining of MOOCs, it is anticipated that in the future, structural improvement in infrastructure could positively impact the MOOC enrollments.
Launching of MOOCs in regional language is a daunting task which requires ceaseless effort from all the stakeholders. Particularly, it needs to draw special attention and interest from academicians and HEC.
MOOCs have a very low completion rate as the participants' interest diminish with the progress of courses. Hence, for future research studies, factors responsible for inhibiting MOOCs could be investigated. The explanation of these factors along with innovative approaches could make MOOCs more influential and successful.
A multi-objective evolutionary algorithm-based soft computing model for educational data mining outcomes (dependent variables), our understanding of the relationship inter-played among different factors, for instance, student-teacher interactions (Allen et al., 2011) and teacher quality (Kraft, 2015) can lead to intervention instruction design that improves academic outcomes. An educational system is a complex system which comprises the pedagogy, human assets (e.g. students and teachers), supportive tools (e.g. course materials and infrastructure), social and cultural influence, and government policies, among others. At the same time, the advent of technologies, in particular the World Wide Web (as known as the internet) has also given rise to new waves in the educational domain.
In this context, distance learning spurred by the emergent of the internet, has become one of the key components in higher education. It offers adult learners the opportunity to pursue their educational dreams without the barrier of distance. Distance learning practises the delivery of course materials without needing both student and teacher to be in the same physical room. The convenience is usually made possible by computer-mediated learning or two-way interactive videos (Tan, Bong and Natarajan, 2015). As such, pedagogic strategies designed to match the new level of instructions with the level of distance learners are rapidly gaining attention among modern educators.
Additionally, the internet has also opened up new opportunities for open education in the form of open educational resources (OER). OER include books, videos, journals, articles, podcasts, lesson plans, open software, and so on at the discretion of openness and sharing (Smith, 2009). Consequently, the increase in e-learning resources and student databases leads to massive repositories of data. Although there has been considerable research on the use of data mining (DM) techniques in discovering potentially useful information from large sets of data in the fields of healthcare (Wang et al., 2013), manufacturing (Durán et al., 2010), market analysis (Fiol-Roig and Miro-Julia, 2011), and many others; only recently has researchers begun to apply them to issues of educational realm through a discipline of known specifically as the educational data mining (EDM). The plausible revelation comes from the realisation that given any context, additional hidden information may be revealed if one invests in trying. EDM involves the development, research, and application of computerised methods in detecting potential patterns from huge pools of educational data (Romero et al., 2010).
With the aggressive push towards sophisticated technology and boundary-less education, more and more factors interact within the already complex educational systems. Unfortunately, as remarked by the Science journal (Smith, 2009), we are to some extent still lacking of the understanding about the effectiveness of value-added components (e.g. OER) introduced to the educational arena. Many educational research works employ EDM techniques that rely on statistical methods (with exception a few exceptions). However, these traditional ad hoc combinations of data management tools and statistical methods are now far from adequate in analysing the vast pool of data. Therefore, it is time to seek new computing approach that can be applied to the highly non-linear, complex, and large volume of data environment. Moreover, the approach ought to be able to handle incompleteness and shortage of data. Fortunately, soft computing lends the solution. Soft computing is a collection of methodologies capable of handling imprecision and uncertainty with the aim of achieving tractable, robust, yet low-cost solutions. Such characteristics of soft computing models make them invaluably attractive when applying to DM applications.
To give an immediate example, soft computing method, in particular the genetic programming complements existing educational research by providing better insights into how student participation affects achievement (Xing et al., 2015). Furthermore, empirical research using DM technique to classify early dropout from selective Mexican high schools have been carried out with success (Márquez-Vera et al., 2016). Yet another recent application of DM technique (i.e. cluster analysis) in exploring the correlation between the students' online interaction patterns and achievement is reported in Cerezo et al. (2016). While these approaches certainly have their strengths, the generic applications are 107 MOEA-based soft computing model for EDM contested when handling multi-objective optimisation problem (MOPs). In principle, MOPs require simultaneous optimisation of conflicting objectives. As in the classification of student achievement model, one would consider improving the classification accuracy rate of the predictive results using the least numbers of available features (e.g. demographic data, social data, past examination grades, and other school-related data) from a pool of data since it is often true that the latter is more difficult to obtain or even if they do, may not be in complete set. Therefore, a robust model should hold high accuracy rate at the presence of uncertainty or data incompleteness. This paper presents a soft computing technique, which comprises an evolutionary algorithm (EA), i.e., modified micro genetic algorithm (MmGA) , coupled with a decision tree (DT)-based classifier, namely, C4.5 for the classification and optimisation of system. The MmGA works well in MOP context as shown by a series of previous successes Tan et al., 2014;Tan, Hanoun and Lim, 2015;Tan et al., 2017). In order to evaluate the proposed soft computing model, an empirical-based case study is conducted. In particular, we aim to answer the first research question raised at the beginning of the paper. That being said, the case study attempts to uncover possibly appealing activities of students' online interaction against test score achievement in an open distance learning institution in Malaysia. Student performance modelling is noted as the second most popular areas in EDM research (Peña-Ayala, 2014). The implications of the present work can be translated into helpful information that may assist educators in designing instructions suitable for students customisable to their learning behaviours so as to improve achievement. Kremer et al. (2013) share the similar view that technology can be used to tailor learning to student's level of knowledge.
The remaining of the paper is organised as follows: the second section covers the background review of EDM and EA; followed by a short overview of C4.5 classifier. Third section is devoted to the description of our proposed methodology. The proposed model is first verified using a set of benchmark problems. Next, a case study for demonstrating the true efficacy of our model is described in fourth section. Finally, the paper concludes with some promising avenues for future work in fifth section.

Preliminaries
Review on EDM and learning analytics (LA) In recent years, the potential of DM and analytics has transformed field after field. DM is the process of analysing and gleaning useful but hidden information from huge data sets (Mukhopadhyay et al., 2014a, b). DM is popular due to its ability to discover data patterns, classify objects, cluster homogeneous objects, and unveil numerous kinds of new findings (Peña-Ayala, 2014). In the educational domain, a specific form of the DM is known as EDM. EDM emerges as an approach to explore educational massive data in enhancing the educational sector. It leverages the data mined through DM to improve learning, cognition, and assessment (Sachin and Vijay, 2012).
On the other hand, LA refers to the use of learner-data in reporting and analysing models for the purpose of predicting and advising learners (Ferguson, 2012;Hwang et al., 2017). Though LA is commonly identified with EDM, they differ in terms of goals and scopes. Baker and Inventado (2014) contrast between EDM and LA in a recent review: from the technical perspective, EDM deals with the development of methodologies for analysis of learning data, while LA focusses on the interpretation of these data for optimising learning and its environment. EDM also emphasises the modelling of relationships among specific constructs, whereas LA relates the interplay among constructs from a holistic view of the system. Additionally, EDM research works concentrate mostly on the development of automated support for learners. LA, on the other hand, puts more effort in informing and empowering instructors about learners' performance progress.

108
AAOUJ 12,1 To date, EDM has taken on and extended many other related fields including text mining, machine learning, statistics, and psychometrics.  propose a prediction model of student performance using DM techniques such as clustering and class association rules. The model has been claimed to be more representative of student groups (clusters) compared to previous rule-based model. EDM is also popularly used for student modelling (Lemmerich et al., 2011;Nandeshwar et al., 2011), which aims at characterising student by emotion, achievements, skills, learning preferences, and fulfilling individual's learning requirements through adaptation of teaching experiences. Another area of EDM application is on student assessment and evaluation, which enables student proficiency to be distinguished at a fine-grained level (Lopez et al., 2012). EDM also facilitates student feedback and support (Leong et al., 2012). More generally, EDM can be applied to educational problems with regards to emotion in context, engagement, meta-cognition, and collaboration tasks (Baker, 2014).
From the lens of LA, students' engagement and learning outcomes can be improved with proper intervention by instructors. To be effective, the intervention should be provided at the right time. LA and DM techniques are helpful in this case. A commonly used approach for discovering sequential patterns among events is known as sequential pattern mining. Chen et al. (2017) adopt the frequent sequent mining and lag sequential analysis (LSA) in order to study how learners collaborate in knowledge-building discourse. Similarly, LSA facilitates the exploration of learners' sequential patterns in other learning settings such as online interactions behaviour (Cheng et al., 2017) and problem-solving behaviour (Chiang, 2017;Hu et al., 2017). It is also not uncommon to adopt LA in analysing learners' behavioural patterns as a result of interaction with strategies or technological tools. For instance, Kizilcec et al. (2017) investigate various self-regulated learning strategies in MOOCs environment in hope of uncovering the most effective strategies and how they manifest in online behaviour. Meanwhile, Van Leeuwen et al. (2014) examine how teacher supporting tools in the context of computer-supported collaborative learning affect teacher guidance behaviour.
In general, more and more educators are turning to both EDM and LA for improving the educational outcomes. As shown by Xing et al. (2015), synthesising LA approaches and EDM techniques supplemented by genetic programming produces an effective student performance prediction model. The model has been claimed to possess higher prediction rate and interpretability compared to traditional models. Whether EDM or LA, educators can continue to benefit from the various scientific and systematic analysis methodologies available.
Review on EAs and multi-objective optimisation Natural evolution provides a promising collection of inspirations for computational algorithms. The group of computing methodologies, which analogises the evolutionary process of biological population in finding optimal solutions to optimisation problems, is known as EA (Golberg, 1989). Generally, EA can be divided into four major classes: genetic algorithm (Holland, 1992), evolutionary programming (Fogel, 1966), evolution strategies (Schwefel, 1993), and genetic programming (Koza, 1992).
Unlike traditional methodologies, EAs are distinguished mainly by the use of a population of search space. Each member of the population corresponds to a potential solution. The quality of the solution is determined by a fitness value associated with each member. During each iteration step (generation), better fitness members receive higher chances of survival or become the parents of the next generation. Offspring which are the new population members are generated using some variation operations, like mutation and/ or crossover. The evolutionary process ends after some termination criteria are met. These synergetic combinations of population-based, fitness-based, and variation-driven search have reported success in many complex optimisation problems Lim et al., 2015aLim et al., , b, 2016Tan et al., 2017). Meanwhile, the literature of GA runs

Proposed model
In this work, a soft computing model to classify and optimise students' online behaviours in a distance learning environment is presented. Students' online behaviours as characterised by a set of web data, forms the input to our proposed model. The web data represents the frequency of students' interactions with courses within the distance learning environment. Our aim is to classify students' frequency of access to the learning repository against their examinations achievement at the end of a semester. Followings are elaboration of the proposed model.
Initially, a standard C4.5 classifier (Quinlan, 1993(Quinlan, , 1996 is applied. It uses a divide-andconquer approach to growing DTs from a set C of cases. Suppose that C fulfils a stopping criterion of decision making. The tree of C is a leaf associated with the most frequent target class in C, which contains only cases of the similar target class. Meanwhile, the proportion of cases in X of jth class is identified. The uncertainty about the class for a case of X, and its corresponding information gained by a test T with k outputs are computed. Next, a specific MOEA, namely, the MmGA  is deployed. The MmGA performs optimisation on two objective functions, i.e., maximising the classification accuracy rate (α) and minimising the number of features (β) of classification process. Note that α describes the systematic errors and measures the statistical bias in handling predictors and outcomes of C4.5 classifier processes. As articulated earlier (recall section "Review on EAs and Multi-objective Optimisation"), the MmGA is able to achieve good convergence rate as indicated by the GD metric. MmGA's search process terminates when objective functions has reached the maximum round of evaluation or achieved convergence as measured by true Pareto. Details on C4.5 classifier as well as objective functions α and β with relation on MmGA are presented in the Appendix.

Benchmark tests
The proposed model aims to yield a solution set, which fulfils the objective functions f 1 and f 2 such that the classification accuracy rate is maximised, while minimising the number of features during the classification stage. Prior to application on a real-world case study, we first examined the proposed model's performance on a set of benchmark data obtained from the University of California at Irvine (UCI) machine learning repository (University of California, 2017). The benchmark data set comprises students' achievements in mathematics and Portuguese in two Portugal secondary schools. The data attributes include student 111 MOEA-based soft computing model for EDM grades, demographic, social, and school-related features. They were collected by using both school reports and questionnaires as published in Cortez and Silva (2008).
Note only mathematical achievements, which were modelled as binary classification, but five-level classification and regression tasks were adopted in this study. We adhere to the original performance evaluation of Portugal education. That is, students are evaluated in three periods during the school year based on a 20-point grading scale (with values between 0 ¼ lowest score to 20 ¼ perfect score). Hence, the data set is split into three classes according to period grade, i.e., first period grade (G1), second period grade (G2), and final grade (G3). As a result, each newly created data set has originally 30 features, which correspond to variable x in the Equation (A5) for each target class G1, G2, or G3, separately. To begin, the collected grades for each class were binarised prior to classification processing: student grades were re-categorised into two groups, namely, well performed (those above or equal to score 8) and not well performed (those below score 8).
For comparison purposes, the proposed model first uses only a standard C4.5 classifier (note: in the remaining of this paper, we merely refer this model by C4.5 classifier). Subsequently, an enhanced model which incorporates the MmGA coupled with standard C4.5 classifier is deployed. It is coined as the MmGA-based classifier. The MmGA analogises the evolutionary process of biological population in finding optimal solutions for MOP. In this context, by maximising α (Equation (A3)) and minimising β (Equation (A4)). Each member of the population corresponds to a potential solution, which is created with MmGA extended population formation. We also employ a ten-fold cross-validation method in producing the experimental results. All experiments involving both methods are repeated over 30 runs with randomised seed. Figures 1 and 2 depict the performance of the proposed model as compared to the standard C4.5 when simultaneously optimising the objective functions f 1 (x) and f 2 (x). Apart from a lower β achievement, our proposed model reported a higher α relative to the standard C4.5 classifier. For completeness, the mean and standard deviation values obtained for each experiment are tabulated in Table I. Mean values marked in italics indicate best statistical significance results at 95% confidence interval under the pairwise t-test (Hall and Holmes, 2003;Götz et al., 2008) comparison. The obvious yet encouraging results obtained inform us that our proposed model is superb in optimising the given data set using lesser number features but at the same time yielding much higher accuracy rate of classification. We attribute this to the superiority of MmGA in performing multi-objective optimisation. Consider a population of probable solutions (aka members) in our proposed model. Each member is represented as a variable x following Equation (A5) and is further associated with multi-objective-based fitness values, in this case α and β. The quality of the member is determined by its fitness values. Like all EAs, MmGA biases members with better fitness: At each iteration step, better fitness members receive higher chances of survival or become the parents of the next generation under an elitism strategy.

Results and discussion
Offspring, or new population members, are generated using mutation, crossover, and selection operators. The evolutionary process ends with both objectives converging in MmGA nominal and outlier evolution cycles; yielding p (Equation (A5)) in response to α and β.

MOEA-based soft computing model for EDM
A case study Satisfied with the preliminary results, let us now consider applying the proposed model to a real-world case study encompassing a Malaysian private institution of higher education with more than a decade of history in open distance education. The institution offers tertiary education to working adults via open distance learning mode. The learning environment has been catered for adult learners seeking to purse tertiary qualifications for professional development and self-enrichment in a flexible manner. Furthermore, students and tutors come from different regions across Malaysia and worldwide.
Rather unique in its kind, the open distance learning institution provides five face-to-face tutorial classes that are spread over a period of five months to its students every semester. It also offers learning-support services via an open source learning management system (LMS). The LMS is an important platform for collaborative learning involving massive teaching-learning activities among course instructors, tutors, and students. For example, apart from the face-to-face classes, students and tutors continue to interact via video conferencing tools supported by the learning platform. Students are also free to engage in online activities such as downloading course materials, posting discussion in forums, participating in online quizzes, submitting assignments, and many more at any time anywhere convenient to them. On the other hand, instructors and tutors often play the role of system administrator in the online platform by uploading course materials, initiating discussion groups, setting up quizzes, marking assignments, answering posts, and others. It should be noted that, throughout the semester, students are generally assessed using three instruments on three periods: assignment 1 (T1), assignment 2 (T2), and final examination on the second, fourth, and fifth month, respectively.

Experimental setup
Moving on, the proposed MmGA-based soft computing model is depicted in Figure 3. Initially, data extracted from the LMS go through a pre-processing stage. It involves gathering various students' interaction data from courses and converting their frequency into required raw data in a tabular format. Noise from the raw data are removed and transformed into a structured data format, i.e., an Extensible Markup Language file format, so that the C4.5 classifier may perform further processing. The processing stage involves employing MmGA-based soft computing model. Lastly, the output of the processing stage is made available for interpretation. In most complex systems, the interpretation may involve end-users and incorporation of other tacit knowledge to uncover the existence of any  possible relationship between the trends of students' online interaction activities with the e-learning platform and their examination performance, for instance. Students' daily online interaction activities for every course are captured in LMS. In this study, a total of 46 courses offered in the said institution are examined. The data are collected throughout the entire semester for 24 consecutive weeks, including two weeks prior to the start of semester and two weeks after the end of semester. This contributed to 24 features, which are further grouped into two targeted classes: well-performed and not well-performed student classes corresponding to examination scores above or equal to 60 marks and below 60 marks, respectively. Note that the number of features is determined by a fixed interval of seven days. They form the input features for the classification and optimisation processes in the subsequent experimental studies carried out within the institution's computational-based server farm.

Results and discussion
As depicted by Figure 4, there were initially many chosen features (i.e. weeks) resulted from the application of C4.5 classifier. After applying MmGA, a significant reduction in the number of features is observed. Worth remarking that the effect is achieved without reduction in the accuracy rates as shown in Figure   i.e., a reduction of up to 57 per cent relative to the standard C4.5 classifier. On closer inspection, the box plot distribution also reflects that the proposed model is more consistent and stable as compared to the standard C4.5 classifier since the former has narrower box and shorter tails. Reader is referred to Table II for the numerical results comprising mean, standard deviation, and p-value of the pairwise t-test comparisons. The best mean values marked in italics are statistically significant at 95% confidence interval.
To take a step further, let us examine the major determinants for the proposed model more closely. As illustrated by Figure 6, ten most prominent features (shaded in black) have been identified by our proposed model after optimisation. They represent the most significant weeks with student interaction activities that are adopted by the proposed model in classifying students achievement (recall Figure 4 and Table II). The captured interaction activities are not limited to students' post-reply inquiry on tutorials, technical hands-on, examination-related discussions, online quizzes, manipulation of teaching-learning materials, academia-related consultation, and clarification.
At second glance, the emergent patterns unfold several interesting events. First, students are actively involved in pre-semester activities before the start of a course (weeks 1 through 2). An obvious example of such activity includes exploration of course materials by students. The trend extends towards the second week just before the commencement of a new semester, which in turn corresponds to the closing date of course registration. This has come with little surprise as students are naturally more curious and eager to know about a new course being enroled in. But the obvious may have yet gotten the attention of educators. As evidential here, educators eager to improve students' first perception about a course, and subsequently leading to higher motivation in continuing the course (student retention strategy) should at least invest more time in the preparation work. Early content availability and accessibility, for example, would promote first positive impression and invite future interaction. A wide range of research works in cognition and social psychology attest to how initial impressions influence human interpretation of later information (Dougherty et al., 1994). Second, weeks 7 through 10 have been recognised as other significant indicators of student performance. Inherently, plenty of practical labs preparation and T1 discussions take place throughout the second month of the semester. To educators, this is likely the best time to engage more with students in ensuring that they are well with the course. The notion arises from the assumption that increasing the two-way interactions between tutor and student will enhance both student motivation and achievement. To this end, Allen et al. (2011) find strong correlation between high-quality tutor-student interactions and improved student achievement gains.
Whereas, the third week of the fourth month and the first week of the fifth month are the remaining two indicators identified for student performance. The former reflects the presence of discussion activities following T2, while the latter arises as a result of revision activities in conjunction with final examination. The preceding makes it clear that revision is important in boosting student performance. Educators may plan revision activities on gradual basis. Revision may be treated as a form of reinforcing previously learned knowledge through practises. Intuitively, educators should pay more attention to retrieval practice in consolidating learning (Karpicke and Roediger, 2008;Karpicke and Blunt, 2011) as part of the revision process. Finally, the common activities involved in the remaining two weeks (weeks 24 through 25) may include closing of posts and reporting of web statistics. They reflect the closing of the semester.

Conclusions
Many of the developing and potential EDM research on student achievement are contended to statistical methods (with a few exceptions) and lack the knowledge in handling multi-objective optimisation problems. To this end, this paper fills the gap by proposing a soft computing model with MmGA coupled with a DT-based classifier, namely, C4.5 classification and optimisation of system. Our model has shown confident results in student achievement classification under the UCI benchmark problem and real-world distance learning case study in Malaysia by simultaneous optimising multiple objectives (performance factors), i.e., maximising accuracy rate, α and minimising the number of features, β. Whether benchmark or real-world case study, the proposed model successfully reduced β by at least 48 per cent while achieving higher α.
We believe that this work can expand access to knowledge and insight into understanding student interaction activities and their achievement based on our empirical results. It may serve as a potential platform to inform educators seeking to reform educational policy by enhancing its provision of learning-support services and create a better learning experience for the students. To this end, the results presented could be easily translated into useful information such as when and what should be done in order achieve the target research goal. In the case study, early educator preparation work, improving tutor-student interactions, and investing in retrieval practices may well improve student motivation and achievement. This is only the beginning of a study that can lead to more elaborative outcomes for the educational arena. The results thus far could very well be true for the case study e-learning environment, but the proposed model is transferable to any optimisation and classification problems. We also plan to deploy other types of MOEAs and classifiers for other experiments in the near future. Finally, another promising work is investigation into the behaviour of the proposed model in response to the nebulous data covering different domains of interest to educators.
We need more rigorous research in the educational arena and soft computing models have opened up a new route. We believe that the future is bright and the vacuum of empirical evidence shall continue to be filled by the enthusiast research works of EDM and the alike.

117
MOEA-based soft computing model for EDM