HYBRID MODEL FOR E-LEARNING QUALITY EVALUATION

E-learning is becoming increasingly important for the competitive advantage of economic organisations and higher education institutions. Therefore, it is becoming a significant aspect of quality which has to be integrated into the management system of every organisation or institution. The paper examines e-learning quality characteristics, standards, criteria and indicators and presents a multi-criteria hybrid model for e-learning quality evaluation based on the method of Analytic Hierarchy Process, trend analysis, and data comparison.


INTRODUCTION
Education and knowledge are increasingly becoming primary developmental resources for the competitive advantage of an organisation (ranging from a company, nation, country, to a region and economic integrations). Therefore, the challenges of higher education development in the 21 st century can be understood as a threat or as an opportunity. If they are seen as a threat, universities will put emphasis on preserving the past. If they are regarded as an opportunity, the emphasis is on the future, which demands the use of best practice from the past and the development of high quality higher education institutions [12]. While estimating the quality of higher education institutions, it is necessary to use the systems approach. This means that a higher education institution must be viewed as a complex system which is a part of a dynamic, changing environment and which interacts with the environment in complex ways through educational, scientific and applicable processes [16].
Online education contributes to the quality of higher education institutions. Today, a number of expressions are used to refer to online education, such as Internet-based education, web-based education, education via computer communication. However, the term electronic learning (e-learning) is often used as a generic term and as a synonym for online education. Electronic learning refers to the use of electronic devices and software for learning, including the transmission of content by means of electronic media such as the Internet, audio or video media, satellite transmission, interactive TV, CD-ROM etc. [7].
The Open and Distance Learning Quality Council (UK) describes e-learning as an effective learning process created by combining digitally delivered content with (learning) support and services, while the European Commission defines e-learning as the use of new multimedia technologies and the Internet to improve the quality of learning by facilitating access to recources and services as well as remote exchanges and collaboration [8].
The paper examines e-learning quality characteristics, standards, criteria and indicators, and presents a multicriteria hybrid model for evaluating the quality of elearning based on the method of Analytic Hierarchy Process -AHP, trend analysis, and data comparison.

E-LEARNING QUALITY
International documents adopted by the United Nations (The Millenium Development Goals) and UNESCO (Education for All), as well as documents about the Bologna Process (declarations, communiqués, reports), emphasise that e-learning will have an important role in achieving certain global aims. This primarily refers to a need for lifelong education, market internationalisation and globalisation, e-business, e-governance, as well as sustainable development in different economic and social fields.
Mc Loghlin and Lee [7] believe that participation, production and personalisation, or the three Ps, point to a new direction in e-learning in the 21 st century. The three Ps are crucial within the concept of collaborative learning and networking.
Different aspects of e-learning research include qualifications and competences of students and teaching staff, the culture of teaching and education, new forms of student-teacher interaction, flexibility of curricula and education, personalization, student awards etc. As a concept, flexibility should not be viewed only in terms of geography and modes of learning, but also in terms of language and accessibility. In other words, flexibility refers to much more than time and space in the digital world [7].
Lerning through technology and learning to use technology in a society based on knowledge require a new approach to formal and informal higher education. Universities will have to include e-learning in their development policies and strategies. They will also have to pay special attention to e-learning quality and to integrate e-learning into the institution"s quality management system. E-learning quality development involves defining a quality strategy, as well as defining the processes of quality analysis, design, realisation, evaluation, and continuous improvement within the system of e-learning [21]. It also involves a long process of establishing the system of e-learning and its integrating into other processes of an educational institution.
The process of adopting, realising and adjusting elearning quality can be viewed at three levels: the level of the individual, the level of the institution and the level of the integration of all stakeholders.
The importance of e-lerning quality assurance has been recognised by the European Commission, which has introduced three different initiatives involving the policies of e-learning development [8]:  eEurope,  Education and Training 2010 and  eLearning Initiative.
In its action plan for e-learning which involves four fields (infrastructure and equipment; quality, content and services; training at all levels; European co-operation and networking), the European Commission emphasises the importance of e-learning. Accordingly, four strategic projects analysing different aspects of e-learning quality have been carried out:

E-LEARNING QUALITY STANDARDS
International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC) have been working intensively on e-learning standardization since 2004. 18 international standards related to elearning had been introduced by 31st December 2010 [13].
The application of ISO/IEC standards requires a certain level of e-society, e-educational institutions and an appropriate national standardization.
Quality standards offer special benefits for organisations, processes and products. Generally, seven main advantages of e-learning quality standards can be identified [21]: 1. Competitiveness (quality standards can increase competitiveness by making it possible to compare elearning performances); 2. Cost-effectiveness (by clearly defining processes, quality standards can reduce failure in the process of elearning analysis, design, implementation and realisation); 3. Motivation (it can be improved through the transparent participation of all stakeholders in formulating the demands of the quality system); 4. Image (quality standards facilitate international acceptance and recognition and increase the reputation of the institution and the e-learning programme); 5. Planning reliability (quality standards enable quality testing and evaluation and business excellence of the elearning system, as well as their re-evaluation); 6. Customer orientation (e-learning based on quality standards makes it possible to establish an equal partnership between teaching staff and students, better understanding and greater customer satisfaction); 7. Continuous improvement (e-learning organisations, processes and products should continuously strive for high quality and business excellence in e-learning).
Although a number of e-learning standards have been established, only one series of standards is related to elearning quality (ISO/IEC 19796).

ISO/IEC 19796 series of standards
The ISO/IEC 19796 series of standards provides a framework for identifying types of data, harmonizing different quality management systems, identifying quality metrics and methods, and providing examples of the best practice for quality e-learning.
This series of standards was published as ISO/IEC 19796: Information technology-Learning, education and training -Quality management, assurance and metrics. It consists of five parts. The ISO/IEC 19796-1 standard is a framework for describing, comparing, analysing, and implementing quality management and quality assurance approaches. It will be used for comparing different existing approaches and harmonizing them towards a common quality model [9].
The main aspects of the RFDQ model are [11]:  description scheme for quality management;  process model defining the basic processes to be considered when managing quality in the field of ICT-supported learning, education, and training;  conformance statement for the description format. The reference process model involves the whole lifecycle of learning, education and training, including e-learning and blended learning. Therefore, it can be used to describe any learning scenario, as well as any kind of educational or vocational training. The reference process model serves as an open descriptive framework which should always be adjusted to the organisation and the learning context, i.e. to the actual situation. The reference process model is an integration of two reference models: the generic process model and the generic description model [20]. The generic process model contains 38 processes grouped into seven categoties (Needs Analysis, Framework Analysis, Conception/Design, Development/Production, Implementation, Learning Process and Evaluation/ Optimization). This model describes the structure of the learning process lifecycle, but it does not contain any instructions or procedures for the realisation of the above mentioned processes. The generic description model defines a standardised way of describing all the 38 processes which compose the elearning process. The description of processes, according to this model, should contain: ID (unique process identifier), category, process name, description, relations, sub-processes/sub-aspects, objective, method, result, actors, metrics/criteria, standards and annotation/example.
Another standard from the ISO/IEC 19796 series was developed by ISO/IEC JTC1 SC36/WG5 and published in 2009 as ISO/IEC 19796 Part 3: Reference Methods and Metrics. The standard is also known as ISO/IEC-3. It broadens the reference framework for the description of quality approaches (RFDQ) defined in ISO/IEC 19796-1 by providing a description of the methods and metrics necessary for the implementation and application of quality management and the system of quality assurance in the processes of learning, education and training. ISO/IEC 19796-3 is an important instrument for the application of the ISO/IEC 19796-1 standard, especially the part referring to the description of certain processes [10]. Image 1 shows the relations between parts of the ISO/IEC 19796 series of standards, and Image 2 -the relations between the quality management/assurance model and the reference process model (RFDQ).

E-LEARNING QUALITY CRITERIA AND INDICATORS
The SEEL project defines a criterion as a principle enabling the assessment of the state of development of each area identified Also, SEEL defines an indicator as an objective element of information which facilitates the analysis of and comparison between services/systems according to the criteria established, while ISO defines an indicator as an objective attribute or characteristic of a practice or work product that supports the judgement of the performance of, or capability of, an implemented process There are three types of indicators [4; 6]:  structural indicators,  practice indicators and  performance indicators.
Structural indicators assess what are sometimes termed "enablers". Enablers are essentially the resources available to the institution to enable it to carry out its mission and objectives. They include: institutional and human competences; technology platforms and tools; governance and management structure.
Practice indicators evaluate the ways in which the institution utilizes its resources. They assess the work practices and processes of the institution. They focus on: the business strategy of the organisation; its targeting and access policies; its pedagogic approach.
Performance indicators assess the results of the interaction between work practices and enablers. They focus on outcomes and impacts, such as: learning outcomes; cost-benefits; technical effectiveness.
The relationship between the three types of indicators is shown in Image 3. Table 2 presents the main criteria for evaluating elearning quality defined during the research on the following European projects: BENVIC [6], SEEL [18], SEEQUEL [19] and CHIRON [2]. It also presents the criteria of the E-Learning Quality model -ELQ, developed by the Swedish National Agency for Higher Education, based on the analysis of the existing European policies and projects, practices of national organisations and research related to e-learning quality [5]. For the projects which define e-learning indicators (BENVIC, SEEL, SEEQUEL, CHIRON), the table gives the number of indicators defined for each criterion. Since the ELQ model does not contain indicators, the number of subcriteria is given for each criterion.

MODEL FOR E-LEARNING QUALITY EVALUATION
One of the models that can be applied to evaluate elearning quality is a hybrid model based on the AHP method, trend analysis and data comparison. This model involves three steps:  Application of the AHP method  Trend analysis and data comparison  Quality index determination

Application of the AHP method
The AHP method was developed by Thomas L. Saaty in the 1970s [15]. It offers a flexible and relatively easy way of analysing and decomposing the problem of decision making. It is a multi-criteria decision making methodology that considers both subjective and objective factors in the evaluation process.
The AHP method involves the following steps: 1. The overall goal (objective) is identified and clearly defined; 2. The criteria, sub-criteria and alternatives which contribute to the overall goal are identified; 3. The hierarchical structure is formed; 4. Pairwise comparison is made; 5. The priority weights vector is estimated using the eigenvalue method; 6. The consistency of judgements is checked; 7. The global priority vector is calculated.

Goal identification. The goal is e-learning quality evaluation.
Identification of criteria and alternatives. Criteria can be identified according to the national standard for the accreditation of e-learning curricula, or they can be taken from international publications, that is, technical reports about the projects involving research on e-learning quality ( Table 2). As there are a lot of indicators of e-learning quality (Table 2), it is necessary to define the key performance indicators, which can be done by the expert group.
Hierarchical structure formation. The AHP method presents a problem in the form of hierarchy. Generally, a hierarchy is structured from the top level (goal or objective), through intermediate levels (criteria and subcriteria) to the lowest level (alternatives). In order to evaluate e-learning quality, it is important to define the hierarchical structure which has three levels: the first level or the top level represents e-learning quality; the second level considers relevant criteria; the third level defines key performance indicators. Image 4 shows the hierarchy scheme for e-learning quality evaluation.

Image 4: Hierarchy scheme for e-learning quality evaluation
Pairwise comparison. Pairs of elements of a problem at each level are compared according to their relative contribution to the elements at the hierarchical level above. The decision maker or expert group estimates the relative contribution of each pair to the objective, or to the criterion, using the 1-9 comparison scale, as shown in Table 3. If action i has one of the above non-zero numbers assigned to it when compared with action j, then j has reciprocal value when compared with i  (1) where A is an nxn matrix, (n is the number of compared elements). Elements of A, a ij , are the judgements about the relative importance of alternative i over alternative j. They have the following characteristics: a ij =1 for i=j and a ij =1/a ji for i≠j.
Relative weights determination. The mathematical basis for determining weights based on matrix theory was proposed by Saaty [9]. The procedure, which is called an eigenvector approach, is based on a special type of matrix called a reciprocal matrix. The objective is to find w: where w is an eigenvector and a column matrix. An eigenvector can be generated in different ways. This paper presents the most frequently used method for determining the eigenvector for matrix A.
The procedure for obtaining the eigenvector involves the following steps: (1) The sum of all elements in each column is calculated; (2) Elements of each column are divided by the sum obtained in the previous step; and (3) The average value of each raw is determined. A column consisting of the average values is a normalized vector (or an eigenvector, or a priority weights vector) [3].
As a result of applying the procedures for determining relative weights, the following vectors are defined: where w i is the weight of i-th criterion, n is the number of criteria, and w 1 +…+w i +…w n =1.
where S i is an alternative eigenvector for the i-th criterion, S i,k is the local priority (score, weight) of the k-th alternative in relation to the i-th criterion and m i is the number of alternatives for the i-th criterion.
Checking results consistency. Consistency means that the decision making procedure involves coherent judgements in the process of specifying the pairwise comparison of the criteria or alternatives.
However, given the above characteristics of the matrix (reciprocal and regular diagonal matrix with elements a ii =1), small changes in the values of a ij retain the highest eigenvalue, λ max , while other eigenvalues are nearly zero. Therefore, the deviation of λ max from n is used to determine the level of consistency. The procedure for obtaining a consistency value is as follows [16]: 1. The matrix equation is solved: i.e.
where RI is a random index. The procedure for calculating this index is described below.
The consistency check of pairwise comparison is done by comparing the calculated consistency index with the average consistency index of randomly generated reciprocal matrices using the 1-9 comparison scale. The consistency index calculated in this way is called a random index. Table 2 shows random indices for matrices nxn, where n = 1,…,8. If a consistency ratio is 0.10 or less, it can be considered acceptable; otherwise, the judgements should be improved. This can be done by double checking the data entry and by omitting bad judgements which have high inconsistency ratios [1]. Global priority determination. The final stage of the AHP method involves finding a composite normalized vector (or a vector of global priority), which shows the contribution of certain alternatives to the achievement of the goal. However, this step is replaced by trend analysis and data comparison in the suggested model.

Trend analysis and data comparison
In order to measure e-learning quality indicators, trend analysis is done and data are compared. The weight (or the contribution) of each indicator is modified by a coefficient (score) whose value depends on the indicator trend compared to the previous year, as well as on its current value which is compared to the value of the benchmark.
The following decision rule is basic for the scoring mechanism [23]: (1) If the indicator trend is growing and the current level is higher than the benchmark, then score is 100.
(2) If the indicator trend is growing and the current level is lower than the benchmark, or if the indicator trend is declining and the current level is higher than the benchmark, then the score is 50.
(3) If the indicator trend is declining and the current level is lower than the benchmark, then the score is 0.

Quality index determination
E-learning quality index is calculated by the following formula: where i is the index for e-learning quality criteria, k is the index for alternatives (e-learning quality performance indicators), w i is the weight of the i-th criterion (the second level); S i,k is the weight of the k-th alternative (elearning quality performance indicator) related to the i-th criterion (the third level); s ik is the score which modifies the weight of the k-th alternative (e-learning quality performance indicator) related to the i-th criterion.

CONCLUSION
E-learning is increasingly becoming a necessary aspect of education and an important aspect of quality of higher education institutions. Universities will have to include elearning into their development policies and strategies. They will also have to pay special attention to e-learning quality and integrate it into the institution"s quality management system.
A number of international and national documents and projects emphasise the importance of e-learning quality. The analysis of these documents and project results shows different aspects of researching this problem, a number of different criteria and sub-criteria, and especially a large number of e-learning quality indicators. This implies a non-systems approach to research, that is, the decomposition of a problem and partial research on its individual parts. However, e-learning quality requires a systems approach to research and problem solving within knowledge management quality, and more broadly, within the institution"s quality management [17]. A good basis for this kind of approach is provided by the ISO/IEC 19796 series of standards. It defines e-learning processes, the structure of the learning lifecycle, as well as methods and metrics necessary for the implementation and application of quality management and the quality assurance system in the processes of learning, education and training.
The paper presents the methodology for estimating elearning quality based on a hybrid model which involves the AHP method, trend analysis and data comparison. The fuzzy AHP method can also be used for e-learning quality evaluation [20] , but it isn"t dealt with in the paper.