A Multi-Dimensional Assessment Model to Evaluate Web Based Learning Systems

The growth of communication technology in the last decade has prompted researchers to develop web based learning systems for various types of applications. The entities involved in the development of such systems are learners, instructors, system designers, administrators to name a few. With varying nature of applications and entities involved, the task of evaluating these web based learning systems appears quite challenging. The present study uses factor analysis to develop an assessment model for evaluating the web based learning systems developed by the authors. The analysis is performed on a data set derived by applying a survey instrument on a set of 140 learners using these systems. These learners ranked 20 items used for evaluation in order of their perceived degree of importance. It was found that the most important dimension was learning outcome which coincides with the general outcome of any mode of learning.


Introduction
In recent times the most important innovation that has changed the face of educational technology is web-based education.This has been possible due to tremendous advancement in the field of computer networks and communication technology.There are two ways in which web based learning may be implemented.The primitive form of this type of learning is Electronic Learning (E-Learning).E-Learning is a general term that is used for learning from any electronic device such as radio, television, computer and all other devices that may be invented in the near future that are electronic in nature [8].A major disadvantage of this form of learning is lack of ubiquity and idle time utilization.The advancement in the field of mobile communications in late 90's gave birth to Mobile Learning (M-Learning).Mobility added to E-Learning is M-Learning.Mlearning enables learning independently of place and time as is ubiquitous through the use of wireless networks and mobile devices [21].
The effectiveness of web based learning is enhanced when several learners attempt to learn something together.In this form of learning, learners can capitalize on one another's resources and skills.Thus Collaborative learning (C-Learning) [7], [12] techniques may be used in E-Learning and M-Learning environments to increase their efficacy.Web learning systems are multidisciplinary in nature.They are used by learners from various domains who have varied experiences of using these systems.The design goals of these systems are thus highly dependent on the nature of the application and the knowledge of the learner.Efficient design of web learning systems is also dependent on the quality of the instructors, administrators and system designers.Thus there are several factors upon which the success of a web learning system depends.The aim of this study is to develop a multidimension model to evaluate web learning systems.It is to be noted that this assessment model is not general in nature.It has been derived on the basis of a survey performed to assess the web learning systems developed by author [2], [3], [4].Various methods have been employed by the researchers for assessing web based learning systems.A method typically used by several DOI: 10.12948/issn14531305/19.3.2015.03researchers is to develop a set of quality metrics for this purpose.Gafani [13] has used the ISO/IEC 9126 model to propose a set of quality metrics for evaluating Mobile Learning Systems.These metrics were defined and empirically validated for mobile wireless systems.Parsons et al. [24] proposed an assessment of mobile learning quality in terms of learner experience.A conceptual frame work for a mobile learning application was constructed and quality metrics were developed for this purpose.Wang [26] developed a comprehensive model and instrument for measuring learner satisfaction with asynchronous e-learning systems.This study performed different reliability and validity tests for analyzing data from a sample of 116 respondents of a survey.The norms of the evaluation instrument were developed from these.In a slightly different vein, Hwang et al. [17] developed a computer assisted website evaluation system for evaluating educational websites.Soft computing techniques like fuzzy theory, grey systems and group decision methods like Analytic Hierarchy Process (AHP) have been used for this purpose.It has been found that the developed system is capable of selecting proper criteria for evaluating websites.Alkhalaf et al. [5] has performed a survey on students of two different universities of Saudi Arabia to find the impact of e-learning systems on the learners.Statistical analysis employed on this survey data found that e-learning systems show positive impact on student learning.In light of the above discussion the organization of this paper is as follows: The following section gives a brief description of the web learning systems developed by the authors and the items used to evaluate these systems.These items are ranked by the users of these systems using a survey.The methodology used for conducting the survey and selection of participants are then described in the next section.Factor analysis is next performed on this survey data set to derive the dimensions of the assessment model with associated item sets along with relative importance of each factor.Trends noticed in the assessment model are then discussed.The study concludes with a parametric comparison of the current study with certain pioneering research works done in this field.

Origin of the Assessment Model
This section is described into two subsections.The first subsection describes the background behind development of the assessment model whereas the second subsection proposes the set of items that are used in the assessment model.

Web Learning Systems used for Assessment
This section describes the web learning systems developed by the authors that have been used in developing the assessment model.Concept Maps (CM) [22] have been used as a mind tool for structuring and organizing knowledge in these systems.A concept map is a directed graph that shows the relationship between the concepts.The directed arcs indicate the sequence a learner should follow to learn a subject.Fig 1 shows an example of a concept map of learning corresponding to a subject that has eight concepts C1, C2, C3, ….,C8 shown the form of the vertices of a graph.The directed edges of the graph indicate the sequence in which lessons are to be delivered to the learner.Thus in Figure 1 a student should first learn the concept C1 first followed by the concept C2.The concepts C3 and C5 can then be learnt simultaneously.Associated with the edge C1C2 there is a confidence level of 0.12 which states that if the student fails to understand C1, then the probability for him failing to understand C2 is 0.12 [16].A brief description of the web learning systems developed by the authors using concept maps is next discussed.DOI: 10.12948/issn14531305/19.3.2015.03

Fig. 1. A Concept Map of learning
The authors [3] proposed an e-learning system where concept maps were generated in a automated fashion for remedial purpose.Direct Hashing and Pruning (DHP) Algorithm [28] was applied on student historical test records to generate a set of association rules and relative weights between the concepts.The process worked by first generating the 1-itemsets and then 2itemsets between the test items and then deriving the association rules between the concepts using these and Test Item Relationship Table (TIRT) (Hwang,2003) as input.Once the concept map of learning is constructed the Remedial Learning Path (RLP) can be computed from it.The proposed method was tested with a set of students enrolled in an introductory 'Java' course in some under graduate colleges in Kolkata, India and found to diagnose their learning problems satisfactorily.This system is acronymed as CM-DHP for farther reference.A major drawback of this method is that relative weights of the concepts are not taken into account while computing the Remedial Learning Path (RLP).Thus the authors [4] proposed an extension of Concept Map which were called Weighted Concept Map (WCM).In this study relative weights were assigned to the concepts based on their degree of importance.Corresponding to a concept which is not properly learnt by a student, several RLPs were generated.The path for which the sum of products of weights and corresponding probability is maximum gives the best RLP.This RLP is to be used for remedial learning.This approach was tested with a set of middle school students and was found to diagnose their learning problems satisfactorily.This system is acronymed as WCM-DHP for farther reference.The concept map generated by CM-DHP is used as a sequence to develop an architecture of a learning system in mobile environment [2].In this work an Intelligent Diagnostic and Remedial Learning System (IDRLS) was proposed which helps the learner identify the concepts he is deficient in and what are the related concepts he should revise.The architecture uses an inference engine to generate association rules.The architecture also uses a learning portfolio to generate learning guidance which is sent as a form of SMS to the learner.A prototype of the system was implemented using Android Emulator [25].t-test was used to compare the pretest and posttest marks of a set of learners for an elementary 'Java' course and results found satisfactory.Finally collaboration techniques were applied on a set of college students for studying an elementary course in Java.The students were divided into three groups.The first group was given to study the subject in the theory class and the second group was given to study the subject in the practical class.These two groups collaborate between themselves using mobile devices to finalize the concept map of learning.The finalized concept map is sent to the third group in the form of short messages.The task of the third group was to prepare the learning and evaluation system.A prototype of such learning system for the group was then developed using Android Emulator.t-test between the marks secured DOI: 10.12948/issn14531305/19.3.2015.03by students before and after collaboration indicates that this form of learning has been effective.The development of this Collaborative Learning System (CLS) is currently under progress.

Item Set in Assessment Model
Based on these systems a set of 20 items are proposed to develop a model for assessing web based learning systems.The items along with their purpose are now discussed.In any learning system the most important entity is the learner.Thus in web based learning systems learner's computer aptitude (I5) is the foremost requirement.Learner will also require good communication skills (I14) to communicate with his course instructor.In this respect instructors' friendliness (I20) may be an added boost for the learner.In web based learning systems learners should be able to assess the learning system at any time from any location.Thus the instructor should be available (I11) to the learners most of the times.Communication skills are also required when several learners collaborate (I19) with each other to perform a certain task.After learners complete the learning process they are to be evaluated (I2).Another mode of evaluation in this regard is to understand group dynamics in regard to collaboration process.This aspect is surveyed in System Evaluation (I17).Learners should be able to evaluate themselves ubiquitously i.e. they appear for a test and the results are immediately messaged to them (I6) irrespective of their location.Yet another way of evaluating the quality of learning is to check learner satisfaction (I12).This is often done using a survey.A learning system should have a sequence in which lessons are to be delivered to the learners.Some form of knowledge management tools (I3) is often used for this purpose.Lessons are stored in the form of learning objects.Designing good quality learning materials require instructor knowledge (I1).A lot of effort is saved if the Learning Objects (LO) are reusable (I9).In certain applications learners are first given to learn the subject in traditional manner.Based upon their performance in the subsequent examinations, some form of remediation (I15) may be offered to weaker students.A learning application is often evaluated on the quality of the system.A computer system should not be assessed by all users i.e. it should be secure (I4).It should also be easy to use (I7) by all users.It should support increase in the number (I16) of learners.However this increase should not slow down (I8) the system.It should also yield results which the learner can depend (I10) on.The system may require some form of maintenance (I18) by the system administrators from time to time.Finally, the ultimate objective of any form of learning is satisfactory employment (I13).The entire set of items and their reference code are given in Table 1.A survey sheet was distributed to all the 140 students for obtaining their feedback.The survey sheet was divided into three parts.Part I queries about the background of the student, Part II asks the learners to rank the 20 items of Table 1 based on the degree of importance whereas Part III asks the students to describe their learning experiences using either of the Web based learning systems described in Table 2. Out of the 140 students 113 responded by submitting the survey sheet.Of them 97 survey sheet did not contain any missing fields.To ensure uniformity these survey data were considered for analysis purpose.A sample survey instrument form is attached in Appendix A.

Statistical Analysis
The data set obtained from a sample count of 97 students was analyzed for developing the assessment model.This process consists of four steps.Firstly factor analysis was performed on the data set to represent the data in suitable dimensions.In the second step the reliability of this form of factorization is checked using Cronbach Alpha.t-test is then applied between the derived factors to find the degree of significance between the factors.Finally, regression analysis was used to find the degree of importance of each items and factors.Part II of the survey data was used for analysis purposes.The entire simulations were carried out using Statistical Package for Social Sciences (SPSS) [6].DOI: 10.12948/issn14531305/19.3.2015.03

Generating Factor Structure
In this study Varimax rotation was used to find the factor structure.This method generates a small number of factors with large loadings and a large number of factors with small loadings.This method also simplifies factor interpretation since each variable tends to be associated with one factor and each factor tends to be associated with small number of variables [1].Table 3 shows the values of factor loadings for each of the 20 items specified in Table 1.Costello and Osborne (2005) suggested that factor loadings greater than 0.3 may be considered significant.Thus the factors I8, I11 and I12 are deleted as they do not satisfy this criterion.Mela and Kopalle [20] has found that multi collinearity can reduce parameter variance estimates.One way of checking multi collinearity is to compute the determinant of the correlation matrix (0.0032) which is greater than 0.0001.Thus all the items used for developing the proposed assessment model are well correlated.
Another way to assess whether multi collinearity affects the survey data is to compute Kaser-Meyer-Oklin (KMO) statistic.Minimum value for this statistic should be 0.5, values between 0.5 and 0.7 are mediocre, values between 0.8 and 0.9 are great whereas values greater than 0.9 are superb [15].Bartlett's test for sphericity says that if the value is less 0.5, then the results may not be considered significant.For the survey data set this value is 0.734.Thus it may be concluded that survey data factors quite well.Also total variance explained by the four factors is 58.99%.This value again indicates that the factors generated by factor analysis are consistent with the survey data.MacCallum et al. [19]

Reliability of the Factor Structure
The purpose of the reliability test is to determine whether the same survey instrument with a different data set would generate the same factor structure.The test used by most researchers in this respect is Cronbach alpha [10].Cronbach alpha generally increases with inter correlations among survey items and thus its value is maximized when all items measure the same latent construct.A value of alpha greater than 0.9 indicates excellent degree of internal consistency, a value of 0.7 to 0.9 indicates good deal of internal consistency, a value of 0.6 to 0.7 indicates acceptable degree of internal consistency whereas a value less than 0.6 indicates poor degree of internal consistency [23].For the factors structure shown in Table 4 derived from survey data set, it is found that Cronbach alpha is 0.782 which indicates high level of internal consistency.The Cronbach alpha for each of the four factors were found to be 0.731, 0.764, 0.711 and 0.694 which again indicates good deal of internal consistency within the factors.The item total statistics do not show any significant increase of Cronbach alpha if any of the factors are deleted and hence no further changes are made to the factor structure.

Checking the Degree of Significance between the Factors
Unpaired t-test [14] was applied on the average values between the items factor wise.It is observed that in most cases the degree of significance between the factors is either very or extremely significant except between F3 and F4.Thus it may be deduced that these factors are by and large independent of each other and may be assumed as a basis for evaluation of Web based learning systems.

Estimating the Degree of Importance of each Factor
Factor Scores [11] corresponding to each item loadings was used to compute the degree of importance corresponding to each item and factor.For this the Component Score Coefficient Matrix was used which was derived from linear regression model.Each item was assigned a score derived from this matrix.Each factor score corresponding to an item is divided by the sum of scores for the corresponding factor.This score in terms of percentage gives the degree of importance of each item.As an example, factor scores corresponding to I2, I13 and I17 are 0.152, 0.314 and 0.123 respectively.Thus the degree of importance for I2 in percentage is given as (0.152/(0.152+0.314+0.123)X 100) = 25.80 %.Similarly the degree of importance for I13 and I17 are computed as 53.32 % and 53.32 % respectively.The degrees of importance corresponding to other items for each factor are computed similarly.After these computations, the degree of importance of each factor is computed.Similar methods are employed for computing these from their respective factor scores.The final model along with the items, factors and their relative degree of importance is enumerated in Table 6.In IDRLS students were first given to study the course in a conventional manner.Exams were then conducted to evaluate these students.Based on these exam results, the Concept Map of learning was constructed.This Concept Map was then used as a learning sequence.Thus the learning systems developed are remedial in nature and hence this factor was given high degree of importance.Collaborative techniques have also been employed to construct the Concept Map of learning group wise (CLS).High degree of importance to this item indicates that students are satisfied with this collaborative learning scheme.Although Learning Objects were constructed for all the learning systems but they were not reused indicating a poor degree of importance (11.47%).The factor System Requirements has poor degree of importance (17.21%).System designers and software engineers are mainly behind system construction and hence this factor has failed to ignite the interest among the learners.Within these the items Security (26.43%) and Scalability (23.18 %) has attracted higher degree of attention.IDRLS and CLS have used login-id and password as security mechanism.The effect of scalability has been examined in details in CLS.It has been found that learner performance varies with change in group size.The items Ease of Use (15.14%),Reliability (17.13 %) and Maintainability (18.12 %) attract poor degree of importance as they fail to appeal to the learners.The factor Learner and Instructor Attitude (27.17%) is given moderately high degree of importance.Learners who are not conversant with web based learning techniques would obviously like the instructor to be friendly (30.26 %) and hence its high degree of importance.Learner Communication skills (27.02 %) are also important as it enhances collaboration (CLS).It has also been found that certain learners are the first time users of computers or mobile devices for learning purpose.This is perhaps suggests moderate degree of importance of Learner Computer Aptitude (23.14 %).Finally most of the subject materials have been stored in the form of Learning Objects.The learners have mostly used these for learning and hence much importance has not been given to Instructor Knowledge (19.58 %).Finally as it turned out most important factor is Learning Outcome (30.31%).Students are mostly interested in the Employment (53.32%) opportunities they would get after completion of the course and hence very high degree of importance of this item.Since the systems developed were mainly used by students who had once performed poorly in the examinations, students were perhaps concerned with the methods of Evaluation (25.80%) as well.Thus a moderately high degree of importance is given to this factor.Finally, group dynamics of the collaborative process were evaluated in System Evaluation (20.88 %).Poor degree of importance to this item suggests that users are not interested in evaluating collaboration dynamics.

Conclusion
The current work proposes a set of 20 items used for evaluating the web learning systems developed by the authors [2,3,4].These items are ranked by a set of 140 learners who used these systems by a survey.Factor Analysis is then applied on this survey data to develop a multi-dimensional model for assessing web based learning systems.It has been found that the most important dimension is PART II (Item Rating Information) 7. Rank each of these items (1-20) based on your perceived degree of importance.

Item Name
Rating

Table 1 .
List of items used for assessingWeb based Learning Systems Based on the Concept Map obtained from this data set IDRLS was constructed.The experiments conducted in CLS consisted of under graduate students from three colleges in Kolkata.These colleges are acronymed as College A, College B and College C respectively.24studentswere chosen from college A, 38 students from College B and 32 students from College C. However the 24 students of college A were a subset of the 60 students in CM-DHP and IDRLS.Thus the total number of students who have used at least one of the web learning systems is 60+38+32=140.This is thus the sample count for the proposed survey.The details of the Web Learning systems along with their users and survey participants are summarized in Table2.

Table 2 .
Summary of System and Survey participants

Table 3 .
Factor Analysis of survey results

Table 6 .
Factor Structure with degree of importance

Table 7 .
Limitations of the model: The efficacy of the developed evaluation model is greatly dependent on the quality of the learners on whom the survey instrument is applied and the statistical techniques used in deriving the model.The comparative study of the papers listed above with the current study on the above proposed parameters is presented in Table7.Comparison of current study with related works