Assessment of the Effectiveness of Internet-Based Distance Learning through the VClass e-Education Platform

This study assesses the effectiveness of internet-based distance learning (IBDL) through the VClass live e-education platform. The research examines (1) the effectiveness of IBDL for regular and distance students and (2) the distance students’ experience of VClass in the IBDL course entitled Computer Programming 1. The study employed the common definitions of evaluation to attain useful statistical results. The measurement instruments used were test scores and questionnaires. The sample consisted of 59 firstyear undergraduate students, most of whom were studying computer information systems at Rajamangala University of Technology Lanna Chiang Mai in Thailand. The results revealed that distance students engaged in learning behavior only occasionally but that the effectiveness of learning was the same for distance and regular students. Moreover, the provided computer-mediated communications (CMC) (e.g., live chat, email, and discussion board) were sparingly used, primarily by male distance students. Distance students, regular students, the instructor, and the tutor agreed to use a social networking site, Facebook, rather than the provided CMC during the course. The evaluation results produce useful information that is applicable for developing and improving IBDL practices.


Article abstract
This study assesses the effectiveness of internet-based distance learning (IBDL) through the VClass live e-education platform. The research examines (1) the effectiveness of IBDL for regular and distance students and (2) the distance students' experience of VClass in the IBDL course entitled Computer Programming 1. The study employed the common definitions of evaluation to attain useful statistical results. The measurement instruments used were test scores and questionnaires. The sample consisted of 59 first-year undergraduate students, most of whom were studying computer information systems at Rajamangala University of Technology Lanna Chiang Mai in Thailand. The results revealed that distance students engaged in learning behavior only occasionally but that the effectiveness of learning was the same for distance and regular students. Moreover, the provided computer-mediated communications (CMC) (e.g., live chat, email, and discussion board) were sparingly used, primarily by male distance students. Distance students, regular students, the instructor, and the tutor agreed to use a social networking site, Facebook, rather than the provided CMC during the course. The evaluation results produce useful information that is applicable for developing and improving IBDL practices.

Introduction
Distance learning involving communication technology such as internet-based distance learning (IBDL) enables institutions to conduct classes on limited budgets and with limited teaching staff while providing the same education quality to both distance and regular students. Royalty-free software applications that are designed for education, such as the VClass learning management system (VClass LMS), are ideal for circumventing budget and teacher shortages, and many universities have adopted IBDL.
The effectiveness of online learning environments must be examined, and methodologies for evaluating this effectiveness are necessary. Therefore, this study assesses the effectiveness of the IBDL pedagogy provided by the VClass e-education platform in live mode via a designed evaluation process and discusses its implementation.
The following sections describe the research goal and objectives as well as the design and details of the evaluation process and its implementation for IBDL. Subsequently, a discussion of the results and a summary and conclusion are presented.

Research Goal and Objectives
IBDL technology is vital to educational institutions, regardless of budget and teaching staff. This technology provides the same quality of education to regular and distance students. Evaluation plays a key role in the utilization of IBDL in educational institutions.
This study assesses the effectiveness of IBDL implemented through the VClass eeducation platform in an educational institution to aid in determining its feasibility in universities. Specifically, this study has the following three research objectives: 1. to design an evaluation process to assess the effectiveness of IBDL, 2. to investigate the effectiveness of VClass IBDL for regular and distance students by utilizing the designed evaluation process, 3. to further analyze the distance student experience with IBDL.

The Study
Several studies have proposed evaluation models for IBDL (Baker, 2003;Passerini & Granger, 2000;Sims, Dobbs, & Hand, 2002). Passerini and Granger (2000) proposed a development framework that comprised a behaviorist step-by-step development process that focused on the constructivist paradigm. This framework consisted of analysis, design, development, evaluation, and delivery processes, which were similar to the phases of the system development life cycle. In evaluating the framework, a variety of methods (e.g., questionnaires, user focus groups, or interviews) could be utilized as formative evaluations, and summative evaluations occurred after the instruction was implemented. These evaluations were designed to assess the overall effectiveness of the instructional layout. Feedback on a variety of criteria, particularly the asynchronous and synchronous communication experience, could be refined from both formative and summative evaluations. Sims, Dobbs, and Hand (2002) proposed a proactive evaluation framework by integrating the production process. The framework identified critical online learning factors and influences in planning, designing, and developing learning resources during formative assessment by focusing on the decision-making process, which was based on the complex interaction between subject-specific content, learning outcomes, and online computer-based learning environments. The online learning factors of the proposed framework, which were considered to be critical to effective online learning, comprised strategic intent, content, learning design, interface design, interactivity, assessment, student support, utility of content, and outcomes. For the assessment criteria of this framework, five elements (i.e., assignments, examinations, project work, work placement, and authentication) were suggested to assess teacher-, peer-, or student-directed factors within the context of critical factors, including the assessment of new environments in which the performance data were collected (e.g., real-world workplace environments). Baker (2003) proposed a framework of K-12 curriculum components to evaluate outcomes that are specific to online distance learning courses; this framework integrated an adaptation of Tyler's principles with the level of cognitive learning in Bloom's taxonomy. Baker (2003) indicated three factors by which distance learning courses differ from traditional in-class courses: 1) a personal interaction factor, which refers to the instructor's opportunity to provide feedback and direction and to observe learning activities in real time; 2) an interaction factor, which refers to interactions among learners as they share experiences; and 3) a class attendance factor, which refers to learners' responsibility and motivation to perform the required tasks. These factors should be considered in selecting distance learning tools. Ferguson and Wijekumar (2000), as cited in Baker (2003) Comparative studies of the effectiveness of online learning remain to be reported.
Several studies that assessed the effectiveness of distance learning by comparing regular and distance students found no significant differences in learning outcomes (Abraham, 2002;Shachar & Neumann, 2003;Smith & Dillon, 1999;Zhao, Lei, Yan, Lai, & Tan, 2005). Other studies found that internet-based student performance was significantly higher than classroom-based performance for four examined items (i.e., knowledge, comprehension, application, and overall exam scores) because of the increased studentcenteredness of IBDL and because students prepared more effectively for the internet-  (Thomas, Simmons, Jin, Almeda, & Mannos, 2005).
Several meta-analyses have reported that significant differences in several factors were found in studies that compared the effectiveness of regular and distance student performances (Shachar & Neumann, 2003;Zhao, Lei, Yan, Lai, & Tan, 2005).
Additionally, several studies have reported that "students who had the opportunity to interact with each other face-to-face performed better than those lacking that opportunity" (Koch & McAdory, 2012).

Evaluation Elements for IBDL and VClass
In general, evaluation methods appear to be acknowledged by scholars in the field.
Nevertheless, certain models may not clearly present the meanings or definitions of terms involved in educational evaluation. Therefore, this paper restates the primary terminology used in the evaluation processes as follows.

Measurement.
In relation to evaluation, "measurement" is the act or process of measuring something (Hornby, 1995;Levine, 2005). Educational measurement is the process of calculating the success of an instructional activity using a data set (e.g., test scores, midterm scores, or dropout rate).

Assessment.
Educational "assessment" refers to a process that attempts to understand and improve student learning. This process includes clarifying instructor expectations to students and setting appropriate outcomes for learning by using relevant information (Huitt, Hummel, & Kaeck, 2001;Levine, 2005).

Computer-mediated communication.
"Computer-mediated communication" (CMC) is the process by which people create, exchange, and perceive information when using networked telecommunications that facilitate encoding, transmitting, and decoding messages. CMC can be synchronous, for example, chat rooms, or asynchronous, for example, discussion boards and emails (Romiszowski & Mason, 1996). VClass (DEC, 1998) in live mode uses three servers, VClass LMS, conference, and session initiation protocol (SIP). The VClass LMS server stores learning resources, student and instructor profiles, and recorded video streams, which students and instructors utilize before and after class sessions. The conference and SIP servers are used for video conferencing, streaming multimedia distribution, and instant messaging from the sender to the remote location over the internet networking system. Thus, VClass in live mode allows instructors and distance students to communicate by microphone and instant messaging in real time.

Methodology
As certain studies have suggested (Ferguson & Wijekumar, 2000, cited in Baker, 2003, the use of the educational technology of this study is based on the idea of using an educational technology that was developed and available in our country. Thus, a royalty-free software application namely VClass which was developed by DEC at AIT in Thailand was utilized.

Learning environments.
The learning environment for VClass in live mode, as illustrated in Figure 1, is divided into two parts, regular and distance classroom settings. Both settings must have internet access.  In the regular classroom, a few components are added: a switching hub, delivery computer, video camera, and staff. The staff member connects the equipment, logs into VClass 15 minutes before the class starts, and coordinates with the IT department regarding problems with the internet networking system. Moreover, the staff member helps the instructor monitor the communication signal via the audio device or instant messages from the remote location. After each class session, the staff member must convert the recorded video lecture into a compatible format (e.g., flv or mp4) for uploading to VClass LMS. Distance students can review the previous class on demand; this ability represents part of the subject group treatment.
The distance classroom is a normal classroom that connects to the Internet. A tutor is vital to the proper operation of the distance classroom to coordinate among the instructor, distance students, staff of the regular classroom, and IT department on the remote side. The tutor also facilitates classroom operations and solves general problems (e.g., clarifying uncertainties regarding course content, coordinating assignment submissions, and proctoring tests).
For this study, class sessions were delivered from the sender to the remote location via the internet networking system. The remote location received and distributed the live class using the equipment in the distance classroom, including a computer with a web camera and audio device for communication with the sender side. The components of these learning environments were the inputs of the system process. The evaluation process of IBDL through VClass gauges the effectiveness of IBDL in live mode. Figure 2 illustrates how the model measures the effectiveness of IBDL. The evaluation's design was based on the definitions above for understandability, practicability, and applicability, especially for learning through IBDL technology, and applying to the evaluation of instructional activities. The process considers three main parts-IBDL technology in live mode, the instructional process of IBDL in live mode, and the evaluation of the effectiveness of IBDL-which are discussed below.

IBDL technology in live mode.
IBDL technology is evaluated regarding its efficiency or effectiveness (i.e., economy, speed, and quality). Regarding efficiency, scientific approaches can be used to evaluate the performance of IBDL technology. In contrast, the effectiveness of IBDL technology can be evaluated by measuring users' attitudes toward its features. Herein, this study adopted the existing resources and equipment of the pilot project, but they were not the primary focus of the evaluation. Nevertheless, open-ended survey questions were provided to measure users' attitudes concerning the existing resources of the pilot project. The teaching and learning process is crucial in IBDL activities and consists of input, process, output, and feedback. To evaluate the instructional process, the entities involved should be considered in identifying the input, process, output, and feedback for data collection based on the research goal and objectives. Figure 3 illustrates the entities that are involved in each part of the instructional process of IBDL in live mode. The input component involves several entities: 1) persons, including students, the instructor, and the tutor; 2) the equipment, comprising the computer that sends and receives the video/audio conference, internet connection equipment, projector, digital camera, and microphone; 3) the learning environment, including the lighting, air conditioning, and size of the distance classroom; and 4) the IBDL treatment plan for distance students, comprising the schedule of learning and communicating, grade/score level, assignment submission, and digital learning materials, including recorded video lectures and flexible office hours.
The process component refers to the methodologies or procedures (e.g., teaching and learning through VClass in live mode, selected CMC systems) that operate on input entities through the VClass system. The suggested treatment for the subject group is the adoption of CMC system technology that is free and widely available (in VClass), such as e-mail, chat rooms, and discussion boards. Recorded video lectures, another treatment for distance learning that can be used to review material throughout the course, are also suggested. learning experience is an optional element. Feedback on the system process (e.g., assignment, test, and midterm scores) is sent to stakeholders as formative results for improving the input and the overall process.
The survey questions on the system process collect the general information (e.g., age, gender, educational level, internet experience, and prior IBDL course) and test results of both student groups. Distance students are also asked to complete a self-assessment and share their IBDL experience; this information is used for comparing the learning behaviors of regular and distance students. A Likert scale is employed for this assessment. The learning behaviors gleaned from the self-assessment are sorted into two categories, before and after class attendance. In the "pre-attendance" survey, the distance students are asked about the frequency of attendance, use of the CMC,  The designed evaluation process for IBDL measures, assesses, and evaluates a distance learning system to acquire information that can help to develop, improve, and affirm its effectiveness. However, before the processes are applied, the goal and objectives of the experiment must be defined, and hypotheses must be proposed. Subsequently, inputs should be identified, and a plan should be formed to describe how the system will operate on the inputs to render the desired outputs. For instructional practice, the desired outputs should be identified through testing, and indicators should be prepared to assess the measurement results (e.g., pass or fail), which should then be distributed to stakeholders as formative evaluation feedback to improve teaching and learning.
Finally, the experimental hypotheses must be tested with statistics to arrive at the results of the evaluation. Data collection instruments (e.g., questionnaires, interviews, or focus groups) should be prepared accordingly. Questionnaires were used because of the ease of surveying distance students by this method. The quantitative and qualitative evaluations were based on the same measurement instrument. Validity methodology and reliability testing demonstrated the reliability of the questionnaire. Statistical analyses were then used to test the hypotheses. Once evaluation results are attained through statistical analysis, summative evaluation feedback must be given to stakeholders. The following section describes the utilization of the designed evaluation process.

Utilization of the Evaluation Process
To evaluate the effectiveness of IBDL through VClass in live mode, the second and third research objectives were examined by implementing the designed evaluation process and statistically analyzing its results. However, this study does not evaluate the efficiency of IBDL technology in live mode or the efficiency of the instructional process; only the effectiveness of IBDL is evaluated and reported. Hence, the instructional process of IBDL in live mode component focuses solely on its input, process, and output to assess statistically the effectiveness of IBDL.
The course described in this study represents the first application of IBDL at

Hypotheses.
The hypotheses were divided into the following two categories based on research objectives: 1) the effectiveness of IBDL, comparing regular and distance students, and 2) the distance students' experience with IBDL through VClass. Each category was subdivided into sub-hypotheses to examine the variables in each research objective.
The expectations of this study were to acquire new knowledge from the collected information. However, only the null hypotheses are presented.
In the first category, the average score for the effectiveness of learning is an essential variable in evaluating the normal distribution of sampling. This study examines variables such as the proportion of students passing an exam based on an assigned indicator, differences in learning behavior (e.g., a student's behavior before and after class attendance), the effectiveness of learning, the relationship of overall average scores for learning behaviors with the effectiveness of learning, and the independence of gender and learning behavior from the effectiveness of learning. Therefore, seven hypotheses are set with a 0.05 significance level: H1.10: The average scores of distance and regular students have a normal distribution.
H1.20: The proportion of distance and regular students who score 50 or higher is 85%.
H1.30: The effectiveness of learning is independent of gender for distance and regular students.

Data collection instrument.
Questionnaires were used to assess the performance and opinions of distance and regular students. Punpinij (2010) stated that a questionnaire response rate of 85% is adequate for the analysis and reporting of evaluation results and that the error rate should be no more than 5%. A response rate of 90% or higher is extremely good. In the present paper, online questionnaires were produced using Google Docs. The response rate was 100%; therefore, the responses for both groups can be used for analysis.

Category of measurement instrument.
For distance students, the measurement instrument is separated into three parts, corresponding to survey question indicators: 1) participant characteristics; 2) selfassessment of learning behaviors; and 3) previous experience with IBDL. For regular students, the measurement instrument is separated into two parts: 1) student characteristics; and 2) self-assessment of learning. Quantitative and qualitative data collection utilized both closed-and open-ended questions to elicit useful information and suggestions from participants.

Validity.
The validity of each questionnaire was controlled using the content validity method.
Questionnaire items were designed based on learning behavior and distance student experience with IBDL. Three experts reviewed the questionnaire items of distance students for content validity. Valid questionnaire items were selected by using the index of concordance or content validity index formula (Pongvichai, 2011) and, therefore, had values greater than +0.50. The items were modified based on the information obtained by the experts. The distance students answered the modified items, and the regular students answered all items except those pertaining to IBDL.

Reliability.
Reliability refers to the consistency of a set of measurements by a given instrument.
Cronbach's alpha (Cronbach, 1951) was used to measure the reliability of the internal consistency of the survey instrument because using the test-retest method was inconvenient (Pongvichai, 2011). Cronbach's alpha is a coefficient that is used for ordered rating scale survey instruments, such as a Likert scale that measures participants' attitudes. Wasserman and Bracken (2003) suggested that Cronbach's alpha of 0.60 or higher indicates acceptable coefficient of internal consistency for group assessment. George and Mallery (2003) proposed that Cronbach's alpha of 0.7-0.8 is acceptable; 0.8-0.9 is good; and ≥0.9 is excellent. In the current study, Cronbach's alpha ( ) was computed for each set of survey question indicators using SPSS.

Reliability of the measurement instrument of distance students.
Part 2, "the self-assessment of learning behaviors," is divided into two categories that represent the "before" and "after" of student attendance in the IBDL class. Quantitative data were collected from distance students using a Likert scale. Cronbach's alpha is Assessment of the Effectiveness of Internet-Based Distance Learning through the VClass e-Education Platform Pukkaew Vol 14 | No 4 Oct/13 268 0.8612 and 0.9487 for the "before the class" and "after the class" data, respectively, both higher than the values suggested by Wasserman and Bracken (2003) and George and Mallery (2003).

Reliability of the measurement instrument of regular students.
Part 2, "the self-assessment of learning behaviors," is divided into the following two categories: the "before" and "after" of student attendance in the regular class.
Cronbach's alpha for "before" (0.6870) is higher than the value suggested by Wasserman and Bracken (2003). Cronbach's alpha for "after" (0.9332) exceeds the values suggested by Wasserman and Bracken (2003) and George and Mallery (2003). A number of items could have been deleted without any substantial negative effect on Cronbach's alpha, but none were deleted.
The value of reliability might depend on the items in each part of the questionnaire, such that a greater number of items results in higher reliability. Moreover, participants may not answer properly, especially if there are many items. Hence, each part of the questionnaire should include only those items that are necessary to evaluate the research hypotheses. Therefore, validity and reliability play an important role in the questionnaire.

Quantitative Evaluation Results
The three groups of hypotheses were tested using the Pearson chi-square test, independent samples t-test, Levene's test for equality of variances, frequency, and percentage. The Pearson chi-square test was used to calculate the independent scale. Their data were separated to evaluate individually to keep the evaluation of the remaining distance student scores correct and normal. Therefore, the questionnaire responses in the final analysis comprised 29 distance and 23 regular students. Hence, the normal distribution result of H1.10 is accepted at 0.080 for distance student and at 0.200 for regular student. For the rejected hypothesis, one null hypothesis, H1.20, was accepted for distance students but rejected for regular students. Thus, the proportion of distance students who could pass to the next subject was 85% or greater, whereas the proportion of regular students was less than 85%. For hypothesis H1.40, differences in the overall averages of each group's individual learning behaviors (e.g., before and after attending the class, asking questions in class) were evaluated. The null hypothesis H1.40 was rejected for both student groups. The indicators of each learning behavior of both groups were tested to identify differences between the groups. For "before the class," the t-test for equality of means indicated that the frequency of class attendance (0.002), downloading learning material (0.000), and reviewing lessons (0.001) differed between distance and regular students. That is, regular students engaged more frequently in nearly every learning behavior. For "after the class," the t-test for equality of means showed that reviewing lessons (0.000) and asking the instructor questions on live chat (0.023) or the discussion board (0.022) also differed between the groups. Again, regular students engaged more frequently in nearly every learning behavior. 6. Distance students should adapt to distance learning.
7. The instructor should teach more slowly, talk to the students, and motivate them to learn.
8. Certain distance students would prefer to learn in a regular classroom.

Summary and Conclusion
An assessment of the effectiveness of IBDL requires an intensive plan to assess the effectiveness of IBDL pedagogy. Designing an evaluation process is essential and needs to be done before working on assessment. The proposed evaluation process performs the function of assessment guidelines. This study focuses on the instruction process of IBDL in the live mode component of the evaluation process. This component represents the input, process, and output elements. Each element indicates items concerning the instruction process. This allows the evaluator to see the item involved in the instruction process as a whole and to identify the outputs required for an evaluation. Once the requisite outputs are rendered, an assessment of effectiveness can be conducted. This study used mean of statistics to assess the effectiveness of IBDL. The starting point of the effectiveness part is measurements component. Input data on the measurements (e.g., test scores and completed questionnaires) were processed using mean of statistics.
Indicators (e.g., grade and assigned significance level) indicate the output of the effectiveness that is assessments component. To obtain the evaluation result, the researcher analyzes and interprets the assessment results. Then the evaluation results are sent to stakeholders to improve and approve the IBDL. An evaluation should be conducted not only at the end of the study (summative) but also during the ongoing process (formative) in order to improve the process and obtain better results at the end of the study. Regardless of whether the evaluation results are positive, they provide useful information for improving and developing the IBDL pedagogy. Hence, after the instructional process of IBDL in live mode produced its outputs, mean of statistics was used to assess the value of systematically acquired information. Two areas for improvement were found, which support the second and third objectives of this study. First, according to the second objective, distance students engaged in learning behavior only occasionally. Surprisingly, the effectiveness of learning was the same for both groups, possibly owing to the helpfulness of the tutor, who scheduled extra classes for the distance students before examination day; the distance students' request for assistance from instructors at the branch campus; and their fear of failing the exam, which motivated them to study. Second, CMC, such as live chat, e-mail, and discussion boards, was sparingly used by distance students; most of the use was by male distance students. Moreover, social media and MSN messenger were utilized as communication channels to reduce the problems of distance and stress in communication between instructors, students, and peers. As described in several studies, teachers are highly respected and typically considered as being knowledgeable and authoritative because traditional Thai culture places a very high value on learning.
Additionally, Thai students may not feel as comfortable asking questions and/or voicing their opinions as Western students; however, they listen attentively, take notes very carefully, and are well behaved (Nguyen, 2008;Pagram & Pagram, 2006). Moreover, [t]he high power distance characterizing Thai culture shapes the behavior of administrators, teachers, students and parents to show unusually high deference (greng jai) toward those of senior status in all social relationships which results in a pervasive, socially legitimated expectation that decisions should be made by those in positions of authority. (Hallinger & Kantamara, 2000) These factors may explain why the interaction quality between the instructors and students was not sufficient and must be improved. However, a social network, such as Facebook, was found to be utilized by the instructor and students. The two areas, ultimately, become invaluable feedback of the study for stakeholders to analyze and to improve IBDL in live mode for better results.
In conclusion, assessments of the effectiveness of VClass IBDL did not show significant differences between regular and distance students. The utilization of educational technologies over the internet networking system may help solve the teacher shortages and tight budget problems to maintain equal instructional quality at the main and branch campus. VClass is an example that can be used as an educational and communicational technology in live mode. However, other educational technologies must also be studied. Moreover, modern social media networking sites such as Facebook play an important role in the communication between instructor, tutor, and students when they are in different times and places. Furthermore, social media provide an opportunity for students to communicate under usage agreements for academic purposes, including reducing the gap of high respect that students have for instructors, particularly in Asia. Therefore, these two observations are useful to