Mastery of Course Learning Outcomes in ODL A Case Study of the Pearson eCollege Learning Outcome Manager

The constant emphasis on maintaining a high level of quality in the Open Distance Learning (ODL) self di - rected courses delivered at Wawasan Open University (WOU) demands the accurate assessment of course learning outcomes (CLO). The summative and formative assessment components of each course module are, in theory, designed to effectively measure the mastery of a set of CLO by an adult learner. However, in reality, there is no real scrutiny of the assessments at WOU to ascertain whether a learner has achieved the expected mastery of a CLO. The general assumption is that the assessment marks are an indication of the mastery of a particular set of CLO measured in a specific assessment component. With the aim of identifying whether assessment marks are indeed reflective of the mastery of a set of CLO, a study was conducted to implement the “Person eCollege Learning Outcome Manager” software application. This paper discusses the methodology and findings of this study and provides several suggestions for the smooth implementation of the Learning Outcome Manager.


Introduction
Being an Open Distance Learning (ODL) institution, Wawasan Open University (WOU) adopts a blended method for course delivery to its adult learners. At the undergraduate level, the learners undertake courses of five credit hours in self directed mode using purpose designed course materials supplemented by brief face to face tutorial sessions and an online learning management system. Due to the lifelong learning aspect of the business where the focus on learning outcomes is considered crucial (Hussey & Smith, 2003), WOU invests heavily in the development and continuous quality improvement of the self directed course materials which undergo a comprehensive instructional design process (Abeywardena, 2013). This, in turn, makes the materials suitable for learner centric Outcome Based Education (OBE) (McNeir, 1993). Each course is specifically designed to promote mastery of a particular set of course learning outcomes (CLO) by a learner. The CLO are further subdivided into unit learning outcomes (ULO), which allow learners to self assess their mastery using formative assessment components. The combination of CLO and ULO contribute holistically to the achievement of the program learning outcomes. Despite the insights provided by the formative assessment component into the learners' mastery of CLO (Black & Wiliam, 1998) the lack of class room based activities in ODL makes the summative assessment component critical in assessing the mastery of CLO.
The summative assessment at WOU is twofold: (i) continuous assessment in the form of Tutor Marked Assignments (TMA); and (ii) final proctored exam. However, this summative assessment component is not scrutinised in detail to assess the mastery of the CLO by the learners. In this regard, the general assumption being made is that the assessment marks are reflective of the Open Praxis, vol. 5 issue 3, July-September 2013, pp. 239-248 learners' mastery of the CLO; which implies that the higher the marks the better the mastery of the CLO. Even though there have been studies conducted such as the one by Gijbels et al. (2005) which identifies a correlation between the mastery of CLO and the Grade Point Average (GPA), no such study has been conducted to identify the validity of this assumption in the context of the ODL environment at WOU. For the purpose of gaining an understanding of the correlation between the marks and the mastery of the CLO by WOU students, a study was conducted using the Pearson eCollege Learning Outcome Manager-which will be referred to as the Learning Outcome Manager for the remainder of this paper.
The learning outcome manager is a web based software platform that allows academics to monitor the mastery of CLO by their students in both formative and summative assessments. The experiment was conducted as a retrospective study by superimposing the mastery of CLO on the assessment marks. The major contribution of this paper is the insight it provides into the correlation between summative assessment marks and the mastery of CLO by adult ODL learners. The rest of the paper is organised into four parts, which consist of methodology, results, discussion and conclusion.

Methodology
This particular study was conducted over a six-month period in the beginning of 2011. An independent academy was created for WOU on the Learning Outcome Manager that allowed the detailed tracking of students' progress for a particular ODL course. TCC123/05 Visual Programming, a five credit hour lower level technical course, equivalent to a first year course of a conventional university, specialising in Visual Basic.Net (VB.Net) was used as the test case for the study. This course was purposely chosen as the test case taking into account the total of 71 learners who had completed the course in the July semester of 2010. These learners were geographically distributed among four learning centers located across Malaysia.
In theory, upon successful completion of the course the learners would have mastered five CLO as shown in Table 1. The summative assessment for this particular course comprised of (i) three TMA, which are a combination of theory and laboratory exercises contributing 50% to the final score; and (ii) one proctored examination contributing the remaining 50% to the final course score. The learners were required to obtain a minimum mark of 40% for both components to pass the course. In an attempt to ensure that the use of the Learning Outcome Manager would not interfere with the assessment and feedback provided to the students, the study was conducted retrospectively. As such, the Learning Outcome Manager was setup to superimpose the mastery of CLO on top of the assessment marks that had already been awarded through an independent exercise conducted prior to the study. Four classes were created on the Learning Outcome Manager representing each of the four learning centers. Additionally, student accounts were created for each student enrolled in a particular class. In a real-world scenario, the students would have been able to track their progress through the Learning Outcome Manager throughout the duration of their study. However, this feature was disabled for the purposes of the study as it was conducted retrospectively. Only the Student ID was used to identify the student in the system to ensure anonymity and unbiased evaluation.
For the effective measurement of the mastery of CLO (i) each CLO was granulated into smaller ULO; (ii) a rubric comprising of a three point Likert scale was identified which measured the mastery of ULO in terms of needs improvement, meets the requirement or exceeds the requirement; and (iii) learning statements were identified which measured the mastery of a ULO against the rubric. Following this exercise, the rubric (Appendix A) was setup on the Learning Outcome Manager. Subsequently, the corresponding CLO in the rubric were mapped against the summative assessments as shown in Table 2. An assumption was made from the outset that the average of the mastery of each ULO would determine the overall mastery of the corresponding CLO. i.e. an average mastery of 2/3 would be deemed as meeting the requirement with respect to the mastery of the CLO.
Upon setup of the system, the academic in charge of the course did the data entry of the assessment marks. Following it, the TMA and answer scripts of the exam were re-evaluated to determine the mastery of the ULO by the learners. This re-evaluation was conducted against the rubric already setup on the Learning Outcome Manager. The numerical marks plus the letter grades were recorded in the Learning Outcome Manager for further analysis. Once the re-evaluation was complete, detailed reports were generated using the system in MS Excel format. The Pearson product-moment correlation coefficient (r) was used to determine the correlation between the assessment marks and the mastery of CLO. The statistical analysis was conducted using the MS Excel software application.

Results
The mastery of the CLO was mapped against the assessment marks for the exam and the three TMA in a scatter plot as shown in Figure 1. The Pearson product-moment correlation coefficient (r) between mastery of CLO and assessment marks was calculated for each assessment as shown in Table 3.

Discussion
According to the results of the statistical test (Table 3), there is a strong positive correlation (r ≈ 1) between the mastery of the CLO and the assessment marks. This provides some indication that the marks of an assessment can be used to ascertain the mastery of the CLO. However, the correlation does not indicate which range of marks would be representative of meeting the requirement with respect to the mastery of CLO. To identify the possible correspondence between the assessment marks and the mastery of the CLO in relation to the rubric, the summative assessment marks were plotted against the mastery of the CLO in a scatter plot (Figure 1). By analysing the scatter plot, it was identified that the marks ranging from 60% to 80% are approximately indicative of the  learner's mastery of the CLO. It is further approximated from Figure 1 that marks less than 40% are indicative of the need for improvement. This bears a resemblance to the passing mark of the summative assessments at WOU, which is set at 40%. Assessment marks beyond 80% are approximately indicative of the learner exceeding the requirement as far as the mastery of CLO is concerned. Referring to Table 4, it can be seen that the overall grade obtained by WOU learners in their summative assessment has an approximate correspondence to the level of mastery of CLO. Despite the fact that the statistical analysis is indicative of a correlation between the summative assessment marks and the mastery of CLO, it must be noted that the correspondence between the rubric and the assessment marks is only an approximation. This is mainly due to the breadth of the three point Likert scale used in the rubric to measure the mastery of the CLO. In the context of the Visual Programming course, given the technical nature of its content, mastery of certain ULO can only be measured as either meeting or not meeting the requirement. This skews the measurement to 1/2 instead of 2/3. In contrast, the achievement of certain ULO requires a scale larger than three points to identify the mastery in between meeting the requirement and exceeding the requirement. Therefore, it can be concluded that a more granular CLO assessment rubric is needed for effectively identifying a potential correspondence between the assessment marks and the mastery of CLO. This argument is further strengthened by the conclusions of the parallel study that used the three credit hour lower level foundation course Advanced Writing Skills for University Studies, which deals with content in the social science domain. In her paper detailing this study, Emmanuel (2011, p. 6) states ". . .when the three point Likert scale had to be created, it led the course coordinator (academic) to examine in detail the criteria for each category to a great extent. . . to be able to show clear lines of delineation among the categories. This involves the ability to write good rubrics." Emmanuel (2011, p. 6) further states: ". . .for course coordinators who come from industrial backgrounds, workshops need to be organised in this area otherwise the LOM (Learning Outcome Manager) efforts will be hampered." As such, it can be noted that the accurate creation of the rubric for measuring the mastery of the CLO is a critical factor in the success of the whole Learning Outcome Manager implementation process. However, it must also be noted that while the increased granularity of the rubric contributes to the increased accuracy of the measurement, it also contributes to the fatigue of the assessor. Thus, the key is to find the correct balance between the two.

Conclusion
A study was initiated at Wawasan Open University (WOU) in collaboration with Pearson eCollege to identify a possible correlation between assessment marks and the mastery of course learning outcomes (CLO) in a self directed adult Open Distance Learning (ODL) environment. During the course of the study, it was realised that the implementation of the Learning Outcome Manager software application needs to be done holistically from inception to conclusion of an ODL course in order to maximise the return on investment. As such, the implementation plan needs to take into consideration the design, development, delivery, assessment, feedback and revision stages of the course development lifecycle. The study further exposed the strengths and weaknesses of the present assessment strategies at WOU especially with respect to measuring the mastery of CLO. From the results, it can be seen that the summative assessments implicitly take this aspect into consideration. However, the need for an explicit attempt at measuring the mastery of the CLO needs to be seriously considered in the spirit of Outcome Based Education (OBE). Furthermore, the effectiveness of using a detailed rubric for measuring the mastery of CLO was also brought to the attention of the stakeholders. Another point that was brought to view was that certain CLO were being assessed multiple times in the summative assessment. In this regard, the study urged the academics to seriously re-evaluate their assessments taking into consideration the concepts in OBE.
Overall, the Learning Outcome Manager was found to be a more comprehensive and effective method of assessing the mastery of CLO. The ability of the system to track the mastery of the smaller ULO in addition to the larger CLO provides all stakeholders a real-time view of the progress made by the students and the class as a whole. This, in turn, acts as a continuous quality improvement mechanism whereby the key stakeholders are provided with the ability to tailor learning activities and assessments to ensure the mastery of most CLO by the learners. Furthermore, the Learning Outcome Manager provides a usable yardstick for the accurate assessment of OBE in ODL environments which doubles as a valuable enabler especially in accreditation exercises.

Rubric for measuring the mastery of Unit Learning Outcomes (ULO) for a particular Course Learning Outcome (CLO).
CLO1: Discuss the principles of object oriented programming

Meets the Requirement Needs Improvement
ULO1.1 Identify the differences between procedural and object-oriented programming.
Being able to differentiate between procedural and object-oriented programming.
Not being able to differentiate between procedural and object-oriented programming.
Being able to explain the role of events.
Not being able to explain the role of events.

ULO1.3
Recall the basic concepts of object-oriented technology.
Being able to explain information hiding, inheritance and polymorphism.
Being able to explain information hiding and inheritance.
Not being able to explain information hiding and inheritance.
Being able to create a project in the IDE.
Not being able to create a project in the IDE.

ULO1.5
Develop a simple GUI event-driven program using Visual Basic.
Being able to create a button click event.
Not being able to create a button click event.

ULO1.6
Apply the Visual Basic program to create a simple application.
Being able to create a simple program to display a word when a button click event happens.
Not being able to create a simple program to display a word when a button click event happens.
CLO2: Write the basic elements of subroutines and functions in Visual Basic programs.

Needs Improvement
ULO2.1 Write a simple program using Visual Basic.
Being able to create a console application to output a string.
Not being able to create a console application to output a string.
ULO2.2 Explain and apply primitive data types including integers, strings, and dates.
Being able to use integer, string and date in a console application.
Not being able to use integer, string and date in a console application. Being able to get data from a database using the SELECT query.
Not being able to get data from a database using the SELECT query.
ULO3.5 Generate database connections and create LINQ to SQL objects.
Being able to create a database connection, create a LINQ to SQL object and use the LINQ to SQL object in an application.
Being able to create a database connection and create a LINQ to SQL object.
Not being able to create a database connection and create a LINQ to SQL object.

Needs Improvement
ULO4.1 Develop a Web application using ASP.NET.
Being able to start an ASP.Net project and create multiple pages.
Being able to start an ASP.Net project and create a default page.
Being able to start and ASP.Net project but not being able to create a default page.
Being able to insert a from into a page with form controls for gathering information and adding the code behind to submit the information to the server.
Being able to insert a from into a page with form controls for gathering information.
Not being able to insert a from into a page with form controls for gathering information.
Being able to bind the data validation controls to the form and output informative information to the user.
Being able to bind the data validation controls to the form.
Not being able to bind the data validation controls to the form.
ULO4.4 Use cookies to obtain information about users.
Being able to create, deploy, update, retrieve and delete cookies and use the cookie information in the application.
Being able to create, deploy, update, retrieve and delete cookies.
Not being able to create, deploy, update, retrieve and delete cookies.
ULO4.5 Connect to a database in ASP.NET.
Being able to add a LINQ to SQL object into an ASP.Net application and use the data from the database in the application.
Being able to add a LINQ to SQL object into an ASP.Net application.
Not being able to add a LINQ to SQL object into an ASP.Net application.