Insights into Engineering Education Learning Outcome ’ s Assessment with Rasch Model 1

Education is a process in which people learn. Knowledge, skill or ability is gained during the learning process to better equipped students on career opportunities. Learning process takes place in academic institutions in which programs are developed for the various fields. Accreditation to these programs requires the academic institutions to adopt qualification framework. The Engineering Accreditation Council of Malaysia (EAC) adopts the American Accreditation Board of Engineering and Technology 2000 (ABET) requirements which promote Outcome Based Education (OBE) learning process. OBE calls for the evaluation of the subjects Learning Outcomes (LO) as specified in the Program Specification. Evaluation method has been largely dependent on students’ performance carrying out tasks in tests, quizzes or submission of assignments. The evaluation on the students’ performance output gives an indication on the students’ achievement of the subject’s LO. However, measurements of the students’ achievement from the observed outcomes remain vague when raw scores are the only means of assessment. Raw scores may not provide true conception on difficulty of tests and or the true ability of students. This article described the measurement and analysis of the Engineering Mathematics LO’s achievement using Rasch Measurement Model. It gives a more meaningful analysis on the students’ achievement.


INTRODUCTION
Learning is a complex process where it involves cognitive, emotional and environmental influences and experiences in acquiring knowledge or skill (Illeris, 2004;Ormrod, 1995). Assessment should reflect these understandings by employing a diverse array of methods, including those that call for actual performance, using them over time so as to reveal change, growth, and increasing degrees of integration. Such an approach aims for a more complete and accurate picture of learning, and therefore firmer bases for improving our students' educational experience (Astin et al., 1991).
A good assessment recognizes the value of information for the process of improvement (Deming, 2000;Mok and Wright, 2004;Wright, 1997). Assessment approaches should produce evidence that relevant parties will find credible, suggestive, and applicable to decisions that need to be made. The point of assessment is not to gather data and return "results"; it is a process that starts with the questions that involve data gathering and subsequent analysis. It is of utmost importance on the onset that this fundamental of measurement must be clearly understood. Analysis must be based on valid data and duly interpreted to generate a reliable report with meaningful information for prudent decision making towards continuous improvement of teaching and learning. Rasch analysis provides more meaningful analysis enabling better inference be made. Rasch measurement model will be used in this study to assess students' achievement on basic mathematics for engineering. The findings will also lead to students' classification based on their ability-achievement hence better management in improving their achievement towards meeting the targeted learning outcomes (Azrilah et al., 2007).

METHOD OF ASSESSMENT
This paper explores the Learning Outcome (LO) on an engineering mathematical subject which exposes students to the basic concept of mathematical engineering. The subject is expected to be able to balance between practical applications of mathematical equations and in-depth understanding of real situation (Wright, 1989). The LO is the specification of what students should learn as a result, after a specified period  Overall  total  LC008  90  90  100  40  80  45  100  90  70  87  LP001  88  100  92  35  80  75  82  90  75  85  LC007  88  100  100  35  60  70  95  80  75  85  LC003  85  90  92  50  80  75  76  90  80  83  LC005  82  100  100  70  60  80  50  83  70  78  LC004  77  100  92  40  60  75  71  80  70  75  LM002  73  100  75  60  50  75  56  67  75  71  LC006  73  100  100  40  70  0  64  83  70  71 of time in the program. Apart from the technical knowledge, students are introduced on report writing and logical thinking. The data include several tests and quizzes taken during the first semester of their engineering program. Apart from the test results, the assessment also includes assessment on report writing, students' skills on Microsoft Excel and mathematical knowledge. Common practice of assessment would take percentages of each assessment components, including the final examination for the semester. The summation of the percentages will be the overall marks indicating students' performance. Students' with high marks are of good performance compared to those with low marks. However, it is always a difficult situation deciding who is better than another when any two or more of the students having similar total scores. Table 1 show example of students' performance result and decision have to be made on who has better ability when it involved similar total score.
Raw-score method of assessment provides unreliable facts about students' performances. The Overall Total is the count of all correct answers provided by the students when answering the tests and quizzes called raw scores. The scores do not take into consideration the students' ability on answering each task and do not consider the difficulty of each task (Azrilah et al., 2007;Wright, 1989). It would not be fair for those students with high marks but only able to answer easy tasks compared to those with high marks on difficult tasks. Surely the next question comes to mind is, how do we determine a task as easy or difficult? Rasch Model theorized (Rasch, 1960(Rasch, , 1961 that those tasks or what Rasch referred to as items, are of easy items when it can be answered by all students. The second theorem of Rasch Model is that, students with high ability should be able to answer all questions, while students with less ability may have some problems with difficult items (Azrilah, 2009;Bond and Fox, 2007). Rasch model will rearrange the pattern of responses according to Guttman pattern. In columnwise, data are sorted from most easy to difficult items and in row-wise, from most competent to less competent students. The probability of answering a yes or a no for each task (right/wrong responses) is modeled as a logistic function of the difference between the person and the item's parameter (Mok and Wright, 2004;Wright, 1989;Andrich, 2004;Zamalia et al., 2010). Refer to (1): The relationship between total scores with ability estimates which is determined by the probability of success of that person to achieve an item, reveals a nonlinear relationship. The shape of the Test Characteristics Curve (TCC) is ogival as depicted in Fig. 1.
Then Rasch model would scale the ability estimate according to the portion of correct responses. The smaller portion of the correct responses, the higher the difficulty of an item hence the higher the item's scale location. Both item and person are being scaled on the same or single scale. The higher a person's ability in relation to the difficulty of an item, the higher the probability of a correct response for that particular item. Person's location on the latent trait is equal to the difficulty of the item, when both the person and the item are located at the same location on the scale, there is a probability of 0.5 chance for a correct response according to Rasch model (Mok and Wright, 2004;Azrilah, 2009).
The purpose of applying Rasch model is to obtain measurement from categorical response data. In their raw form, categorical data cannot have mathematical     operations; addition, summation, and even mean. Therefore meaningful inference cannot be made upon categorical data. It only allows for data summarization using mode and median in reporting of the data (Leedy and Ormond, 2009). Rasch Model embodies certain criteria to be met in order to obtain measurement (Mok and Wright, 2004;Rasch, 1961;Bond and Fox, 2007;Andrich, 1988) and that the data should fit the model (Andrich, 2004) and not from the perspective of statistical modeling, model to fit the data.

ANALYSIS AND DISCUSSION
The Cronbach-α value is 0.33 reveals that the instrument might have some problems in assessing the students' performance. Rasch analysis allows further investigation, checking on the reliability of the person and item (Table 2). Item reliability is at 0.99, an excellent reliability value (Fisher, 2007). This indicates that the item size is sufficient to provide an excellent item spread with ranges of difficulty; there are items which are easy and some are difficult and some items in between.
It also reveals that there is one (1) item with minimum extreme score, pointing out that there is one (1) very easy item located at the end most of the measurement scale at -1.71 logit. The item is unable to demonstrate the variation of person ability; all those students who managed to answer the items correctly cannot be discriminated between a smart and a poor student.
The person reliability is rather low at 0.31, which indicate a poor spread of person; indicate a high level of inconsistencies in the students' responses (Table 3). This is further reveal by the Person Separation 0.66 which points out that the 243 person sample size has almost similar abilities but the instrument does not have enough good items to separate them into distinct group. More items is needed to be constructed to make the instrument better hence provide a laser cut to discriminate the students.
The poor value of person reliability however does not reveal that the instrument is not valid and analysis cannot be done. The Person Mean is located at -0.03 logit, indicating that the students have almost the same ability with the item difficulty (Item Mean = 0.00 logit) and the items in general are difficult to this group of students. Figure 2 depicts the location of items on the measurement scale, indicating their level of difficulties.
Observation made from the map in Fig. 2 showed that 4 of the items located above the mean item and the other 4 items located below the mean. Item KP which refers to chemical engineering appears to be the easiest item among those eight (8) items. KP is the minimum extreme item and is shown to be located at the lowest position on the logit measurement ruler in Fig. 2. ES-Electrical system, Excel-Microsoft Excel application, and Laporan-Report writing, are other items that are considered easy which is located below the item mean.
Maths -Mathematical concepts are the easiest among those items that are difficult and all four (4)   Quiz 5 and AS -Structure and civil which is the most difficult item among the four (4). Figure 3 is the person map where it depicts the location of each student according to their ability measure. In general, there is almost equal distribution of students above and below the item mean at 0.00 logit. The initial observation made to the summary statistics from Table 3, reveals that person separation is 0.66 near 1.00, indicating these group of students have almost the same ability thus a very strong homogeneity.
In common practice of assessment, those with high total marks are deemed have good command of the subject. The data used in this paper included the total marks of overall test percentages, and converted it into rating of 1<60, 2>60, 3>70, 4>80, 5>90. Logically, those with a '5' and '4' should be located on the upper location, above mean item = 0.00 logit, of the measurement scale on the map in Fig. 3. A detailed scrutiny of the Person-Item map is shown in Fig. 4 segregated between male and female student. Figure 4 showed female students' ability location. The achievement overall test rated from 1 to 5 is the last character (6 th character) coded in the person demographic. There are some students with rating '2' and even '1' located on the upper portion of the map; above the mean item. Refer to person in the square in Fig. 4. Even though by summing up all the score of the test and quizzes gives those students low total score, to the contrary, Rasch finds them a more able student.
In this article, the data are of the scores for each tests and quizzes. It will be a more meaningful analysis if further investigation can be conducted on the items sub-construct of each of the tests and quizzes. Each tests or quizzes would have several sub-items (questions) within them, and usually it is based on the learning outcome of the Table of Test Specification, normally defined according to Bloom's level of knowledge. Then further findings can be explained according to the respective Learning Outcomes (LO); either the hierarchical achievement of the Bloom is according to the taxonomy or the reverse and confirms whether the students met the specified LO or not.

CONCLUSION
The basic engineering mathematics subject introduces the students to the basic mathematics in engineering for all engineering students. Engineering students are exposed to the initial and necessary concepts for their preparation towards focused engineering field later in their engineering program.
Besides assessing the students' knowledge on the subject, the assessment includes report writing, their knowledge on the Microsoft excel application and also on their mathematical skill.
The assessment reveals that the students find it easy in the area of chemical engineering and electrical engineering. The students also find it easy to handle report writing and have no problem with Microsoft  excel application. However, the students have some problem in basic mathematical knowledge, material science, Quiz 5, and find the final examination to be difficult. The most difficult item is on the area of civil and structural engineering, where all students are unable to solve the tasks given under this topic. There is definitely a need to identify the reason on why such preference is shown by this group of students.