Teaching Performance Evaluation Framework: An Analytic Hierarchy Process Approach

Teaching Performance Evaluation Process, Multi Criteria Decision Making, Analytic Hierarchical Process Teaching performance evaluation process in higher learning institutions (HLI) is of the most importance in building a higher education policy which purposes for teaching improvement and administrative decision making. Several issues have been raised such as 1) instrument characteristic used to detect values is not sufficient to handle subjective evaluation criteria’s and, 2) the most techniques used in handling survey data is statistical method. In conventional practice, ordinal Likert scales used for marking schema and statistical techniques are implemented for analyzing the data. In order to manage the issues, this research presents a multi-criteria decision making methodology, called analytic hierarchical process (AHP) to evaluate HLI instructors teaching performance. As a well known decision making approach, AHP can systematically cope with highly subjective problems in social system. An AHP implementation framework for teaching performance evaluation will use the possible internal factors; preparation, organization, delivery and effectiveness that are evaluated as teaching evaluation weight. The future work is also suggested, this evaluation system should be implemented in real time situations. It becomes the best platform to evaluate teaching performance and categorize instructors into appropriate bands.


Introduction
The implementation of performance evaluation is a necessary and beneficial process to provide annual feedback to staff members about job effectiveness and career guidance.The performance review is intended to be a fair and balanced assessment of an employee's performance.Currently, there are a lot of instruments that can be used in order to assist supervisors and department heads in conducting performance reviews process in higher learning institutions (HLI) world.It is important to implement teaching performance evaluation process especially in academia.Teaching and learning are a core business and continuous improvement process must be effectively done in HLI.Otherwise those HLI will not meet their standard.In addition, HLI instructors are a back bone of this education organization.Nowadays, the evaluation criteria of teaching performance of HLI instructors usually consists of several components and parameters, each involving a number of judgments often based on imprecise data where most used instruments are an optical mark reader (OMR) answer sheets to select ordinal Likert scale answers.
In addition, statistical methods are used for aggregating information from these assessment parameters.These methods are accepted especially in many HLI around the world although it is argued that this statistical technique does not necessarily offer the best way to evaluate human knowledge and skills.Hence, this study has the main objective of proposing a mechanism to evaluate the HLI instructors teaching performance evaluation.The proposed mechanism can often reflects the way people think and make judgments.
With the ability and features of an MCDM approach especially AHP, in decision making process, the quality of the teaching performance evaluation will result, more reliable and validate one with highest confidences.
The rest of the paper is organized as follows.In Section 2, some literature is reviewed about teaching performance evaluation process including its guideline, MCDM theory and AHP approach and also related research efforts to this research topic.Section 3 presents a suggested design model and methodology for teaching performance evaluation based on MCDM method.The final section, Section 4 presents some concluding remarks.

A. Teaching Performance Evaluation Process
Teaching performance evaluations are often designed to serve two purposes; to measure instructor competence and to foster professional development and growth.A teaching evaluation system should give instructors useful feedback on classroom needs, the opportunity to learn new teaching techniques and counsel from principals and other instructors on how to make changes in their classrooms.Evaluators are only mostly students.Specific procedures and standards must be first set to achieve these goals evaluations.(Boyd, 1989).
Although, there are several of practices are currently applied for conducting the teaching evaluation process and with those mechanisms, there are a lot of controversies especially on the resulted reliability and validity.Furthermore, evaluation of teaching in HLI involves collecting evidence, from various stakeholders (mainly students), for the purpose of improving the effectiveness of the teaching-learning implementation process.A successful evaluation generates valid and reliable outcomes that indicate directions and actions to improve.This is a main objective of an evaluation process.Fig. 1 shows an example of an OMR form for teaching performance evaluation process with Likert scale answers.This standard evaluation form is categorized as close-ended type of questionnaires.Beside that, there are certain HLI still used open-ended questions which are including highly subjective form of evaluation criteria and need a pedagogical expertise to analyze and interpret the conclusions results.
Based on stated mechanism above, OMR form notation (multiple-choice questions where the students indicate their answers by marking on the form which can then be automatically scanned and scored) are widely used because it is recognized as an efficient and cost effective data entry method for the large scale processing of category data.From its origins in the purposes of examinations, it is now widely used in other large-scale evaluations and exercises.Traditionally, questionnaires are evaluated using copies of paper questionnaires and/or OMR answer sheets.An optical mark scanner will then be used to read the marks made on the OMR forms.The process will produce an electronic data file containing responses to the questionnaires.The data file can then be analyzed using software programs such as Microsoft Excel, SPSS or SAS.With the availability of web technology, there is now the option of collecting evaluation data online.Recently, a number of IHL use Internet facilities for teaching performance evaluation purposes which is more convenient and effective to generate the final results.
The conventional method recently used on teaching evaluation process especially in HLI is a statistical approach which is applied mean or average score calculation.This calculation method is synonym with the usage of an OMR form which is most often used during an evaluation process.On the other hand, there are several methods that are also applied to Cite this article as: Azizul Azhar Ramli, Shahreen Kasim, Mohd.Farhan Md.Fuzzee, Hairulnizam Mahdin, Teaching Performance Evaluation Framework: An Analytic Hierarchy Process Approach/ Acta Inform.Malay.(1) (2017) 01-06 evaluation data such as statistical method includes studentized range or coefficient of variation, reliability analysis and so on.

HLI Instructors Teaching Performance Evaluation Guideline
Teaching performance evaluation or assessment provides instructors with starting-points for their teaching and with advices on how to gather feedback on their teaching practice and effectiveness as a part of a systematic program of teaching development.
All evaluation techniques contain implicit judgments about the characteristics that constitute quality of teaching.These judgments should be made explicit and indeed should become part of the evaluation process itself in a manner which recognizes instructors' privileges to be evaluated within the context of their own teaching philosophies and goals (Geis, 1977).The criteria for evaluating teaching performance vary among disciplines and within disciplines and should take into consideration the level of the course, the instructor's objective and style, and the teaching methodology employed.Nonetheless, the primary criterion must be to improve student learning.Concretely, indicators of quality of teaching in IHL can include the following.
i. Effective choice of materials, ii.Organization of subject matter and course, iii.Effective communication skills, iv.Knowledge for the subject matter and v. Availability and responsiveness to student concerns and opinions.

Fig.1 A section of a typical closedquestionnaire used for teaching evaluation process
In addition, the value added for teaching quality can be enhanced by providing the following criteria.i.To establish a positive learning environment, ii.To motivate student engagement, iii.To provides appropriate challenges, iv.To be responsive to students learning needs and v.To be fair in evaluating their learning.
Regarding the teaching assessment methods, there are two general categories include formative and summative purposes.The formative assessment of teaching performance can be carried out at many points e.g. during instructional period either in formal classroom or virtual environment.The main purpose of the assessment is for instructors to find out the certain part that can be improved such as teaching style and student's evaluation procedure.The assessment is initiated by the instructors and solicited from many sources such as self and students using a variety of instruments including survey or online forms.
On the other hand, the summative assessment is usually conducted at the end of a particular course or at the specific points in instructor's career.The purpose is to form a judgment about the effectiveness of a courses and/or instructors.The judgment may be used for tenure and promotion decision to reward success in the form of teaching awards or merit pay, or to enable departments to make informed decisions about changes to individual courses, the curriculum or teaching assignments.At most IHL, the summative evaluation is implemented and it includes the results of teaching evaluations regularly scheduled at the end of academic terms.Furthermore, the famous evaluation instrument for this purpose is OMR form and statistical calculation method.

B.Multiple Criteria Decision Analysis (MCDM)
MCDM techniques are often selected arbitrarily: sometimes an analyst is already familiar with a procedure, other times a technique is developed in an ad hoc way of the following; it also happens that a technique is chosen simply because its software is available.In fact, researchers highlight that, each situation or problem demands specific MCDM technique.The impact of the choice of a technique on actual decisions is also well known, as well as the consequences of poor decisions.
Basically, there are three generic types of solutions for MCDM problems.Table 1 shows the summary of particular types of MCDM method and this paper focus on implementation ranking solution of MCDM method.
An alternative approach that categorized as a family of MCDM problem is an AHP approach.Regarding this popular approach, there is a long period of debate, in fact on the effective value of the AHP approach.Perez (1995) proved that the AHP approach is based on a firm theoretical foundation.
The AHP has widespread use due to its flexibility, easy to use and Kirkwood (1997) mentioned that AHP can be even implemented in spreadsheet environment.However, ambiguity in relative importance, inconsistent judgments by decision makers and the use of 1 to 9 scales can be thought as the disadvantages of this approach.The ratio scale makes sense when dealing with something like distance or areas which are natural ratio scales but not when dealing with comfort, image or quality of life, for which no clear reference levels exists.Furthermore, for large problems, too many pair-wise comparisons must be performed (Malczewski, 1999).
Discuss below the general steps of AHP approach will be discussed below.
Steps are as shown below the following procedures.
In addition, simulation result will yield a set of consistency ratios for the corresponding matrices as stated in Step 6.The distribution of the consistency ratio should be studied and corresponding changes in the pairwise comparison matrices should be made (Saaty, 1988).

C. Related Research Efforts
These studies are carried out to improve the statistical results of a method used in a real teaching evaluation process.This traditional analysis or synonym with Likert scale was compared with selected MCDM approach, called AHP.In general Likert scale is a method of ascribing quantitative value to qualitative data, to make it amenable to statistical analysis.Used mainly in training course evaluations and market surveys, Likert scale usually have five potential choices such as strongly agree, agree, neutral, disagree and strongly disagree, but sometimes go up to ten or more.A numerical value is assigned to each potential choice and a mean figure for all the responses is computed at the end of the evaluation or survey.The final average score represents the overall level of accomplishment or attitude toward the subject matter.
The statistical technique most frequently used is descriptive techniques which include i. Summaries using a median or a mode.The mode is probably the most suitable for easy interpretation.
ii.Expression variability in terms of the range or inter quartile range.
iii.The distribution of observations in a dot plot or a bar chart or histogram.
Related to the performance evaluation process in various fields, we have found several papers that were implemented AHP approach as an evaluation method.Kadarsah (2007) developed a framework of measuring key performance indicators for decision support in higher education institution.
The framework is based on key success factors related to education institution sustainability which includes academic, research and supporting key performance indicators (KPI).AHP is used for weighted each KPI.As results, measurement values that reflect KPI scores are visualized in form of "wheel-shape".The developed framework contributes in measuring and explaining HLI success using multi dimensions of KPI.
In order to improve human resources management, Albayrak and Erensal (2004) applied AHP approach to solve the human performance improvement problem.This research presents a model which illustrates the relations and importance between human performance improvement and the style of management.The AHP was used for the purpose of structuring and clarifying the relations and important for both stated components.At the end of this research, they concluded the best management style in improving human performance is management by values.
Rafikul and Shuib (2005) also discussed about measurement of human performance system.An applied research type is discussed about the performance of individuals against organizational goals determines whether the organization meets its goals or otherwise.The criteria are measured based on quantity/quality of the work, planning, initiatives, teamwork, communication and several related external factors.Each of these criteria has been divided into three subcriteria and this developed framework was implemented to analyze real evaluation data.Overall ranking of employees has been obtained using the absolute measurement procedure of AHP.
Another research related to AHP application was innovated by Frair (1995).This research pursued an employment of AHP to assess the contributions of engineering student team members to the team effort.On the other hand, a description of AHP for teams within a production engineering class is also clearly illustrated.The minimal success of traditional student questionnaire to asses team performance is describe followed by a description of the what appears to be more meaningful results when AHP approach is used.

General MCDM Design Model and Methodology for Teaching Performance Evaluation
As stated in the comprehensive literature review in the previous section, it can conclude that MCDM is a powerful tool used widely for evaluating and ranking problems containing multiple, usually conflicting criteria (Pomerol and Barba Romero, 2000).
On the other hand, the MCDM approaches generally enable us to structure the problem clearly and systematically.With characteristic, decision makers have the possibility to easily examine the problem and scale it in accordance with their requirement.
During the literature survey, we have encountered few researches on the IHL instructors teaching performance evaluation process which is used any related MCDM method, although there are many studies related to the teaching performance evaluation procedure.Therefore, we propose a formal MCDM based approach, an AHP.The widest used tool for MCDM and also its features and ability to produce the most appropriate analyzed results (Liu et al. 2007) are the main factors of an AHP were chosen.On the other hand, this research focuses on implementation of AHP approach in HLI instructors teaching performance evaluation process which considering highly subjective of multiattribute criteria.
The teaching performance evaluation procedure of this particular study consists of three major steps as summarized in Fig. 3.They are includes Regarding the suggested evaluation procedure above, the identification criteria and sub-criteria are most of the important part and must be clearly selected.As shown in Fig. 4, there are four determined criteria that must be considered during the teaching evaluation process for IHL instructors; preparation, organization, delivery and effectiveness.
Based on Fig. 4 below, there are criteria levels which include four criteria.Beside that, there is also an alternative level which may consist of a group of HLI instructors which become as alternatives or candidates.This group may vary from certain departments, faculties/schools or as a whole of instructors groups in HLI.

A. Teaching Performance Evaluation through MCDM Method
Evaluating process for IHL instructors can be classified as MCDM problem.
There are two kinds of methods for solving a MCDM problem includes compensatory and non-compensatory ones (Hwang and Yoon, 1981).The non-compensatory method does not permit tradeoffs among attributes.The MCDM techniques in this category are simple, but may not be suitable for the teaching performance evaluation.In contrast, the compensatory method allows tradeoffs among attributes.A slight decline in one attribute is accepted if it is compensated by some enhancement in one or more other attributes.These are three subgroups in this category; scoring, compromising and concordance methods.
We have selected the most appropriate approach from compensatory method as a guide line for developing suggested evaluation model, that is, an AHP approach.The next subsection is spent to discuss an implementation of selected method.
In order to illustrate on how to evaluate the teaching performance of IHL instructors and rank each instructor to the appropriate band, we use a sample situation below as a case study.

B. The AHP Approach
Based on the discussed system design in the previous section, AHP approach is used as an MCDM method to design teaching performance evaluation framework.In this section, the suggested teaching performance evaluation system will be presented general steps of AHP approach as a guide line.
Considering the general procedure that was discussed in the literature review section, we used the hierarchy of criteria for teaching evaluation procedure which was previously designed in Section 3.
After decomposing the problem into a hierarchy notation, alternatives at the given hierarchy are compared in pairs to assess their relative preference with regard to each criterion at the higher level.A scale is needed to represent the varying degrees of preference.Saaty (1980) established a scale in Table 4, where 9 is upper limit and 1 is lower limit and a unit difference between successive scale values is used.Adapted from Table 2, pair-wise comparison scale for AHP preference is summarized as shown below.
A comparison is done among the elements on the same level of the goal hierarchy.In a comparing process, a value V from the scale is assigned to the comparison result of two criteria P and Q at first, and then the value of comparison of Q and P is reciprocal value of V.The value of the comparison of P and P is 1.Following these rules, a matrix of attributes is calculated through finding the eigenvector associated with the maximal eigenvalue of this matrix.A practically used algorithm is the following one.i. Normalize each column by dividing each cell by the column total.
ii. Sum each of the rows into a new column.
iii.Normalize the new column by dividing each value by the sum of the column.iv.Represent of the normalize column eigenvector, which contains the weight of each attribute.
The pair-wise comparison value of selected criteria is presented in Table 5 below.
Table 6 shows a normalized pair-wise comparison value matrix of criteria and Table 7 shows the weight ranking for each criterion.
The following step is to ranks the relative importance between each pair of alternatives in term of a criterion.Based on the calculated results, we notice that, a preparation criterion with 0.553 is the highest weight and followed by delivery with 0.312, organization with 0.084 and effectiveness as a less weigh which is 0.051 scores.The illustrated graph is shown in Fig. 6.
An index of consistency ratio (CR) can be used to measure consistency of an n-order square decision matrix.In AHP, the recommended threshold for CR is less than or equal to ten percent or 0.1, when the value of a CR is lower then this value; the decision matrix is accepted and can be applied to making decisions.Otherwise, the matrix is inconsistent and the pair-wise comparison weights must be revised.
The calculation of CR of a pair-wise comparison matrix is implemented by taking the following detail method innovated by Saaty (1980).Beside that, a priority vector for each row also must be calculated using the following formula: Based on the pair-wise comparison value on Table 5,  Similarly, we make a pair-wise comparison of the alternatives (instructors-INS) with respect to the four criteria in the higher level.The comparison and weights results are illustrated in Tables 7-10.

C. Results of the AHP Application
In order to determine the best performance of instructors, Saaty's (1980) AHP was applied to determine performance point values for this applied research as in Fig. 4. The priority rankings for each instructor were determined from a hierarchy based on four criteria and seven alternatives.
The selected criteria were compared on a pair-wise basis.This produced a ranked score for each instructor on each of the criteria and ranked the score for each criterion.Multiplying these scores provides a summary score for each the criteria are determined, each of the alternative was compared on the basis of the criteria.The eigenvalue of the square matrix, the preference vector, is then computed to determine the relative ranking of the four criteria in selecting the best evaluation point.
Based on the preference vectors given in Table 11 above, the four evaluation criteria are ranked, thus, preparation is the most important criterion for teaching performance evaluation whereas effectiveness is of least consider.
The final results which show the evaluation ranking are illustrated in the graph below which shows the best performance in HLI instructors teaching performance activities for this case is Instructor3 which 0.267 scores and Instructor7 with 0.052 which is the lowest point of teaching performance evaluation results.With an implementation of the AHP approach to the sample case, which is considering four evaluation criteria includes; preparation, organization, delivery and effectiveness, we find that the produced results are consistently calculated.It is proof by a consistency ratio value which is less then 0.1.On the other hand, the listed teaching performance evaluation criteria's of HLI instructors are highly in subjective evaluation term.Based on the constructed sample of case study, we find that the discussed ability and features of AHP are truly presented here.With two levels of hierarchical process; criteria's level and alternative level, the main objective of this research is achieved which preference ranking of alternatives (in this case, instructor's) is clearly concluded and highly meaningful.

Conclusion
This paper introduces a model, based on Saaty (1980)  logically and to assist in making sensible decisions.In addition, the proposed AHP framework that might assist in providing direction for evaluating instructors teaching performance point which almost becomes as needs in HLI.
Based on the eigenvalues computed by the proposed framework, we can conclude that preparation criterion is the most important in order to determines the IHL instructors teaching performance evaluation.The preparation includes several activities such as discussing teaching activities plan and clear assessment, and exhibited knowledge of the subject.However, effectiveness criterion was the minimum consideration for teaching performance evaluation process.In fact, the relative ranking is preparation, organization, delivery and effectiveness.
Based on the discussed case study which experimentally used AHP for evaluating HLI instructors was teaching performance, we found that Instructor3 shows the best instructor performance on the basis of the four discussed criteria's.On the other hand, the minimum performance point for this case study is Instructor7 which may to improve his/her related teaching activities.
The findings of this study point to the need for HLI instructors itself and also their management to understand the impacts that have to consider for improve the overall institution performance.Through the human performance analysis process, especially related to the teaching and learning environment, managers or persons who are in the top management will be able to make better decisions about the work organization and management strategies in order to enhance their education business.As strategy tool, the understanding of human performance will help to identify the right management technique for managing valuable human resources group that fit into intuitional vision.Effective AHP method supported teaching performance evaluation process can play a significant role in helping intuitions retain and to improve their best critical talent.
This study is done to consider only internal criteria related to the teaching performance evaluation process.Therefore, also external criteria's can be considered for enhancing this research such as size of student and class, instructor's gender, class time slot and so on.Furthermore, artificial intelligence techniques can be used in this kind of applications.For instance, expert system may be used to enhance the explanation facilities or neural network techniques to evaluate and forecast the potential of instructor's teaching performances relating to different cases.Hence, it is important to note that the aim of the proposed evaluation system is not to replace the current system of evaluating performance but it may be used to strengthen and improve the present system of evaluation by providing additional information for appraiser to make decision in teaching performance evaluation and human resources management in an organization especially HLI.
Cite this article as: Azizul Azhar Ramli, Shahreen Kasim, Mohd.Farhan Md.Fuzzee, Hairulnizam Mahdin, Teaching Performance Evaluation Framework: An Analytic Hierarchy Process Approach/ Acta Inform.Malay.(1) (2017) 01-06 i. Lam max=the maximum eigenvalue (Perron root) of the matrix = Lmax = ii.C.I. = Consistency Index = ( -n)/(n-1) iii.R.I. = Random Index.For each matrix of size n, Saaty's team generated random matrices and computed their mean C.I. value and called it the Random Index.iv.C.R. = Consistency Ratio = (C.I.)/(R.I).A value less than or equal to 0.1 is acceptable.Larger values require the decision maker to reduce the inconsistency by revising judgments.Cite this article as: Azizul Azhar Ramli, Shahreen Kasim, Mohd.Farhan Md.Fuzzee, Hairulnizam Mahdin, Teaching Performance Evaluation Framework: An Analytic Hierarchy Process Approach/ Acta Inform.Malay.(1) (2017) 01-06Before this following method can be done, the computation of the n-th root of the product of the values in each row must be calculated first.This technique called the geometric mean based on the following, sample data, [a1, a2, ……, an],

Fig. 7
Fig. 7 Teaching performance evaluation results and each weight