A decision support framework for performance evaluation of Indian technical institutions

Article history: Received May 12, 2013 Accepted June 22, 2013 Available online June 3


Introduction
Education and health are two primary ingredients for the development and growth of a Nation.To make India a knowledge based society the most important area that needs to be addressed first is education.As envisaged by Indian erudite scholars around 1000 years ago, education is a never ending journey from less light to more light (Tamaso Ma Jyotirgamaya).It is the manifestation of perfections already in man.
The primary objective of education is to enable us to know things we did not know earlier, so as to improve the quality of life.In 1993 United Nations development programs (UNDP) under the leadership of Prof Amartya Sen and Prof M. Haque devised a composite index called human development index (HDI) to measure the quality of life.The components of HDI are life expectancy, literacy rate and gross domestic product (GDP) of the country.According to 1997 HDI value India ranks 139 th out of 174 countries.Education can improve the HDI value.We need education to be educated and thereby to be enlightened.Indian patriotic monk Swami Vivekananda said, 'we want that education by which character is formed, strength of mind is increased, the intellect is expanded and by which one can stand on one's own feet'.To ensure education for all (EFA), Government of India implements a number of programs for the achievement of the EFA goals, including, inter alia, sarva shiksha abhiyan (SSA), midday meal scheme (MDM) and national literacy mission (NLM).Indian technical education system is one of the largest educational systems in the world.Engineering education in India started during the British colonial rule and it focused mainly on civil engineering.Gradually few engineering colleges namely the Engineering College at Roorkee, Poona Civil Engineering College at Pune, Bengal Engineering College at Shibpur etc., came up in the mid 1850s.Presently, the technical education system in the country can be broadly classified into three categories like, Central Government funded institutions, State Government/State funded institutions and Selffinanced institutions.In In 2009-10 there were 65 centrally funded institutions in the country.The 65 centrally funded institutions of technical & science education are shown in Table 1.These institutions function following the guidelines stipulated by All India Council for Technical Education (AICTE) and the Council of architecture.As of now 2300 engineering colleges are running in India and 600,000 students are passing out in each year (Biswas et al., 2010).In the list of the best technical institutes in India, the first name comes into mind is a group of institutions called Indian Institute of Technology (IITs).The purpose of this paper is to assess the relative performance of these IITs based on multiple criteria.In this paper, we have considered seven Indian Institutes of Technology (IITs) located at Kharagpur, Bombay, Madras, Kanpur, Delhi, Guwahati and Roorkee for study and these are coded as A, B, C, D, E, F and G respectively.These institutions are declared as "institutions of national importance".The main objective of IITs is to impart world class education in engineering and technology, to conduct research in the relevant field, and to further advancement of learning and dissemination of knowledge.
As the uniform quality output has become the prime concern today, therefore, performance evaluation and ranking of these technical institutions have become a research issue.All the stakeholders want to get optimum benefits at shortest period of time and at an economical cost to improve the quality of life.Therefore, this is high time to do performance evaluation of the technical institutions.During the past two decades considerable volume of research has been conducted worldwide regarding performance evaluation of universities, libraries, research institutes etc.Most of the studies have focused on UK or Australia.The use of league table (Herbert and Thomas, 1998;Yorke, 1997Yorke, , 1998) ) is found to rank academic institutions in UK.League tables are generally used to compare academic performance of various institutions by considering a set of well-defined criteria such as-student satisfaction, research assessment/quality, entry standards, student-staff ratio, academic services spend, facilities spend, good honours, graduate prospects and completion rate.A statistical technique known as Z-transformation is applied to each criterion to create a score for that criterion.Weighted Zscores on each criterion help to determine the final rank of the institution.
Apart from the concept of league table, researches on universities in UK include those by Athanassopoulos and Shale (1997), Glass et al. (1995aGlass et al. ( , 1995b)), Johnes (1996Johnes ( , 2006)), Casu and Thanassoulis (2006), Flegg et al. (2004).In UK Portela and Thanassoulis (2001) have investigated the efficiency of schools also.Plenty of studies have been reported on efficiency analysis of Australian universities.Among the authors that have written about it we can mention Avkiran (2001), Abbott and Doucouliagos (2003), Worthington and Lee (2008) etc. Kao and Hung (2008) have concentrated on performance evaluation of academic departments in Tiwan.Fandel (2007) makes a study on German Universities.Korhonen et al. (2001) analyse 18 research units at Helsinki school of Finland.Elsewhere Hashimoto and Cohn (1997)  In this paper, we suggest an integrated fuzzy multi-criteria decision making model consisting of fuzzy analytic hierarchy process (FAHP), data envelopment analysis (DEA) and technique for order preference by similarity to ideal solution (TOPSIS) to assess the relative performance of the IITs.The contribution of the present work is that, this model is robust; it is easy to deal with; complex mathematics is not required and the evaluation criteria encompass stakeholders' preference.
Computation of the degree of relative importance for evaluation criteria is made through fuzzy AHP.
For each alternative, DEA method helps to compute local performance score which is expressed in linguistic terms for each criterion.Finally TOPSIS method is used to aggregate performance scores under different criteria into an overall performance score for each institution and ranking the institutions according to their overall performance score which has been called as relative closeness to ideal solution.
The paper is organized as follows: Section 2 describes application procedure of proposed model used in the paper.Section 3 gives information about data and computation.In the last section conclusion is given.

Selection of evaluation criteria
The performance of technical institutions in absolute sense is very difficult to measure.There are lot of factors/criteria/attributes/objectives those affect the performance of the institutions and the measurement result is very much sensitive to the selection of the criteria.In the literature mentioned above, the criteria are categorized either inputs or outputs to conform to DEA algorithms.Thus, the selection of criteria plays a crucial role in performance evaluation.According to Barros (2005) ''. . . the criterion of available data is frequently used, since it encompasses the other two criteria applied to the selection of the determinants.The first of the two is the literature survey, which is a way to ensure validity of research and therefore a criterion to take into account.The remaining criterion for measurement selection is the professional opinion of senior management''.For our research we prepared a questionnaire containing fifty questions related to criteria selection and the same was circulated among the experts.Aggregating their views by doing Pareto analysis following criteria are short listed for the study.Instead of classifying the criteria into output and input, in our study we call these either beneficial or non beneficial criteria respectively.In most of the literatures faculty strength and student intake are considered as inputs whereas we consider them beneficial criteria i.e., higher the better according to Taguchi's concept.The reason is that India is the second most populous country in the world.This country is blessed with the availability of human resources in the working age group.The challenge before the country today is that this available manpower has to be made employable by imparting necessary training and skill through technical education to cater to the need of expanding economy.It is understood from the published news that the IITs are suffering from acute shortage of qualified faculties.Desired teacher student ratio, as prescribed by AICTE (an autonomous body under ministry of human resource development) should be 1:15.Many institutions are running with a lower value of this ratio.Therefore these two criteria are considered as beneficial criteria from socioeconomic point of view.In this paper we consider tuition fee per semester (TF) in rupees as the only non beneficial criterion.Fig. 1 shows the hierarchy of the criteria for the assessment of seven IITs using proposed model.

The proposed model
Multiple criteria decision making is not an esoteric subject.Irrespective of field, it can be employed to select and prioritize the alternatives in the set.Lot of multiple criteria analysis tools like-AHP (Saaty, 1980), TOPSIS (Hwang & Yoon, 1981), DEA (Charnes et al., 1978), ELECTRE (Roy, 1968), MOORA (Brauers & Zavadskas, 2006) etc. are available for performance evaluation and ranking of In this paper we use fuzzy AHP to determine the weights of the evaluation criteria, DEA method is used to compute local performance score which is expressed in linguistic terms for each criterion.Finally TOPSIS method is used to aggregate performance scores under different criteria into an overall performance score for each institution and ranking the institutions according to their overall performance score.The output of FAHP and DEA are input to the TOPSIS method.Fig. 2 presents a schematic view of the proposed model which can be divided into three phases.The phase-I deals with team working.The weights of the criteria are determined in phase-II and finally the alternatives are ranked in phase-III.The proposed method starts with the definition of decision matrix which has in general four components, namely: (i) alternatives (ii) criteria or attributes (iii) subjective weights or relative importance of each criteria and (iv) measure of performance of alternatives with respect to the criteria.The decision matrix can be expressed as follows: (1) where A i represents the alternatives, i = 1,2, . . .,m; C j represents j th criterion or attribute, j = 1, 2,. . ., n.The subjective weight of the j th attribute is denoted by W j ; and x ij indicates the performance of each alternative A i with respect to each criterion C j .

Fuzzy AHP
The analytic hierarchy process is a multi-criteria decision making (MCDM) tool to render subjective judgment on one criteria over another.This tool which was first introduced by Saaty (1980), works on eigenvalue approach to the pairwise comparison.In this method the relative preference of the qualitative factors are expressed in terms of Saaty's 9-point scale.The AHP method is based on three principles: first, structure of the model; second, comparative judgment of the alternatives and the criteria; third, synthesis of the priorities.As observed in the review paper on AHP (Ho, 2008), we find that due to its simplicity and wide applicability, it has been successfully combined with other tools, like-mathematical programming, quality function deployment (QFD), meta-heuristics, SWOT analysis and data envelopment analysis (DEA).
Though AHP is very much able to deal with the expert's knowledge and experiences by perception or preference, it still cannot reflect the human thought totally with the crisp numbers.Therefore, the fuzzy-AHP that integrates the fuzzy theory into AHP environment is applied here to solve the performance evaluation problem of Indian technical institutions.Fuzzy set which is an extension of crisp set deals with ambiguous or imprecise data.It was first introduced by Zadeh (1965).A fuzzy set is characterized by a membership function which assigns to each object a grade of membership ranging between zero and one.Triangular and trapezoidal fuzzy numbers are normally used to capture the vagueness of the parameters related to selection of the alternatives.The triangular fuzzy number (TFN) is very simple to use and calculate.It helps in decision making problems where the information available is subjective and imprecise (Zimmerman, 1996).In practical applications, the triangular form of the membership function is used most often for representing fuzzy numbers (Kahraman et al., 2004;Xu & Chen, 2007) that can be defined by a triplet M=(l,m,u), m is the median value of fuzzy number M. l and u are the left and right side of fuzzy number M respectively as shown in Fig. 3. Some basic important definitions of fuzzy sets can be found in the followings (Chen et al., 2006;Lee et al., 2009;Önüt & Soner, 2008;Wang & Chang, 2007;Zimmerman,1996).
Let us consider two TFN M 1 and M 2 , M 1 = (l 1 , m 1 , u 1 ) and M 2 = (l 2 , m 2 ,u 2 ).Their operations laws are as follows, Fuzzy scale for pairwise comparisons of one criterion over another is shown in Table 2.This scale is used to develop pairwise comparison matrix.In this paper the extent fuzzy AHP is utilized, which was originally introduced by Chang (1996).Let A = (a ij ) mxn be a fuzzy pairwise comparison judgment matrix.Let M ij = (l ij , m ij , u ij ) be a triangular fuzzy number (TFN).The steps of fuzzy AHP are as follows: Step 1: Form the pairwise comparisons of attributes by using the fuzzy numbers, which is composed of low, median and upper value, in the same level of hierarchy structure.
Step 2: The value of fuzzy synthetic extent with respect to the i th object is defined as subject to By the above set of formula the TFN value of S i is calculated.
Step 3: We compare the values of S i respectively and calculate the degree of possibility of S j = (l j ,m j , u j ) ≥ S i = (l i ,m i , u i ).That can be equivalently expressed as follows: where d is the ordinate of the highest intersection point D between µS i and µS i (see Fig. 4).To compare S i and S j both the values of V(S j ≥ S i ) and V(S i ≥ S j ) are required.
Step 5: We normalize the weight vectors.That is as follows,

Data Envelopment Analysis (DEA)
Data Envelopment Analysis (DEA) has been recognized as an analytical technique for measuring the relative efficiency of a group of similar organization.This technique was first put forward by Charnes et al. (1978).It deals with the efficiency problem of complex organizations that utilize multi inputs to produce multi output.The set of similar organizations like banks, post offices, schools, colleges, police stations, agricultural firms, departments etc., assessed by this method are known as Decision Making Units (DMU).For our research, to characterize the relative importance of each alternative with respect to each criterion, we define for each criterion a set of evaluation grades: G j ={H j1 ,……….H jkj } (j=1,……,n) where H j1 ,……….H jkj represent the importance from the most to the least important and k j is the number of evaluation grades for criterion j.
This definition allows for different criteria to be evaluated using different numbers of assessment grades and provides flexibility for setting up linguistic grades.We then ask the experts from different domains to assess the decision alternatives and classify them into their corresponding assessment grades in terms of their relative importance with respect to the criterion under consideration.Without loss of generality, assume that criterion j will be assessed by N j experts (j = 1,. . .,n).Then, assessment results can be characterized by the following distribution assessment vectors (Yang 2001).
Now let us assume s(H jk ) be the scoring of grade H jk (k = 1, . . .k j ).Then, the local weight of each alternative with respect to every criterion can be defined as follows, , i=1, . . . ,m; j=1,. . . ,n. (15) To determine the local weight of each decision alternative with respect to every criterion, we choose the following DEA model (Wang et al., 2008;Amiri et al., 2010).
where s(H jkj ) are decision variables and s(H j1 )≥2s(H j2 )≥……≥k j s(H jkj )≥0, is the strong ordering condition imposed on evaluation grades, which is similar to strong ordering condition on different ranking places in voting system proposed by Noguchi et al. (2002) and adopted by Liu and Hai (2005) in their voting AHP.By solving model ( 16) for each criterion s * (H jk ) (j=1, . . .,n; k=1, . . .,k j ), we get optimal scorings of the evaluation grades determined by the model.Then the local weights which can be considered as local performance score, θ ij of each alternative with respect to the n decision criteria can all be generated by Eq. ( 15).

Technique for Order Performance by Similarity to Idea Solution (TOPSIS)
TOPSIS method was first introduced by Hwang and Yoon (1981).This technique suggests that the best alternative would be the one that is nearest to the positive-ideal solution and farthest from the negative ideal solution (Ertuğrul and Karakaşoğlu, 2007).The positive-ideal solution is a solution that maximizes the benefit criteria and minimizes the cost criteria, whereas the negative ideal solution maximizes the cost criteria and minimizes the benefit criteria (Wang and Elhag, 2006).In brief, the positive-ideal solution is composed of all best values attainable from the criteria, whereas the negative ideal solution consists of all worst values attainable from the criteria.The general steps of the TOPSIS method are as follows.
Step 1.A decision matrix is formed and expressed as follows,  (17) where A i represents the alternatives, i = 1,2, . . .,m; F j represents j th attribute or criterion, j = 1, 2,. . ., n, related to i th alternative; and θ ij indicates the performance rating of each alternative Ai with respect to each criterion F j .
Step 2. Calculate the normalized decision matrix R=[r ij ].The normalized value r ij is calculated as, Step 3. Calculate the weighted normalized decision matrix by multiplying the normalized decision matrix by its associated weights.The weighted normalized value V ij is calculated as: i=1,2,……m and j=1,2,……..n; (19) where w j represents the weight of the j th attribute or criterion.The weights are determined using fuzzy AHP.
Where K is associated with beneficial criteria and K ' is associated with non-beneficial criteria.
Step 5. Measure Euclidean distances of each alternative from the positive-ideal and negative-ideal solutions.The two separations for each alternative are respectively calculated as: Step 6. Calculate the relative closeness to the positive-ideal solution.The relative closeness to the positive-ideal solution can be defined as: The higher the closeness means better the rank.

Priority of criteria
Considering the feedback from the experts from the academic, government, industrial and research sectors, we form pairwise comparison matrix of 6 criteria to get their relative weight over other.
Table 4 shows the fuzzy evaluation of the criteria.

Generating the local performance score
We construct the performance evaluation table (Table 6) taking opinion from 10 experts.Then, using LINDO program we solve model ( 16) for each of the six criteria to generate the local performance scores of the seven institutions with respect to the six criteria.Note that local weights can be interpreted and understood as local performance scores in performance evaluation applications.For each criterion, the optimal solutions to model ( 16) as obtained through LINDO program are shown in the following table (Table 7) Based upon the above optimal solutions, the local performance scores of the seven institutions with respect to each of the six criteria are calculated by Eq. ( 15) and presented in Table 8.The square root of squares of elements for each column is also shown in the last row of Table 8.This value is required for normalization of local performance score matrix according to formula (18).

Aggregation of local performance scores into overall performance score
Once we get the local performance scores of the seven institutions with respect to six criteria, we apply TOPSIS method to aggregate the weights of criteria into an overall performance score.To comply with this method, using formula ( 18) and ( 19), we prepare the normalized and weighted normalized performance score matrices which are shown in Table 9 and 10.In Table 10 last two rows indicate positive and negative ideal solutions (PIS and NIS).While determining PIS and NIS, except the criterion tuition fee per semester (TF) all the remaining criteria are considered to be higher the better type (beneficial criteria).In the next step we compute the Euclidean distances of each alternative from the positive-ideal and negative-ideal solutions (D i + , D i -).Using formula (21) we calculate relative closeness of each alternative to the positive-ideal solution.Results are shown in Table 11.The same table also exhibits FAHP-DEA-TOPSIS method based priority ranking of institutions as A  B  E  C  G  D  F when arranged according to descending order of their relative closeness value.The graphical view of the performance is also presented in Fig. 6.In Fig. 6 we see that the closeness values of institutions A, and B are above the mean closeness value.Institutions C and E are marginally above the mean line and D, F and G are below the mean line.Therefore, institution A can be considered as benchmark or improvement target for B, C and E which form the second group.Similarly, this second group can be considered as the improvement target for the rest institutions i.e., D, F and G.

Conclusion
The primary mission of a technical institution is to explore and transmit knowledge.The former is achieved through research while the later is achieved through teaching.In this age of the knowledge economy, technical institutions play a key role in the development of a country.As the number of young people enrolling in technical education courses in India is increasing, the resources received from the government must be used more efficiently to meet the increasing demand for education.Hence, the institutions should investigate not only the educational outputs produced by the departments, but also the resources utilized in producing the outputs.There has been a national concern of improving the standards of technical education.Many manufacturing and service organizations are demanding to introduce modern research facilities to nurturing a research culture in most of the technical institutions, so that these can produce manpower with very high levels of creativity and innovative abilities and thus help Indian industries to increase competitiveness of their product and services.Many initiatives have been taken in the last few years both by the Government and private sector to improve and maintain the standard of technical education.Since we are in the domain of knowledge, we feel that there is an urgent need to do performance evaluation of Indian technical institutions by considering some relevant criteria.We believe that this kind of study provides useful empirical insights into an important public policy issue.In this paper, we have proposed a new fuzzy AHP-DEA-TOPSIS methodology to evaluate the relative performance of Indian technical institutions.The proposed methodology uses fuzzy AHP method to determine the weights of criteria, linguistic terms such as high, medium and low to assess performance under each criterion, DEA model with common weights to determine the values of the linguistic terms, and the TOPSIS method to aggregate the performance score under different criteria into an overall performance score of each institution.This technique is sound surrogate to the traditional techniques.
The study reveals that the institution A is the best and F is the worst.However the institutions D, F and G need special attention to improve their performance in respect to the criteria of Ph.D and Patent.According to the method the ranking we get is in the order of A  B  E  C  G  D  F. The proposed model is generic in nature.It can be applied for performance evaluation and ranking of a group of organizations irrespective of field.However, evaluation criteria should be selected very carefully because the criteria play a vital role in the determination of relative ranking.

Fig. 6 .
Fig. 6.Graphical view of overall performance score of the alternatives

Table 1
List of centrally funded institutions

Table 2
Fuzzy Scale

Table 3
Distribution decision matrix for decision alternatives .

Table 4
Fuzzy evaluation of the criteria

Table 6
Performance evaluation data for 7 institutions

Table 7
Optimal solutions to model (16)

Table 8
Local performance score matrix for 7 institutions

Table 9
Normalized performance score matrix

Table 10
Weighted normalized performance score matrix