Applying DEA sensitivity analysis to efficiency measurement of Vietnamese universities

Article history: Received March 25, 2015 Received in revised format August 16 2015 Accepted September 15 2015 Available online September 15 2015 The primary purpose of this study is to measure the technical efficiency of 30 doctorategranting universities, the universities or the higher education institutes with PhD training programs, in Vietnam, applying the sensitivity analysis of data envelopment analysis (DEA). The study uses eight sets of input-output specifications using the replacement as well as aggregation/disaggregation of variables. The measurement results allow us to examine the sensitivity of the efficiency of these universities with the sets of variables. The findings also show the impact of variables on their efficiency and its “sustainability”. Growing Science Ltd. All rights reserved. 5 © 201


Introduction
Data envelopment analysis (DEA) is a popular tool for measuring the performance of different universities.This is because it is well suitable for a rather special characteristics of these institutions.First, it is hard to assign monetary values to the inputs and outputs like other nonprofit organizations.Second, a university produces multiple outputs using multiple inputs.In fact, higher education is in the fifth place of the most common field of study in DEA literature (Liu et al., 2013).
Using a linear programming technique1 , DEA has overcome the limitations of traditional efficiency measurement methods such as ratio analysis, regression analysis, especially in the application to nonprofit organizations.De La Villarmois (1999) and Chen and Iqbal (2002) discussed in detail the disadvantages of these two approaches compared with DEA.Nevertheless, unlike econometrics, this method has no formal tests of significance in variable choices.Therefore, many authors have applied sensitivity analysis to examine the influence of various variables on the efficiency results of the measured DMUs2 .In fact, the sensitivity analysis technique was first addressed in the study of Nunamaker (1985), and was increasingly developed.Cooper et al. (2011) emphasized that this topic has taken different types of DEA in the literature.One part of this literature studies the responses with given data when DMUs are deleted or added to the set being considered.Another part of the literature deals with increases or decreases in the number of inputs and outputs to be treated.Furthermore, we can also examine the sensitivity associated with the choices of various DEA models.We present only some studies for the higher education and concerning the modification or/and alteration of variable sets.
In contrast with Nunamaker (1985), Sinuany-Stern et al. (1994) examined the deletion of a variable from the DEA and the aggregation of two variables.They concluded that deleting a variable as well as combining two variables might turn an efficient DMU into an inefficient one.Ahn and Seiford (1993) examined the sensitivity of DEA while analyzing the relative efficiencies of public and private doctoral granting institutions of higher learning in the United States for the academic year 1985-86.Their results supported the hypothesis that public schools were more efficient than private schools when visible and closely monitored output variables such as undergraduate and graduate enrollments were used for evaluation.These results were consistent across their four DEA models selected (CCR, BCC, additive and multiplicative models)3 , meaning they were not sensitive to the particular DEA model.McMillan and Datta (1998) ran nine sets of different specification DEA models and concluded that the choice of variables included in the DEA was important.It was valuable to distinguish between master and PhD level graduate students.It was also important to identify separately science and other programs in terms of both inputs and outputs (i.e., faculty and students).They also remarked that universities that were input efficient were not always cost efficient although the two results were generally consistent.
To examine the relative efficiency of 109 higher education institutions in England for the year 2000/01, Johnes (2006) proposed two models: full model including the three outputs and six inputs selected, and reduced model for which two input variables STAFF and LIBCOMP have been removed4 .The author is based on the result of p-values, a test developed by Pastor et al. (2002), to determine the elimination of these variables.Although the application of this test is rare in the higher education sector, it could give alternative options to overcome weaknesses in model specification in DEA application.
More recently, Montoneri et al. (2012) also confirmed the importance of DEA sensitivity analysis.According to these authors, a sensitivity study can help us further identify the characteristics and performance among the efficient DMUs.In addition, the result of the sensitivity study can help us clarify the influences of inputs and outputs on the different evaluated units.Therefore, the authors have also applied this technique in their study.To examine the teaching performance of 18 classes of freshmen students studying a course of English conversation in a university of Taiwan, entering from the academic year 2004, 2005 and 2006, they conducted the study in 2 phases.In the first phase, the authors used the output-oriented CCR model with 4 indicators (2 inputs, 2 outputs) to measure their overall efficiency.In the second phase, they reduced one of the 4 indicators in order to observe its impact on each DMU's CCR score.The study's results confirmed also the assumption that withdrawing an input or output item will decrease the overall efficiency of evaluated units.
These previous studies show that variable selection is a critical part of DEA.It is advisable to examine the results from a variety of variable specifications to see if DMU efficiency is sensitive to variable selection (McMillan & Datta, 1998).Our study, thus, is also going to examine the effects of variable replacement as well as aggregation/disaggregation of variables while analyzing the relative efficiency of universities in Vietnam.Because it is the first one on the efficiency measurement of the system of Vietnamese universities, and there is not yet in Vietnam a university ranking system for the comparison with our results, applying the sensitivity analysis is needed.The article proceeds as follows.The different sets of input and output variables used for DEA model are described in the next section, followed by the analysis of the measurement results, and concluding remarks are given at the end.

The proposed models
The choice of variables included in the DEA model is important.The concern for variable selection is compounded by the fact that as the number of variables increases, the number of DMUs deemed efficient and the efficiency scores of the inefficient units will typically increase (Ahn & Seiford, 1993;McMillan & Datta, 1998).Nevertheless, Boussofiane et al. (1991) indicated that an additional input or output item will weaken the discrimination of DMUs' efficiency evaluated by the DEA model.Hence, it is particularly important that the variables included reflect a valuable component of input or output.In addition, it is advisable to keep the number of variables to less than one-third of the number of observations (McMillan & Datta, 1998).
A number of options are available for choosing input and output variables for DEA analysis in the higher education sector.However, the determination of input and outputs is particularly problematic when it comes to universities.
First, there is sometimes a confusion between input and output variables.For example, many studies have used the number of students or number of enrolments (headcount or full-time equivalent (FTE)) as a teaching output variable (e.g., McMillan & Datta, 1998;Avkiran, 2001;Abbott & Doucouliagos, 2003;Castano & Cabanda, 2007a, b;Bobe, 2009;Guzman & Cabanda, 2009;Cuenca, 2011;Lee, 2011;Agasisti & Lezzi, 2013).Ahn and Seiford (1993) suggested that headcount or FTE enrolment has been popularly used as a major output of institutions of higher learning (IHLs) not only because of data availability, but also because of its direct relation to state funding formulas.In contrast, there are also studies that considered it as input and used the number of degrees awarded or the graduation rate as a teaching output variable (see, e.g., Flegg et al., 2004;Johnes, 2006;Youn & Park, 2009;Katharaki & Katharakis, 2010;Kuah & Wong, 2011;Calhoun & Hall, 2013;Rahimiana & Soltanifar, 2013).Madden and Savage (1997) argued that higher quality teaching would produce a larger number of graduated students.Considering the fact that the number of degrees awarded is the result of the number of student years, Ahn and Seiford (1993) also argued that it would appear to capture effectiveness of education better than enrolment measures could.Nevertheless, comparing the number of degrees awarded, the result of the number of academic years, with the actual resources of universities can produce misleading conclusions in efficiency, since it depends much on the quantity, and then the quality of enrolled students.
Second, outputs of universities are not easily measured.Even though, they are broadly grouped into teaching, research and service, it is extremely difficult, if not impossible, to find a "true" output measure for each of these dimensions (Ahn & Seiford, 1993).Various variables have been used as measures of teaching output.The student enrolments and the number of degrees awarded are the most common, with a distinction between graduate and undergraduate programs.Student credit hours have been also used (Sinuany-Stern et al., 1994;Kao & Hung, 2008) but it can have the problem of credit hours differing significantly among programs of full-time students (e.g., science students versus humanities students).Research output is more difficult to measure.Lacking reliable and easily obtainable measures, many studies substitute research grants, an input, as a proxy for research output (e.g., Ahn & Seiford, 1993;McMillan & Datta, 1998;Avkiran, 2001;Johnes, 2006).Publication counts are sometimes available and used as a measure of research output (Warning, 2004;Bobe, 2009).Service is the most difficult output to measure.Given the diversity and sometimes even amorphous nature of contributions in this area, there is no composite and reliable index.Almost all studies in DEA method, thus, ignored this aspect.
Third, there is sometimes a very high correlation between two inputs or between two outputs, so the number of inputs or outputs can be reduced.For example, Johnes (2006) decided to remove the STAFF variable (total number of full-time academic staff) not only due to the lack of significance of this variable but also due to its highly significant correlation with the remaining inputs.
Additionally, in order to decide the appropriate input/output measures, apart from considering what, conceptually, are the inputs and outputs for a university, it is said to be mutually taking in consideration the availability of data (Beasley, 1990).
After examining the variables used in previous studies as well as available secondary data resources in Vietnam, the input and output variables chosen for this study include: The number of full-time academic staff (headcount) is used due to the unavailability of FTE data.This difficulty was also experienced by Johnes (2006).Therefore, we propose in compensation, in our study, a consideration of the DEA sensitivity to this variable's disaggregation in doctorate (DOCSTAFF) and non-doctorate (NONDOCSTAFF) staff.Regarding the floor area used for training and research (in square meters), Kao and Hung (2008) and Do and Chen (2014)  The number of full-time undergraduate enrolments includes the total number of full-time undergraduate students enrolled annually.This is currently the main and most important part of students in Vietnamese universities.In the number of postgraduate enrolments, we simultaneously consider the aggregate variable or a distinction between master's (MASENROL) and doctoral (DOCENROL) enrolments.The aggregate variable of two variables of enrolments (ENROL and GRADENROL), hence, is the total enrolments abbreviated TOTENROL.In order to assure a higher comparability of this variable, we used the conversion coefficients set by the Ministry of Education and Training of Vietnam, in Decision No. 795/QĐ-BGDĐT for its calculation.In addition to the enrolment variables, numerous studies also used the (total) number of students as a representation of output of teaching.Compared with the enrolment indicators, this variable is more compatible with the annual expenses of teaching resources.Nevertheless, the application this measure would result in problems because it could differ significantly between institutions where training time is different (for example, establishments of medicine or technology compared to others), and it could include students who have repeated.Having obtained the data of the two groups of indicators (number of enrolments and number of students), we will therefore propose two groups of models for comparison.
With regard to research output, there is no quantity data available concerning publications or citations of Vietnamese universities.The only available indicator which can represent this group of output is the research income.It includes the sources of income resulting from scientific research activities, technology transfer and other research-related services.However, in the current reality, there are still too many universities whose research income has a zero value.This makes it difficult, if not impossible, to apply the multi-stage DEA method to identify the efficient projected point (Coelli, 1996).Hence, rather than neglecting this factor we decided to replace it with the total income (TOTINCOME), a general output variable.The total income (in billion VND) includes total sources of income from budget, tuition, scientific research and technology transfer activities, and other.In fact, the incorporation of the total income as an output along with the student number was proposed by Castano and Cabanda (2007a, b), Guzman and Cabanda (2009) and Cuenca (2011).In short, the different variable sets used to measure the efficiency of Vietnamese doctorate-granting universities are summarized in Table 2. Here, the M1 and M2 models are the basic models, the former including the number of enrolments as outputs of teaching, and the latter including the number of students.M1 is then modified into three alternatives (M3, M5, M7) by the replacement as well as the aggregation/disaggregation of variables.Similarly, M2 is modified into M4, M6 and M8.

The results and discussions
In this section, we report the technical efficiency of 30 doctoral granting universities in Vietnam, which have sufficient data for the academic year 2012-2013.We use the output-oriented VRS6 model because this model is the most suitable postulation in the higher education sector (Ahn et al., 1988).The DEAP (Coelli, 1996) and EMS (Scheel, 2000) softwares are simultaneously used to calculate the results of this study.Table 3 presents the VRS scores of these 30 for the eight models proposed.This table shows that their efficiency results are not the same, depending on the models used.Note: (1) These VRS scores were calculated by DEAP 2.1.
(2) Times efficient means the frequency of being efficient of each DMU across eight DEA models, from M1 to M8.
(3) # efficient means the number of efficient DMUs.
First, from the results of models M1 and M2, it can be seen that the efficiency of these universities decreases when the variables of number of enrolments are replaced by those of number of students.Concretely, there are only 12 efficient DMUs with M2 model versus 15 with M1.To try to understand the main causes of this change, we investigate the universities which have changed efficiency (efficient to inefficient or vice versa) between these two models.
-There are 4 efficient universities with M1 model which become inefficient with M2: DMUs 8, 25, 26 and 28.After checking their complete data, we see that thanks to a significant increase in the number of enrolments in 2012-2013 (undergraduate for DMUs 8, 25, 28, and postgraduate for DMU26), these universities are evaluated efficient with the M1 model.However, if they are measured based on their number of students (undergraduate and postgraduate), they become inefficient (their result with the M2 model).This means that their scale of students is not yet proportional to their available resources, and an increase in their number of enrolments would help them improve their technical efficiency.Their number of enrolments for the academic year 2012-2013 can be considered as a reference.
-In contrast, the DMU10, which is not efficient with the M1model, becomes efficient with M2 because its number of enrolments in 2012-2013 was lower than the two previous years.Thus, although this university is evaluated efficient in 2012-2013 (its result with the M2 model), it could be inefficient for the next few years because of its limited amount of enrolments that year.
Then, we compare the results of models M3, M4 with the two previous ones (M1 and M2).M3 and M4 are two models in which we divide the input variable STAFF into two groups: DOCSTAFF and NONDOCSTAFF.This separation is necessary because the contribution of these two groups of teachers (with doctorate or not) in the efficiency result of universities might be different.In fact, when this variable is separated into two, the efficiency of doctoral granting universities in Vietnam improves significantly.Concretely, when comparing the results of the M3 model with those of M1, we see that there are three additional efficient DMUs: DMUs 10, 16 and 19.After checking the weights 7 of their two input variables DOCSTAFF and NONDOCSTAFF with the M3 model, we discover that only DOCSTAFF variable contributes to the overall efficiency of these three universities.That is why when we group this variable and the NONDOCSTAFF into one (STAFF variable), their comparative advantage decreases (justified by their inefficient result with the M1 model).Similarly, if we compare the results of the models M2 and M4, we see that there are four additional efficient DMUs (8,19,25 and 28).
We continue the analysis of Table 3 with another group of disaggregation models: M5 and M6.These are the ones for which we divide output variables relative to postgraduate enrolments (or students) into two: master's and doctoral groups.Thus, in both these models, there are three output variables used to measure the results of teaching, corresponding to the three levels of training: bachelor, master and doctorate 8 .By comparing their weights, we could determine the level of contribution of these systems of training programs to the overall efficiency result of doctoral granting universities in Vietnam.
The results of Table 4 show that both variables ENROL and STU contribute much more to the overall efficiency of universities than the two variables MASENROL and MASSTU, which themselves contribute more than the two variables DOCENROL and DOCSTU.This means that the descending order of the contribution of the three training levels to the efficiency results of Vietnamese doctoral granting universities is: bachelor, master and doctorate.Combining with the results in the M1 and M2 models, we can conclude that the system of postgraduate training in these universities is still much weaker than the undergraduate training.
Finally, we finish the analysis of the previous table (Table 3) by comments on the results of the two final models: M7 and M8.Unlike the M5 and M6 models, these models are formed by grouping all the examined output variables relative to students into one: total number of enrolments (TOTENROL) for M7 and total number of students (TOTSTU) for M8.These are our two aggregate models, which we can utilize to measure and compare the efficiency of all types of universities.Therefore, compared to the two basic models (M1 and M2) and the two disaggregate models (M5 and M6), the efficiency scores calculated from these two aggregate models are lower.This is what researchers of the DEA sensitivity analysis highlighted.For instance, Ahn et al. (1993) concluded that the aggregation of variables reduces the number of dimensions used for relative comparison and thus reduces the chances of DMUs of being selected as efficient units.
7 The technical efficiency of DMU k is defined as the ratio of the weighted sum of outputs to the weighted sum of inputs (Charnes et al., 1978): , where there are s outputs and m inputs: yrk is the amount of output r used by DMUk; xik is the amount of input i used by DMUk; ur is the weight applied to output r; and vi is the weight applied to input i.In this way, for the M3 as well as M4 models, v1 and v2 are the weights applied to inputs DOCSTAFF and NONDOCSTAFF, respectively. 8ENROL, MASENROL, DOCENROL for the M5 model; and STU, MASSTU, DOCSTU for the M6 model.Note: (1) The above input and output weights were calculated by EMS software.
(3) For the M6 model, u1, 2, 3 are the weights of output variables: number of full-time undergraduate students (STU), of master's students (MASSTU) and of doctoral students (DOCSTU), respectively.(4) Contribution frequency: the number of times that each examined variable contributes to the result of the overall efficiency of universities (its weights are greater than 0).
In summary, although the DEA has no formal tests of significance for the selection of variables, one can observe the influence of different variables on the overall efficiency of studied DMUs based on the application of the sensitivity analysis.

Conclusion
This paper has examined the relative efficiency of 30 doctorate-granting universities in Vietnam using DEA sensitivity analysis.The findings show that although this is the group of the largest universities of Vietnam, their efficiency is not yet really high.For all the measurement models, the ratio of establishments achieving the optimal efficiency (equal to1) is only approximately 1/2.There are still a lot of universities having an efficiency score lower than 0.5.Particularly, the efficiency of their postgraduate training is evaluated weaker than the one of their undergraduate training.This problem should be overcome soon in order to improve the competitiveness of the Vietnamese higher education system at the regional and international scale.
By comparing the results of the eight models, we can additionally see that there are ten universities that are always efficient: DMUs 1,5,11,12,13,15,21,23,24 and 27.This means that all the variations of variables have no impact on these units.In other words, these universities are operating satisfactorily in all activities.They are the most representative and thus, need to be developed as the role models.

Table 1
List of variables judged that departments or universities with more space are expected to achieve more in teaching and research activities.Floor area for training and research includes classrooms, libraries, and laboratories.In relation to outputs, we only consider obtainable and quantifiable variables.Because the data on graduated students in Vietnam is not yet complete and accurate, the quality of the universities have not yet been strictly controlled, the enrolment or student measures are preferred in our study.As in other studies, these variables are categorized into two groups: undergraduate and postgraduate.

Table 2
Eight proposed variable sets

Table 4
The weights of output variables relative to students for the M5 and M6 models