Benchmarking Distance Learning Centers with a Multiobjective Data Envelopment Analysis Model

Since the beginning of the 21st century Rio de Janeiro State is concerned with distance learning project. This project is called CEDERJ (for the name in Portuguese). One of CEDERJ’s main goal is to contribute with the geographic expansion of undergraduate public education. CEDERJ’s expansion in terms of number of local centers and types of courses brings up the need to evaluate CEDERJ globally, since the system consumes public resources.


Introduction
Since the beginning of the 21st century Rio de Janeiro State is concerned with distance learning project.This project is called CEDERJ (for the name in Portuguese).One of CEDERJ's main goal is to contribute with the geographic expansion of undergraduate public education.CEDERJ's expansion in terms of number of local centers and types of courses brings up the need to evaluate CEDERJ globally, since the system consumes public resources.
In this chapter we use advanced models in Data Envelopment Analysis (DEA) to undergo the evaluation of the CEDERJ's centers.We shall notice that since its very beginning DEA has been used in educational evaluation (Charnes et al., 1978), mainly because DEA deals with multiple inputs and outputs and does not need financial figures.Our evaluation is limited to the mathematical undergraduate course because it exists since the very beginning of the CEDERJ.
We first perform a standard DEA evaluation of the CEDERJ centers.Due to the structure of CEDERJ, a problem arises when trying to identify benchmarks for the inefficient centers.In fact, CEDERJ has some centralized decisions and some decisions that are independent for each center.In DEA this means that some variables are not controlled by the centers themselves.Therefore, we introduced a new multiobjective DEA model with noncontrollable variables, called MORO-D-ND.This model provides a set of targets for inefficient centers to achieve in order to become efficient.The CEDERJ center may choose among this set the most suitable target.We also introduce a decision rule for this choice.We calculate the Center non radial efficiency index corresponding to each target in the set.To calculate the non radial efficiency we use the efficiency index presented by Gomes Jr et al (2010a).This chapter is organized as follows.After this introduction we present the CEDERJ in section 2. Basic Data Envelopment Analysis altogether with the MORO-D-ND model and the non radial efficiency index can be found in section 3. Section 4, we present the case study and results.Some final remarks are presented in Section 5.

www.intechopen.com
Advanced Topics in Applied Operations Management 184

CEDERJ
CEDERJ is the acronym for Rio de Janeiro Center for Distance Learning (in Portuguese Centro de Educação a Distância do Estado do Rio de Janeiro).One of the CEDERJ's main target is to contribute with the geographic expansion of undergraduate public education.This is also one of the targets of public universities in general.A second main target is to grant access to undergraduate education for those who are not able to study in regular hours, usually because of work.Finally, developing the state's high school teachers and offering vacancies in graduate courses are also targets to be achieved.
In june, 2010 the coursed of the CEDERj included Mathematics, Biology, History, Pedagogy, Chemistry, Turism, Physics, Technology in Sciency Computation and Management.CEDERj has 34 (thirty four) centers convering almost all the Rio de Janeiro State, as shown in Figure 1.In CEDERJ, students have direct contact with tutors, who are of great importance (Soares de Mello, 2003) for they are responsible for helping students with their subjects as well as their motivation.Its pedagogical program is based on advances in the area of information and communication technologies, but also offers practical classes in laboratories.Students receive printed and digital material, which includes videos, animations, interactivity with tutors, teachers, other students and guests.This environment helps creating knowledge.
Its expansion in terms of number of local centers and types of courses brings up the need to evaluate CEDERJ globally, since the system consumes public resources, and also locally, in order to reduce eventual differences.
Gomes Junior et al (2008) evaluated CEDERJ courses using the so called elementary multicriteria evaluation (Condorcet, Copeland and Borda).The authors point out that there is an apparent relation between regions wealth and its position in the final ranking; and a reverse relation between the number of regular universities and the local center's position.In the present study, these variables should be considered when clustering the local centers.Menezes (2007) made a scientific investigation on distance education, focusing on CEDERJ, analysing how new information and communication technologies impact on time and space organization.

Benchmarking Distance Learning Centers with a Multiobjective Data Envelopment Analysis Model 185
There are many other studies on CEDERJ, yet they are mostly qualitative.Qualitative literature allows different interpretations, and it might become clearer with measurable facts.Our goal is with this quantitative approach to complement the existent qualitative literature, with no intention to replace it.

Basic DEA and some new models
DEA classical models, CCR (Charnes et al., 1978) and BCC (Banker et al., 1984), were based on the Farrell (1957) definition for efficiency to evaluate DMUs that use multiple input and multiple outputs.This is done in a radial manner that is reducing inputs or increasing outputs equiproportionally, in which is called radial efficiency.This radial efficiency can be calculated input oriented, reducing all inputs in the same proportion; or output oriented, increasing all outputs in this same proportion.
For inefficient DMU a target is determined, which consist of the inputs or outputs levels the DMU has to attain to become efficient.Models have been developed to present alternative targets.In this section we present the MORO models that provide a set of targets for inefficient DMU.
On the other hand, radial efficiency, suitable in many cases, may not be appropriate for many real cases.As a result, models that deal with different situations have been presented.One situation is relevant for this study: the existence of non discretionary variables.They cannot vary at the discretion of the decision maker.Being part of the analysis they have to be taken into account.Therefore a brief review of these models is also made in this section, pointing out that the main concern of this has been the efficiency evaluation over the targets.

The MORO models
In many cases the target provided by the DEA classical models may not be viable, due to operational or managerial problems, or simply because we have additional information about the variables.Some alternative models have been presented by Thanassoulis and Dyson (1992) and Zhu (1996).These models provide a target that suits the DMU introducing value judgments about the variables in the model.
One common characteristic is that they only provide one target for inefficient DMUs.If we want a different target a different set of weights must be establish and all the evaluation has to be repeated.
As we have mentioned earlier, the MORO models (Lins et al., 2004, Quariguasi Frota Neto & Angulo-Meza, 2007) provide a set of targets instead of one target for inefficient DMUs.This is possible because of the nature of the multiobjective models, which provide a set of non-dominated solutions (Soares de Mello et al., 2003).
The most common of the MORO models is the MORO-D-CRS that is presented in (1).This model is very similar to the envelope version of the CCR model.This model allows each variable to change independently, and not in a radial way as the classical DEA models.
The  r factor is the variation for the output r,  i factor is the variation for the input i.We have one objective function for each factor, and we try to maximize the factor for the outputs and minimize the factors for the inputs.The restrictions guarantee that we will find projections in the efficient frontier, since the variations of inputs and outputs are independent we replace the inequalities by equalities (Quariguasi Frota Neto & Angulo-Meza, 2007, Lins et al., 2004).The last two restrictions guarantee that the outputs will maintain their levels or increase and the inputs will maintain their levels or decrease.In this way we will obtain targets that dominate the DMU o under evaluation, in an approach similar to the Thanassoulis and Dyson mono-objective model (Thanassoulis & Dyson, 1992).Therefore this model is called MORO with dominance, constant returns to scale.
The last two restrictions can be removed.In this any point in the efficient frontier can be a feasible target for the inefficient DMU, in an approach similar to the Zhu mono-objective model (Zhu, 1996).This model would be called MORO CRS without dominance, or MORO-CRS.
To illustrate these situations, we present Figures 2 and 3 from Soares de Mello et al (2003).In both figures a variable returns to scale frontier is shown.In Figure 2, possible targets for DMU o using the MORO-VRS model are shows.In Figure 3, the MORO-D-VRS is used to determine the targets for DMU o.To obtain the variable returns to scale we introduce the convexity restriction (2) in the model.Such a model would be called MORO-VRS or MORO-D-VRS depending whether we consider dominance or not, as in the aforementioned Figures.
In contrast to the mono objective models, MORO formulation completely explores the set of projections, ensuring that every projection according to model (1) is found, and determines the maximum number of alternative targets for each DMU.Thus, instead of running several mono-objective problems with different weights, MORO models find the projections through a structured and non-interactive algorithm (Quariguasi Frota Neto & Angulo-Meza, 2007).We have to point out that not only the extreme points are targets for the inefficient DMU but also the linear combinations of these points that lie in the efficient frontier are possible targets.This will happened depending on the method used for solving the multiobjective problem.For example, in Figure 3, for DMU o, the extreme points, targets, are a, B and b, also any point in the segments aB and Bb are possible targets for DMU o.Therefore, in theory we will have an infinite set of targets depending on the method use for solving the multiobjective problem (Soares de Mello et al., 2003).
According to Clímaco et al (2008) the MORO models can be classified in the group that uses multiobjective models to solve problems in DEA.
An efficient DMU is on the Pareto efficient frontier and thus ** 1 ri    ,  r, i, as the equality restrictions of the model require nil value slacks.If this is not the case, the targets for the outputs are given by (3) and the targets for the inputs are given by (4).In this way, alternative targets can be obtained based on the preferences of the decisionmaker.
Gomes Junior et al (2010b) stated that the equality restriction on the MORO models may be very restrictive and may present computational problems.They proved that if those restrictions were replaced by inequality constraints, the model will determine the same efficient frontier, that is the same set of targets.The authors called this model with inequalities the MORO-D-R model.

Non discretionary models in DEA
As mentioned previously, in some real cases DEA classical models do not take into account non discretionary variables.Those variables cannot be modified due to fixed factors of production or external factors.For example: federal employees in Brazil cannot be fired, so they become a fixed of number of resources.This situation can be complicated to deal with in an input oriented DEA model, when there are other inputs that we would like to reduce.
Therefore to deal with non-discretionary variables many researchers have concerned themselves with DEA models for this purpose.The first model was introduced by Banker and Morey (1986) and the input oriented variable returns to scale model is presented in (5).In the output oriented model ( 4) we can see that inputs are divided into two groups: controllable and non controllable.We also can see that the maximum equiproportional reduction in all discretionary inputs, maintaining the non discretionary inputs constant.Therefore, the only difference between this model and the standard variable returns to scale DEA model (Banker et al., 1984) is the removal of the factor  from the right-hand side of the non-discretionary inputs.
They also provided the output oriented variable returns to scale DEA model for non discretionary variables.Analogous to model (1), in the output oriented version, outputs are divided into two groups and then the factor is only multiplied to the controllable outputs.
As pointed out by the authors, the constant returns to scale version of this model can be easily formulated with the exclusion of the convexity constraint (Cooper et al., 2006, Syrjänen, 2004)   In this model variables are divided in two sets: controllable (C) and non-controllable (NC).
The factor h o is just for C set.As we can notice the equality restrictions are for the non controllable variables and the inequalities are for the controllable variables.The last restriction imposes an upper bound, U, and a lower bound, L on the sum of  j , to take into account the type of returns to scale of the problem (Cooper et al., 2007).
This model represents a different approach when compared to the Banker and Morey model (4), as it takes into account the existence of non-controllable variable in both the input and output sets.
Following these works many other approaches have been proposed.There are alternative models for this variable considering various stages.For example Ruggiero (1996, Ruggiero, 1998), Yang and Paradi (2006).
As Muñiz et al (2006) and Cordero et al (2009) stated, models can be divided regarding the stages needed to perform efficiency analysis, one-stage and multi-stage models.They made a comparison and introduced new models for calculating efficiency with non-discretionary variables.
Camanho et al ( 2009) classified models for non discretionary factors in two groups: external factors, considering the external conditions where the DMU operates, and internal factors, factors that are internal to the production process but not controlled by the decision makers.
They also proposed a method to evaluate efficiency treating the non-discretionary variables according to their classification.
Recently, Estelle et al (2010) presented a new three stage model and made a new comparison of the non discretionary models that uses various stages.Also, Cordero-Ferrera et al (2010) proposed a multi-stage approach based on Tobit regressions.They also used a bootstrap procedure is used to estimate these regressions to avoid potential bias.They illustrated their methodology with an empirical application on Spanish high schools.
All the works above have as main objective to evaluate the efficiency of the DMUs.In the present paper our aim is different; we want to determine a set of targets for the inefficient DMUs, and we do not asses the efficiency of the DMUs in an environment with nondiscretionary variables.

Multiobjective models for target determination with non-discretionary variables
As seen previously, the MORO models determine a set of targets for each inefficient DMU.
We assume that all variables may change their levels in order to be efficient.In some cases, one target of the set may change the level of one variable at a time.See for example target a for DMU o in Figure 3.If the output in that example is a non-discretionary variable, the decision-maker will choose the target a.Unfortunately, there is no guarantee that the set of targets will always contain a target for any specific non-discretionary variable.
Also, the MORO models allow different degrees of changes in inputs and outputs levels.Thus, to ensure that the set found contains targets that take into account non-discretionary variables we present an extension of the MORO models.The resulting model is in (7) and it is called MORO-D-R-ND, the MORO model with dominance and inequality restrictions with non-discretionary variables, or simply MORO-ND.
In this model we have a factor for every discretionary input (1.. m i ) and output (1..s o ).We have divided the restrictions of the inputs and outputs in two groups, the one that deal with discretionary variables (D o for outputs and D i for inputs) and the ones that deal with nondiscretionary variables (ND o for outputs and D i for inputs).For the first group, as the variables are allowed to change independently, we have set equalities, in a similar approach as the MORO models (1).For the second one, the variables that we cannot change, we set inequalities similar to the envelope model, in an approach similar to the Banker and Morey model.The last two restrictions of this model we have the dominance restrictions, so for the output we can increase or maintain its level and for the input we can reduce or maintain its level.
As the other MORO models, we can obtain a set of targets taking into account the variable that are fixed, for any reason, in the analysis.Obviously, the added advantage is that we do not have to specify an orientation (input or output) for the model, because is a non radial model.
We can also account for the variable returns to scale introducing the convexity restriction (2), and find targets without dominance by eliminating the two last restrictions in model ( 7).
We can also identify an efficient DMU when ** 1 ri    ,  r, i, as the equality restrictions of the model require nil value slacks.If this is not the case, the targets for the variables are given by equations ( 3) and (4) in section 2. In this case, the non-discretionary variables will maintain their levels.Once again, the alternative targets can be obtained based on the preferences of the decision-maker.

DEA non-radial efficiency based on vector properties
It makes no sense to deal with efficiency as a scalar, as this quantity depends on the DMU projection point on the frontier.Thus, the efficiency is characterised by a number and by a direction of projection, characterizing a vector.Mello et al. (2005) propose an index of vector efficiency.This index has restrictions regarding its utilisation according to the statements of the authors.

Soares de
In this work we propose the development of a non-radial efficiency index based on the vectorial properties of the problem.These properties define that a DMU must be projected to the efficiency frontier in a direction which is determined by the decision maker, through the choice of the target.
Figure 4 illustrates the concepts which will be used to obtain the vectorial efficiency index.The index was developed for the two dimensional case, as it allows a better visualisation.The input or output oriented efficiency in the classic DEA models is calculated by the ratio between the distance from the projection of the DMU on the efficiency frontier to the coordinate axis and the distance between the DMU and the coordinate axis.For DMU A, the input and output oriented efficiencies calculated by the classic DEA models are given, respectively, by the equations ( 8) and ( 9).On the other hand, the complement of the efficiency is the ratio between the distance between the DMU and its projection on the frontier and the distance between the DMU and the coordinate axis.For DMU A, the complements of the input and output oriented efficiency are given, respectively, by the equations ( 10) and ( 11).
However, we wish to calculate the efficiency index of the DMU when it is projected on the frontier following a non-radial projection.
We supposed that the DMU A is projected on the efficiency frontier on the target determined by the point P.This direction defines an angle  with the horizontal axis.The DMU A has coordinates (x, y).The coordinates of point P are known and denominated (x E , y E ).
The horizontal projection of point P represents the complement of the efficiency of DMU A if we project only with the input orientation and is given by the equation ( 12).The vertical projection of point P represents the complement of the efficiency of the DMU A in relation to orientation to output according to the equation ( 13).
In this way, we calculate the complement of the non-radial efficiency of the DMU A when the DMU A has as its projection the target defined by point P by the equation ( 14).
22 22 " '" When we substitute the coordinate of the points A, E, F, P and their projections P' and P", we have the equation ( 15).
The efficiency index is defined in the interval [0,1 (Cooper & Pastor, 1995).In this way, the efficiency of the DMU A when projected in the target specified by point P is equal to the difference of its complement to the unit and is given by the equation ( 16).
The DMU target on the frontier is point P. The coordinates of point P are defined by the objective functions of the MORO-D model and calculated by the equations ( 17) and (18). . .
Substituting the expressions ( 17) and ( 18) in the expression ( 16), we have the non-radial efficiency of a DMU when projected on point P on the frontier, given by the expression (19).
  Equation ( 19) can be generalized for multiple inputs and outputs.This generalization is presented in equation ( 20).
  The parameters φ i represents the reductions for each input i, the parameter ϕ i represents the increase for each output j, for an inefficient DMU to become efficient.The value m and s are the total number of input and outputs, respectively.

DEA in educational evaluation
DEA has been widely used in educational evaluation.For instance, Abbott & Doucouliagos (2003) measured technical efficiency in the Australian university system.They considered as outputs many variables referring to research and teaching.Abramo et al (2008) evaluated Italian universities, concerning basically scientific production.
The firsts authors went through analysis using various combinations of inputs and outputs, because the choice of the variables can greatly influence how DMUs are ranked, which is similar to what is done the process of variable selection in the present paper.The seconds also verify the importance of choosing the right variables, by comparing the final results with analysis of sensitivity, and observing how different they are.Abbott & Doucouliagos (2003) introduce the concept of benchmarking as one of DEA strengths, though neither of the articles actually calculates it.Finding benchmarks and antibenchmarks is important for the study's applicability, since it is the first step to improving the inefficient DMUs.These authors also propose clustering the universities, according to the aspects of tradition and location (urban or not), which in their work, does not significantly affect results.
A more comprehensive review of DEA in education can be found in Soares de Mello et al (2006).More recent works in the subject involve evaluation in secondary education (Davutyan et al., 2010).Sarrico and Rosa (2009) used DEA to evaluate the Portuguese secondary schools.The performance of academic departments was evaluated by Tyagi et al (2009).Tajnikar and Debevec (2008) used DEA to evaluate full-time higher education.In Brazil higher education performance was evaluated by de França et al (2010).A study using DEA and Self-Organizing Maps for CEDERJ centers evaluation was done by Angulo-Meza et al (2011).

www.intechopen.com
Benchmarking Distance Learning Centers with a Multiobjective Data Envelopment Analysis Model 195

Case study and results
There are many other studies on CEDERJ, yet they are mostly qualitative or using ordinal methods (Gomes Junior et al., 2008).Qualitative literature allows different interpretations, and it might become clearer with measurable facts.Our goal is with this quantitative approach to complement the existent qualitative literature, with no intention to replace it.
The There are other professionals, besides tutors, involved in the CEDERJ system, such as those responsible for preparing the material.However, the Math material is the same in every local center, so these professionals should be attributed to each course, not to each local center.Although 24 local centers offer the Math course, only 13 have had graduates in 1/2009, which will be considered in this study.
The centers have no autonomy to formulate the evaluation tests.They are formulated by the central co-ordination.Therefore, the number of students concluding the course is not easy to be manipulated by the centers administrations, and so this number is suitable to be used as an output.
In this evaluation, the non-discretionary variable is the number of students that have enter the course four years before.Obviously, that is because we cannot change the past.
Of course, the number of tutors is a discretionary variable.The number of students that have finished the course is not directly determined by the center.However, it is a consequence of the center educational practices.So this variable can be use a nondiscretionary variables (Cordero-Ferrera et al., 2008).
The data set is for the thirteen centers under evaluation is shown in Table 1.We use the MORO-D-R-ND model to determine a set of targets for the thirteen centers under evaluation.The TRIMAP (Clímaco & Antunes, 1989) software was used to solve the multiobjective problem.Among the set of target, the decision maker can choose one target from the set, according to many managerial needs.In this case, we have chosen the target which provides the higher efficiency index.In Table 3, we have highlighted the target with higher efficiency index for each Center.
In the case where any these targets are not feasible, other multiple objective methods, for example Pareto Race (Korhonen & Wallenius, 1988), can be used to determine an alternative target from the efficient frontier.

Conclusion
In this paper, we have performed an educational evaluation of the CEDERJ Centers, using an advance multiobjetive DEA model.The multiobjective model was chosen due to its flexibility in providing alternative targets.Moreover, the vectorial efficiency index provides an effective tool to help the decision maker in choosing the most suitable benchmark for each DMU.The proposed model has proved to be usefull in a managerial and operational aspect.
Comparing our model with classical DEA we notice that both models classify the same DMUs as efficient (it does not change the efficient frontier).However, in our model the inefficient DMUs have the flexibility to achieve a higher efficient index.This characteristic may have decision maker to better accept DEA results as a managerial tool.
Regarding the theorical research, future works involved the comparison between the results obtained with the MORO models and the MORO-R models both with non discretionary variables and dominance constraints.
In relation to the case study, we intend to undergo an evaluation of the other courses, and also involving other CEDERJ Centers.

Acknowledgment
To FAPERJ for financial support.

Fig. 1 .
Fig. 1.Map of the CEDERj Centers in the Rio de Janeiro State.
the target chosen by the decision maker and thus we define the values for ** e ri   among the solutions of the MORO model chosen.
www.intechopen.comBenchmarkingDistance Learning Centers with a Multiobjective Data Envelopment Analysis Model 189
DMUs being evaluated in the present research are the local centers that offer Mathematics undergraduate course, therefore each of the following variables are related to the Math course in each local center.The inputs are AI -Number of students enrolled in the course in the first semester of 2005 (this semester was chosen because the course is four years long) and NT -Number of tutors in the first semester of 2009 proxy for the resources used in the center.The output is AF -Number of students that graduated in the first semester of 2009.
Golany and Roll (1993)roduced by the same authors Banker and Morey,Camanho et al (2009),Estelle et al (2010)among others.Golany and Roll (1993)extended Banker and Morey's constant returns to scale model to account for non discretionary variables in both inputs and outputs.
In theCooper, Seiford and Tone book (2007)a non-controllable model is presented.They stated that restrictions involving non-controllable variables, due to external conditions, should be expressed exactly by equality by a nonnegative combination of the corresponding non-controllable variables.This model is in (6).

Table 1 .
Data set for the educational evaluation. www.intechopen.com

Table 2 .
Table2shows the results for all the centers.Set of targets for the centers.In this Table2, we note that there are two efficient centers: Angra dos Reis and Piraí.For each inefficient Center we have a different solutions, leading to different set of targets.In Table3, we present the target for each inefficient Center and also the vectorial efficiency index corresponding to each target calculated according to equation (20).

Table 3 .
Targets and efficiency indexes for the centers.