Proposal for improvement capes evaluation system in engineering III

This paper aims to discuss the criteria and weights used by the CAPES evaluation in the area of Engineering III of the Postgraduate Program evaluating the possibility of incorporating some criteria and/or methodological aspects of international evaluation systems. The Brazilian postgraduate system must be evaluated not only in terms of number of programs, students and notes, but also need to be considered distribution by area of knowledge and quality in research. So we sought to characterize the international rankings to differentiate from each other mainly by their methodological orientation. Thus the authors emphasize the difficulty of assigning appropriate weights to each indicator in order to meet the demand of users who consult the rankings with interests as diverse. Therefore the results of this paper contribute to the improvement of the evaluation methodology of CAPES, when considering the critical identified in the analysis, suggesting the inclusion of new indicators and criteria, as well as the redistribution of weights. It was also observed that the performance in research has been consideration in the analysis and classification of universities in major international university rankings.


Introduction
The search for global education institutions and research have contributed to the structure and membership of such institutions, motivated by a focused vision for the creation of knowledge and the development of practical applications for such innovations and contribute to the social development and economic the nation, become a source of additional financial resource, human and material, forming thus a virtuous cycle of excellence in the areas of operation of these institutions (Etzkowitz & Leydesdorff, 1999;Benneworth et al., 2009;Olcay & Bulu, 2016).
Face of this context, the evaluation system of universities and programs of postgraduate is a problem influenced by several factors, which contribute to the analysis of a wide range of alternatives, this way the multi-criteria approach to support decision fits the problem on that of HE Ranking (Higher Education). The multi-criteria approach in decision problems allows the construction of the preference structure(s) of decision maker(es) that meet several goals, which are represented by criteria such as, for example, teaching, research, citations, social integration, intellectual production etc., and considering the importance of each criterion for the decision problem, they have grounds for the review process of the available alternatives, in this case, the set of universities (De Almeida et al., 2015).

The evaluation system of Brazilian programs of postgraduate is through Higher Education
Personnel Improvement Coordination (CAPES), which assesses and acts as a development agency. Evaluation systems such as Times Higher Education (THE), Quacquarelly-Symonds (QS) and Academic Ranking of World Universities (ARWU) evaluate universities globally, not acting as a financing agency. Thus, the Brazilian evaluation of graduate programs can not be compared with the above evaluation system. Although the indicators and the data base are similar, the assessment systems are different (Yu et al., 2016;Olcay & Bulu, 2016).
The main core of any postgraduate is research, this depends on training and requires full dedication to study, and academic institutions task binding on such activities. The results of this research, when applied, lead to economic and social development (Dias & Rorato, 2014).
In this way, Brazilian postgraduate system must be evaluated not only in terms of number of programs, students and notes, but also need to be considered distribution by area of knowledge and quality in research. They are great challenges of the National Postgraduate System -SNPG in terms of reduction of regional asymmetries and areas of knowledge, new researchers training time, qualification of teaching staff, promotion of scientific growth and increase the country's role in the international arena (CAPES, 2015).
The promotion of Brazilian scientific production and the increase of the country's protagonism on the international stage, combined with advances and the highlight of Brazil in the economic environment as an emerging nation, have meant that the country pass also facing the challenge of preparing to have universities international prestige, the world-class call university has become a slogan, used not only to express the improvement in the quality of teaching and research in higher education, but also to refer to the development of capacity to compete in the global market by acquisition, adaptation and generation of knowledge (Niland, 2007;Salmi, 2009;Altbach, 2010).
Thus, in this context, the study presents the model used by the CAPES system to evaluate brazilian graduate programs, which, according to the context uses a multicriteria method that is based on a mathematical structure to evaluate a problem.
Given this contextual aspect, the study discussed Exacta, São Paulo, v. 17, n. 1, p. 15-34. jan./mar. 2019. Bortoluzzi, M. B. O., Melo, F. J. C., & Muchanga, A. by the model used by CAPES and presented the International Systems to Rank universities, such study discusses a vision perspective to be added to the model used by CAPES.
The contribution of this paper is to discuss the criteria and weights used by the CAPES evaluation of the Postgraduate Program evaluating the possibility of incorporating some criteria and/or methodological aspects of international assessment systems. The evaluation system, both the universities and the postgraduate programs, use the model of deterministic additive aggregation.
Thus, the evaluation of each criterion is an additive function of the sub-criteria and overall rating is a weighted sum of the new criteria. This paper will be divided into five sections, in section 1 introduction was presented with its rationale and objective. Section 2 described the concept of Multi-Criteria Decision Aid -MCDA.
Section 3 describes the evaluation of university and Brazilian graduate program systems. Section 4 was presented the analysis of the impacts of the criteria and weights used by the systems described above. Finally, in section 5 was carried out the relevant conclusions.

Multi-Criteria Decision Aid -MCDA
A multicriteria problem consists in a situation where there are at least two alternative options to choose from, and this choice is conducted by the wish to attend to multiple objectives, which are associated with the consequences of choice for each alternative to be followed (De Almeida et al., 2015).
The modeling of decision problems involving multiple criteria and different decision concerns over alternatives such as choice, ordering or classification. In this way, it is used multicriteria de-cision support methods (MCDA), which cover a wide range of methods available in the literature.
The modeling of decision problems involving multiple criteria and different decision concerns over alternatives such as choice, ordering or classification. In this way, it is used multicriteria decision support methods (MCDA), which cover a wide range of methods available in the literature (De Almeida, 2013).
The main methods of multicriteria decision support are classified into three major groups: Single Criterion Synthesis, outranking methods and interactive (Roy, 1996;Belton & Stewart, 2002 (2) represent the normalization. (1) An important factor for the use of an additive aggregation function, if and only if the criteria are mutually independent in preference (Keeney & Raiffa, 1976). Another issue that must be taken into consideration is about getting the scale constants that can not be based only on the degree of importance of the criteria. As the additive model represents the function value set on the consequences rather than be drawn to the alternative.
The constant scales are associated with the replacement rate, which brings the concept of trade-offs between the criteria, ie the gains com- The program assessment level greater than or equal to 3 has its validated and recognized di- for all aspects taken into account in the procedures adopted by the evaluation (CAPES, 2013). Table 2 shows, respectively, the evaluation criteria for the area of Engineering III and a brief description of the sub-criteria and their respective weights once the criteria may vary among the evaluation areas Through the indices removed from the reports of each program, we evaluate each of these sub-criteria of a qualitative or quantitative way. They are assigned concepts ranging from Very Poor, Poor, Fair, Good and Very Good to each of these subcriteria associated to the six evaluation criteria. The algorithm used to obtain the sub assessment of and relationship between the numerical values and concepts (VG, G, F, P and VP) are shown in Table 3.
Based on an overall assessment arrives at the levels (final concept) from 1 to 5 (CAPES, 2013).  (Vincke, 1992).   Technical production , patents and other productions relevant 20 7 Social Inclusion (10%) Insertion and regional impact and (or) national program 40 4 Integration and cooperation with other programs 40 4 Visibility or transparency given by the program to its performance 20 2 Source: Adapted CAPES (2013).

Quacquarelly Symonds (QS)
The QS is a commercially oriented ranking, produced by consultancy Quacquarelli Symonds, specializing in education and study abroad. The ranking is designed to guide students seeking training in higher education institutions of excellence, as well as corporations and institutions seeking qualified professionals the labor market (QS, 2016b). The main characteristics QS ranking is presented in Table 4.

Quacquarelli Symonds (QS) World University
Rankings, aims to help students make compari- The QS uses an alphanumeric notation to group and compare universities based on four aspects: Size (student population); Comprehensive (areas of operation); Intensity of research; University age in question (QS, 2016a). This information is presented in the ranking with the final score obtained by each university according to Table 6.

Times Higher Education (THE)
The Times Higher Education (THE) is a weekly London magazine, which seeks to inform the public about issues related to higher education.    ). The Table 9 shows the main features of ARWU ranking.
The ranking uses five indicators that measure: scientific production in quantity and quality; the number of researchers with a high level of citations; former students or teachers who received the Nobel Prize or Fields Medal, the equivalent in mathematics; the publication in prestigious magazines. There is a sixth composite indicator that combines the above and considering the number of teachers of the institution with full-time dedica-  tion. Finally, the six indicators are aggregated and assigns a numerical score end in the best institution that receives points 100 (ARWU, 2015a). The It is known that the development and use of rankings will always be subject to criticism by a number of problems and drawbacks, among which are: indicators selection criteria, weighting and weight assigned to them, the standard-  ization of results, many rankings described combine several steps to produce the final score, errors in the collection and processing of data, lack of transparency and reliability of results (Saisana et al., 2011).

International Rankings: Performance of Brazilian Universities
To evaluate the quality of education in higher education rankings of universities has been created, they not only created a visual way to differentiate universities, but also began to promote academic, scientific and educational competition among universities on a global scale (Salmi & Saroyan, 2007). Already the international rank-    The results show that the performance of universities in the THE ranking is very shy. Based     Table 2, and the totals for each criteria that are again weighted, resulting in the overall evaluation of the program, as shown in Table 3. based on the overall assessment arrives at the levels (final concept) from 1 to 5. As the UFPE and UFRGS university and present concept 6. Table 12 shows the compendium of data.
The criterion Faculty and Student Body, theses   tor. In Table 13 shows the criteria and associated indicators for each ranking. For the THE rankings criterion International outlook (7,5%), examines whether there is diversity (foreign students and teachers) on campus, is a sign of how an institution puts into global perspective.  Bortoluzzi, M. B. O., Melo, F. J. C., & Muchanga, A. This criterion is 7,5% relative weight in the rankings and is composed of all three indicators of the same weight (2,5%). As for the QS rankings, this criterion has a final contribution of 10%, and checks the degree of international opening of the institution with regard to foreign teachers and students.  After analyzing the criteria used in the rankings in the study, whose main objective is to rank universities, given the objective of this study aimed Although every attempt to quantify the stratification (high impact production, A1, A2, B1, B2, B3, B4, B5 and C) often causes distortions and may represent a low stimulus to the more differentiated scientific literature. One of the simple ways to control this issue and effectively encourage more differentiated scientific production is to score the scientific production of teachers and PPG leaving the sum of points obtained JCR, as in ARWU ranking.
According to Milk (2010) JCR base is considered a more reliable basis for being more stable and suffer less fluctuation than other bases.
It was found that the criteria used JCR base are generated independently by two groups with different private and commercial interests Thomson Reuters and Scopus, which do not necessarily adequately represent the worldwide scientific publications. In this way it was observed that the CAPES to use the JCR base should rethink your model as to its use, that because these two groups has considerably expanded to include periodic least developed countries in their databases, which creates, among others problems, difficulties in time series analysis. It is necessary to review this dependency and generate alternative criteria, less biased and more "authentic".
Worldwide, the number and quality of citations received (with and without self-cites) and H-index and the like of each of the teachers are highly valued. Something that CAPES has not incorporated this in its evaluation platform. The map assessments could include one criterion such as valuing and quantifying the quotes obtained by the scientific products of a given PPG in the previous four years as well as total number of citations and H-index of the permanent teachers.
There is to propose a different criterion for co-authorships score. Unfortunately, it seems to be common practice that many teachers are listed in a co-authorship without having actually met the criteria for authorship. This does not help the quality of Brazilian research actually generates distortions.
It is necessary to incorporate in the evaluation system CAPES a criterion for internationalization of Brazilian and foreign institutions. This is an important mechanism for exchange and scientific development and promotion needs to be valued, as adopted by THE and QS rankings.
It is also necessary to incorporate the faculty criteria CAPES, activities they value the role as number of views expressed, peer-review in different journals, as a participant of editorial board, or participate in activities such as the dissertation stands, thesis or contests for teachers, administrative positions, the orientation of graduate student, a member of committees and university committees, among many other activities, the evaluation of teachers of PPG, which currently total score.
Among the features observed is that different from ARWU, which employs only bibliometric indicators to identify key universities, THE main characteristic is the weight given to opinion surveys with the academic peers and labor market professionals about the reputable universities in teaching and research. The THE differs from ARWU by employing both qualitative analysis (prestige and reputation) and quantitative (performance indicators) in order to identify the universities that stand out in the world in terms of education and research. The QS uses an alphanumeric notation to group, compare and select universities based on national rankings, academic reputation, geographical balance, direct submission, among others. In turn, the CAPES uses bibliometric indicators in the graduate evaluation system.

Conclusion
In this work we aimed to discuss the QS Although it was observed that with regard to standardization of results, many of rankings described combine several steps to produce the final score.
Thus the results of this paper contribute to the improvement of the evaluation methodology of CAPES, when considering the critical identified in the analysis, suggesting the inclusion of new indicators and criteria, as well as the redistribution of weights. It was also observed that the performance in research has been paramount consideration in the analysis and classification of universities in major international university rankings.