Validation of the academic management evaluation instrument based on principal component analysis for engineering and technological courses

In recent years, the expansion of higher education in Brazil has led to a series of demands related to aspects concerning training at the college level. These processes relate to: academics, professionals, entering in the labor market, among others. In this context, an important aspect is the quality of the courses. Thus, the evaluation becomes a critical diagnostic process of reality and starting point for possible interventions to be put in practice by the coordinators of the programs. This article presents the results of a questionnaire administered at the Federal University of Ceará (UFC), especially to Systems & Digital Media and Engineering Programs professors. This research aims to identify how the professors from each department see the administrative procedures developed by the departments and also investigate the possibility of using Principal Components Analysis (PCA) as a support for management of the higher education training. The methodology included the implementation of Likert scale questionnaire and subsequent mathematical treatment with PCA. The results indicate the potential application of PCA to support the management of higher education; it was possible to extract preliminaries inferences related to management methods and their characteristics. This suggests the possibility of developing the Educametrics field.


Introduction
Issues related to high dropout and failure rates in higher education are historical in Brazil.However, the advent of shares increased supply of higher education, these issues become diagnostic evaluation is emerging as a starting point, becoming prominent for understanding these phenomena and suggesting possible interventions.Throughout this study, we will discuss the process of administration in higher education, specifically the one carried out by the managers of the courses and their role to solve this problem.
From the systemic planning perspective, the academic administration must present a fundamental role for the correct objective to solve problems related to the educational quality provided to the undergraduate students, and their consequent formation for the Labor Market (UNESCO, 1998).This is due to the fact that academic environment takes a broader position than just its instructional services, becoming prominent with regard to the management of development processes (Vanghese, 2007).It is noteworthy that the proposed actions must follow the current public policies (BRASIL, 1996), as well as those recommended by the institution (UFC, 1982).It is noticed that the courses' managers are responsible for: regulations, policies, supporting and guiding students throughout their university education within the limits of the academic and administrative aspects (Laguador, 2013).
These limits confront us with the need to evaluate the academic and managerial skills of coordinations at Brazilian higher education.Generally, education management has been studied subjectively, mainly by evaluating its qualitative aspects.However, this study seeks to contribute to the analysis of the educational context under a quantitative point of view regarding qualitative inferences, according to the researches that have been developed recently in a field called "Educametrics" (Silva, et al., 2012), (Vasconcelos et al., 2013), (Nunes, et al., 2013) and (Nunes, et al., 2015).
This study aims to validate an assessment tool that provides the analysis of academic higher education management skills, from the perspective of the respondents of the courses, who filled a form based in a set of statements (A i , i = 1, 2, … , 14, 15) developed for this purpose.These statements were based on the Federal University of Ceará's internal statute (UFC 1982).The UFC's statute point out some directions that must be followed by the managers of the institution regarding the improvement of the quality in higher education.These statements had been previously presented in Nunes, et al. (2013) and Nunes, et al. (2015).Thus, we sought to investigate the applicability of Principal Components Analysis to achieve educational indicators for management courses and vocational formation in engineering, technology and related areas at Federal University of Ceará, based on the results of these indicators the coordination can manage the processes regarding better practices for the stutend formation.

Evaluation and Management in Higher Education
Currently, the Higher Education Institutions in Brazil had been undergoing a process of renewal and changes in their ways of working, in order to improve their service to society (Schuwartzman, 1992) and (Sguissardi, 2009), which may include the higher education management and its practices in order to improve its function.
It is necessary to understand that educational management has its basis in several areas, highlighting the university administration and education.Thus, the main responsible for conducting the process is the coordination of higher course (Marquesin, et al., 2008).This feature assigns a higher complexity for the activity performed by these professionals, due to the need to manage the processes of teaching and learning at different levels and with different institutional responsibilities, from the planning (educational, structural, financial), execution and evaluation (Valoi, 2007).Therefore, the management of higher education has multimodal characteristics, allied to these evaluative processes, which may contribute to the improvement of educational quality with respect to the formation of higher education level (Gómez, et al., 2014).Among the knowledge that should give management support the evaluation of higher education (Bogoya, et al., 2013), can be used to check various aspects, going through the simplicity of institutional process of teaching and learning, such as: mission, institutional development plan, research and extension policies, graduate, social responsibility, human resources policies, physical infrastructure, and educational materials (Stufflebeam, et al., 2007).
According to Tyler (1942) and(1993), the evaluation provides grants for critical analysis from the perspective of the institution, enabling the investigation of ineffective elements, rearranging the various stages of the process.Still on this model, some aspects should be highlighted: professor/ student interaction, education as a process of establishing standards of conduct, the diversity of instruments in the evaluation process, among others.Thus, it can be seen that the evaluation becomes more significant when we take in to account the analysis of programs, curricula and educational systems, crossing the boundaries of the specific measuring student performance.This gives to evaluation some parametric aspects for intervention in reality/context of the program/ course.This with the intention to further develop the institutional function (Tyler, 1942).
According to the information above, the qualitative and quantitative assessment of the management procedures of undergraduate degrees in engineering, technology and similar fields at UFC, may provide a basis for the quality improvement of the courses (Gómez, et al., 2014) and their graduates; and also enable the adoption of strategies that aimed the reduction in dropout and failure rates, providing an improvement in the academic outcomes of undergraduate students.Thereby, we were looking for a mathematical mo-Sophia Antipolis, France.Affiliation: Adjunct Professor, Universidade Federal do Ceará, Brasil.E-mail: andre@gtel.ufc.br.del that provides the link between the elements of qualitative and quantitative aspects of the assessment proposed.

Mathematical Model
In the present study the multivariate statistics method called Principal Components Analysis -PCA was used, aiming the evaluation and validation of the instrument developed, regarding the extraction of information related to the management processes of Engineering, Technology and similar courses at UFC.The PCA is a method that reduces the space of representation of measures of the information brought by the data, whose most significant information is preserved.
The process starts by discarding linear combinations with small representation of the data characteristics, thus the overlap information is not considered.
The PCA is used to analyze integrated data obtained from statistical measures relationships in order to identify patterns, possible inferences and classifications in the data set for analysis.
In this work, we started from a database in matrix form, ] is a co- lumn vector which components originated in data set.The PCA seeks to relate the entries of X through measures of covariance or correlation coefficients of the variables.This new set of information has some important algebraic characteristics (Green, 2011) and (Tabachnick, et al., 2001).
The PCA-based analysis in this work, part of the correlation coefficients matrix, which highlights the relationships of the data in a domain of observation, is represented by P linearly independent columns of the matrix X, assuming p<m This indicates that the matrix X has full rank p.It should be noted that the eigenvector associated with the largest eigenvalue, of the covariance or correlation matrix of the data, represents the direction of the projection with the greatest intensity information from the original dataset.
The PCA method is calculated from the original database, represented by the matrix X, following the sequence below.
Algorithm PCA , formed by the vectors of observed data � x x P 1 , , … , where P is the number of statements.
(i). calculate the µ i sample mean and the s i 2 sample variance of the variable p-th: Where the value of x lp is the value of x p in the p-th observation, which comprises the statements evaluated by respondents.
(ii).Based on the eq.( 1), compute the entries from covariance matrix  =       × c jk P P between the vectors x j and x k In the statements A j e A k : (iii).Calculating the correlation coefficient between the statements A j and A k , considering the Equations ( 2) and ( 3 Where R PxP is obtained from Equation ( 4) From the Eigenvalue Decomposition -EVD of R, obtaining matrices of eigenvectors and eigenvalues Λ and Q, respectively: and wherein λ i , and , , q i P i , ∈ [ ] are associated eigenvectors and eigenvalues , respectively.P lst largest eigenvalues and its eigenvectors associated are chosen as main components.

Methodology and Research Context
The study was conducted at the Federal University of Ceará (UFC), with professors from six engineering courses from the Centre of Technology and Agricultural Sciences (Teleinformatics Engineering, Electrical Engineering, Chemistry Engineering, Mechanical Engineering, Civil Engineering, Fishing Engineering and Mechanical Producer Engineering) and the Systems and Digital Media (SDM) course at UFC.These courses had been chosen by the fact that the institutional affiliation of the researchers, who has conducted this research, has free access to the professors.It is important to notice that the evaluated courses had starting their curricula reformulation witch this study could be useful to help the managers' decision making.
The research sought to identify the professors' point of view about procedures, methods and actions taken by the coordination of each course with regard to administrative processes.The instrument developed was based on the Institution rules, where the skills and competencies of Course's Coordinator are pointed out (UFC, 1982).About 20 % of professors answered the form.

Instrument design of the research evaluation
As noted, it was developed an instrument in the form of a poll, that containing 15 statements to obtain the informa-tion that were quantified (numerical data) (Nunes, et al., 2013) and (Nunes, et al., 2015).
Regarding the statements, the participants suggested their level of agreement through a 6-point Likert-type Scale, which, at the time of processing data, received the conversion values of: Totally Disagree (1); Strongly Disagree (2); Disagree (3); Agree (4); Strongly Agree (5); and Totally Agree (6).The purpose of this instrument is to identify possible inconsistencies about management processes under the professors' point of view, especially focused on academic and administrative management.Purposely, the statements were randomly rearranged, not allowing any kind of classification based on its structural features, such as: teaching and learning relationship, academic and vocational formation, and management procedures.
The data collection was performed using an electronic type, available at this storage service and sync of the Google Drive ® service, as well as in printed form.In the data analysis, the software Matrix Laboratory -MATLAB and the Statistical Package for Social Sciences-SPSS were used.The plots were obtained by the software OriginPro8 ® .

Validation of results
To ensure that the database was suitable for PCA, some tests of reliability and adequacy were performed as described below We emphasize that all the data variables were used to analyze the eigenvalues obtained a priori (PCA-1), and after this, only the most important variables were selected to the final analysis (PCA-2).
1. α de Cronbrach is an index that seeks to ensure the reliability of the data from the measure of its internal consistency (Hörnquist, et 2002).For the collected data, the Cronbach's α was 0,9402 (Table 1 -PCA1), which ensures a good statistical distribution of the data around the mean, i. e., respondents has following the same thinking to answer the poll.Therefore, the data is reliable for analysis, according to this parameter.2. Kaiser-Meyer-Olkin (KMO) and Bartlett's Tests: Calculate the correlation among all variables analyzed and also verify the hypothesis of correlation matrix of variables is an identity matrix (Tabachnick, et al., 2001).
The value obtained for KMO test was 0,824 (Table 1 -PCA-1), showing that the data are significant, thus guarantees a good adaptation of the sample for the application to use the PCA, since this must be equal or upper than 0,6 (Green, 2011).We find that the Bartlett's test reject the null hypothesis that the correlation matrix would be an identity matrix (Green, 2011).
Application of PCA ) , (Hörnquist, et al., 2002)   and (Pasquali, 2012).The values of the communalities are set in Table 2 (PCA-1), and all the 15 statements presented values higher than 0,6, indicating high representativeness of the statements in the first case (PCA-I).

Selecting the Number of Components:
A Figure 1 (PCA-1) shows the magnitude of the eigenvalues (λ) extracted from the correlation matrix.To select the number of principal components to be considered later, we used the Kaiser criterion, that says that the components to be considered must submit eigenvalues greater than I (λ>1) (Tabachnick, et al., 2001).In Figure 1 (PCA-1) we can see an abrupt decay curve and apparently stabilizes at values between 0 e 1 after the first 3 components, which represent an accumulated variance that exceeds 7 % as seen in Table I (PCA-1).This result shows sufficiency on the representation of the original data (Pasquali, 2012).3. Cluster Analysis By PCA: For the PCA-1, the three principal components obtained, by the Kaiser criterion, does not make any sense in the analysis.Then, the first two Principal Components (PC1 and PC2) were selected presenting great results according with the preliminaries expectations (Figure 2).In Figure 2 the formation of two clusters of statements can be verified: Cluster I -A 1 , A 2 , A 3 , A 9 , A 11 ; and Cluster II -A 5 , A 6 , A 7 , A 8 , A 12 , A 14 , A 15 .We observe in Figure 2 a set of loadings formed by statements A4, A10 e A13, which we considered as outliers.Observing the outliers statements, the statement A4 is related to the National Examination of Students' Performance (ENADE) and the statements A10 and A13 are related to the maintenance of equipment and infrastructure.
The discrepancy regarding A4 can be attributed to the fact that the SDM course has not participated of any external assessment process which contributes, in this study, for 50 % of respondents.According to the items A10 and A13, this apparent distancing of the clusters reveals the opinions of the professors that the coordination has no obligation to act in the maintenance of equipment and physical infrastructure sector, since this work should be done by other university sectors (UFC, 1982).
The occurrence of outliers showed the need to remove their items from the analysis, since this could negatively influence the clustering formation.The removal of items demanded a new validation process where the data in were obtained Table 1 (PCA-2).
It can be observed that the new validation was consistent with higher values than the previous validation step.It is possible to observe that the model selects only one component equal to or greater than 1 (Kaiser Criterion), and this alone represents approximately 61 % of the variance (Table 1.PCA-2), where we can still view the stability in the decay of magnitudes of the eigenvalues in Figure 1(PCA-2).
In PCA-2, the Kaiser criterion selected only one principal component, however, as showed in the first analyze (PCA-1), this criterion was revised to make reasonable inferences about the results obtained and, in this case (PCA-2), we have considered the second principal component in the analysis, as can be seen in Figure 3.A new consistent pattern can be observed in Figure 3, with the formation of three clusters.The statements belonging to each cluster are presented in Table 3.The Cronbach's α was computed to measure the internal consistency of the clusters obtained and all the three clusters have a high internal consistency.
From Figure .3, the statements related to cluster I have in common the fact that they refer to aspects related to curriculum, assessment and academic and administrative leadership, this inference reinforces the importance, attributed by professors, about the inductive role of guidelines for stroke that can be exercised by the coordination.This cluster was named Planning x Assessment (PA).The cluster II presents statements related to the institutional training environment and its external visibility, more broadly in society, with regard to aspects related to meaningful learning and ENADE, or more specifically on their performance in the labor market through his first contact through internships.The cluster was named as Management of Professional Training x Labour Market (PM).
Analyzing the cluster III, the statement points to an affinity in the matters relating to Academic Training of students, es-pecially with the regard to the enviroment to support learning as library and extracurricular activities.This cluster was named the Academic Management Training x Environments to support learning (AE).
The Figure 3 can still provide other information such as: statement of the cluster I present higher loading values on PC1, followed by the other two clusters with almost the same order of magnitude; towards higher values of PC2 loading are presented by the statements of cluster II, followed by the other two, being these of similar values.
We also observed that PC1 and PC2 in the cluster III was identified as the second most important in relation to these two components.We can infer that professors showed an awareness of the important role that the management of higher education has in conducting academic training processes in engineering courses, and related technology.

Conclusions and Perspectives
The aim of this study was the validation of an assessment instrument constructed to identify the skills of academic management, using educational indicators, in the context of professors' point of view of the Systems & Digital Media and Engineering courses at Federal University of Ceará.In this sense, this instrument can be used to provide support for the curriculum design and management process regarding the improvement of the quality in higher education.
The results show the feasibility of using PCA for the analysis and validation of the data collected related to academic higher education management.Results showed the need to remove some items that were corrupting the analysis.This was generated by the professors, understanding the role of coordinators who have no autonomy in the management of financial resources.This result does not correspond to what is practiced in the institution, thus indicating a clear shift of management processes to improve the quality of its courses, this being based on the professors' point of view.
The remaining statements were classified into three groups: Planning x Assessment; Management of Vocational Training and Labour Market; and Management of Academic Training and their support learning environments, and these can be seen as educational quality indicators.We can also highlight the important role of coordination presented in the meaning given to management in major components extracted.Thus, it was established that the management in higher education is more focused on academic training than in professional training.This corroborates that this instrument allows us to make evaluative inferences about actions and management methods.Then, it can support possible interventions proposed by the course coordination, regarding the improvement of teaching and learning and thereby improving the quality of training in technology and engineering areas.
The analysis of results points to the existence of a relationship between the methods of management in higher education, where it was found that this provides important elements for academic and vocational training.This corroborates the pos-sibility to make evaluative inferences about the actions and methods of management.Thus enables the design of interventions aimed at improving teaching and learning in undergraduate engineering education.We could even say that the vision of professors points to the need for decentralization of financial management within the courses.
Although the results obtained are presented fairly consistent and, therefore, promising for the use in evaluations of management processes in undergraduate courses, the validation study showed a limitation with regard to the application context: only 7 courses were evaluated, and the results obtained can not be extended for the whole university.
These studies point to possible perspectives, such as: the possibility of application of the instrument in other courses and areas; can make use of inferences as subsidy for reformulation of policies that aims to decrease dropout and failure rates, contributing to the intervention with regard to the academic success of the learner; investigation of the relationship between the results found in this research with the indices calculated by external evaluations; adaptation of the instrument to implement with students of the courses, that aimed to identify the vision of the graduate student on the level of training received; the application of other mathematical models which exploit the inherent information data, such as multilinear decomposition and tensor analysis (Vasconcelos et al., 2013).
. To obtain the matrix of correlation coefficients, hereinafter called correlation matrix.

Table 1 .
Comparative validation of the indicators in PCA-1 end PCA-2

Table 2 .
Values Communalities of the Statements.