A NALYZING THE IMPLEMENTATION OF AN ERP SYSTEM BY SELF-ASSESSMENT IN HIGHER EDUCATION

Over the last few decades, not only organizations but also Higher Education Institutions should be more responsive to the demands of the changed global business environment and improve their effectiveness. Our motivation to write this paper is to assess the implementation of an Enterprise Resource Planning (ERP) system in higher education and their associated benefits, with a focus on students’ performance when applying an SAP solution. This paper analyses the accuracy of undergraduate students, focusing primarily on the concept of self-assessment as they predict and evaluate their own performance relative to their externally assessed achievement. In the preand post-examination predictions the higher achieving students seem to predict and evaluate their examination results more accurately than their lower-achieving fellows. Although a gender gap cannot be found in self-estimation, we found substantial differences by comparing the selected language. Foreign students seem to overestimate their own examination performance to a greater degree than Hungarians do. Consequently, our results might allow decision makers to identify why self-assessment is important when implementing pilot ERP projects. The result of this study also provide evidence for groups of clients and other stakeholders in order to reduce failure in both higher education and business environment.


Introduction
The globalized business environment has continually changed and is increasingly becoming more complex, resulting in improved competition and continuous pressure to achieve low operational costs and to satisfy the growing needs of stakeholders.Over the last few decades, economies based mainly on manufacturing have moved to knowledge-intensive service industries driven by technology changes and innovation.An important feature of these kinds of services is the constant need for adaptation to clients [9].Consequently, not only organizations should be more responsive to the demands of their business partners, but also the Higher Education Institutions (HEIs), which have been strongly influenced by these global trends and by the growing demands of universities worldwide to improve their performance and efficiency.
The European collaboration in education and training for the period up to 2020 should be recognized in a strategic framework which encompasses a lifelong learning perspective.Indeed, this fundamental principle could support learning IT systems from early childhood education through to higher, vocational education and training and adult learning.The EC report [2] also emphasized that efficient investment in HEIs through ERPs is an essential component of European strategy to deliver high levels of sustainable, knowledge-based economic growth and increasing competitiveness.
However, the increased demand for a business and IT workforce has similarly recognized that HEIs should integrate 21 st century skills, such as familiarity with computers and technology, critical thinking and problem-solving, communication and self-direction [15].In recent decades, the attention of HEIs have turned to Enterprise Resource Planning (ERP) systems, which have been one of the most impressive technological innovations to integrate the different functional business areas of companies, such as Finance and Accounting (FI), Human Resources (HR) and Logistics (Sales and Distribution and Material Management) etc., in order to improve organizational processes [1].As a result of substantial investments in implementation, most of the prevailing ICT systems have been replaced by ERPs in these HEIs, in order to enhance their students' performance by teaching improved managerial tools [20].
According to the terms of the Higher Educational Institution's (University of Debrecen and University of Magdeburg) license agreement the academic use of SAP® Software is permitted for the purposes of teaching by staff and learning by students and also submitting other research work.Through the University Alliances Program (UAP), SAP® provides advanced technologies and several business solutions for hundreds of HEIs to develop highly qualified graduates, who contribute to the core competencies of their future employers [10].However, Mehlinger [12] highlighted that in excess of 60 percent of the implemented ERPs failed to measure up to the expected outcomes.In light of the fact that these applications did not explicitly develop the end-users' performance, it is worth attempting to explain what kind of elements impact on this.Although Zairi et al. [35] and Osei-Bryson et.al. [18] etc. have previously attempted to answer this problem from this perspective and have been able to determine critical factors at an organizational level, recently less research has dealt with ERPs in a university environment.Essentially, universities differ from business enterprises, because they are primarily governmental organizations and operate in generally non-profit circumstances and for academic purposes.Faculties and their staff frequently cooperate on fundamental institutional activities, and students also need accurate information and better e-learning environments.Any lack in these areas can result in a high percentage of implementation failures in ERP projects [24].Although earlier research has aimed at evaluating ERPs' implementation success in terms of the expenditure invested (i.e.[34], several unexplored questions still remain, such as how they can contribute to higher performance of users and how the utility of these applications in HEIs can be evaluated empirically. Our motivation to write this paper is to analyze the implementation of ERPs in higher education and their associated benefits, focusing primarily on the students' performance.Our research derives from the fact that in HEIs a large proportion of students seem to evaluate their own performance irrationally (see [11]).However, there is still no existing consensus on whether students' self-assessment ability is obviously learnable or not (e.g.[3]; [23]; [36] etc.) during studies in higher education, although Watermann and Peters [32] pointed out that specific student groups are exposed to the phenomenon of inaccurate self-assessment.Therefore, our research paper primarily focuses on the measurement of students' ability to predict and evaluate their performance of their SAP examinations, relative to the externally assessed achievement.In this comparison our findings highlight that how can universities support the students' learning effectiveness and also measure the success of ERPs implementation.

Brief literature review of self-assessment of ERPs in higher education
Analyzing the organizational readiness of ERPs is a method of evaluating different dimensions of the success of implementation, which in our study is measured by the self-assessment of applying SAP in practice with the students' performance in an HEI.Measuring the readiness of pilot ERP projects in HEIs is a particularly complicated process, and one of the first steps to indicate the success of implementation of such ERP systems as SAP is the self-assessment when students predict and evaluate their own performance.
Theoretical approaches proposed for the evaluation of the readiness of ERPs are usually based on the McKinsey model (see [29]).In this method, organizational readiness is determined by seven major dimensions, such as managerial and organizational, structural, process, technical, infrastructural and cultural.Since its introduction, two additional dimensions, namely supportive factors and selfevaluation have been added to the traditional (7S) model in order to better identify the readiness of ERPs [19]; [26]; [28] etc.Moreover, another ERP assessment model developed by Saremi et al. [25] also classified appropriate organizational readiness factors for assessment of ERPs (i.e.cultural, organizational, supportive, motivational and IT).Thus, in another model proposed by Razmi et al. [21] the readiness of ERPs is assessed in organizational, project and change management dimensions, while Nazemi and Naderi [16] offered strategic, operational and tactical levels.However, there is no agreement in the literature in respect of a clear relationship between selfassessment and the implementation failures of ERPs in HEIs.Unfortunately, the concept of accuracy in several studies is still confusingly determined by reference to the measurement of self-assessment.In this paper, accuracy is defined as the results of the absolute difference between the student-estimate and the ultimate tutor-estimate exam scores and is used to describe the student's self-estimation ability when applying an ERP, independently of the direction of this mistaken evaluation (i.e.over-and underestimation).
Our research represents an analysis of examinations taken at the University of Debrecen in Faculty of Economics and Faculty of Informatics focusing on business, economics and management, and business informatics students' self-assessment as regards their ability to apply SAP®.All in all, the main purpose of this study is to explore whether students can accurately predict and evaluate their results.In our hypothesis the high-achieving students are more accurate in self-assessing their performance.In this case we assume that the implementation of the ERPs appears to be a success in the HEI.We have also paid particular attention to disparities in gender and language.In the following sections the data available and the methods applied are also presented.Finally, we attempt to trace a brief conclusion from the results of our research, which will hopefully clarify debates on the contributions of ERP implementation in HEIs.
Based on the findings of Karnilowicz [5]; Kruger and Dunning [6]; and Kun [7] etc. we also assume that (H1) higher achieving students assess their examination results more accurately than their lower achieving fellows.Moreover, our study forms two additional sub-hypotheses: H11: Higher achieving students predict their examination results more accurately than their lower achieving colleagues.H12: Higher achieving students evaluate their examination results more accurately than their lower achieving colleagues.
According to Macdonald [11] and Sistrom et al. [26], there can be a gender gap in the direction of the student's self-estimation.In historical data Voyer and Voyer [30] showed that these gender differences favors females in all fields of study and the advantage in school marks has remained stable across the examined years.Meanwhile, Mitchell and Lusardi [11] found that women are significantly less likely to answer questions correctly, and to more accurately declare that they don't know the right answer.However, these disparities are not consistently reported, because there are several studies that could not identify gender related effects of self-estimation, such as Hobohm et al. [4] and Kruger and Dunning [6].In order to demonstrate whether the gender gap in terms of the students' accuracy exists or not, we are also assuming (H2) that female students tend to estimate their own performance less confidently compared to males.In this sense two sub-hypotheses are also formed, specifically: H21: Female students predict their examination results less accurately than their male colleagues.H22: Female students evaluate their examination results less accurately than their male colleagues.This paper also demonstrates the disparities of the student's self-assessments made by different installed languages.Both of the Foreign (international) and Hungarian students use the same clients and the SAP® Easy Access environment, but thanks to the pilot project our courses are taught in English to the Foreign and in Hungarian to the native students.Although, Foreign students officially use English, but some of them came from China, South-Korea and Turkey, etc. where English is not native.In order to compare the linguistic differences to apply SAP® we assume that the Foreign students tend to overestimate their own performance more than Hungarians (H3).H31: Foreign students predict their examination results less accurately than their Hungarian colleagues.
H32: Foreign students evaluate their examination results less accurately than their Hungarian colleagues.

Sample and methods
Our estimations are based on a sample of 102 Hungarian and 57 Foreign Bachelor (BA) and Masters (MA) students (84 men and 75 women) from the Faculty of Economics and the Faculty of Informatics at the University of Debrecen, Hungary.At the time of the examination 74 students were studying on the Business Informatics (BA), 28 on the Management and Controlling (MA) Hungarian majors and 57 in the Management and Business Administration (BA) International majors.Their compulsory Introduction to SAP course, and specialist optional SAP Applications I.The course provided the basic theoretical concepts of ERP systems and also an introduction to the SAP R/3 Systems, i.e. navigating in the graphic user interface (GUI); creating reports and defining jobs, the business workplace (BW) functions and some exercises with Global Bike Industries (GBI) in practice.After having completed these seminars, students will be able to understand the advantages of adopting ERP systems and they will be able to navigate through the SAP User Interface to work on the business processes and case studies on their own to establish their business decisions.The examinations (same tests) were carried out on two specific dates and language, one for the Hungarians and one for the foreign students in Hungarian and in English.Consequently, eliminating infrequent effects deriving from the linguistic differences among the majors, these factors are considered as dummy variables during our analyses.All midterm test versions had the same structure, the first part typically included true or false (T/F) and multiple-choice (MC) questions (one or more correct answers from four choices) and the second part contained mini cases (MINI).The T/F and multiple-choice questions counted for 10 points (each correct response was worth one point) and the mini cases (four mini cases tested the ability to apply SAP®) for 20 in the total test score; thus the maximum score was 30.
Before the students started their examinations they were asked to predict their results.To motivate them to predict more accurately, they were offered a higher percentage in a later test as a bonus if they could estimate well; specifically, 10 percent for a perfect hit for both types of questions, and 5 percent if the approximation was within a ±1 point range.After the tests had been completed, they were also requested to make their final estimation of the same test scores so as to correct the former prediction if they desired.Moreover, students were informed that only their second estimation was involved in the final evaluation process to determine bonus points.In this way, the pre-and post-examination assessments made it possible to research how students are able to reconsider their abilities to apply SAP® after the test has been completed.In order to exemplify the robustness check of our estimations, we measured the self-assessment by using various methodologies.In this paper, besides descriptive statistics, linear regression models and independent sample t-tests are frequently analyzed to highlight the distinctions among our evaluations.

Results
Before testing earlier declared hypotheses, in Table 1 we summarize and provide a brief overview of the ordinary descriptive statistics (minimum, maximum, means and standard deviation values) of preand post-examination self-estimations of students and the tutor-assigned test scores.

. Frequencies (%) of pre-estimated test scores by final grades with bonuses
Note: 1less than 50%, 2 -50-59%, 3 -60-69%, 4 -70-79%, 5more than 80% Figure 1 and 2 both show the frequencies of predicted and evaluated results by final grades with the achieved bonuses.We can assert that those students whose final test performance was higher than 50% can only achieve additional (5% and 10%) test bonuses at the pre-evaluation.Moreover, we also realized that after the exam 89% and 94% of the students whose final test performance was higher than 60%, were able to gain 10 and 5 percent bonuses.
In the following linear models (Equation 1 and 2) the dependent variables represent the accuracy of the students' preand post-estimations.
where ADIFTUTPRE is the absolute value of the difference between the student's pre-examination and tutor assigned estimation.FINALSC variable denotes the tutor-assigned final scores of the true or false and multiple choice questions.SEX equals to 1 if the student is female, 0 if male.LANGUAGE equals 1 if the student is Foreign, 0 if Hungarian.We also use a MAJOR dummy with 1 if the student is on the Master (Management and Controlling) major, 0 if not.
where ADIFTUTPOST represents the absolute value of the difference between the student's postexamination and tutor assigned estimation.
In our models the SEX, LANGUAGE, MAJOR dummies is selected to maximize the 'goodness of fit' of our estimations (R 2 , as the percentage of the response variable variation that is explained by a linear regression model).Consequently, the accuracy of is estimated independently in two model specifications.The first one (Model 1) contains the available independent variables and the other (Model 2) is restricted to those that are significant at least at the 10 percent p-level.Moreover, there are additional coefficients that are not included in our restricted models to evaluate the ability of how students can use ERPs.Hence, the validity of our conclusions is limited by the bias caused by the exclusion of certain of these variables.Note: Heteroscedasticity robust t-statistics are in parentheses.Letters in the upper index refer to significance: ***: significance at 1 percent, **: 5 percent, *: 10 percent.P-values without an index mean that the coefficient is not significant even at the 10 percent level.
Statistics of the regression models are shown in Table 2 and 3 for the pre-and post-test estimations.We found on both models a significant linear connection between the accuracy of students' predictions and tutor's assessments.Essentially, the effect of tutor-assigned final scores on the absolute value of the differences between self and tutor assessments does not seem to be large (ranged from -0.24 to -0.43), but the students' results correlated negatively with accuracy.Consequently, the higher achieving students seem to be able to predict and evaluate their examination results more accurately than their lower achieving fellows.Hence, we can also accept the H11 and H12 sub-hypotheses.
Although, gender (SEX) has a negative effect on accuracy in both models, there are no significant tstatistics we cannot claim that female students predict and evaluate their examination results more accurately than their male colleagues.Hence, the H21 and H22 hypotheses could not be accepted.Moreover, it can be only at post-estimation claimed that linguistic differences are in a significant positive relationship with accuracy in Mini cases and in the Total results.Consequently, ceteris paribus, the foreign students are less likely to predict their examination results accurately than their Hungarian after the exams.Note: Heteroscedasticity robust t-statistics are in parentheses.Letters in the upper index refer to significance: ***: significance at 1 percent, **: 5 percent, *: 10 percent.P-values without an index mean that the coefficient is not significant even at the 10 percent level.
Essentially, more demonstrative methods are needed to analyze the ways in which students estimate their ability to apply SAP® when dividing the sample by gender and language.Therefore, independent samples t-tests are used on pre-and post-examination assessments to determine which student groups tend to estimate their own knowledge more highly than the total tutor-assigned scores, and which do not.Regarding the difference between the students' self-estimation and the tutor assigned results by gender and language we can summarize the following results (see Table 4).
According to the t-tests, there are no significant differences between the two sexes.Hence, there is no gender gap in students' self-estimation of applying SAP®.Moreover, we found that the means of students' preand post-evaluated and tutor-assigned test scores vary significantly by language, as the t-probes indicated in both types of questions.Foreign students seem to overestimate and Hungarians subsequently underestimate their exam results.The average mean differences of Hungarian students at the pre-estimation were (-0.2), (-0.3) and (-0.33) at the T/F and MC, MINI Cases and Total scores.In the case of Foreign students they were (0.35), (0.54) and (0.59).After the exam, the average means of the Hungarian students were (-0.24), (-0.21) and (-0.27), and at the Foreign students were (0.43), (0.38) and (0.51).Consequently, these results validated our regression models, so Foreign students tend to predict and evaluate their results less accurately and also seem to overestimate their own pre-and post-examination performance to a greater extent than those who learned SAP® in Hungarian.In this case, we can claim that substantial differences can be found in self-estimation by language, and both of the H31 and the H32 hypothesis can be accepted.

Conclusion and discussions
Making appropriate decisions is especially critical for today's globalized business environment.These challenges, primary focus on ensuring better education and training to support companies and their clients.Not surprisingly, governments are currently interested in searching for effective solutions to improve the educational level of HEIs to provide up-to-date learning opportunities for their future voters.One of the main problems in investing ERPs is focusing only on the technical and financial aspects of a project and ignoring non-technical issues, such as applying them accurately, while several studies indicate that implementation failures are more likely to be caused by organizational and social issues than technical ones [13].
This research paper tried to explain the importance of self-assessment in ensuring the success of implementing ERPs in HEIs.In our opinion, studying the impacts of ERPs on user performance is a significant way to assess the utility of these applications.When an ERP system is fully realized in business organizations it produces significant benefits, such as improved access to accurate knowledge [24].Meanwhile, when ERPs implementation fails, one cause may be its inability to meet the expectations of its stakeholders.Therefore, providing information about the accuracy of selfassessment in HEIs should support to avoid any failure caused by mismatches between the implemented system and clients' expectations.
In this study the first objective was to analyze how students can estimate their examination results regarding their ability to apply SAP®.In our models the higher achieving students seemed to predict and evaluate their examination results more accurately than their lower achieving fellows.This is in accordance with the conclusions of Karnilowicz [5]; Kruger and Dunning [6]; Sundström [29] etc. Although, we assumed that female students were less likely to predict and evaluate their grades the results of our analysis cannot demonstrate this phenomenon.Consequently, despite the findings of literature [22]; [27], it cannot be disproved that there was a difference in the way the two genders selfassessed.
Finally, from another perspective, we also focused on language disparities, which can vary across specific socioeconomic groups.The results of this analysis provide evidence that substantial linguistic differences can be found in self-estimation.Foreign students obviously overestimated their own preand post-examination performance more than did the native Hungarians.In the literature Hobohm et al. [4] and Wallace et al. [31] supported the existence of such a phenomenon in HEIs.
Generally, these results highlight which student groups are predisposed to assess their end-user ability accurately, and to what extent the selected default language can contribute to increasing the efficiency of HEIs and enhance their preparedness to implement ERPs.Consequently, our results might allow directors and other professionals to understand why not only the technical issues or implementation processes, but accurate self-assessment and suitable didactics are also important in making business decisions about initiating pilot ERPs projects [17].
One of the main limitations of our study is that other important factors, such as leadership, staff training, organizational culture and communication, etc., which can have a direct impact on the success or failure of ERPs implementations [24], are not included in our model, due to the restricted access to data.Moreover, this pilot research was conducted within one university, in two faculties, and in one country, hence these findings would have a limited generalizability.It also appears that universal conclusions of ERPs implementation cannot be drawn only in respect of the accuracy of selfassessment in other HEIs.
As concluding remarks we should mention possible future research directions.Hopefully, it is worth conducting further analysis in the coming years to explore and expand the extent to which additional determinants and methods beside accuracy and native language may explain and make comparable the direction of assessment regarding the enhancing ability to apply ERPs in HEIs in different majors, ethnicities etc. in the long run.Moreover, even if differences were found, it should be interesting to justify how could these disparities provide helpful knowledge for either businesses or universities to reduce implementation failures?

Figure 1
Figure 1.Frequencies (%) of pre-estimated test scores by final grades with bonuses

Table 1 .
Descriptive statistics of pre-and post-estimated and real (tutor) test scores

Table 3 .
Results of the linear regression models for the post-estimation of self-assessment