Testing and Validating DeLone and MacLean IS Model: ERP System Success in Higher Education Institutions of Pakistan

−The universities in Pakistan have recently started replacing their old legacy systems with ERP systems which are commonly used in business organizations to gain a competitive edge over competitors. The Higher Education Commission (HEC) has implemented ERPs systems at eight different universities of Pakistan. HEC has invested a huge amount of money to facilitate the integration, customization, and implementation of ERP systems in these universities. Previous studies have mainly focused on Critical Success Factors (CSFs) and risk factors of ERP systems. This study intends to empirically measure the support of ERP systems in teaching processes. DeLone and McLean model of Information Systems (IS) success is one of the most commonly used models cited in IS literature. In this study, DeLone and McLean model was applied at a University level analysis to access the impact of ERP in higher educational institutions and their support in the improvement of academic processes. Hypotheses were tested on the research model using empirical data collected from 230 respondents, including students and faculty at two selected universities, with the use of a questionnaire. The data were analyzed by structural equation modeling. The model was empirically tested and the findings showed that use and user satisfaction effected most net benefits. Information quality, system quality and service quality accounted for 42.6% effect in use. Whereas, information quality, system quality, service quality and use accounted for 46.5% variance in user satisfaction. Keywords-higher education; Enterprise Resource Planning (ERP) system; DeLone and McLean IS model


INTRODUCTION
The term ERP (Enterprise Resource Planning) was coined in early 1990s. An ERP is a multi-module application software package, which tightly integrates data and business processes for sharing information in an organization [1]. Higher educational institutions have implemented ERP systems to automate and customize their business processes, which include student records, admissions, student finances, and most common academic and administrative services [2]. ERP systems are implemented in higher education to improve the performance and the efficiency of the organization to cope with the changing environment. Despite that ERP promises to benefit the organizations and even though it is a substantial capital investment, it is not necessarily the case that all ERP systems implementations have successful outcomes. In response to that demand, higher education institutions in Pakistan have started implementing ERP systems to gain a competitive edge over the competitors and enhance the efficiency in the processes. ERP has automated and integrated the key processes related to academic and administrative services [3]. The Higher Education Commission (HEC) of Pakistan has invested a significant amount of funding to facilitate the customization and implementation of ERPs in eight selected universities (Dow University of Health Sciences, Karachi (DUHS), University of Engineering and Technology, Peshawar (UET), Quaid-e-Azam University, Islamabad (QAU), Islamia University, Bahawalpur (IUB), Balochistan University of Information Technology and Management Sciences, Quetta (BUITEMS), University of Punjab, Lahore (PU), Sukkur Institute of Business Administration (SIBA), and Institute of Business Administration (IBA), Karachi) as a pilot run in the country. Although there are many studies on the implementation of ERP systems in business, less research has been reported on the impact of ERP systems in higher education. In Pakistan, some small-scale research has focused on the implementation and efficiency issues of the ERP systems, which suggests that there might be a lack of research in the evaluation of ERP systems in the university environment [4,5]. The reported research in the contextual areas mainly identified the Critical Success Factors (CSFs) for the effective implementation of ERP systems [1,6]. The current research study intends to measure the post implementation impact of an ERP system in higher education institutions, focusing on the improvement in the academic and administrative processes [7].
The main reasons that Higher Education Institutions (HEIs) are implementing ERP systems are the needs to replace the legacy manual systems (traditional manual systems), to improve customer service, to increase the transparency in the processes, and to enhance operating efficiency [1,8]. The current study evaluates the impacts of ERP systems at the end user level where the actual benefits and impacts can be measured. The results of the study can be used by the HEC and HEIs to inform further expansion of the ERP systems to other Universities in Pakistan. In particular, this research aims is to explore the impact of ERP implementation in higher education institutions in Pakistan to improve the academic and administrative processes [5]. ERP implementation in HEIs is different from business organizations because they have different environments and conditions. The ERP systems are used by the HEIs to support faculty and students in the fulfillment of a variety of academic needs [5]. To create a high level impact of the research study, the results would be shared with higher education leadership for making decisions regarding the implementation of ERP in different HEIs of Pakistan in the future [9].
II. LITERATURE REVIEW An ERP system is the most widely accepted option for Universities that seek to gain a competitive edge and increase their efficiency and transparency through the integration of all information flowing through the different departments [10,11]. Despite this perceived positive impact of the ERP system in HEIs, implementation and integration is considered to be a costly and complex solution [12]. This has led to the need to evaluate the system from a stakeholder's and technical perspective [13]. Most previous research studies on similar topics are related to pre or post implementation issues related to the stakeholder's perspective or the technical input [14]. Few studies have reported on the CSFs related to the implementation of the ERP system life cycle, or on the academic advantage that the system implementation might foster. There is not a common agreement in which CSFs are most important, but most studies suggest that the issues related to top-management support, technical support, user training, education level of the users and organizational culture are important factors in a successful ERP system implementation [12,15,16].
There are a few studies that have reported the success ratio of the ERP systems implementation in higher education and analyze the factors and reasons of failure of such systems. Authors in [17] reported that the failure rate at the implementation stage is 40% to 60%. However different researchers have different definitions of failure. The critical failure factors reported in literature are high turnover of project team, over-reliance on heavy customizations, poor consultant effectiveness, poor IT infrastructure [10,17,19], and poor expected Return on Investment (ROI) [18]. The studies reported in the literature tested and validated the ERP success factors using different theoretical Information System (IS) models [12,15,16]. However, different theoretical models to measure IS success, have been reported such as: the Technology Acceptance Model (TAM), the Technology-Organization-Environment (TOE) framework, the DeLone and McLean model and the Diffusion of Innovation (DOI). Each IS success model has strengths and weaknesses. The research studies reported that IS success models were used on their own or sometimes in combination with others to measure ERP success in organizations.
The literature review did not find any research work that studied the impact of ERPs in HEIs with particular emphasis on the support on academic processes and higher education leadership decision-making. There are a few studies that measure the efficiency of the ERPs in the context of higher education institutions of Pakistan. Authors in [6] proposed a conceptual model based on few suggested changes in the D&M model. The research study also analyzed the success and failure of the ERP systems implemented in eight Universities in Pakistan based on four CSFs. Authors in [20] conducted a similar research study to evaluate the performance of ERP in Saudi universities. The results showed that system's quality and service quality have positive impact on the stakeholders' performance. Authors in [21] studied the impact of ERP systems in higher education. The study developed an IS successful model based on the key ideas of TAM, TTF, and D&M models. The findings indicated that system quality, task technology fit, and information quality are the key factors that lead to better end-user performance. Author [7] investigated the success of ERP in Pakistan with end-user perspective and identified CSFs such as top management support, effective project management, clear goals and objectives, selection of ERP systems and data accuracy. Authors in [22] measured the effectiveness of ERP systems in the corporate sector. The research combined the D&M and TAM models and the findings suggested that the performance of the employees can be enhanced through high quality ERP usage. It can be seen that there is a noticeable research gap in this area [20].

A. Research Model and Hypotheses
The DeLone and McLean IS success model is the most widely cited model in IS literature [23,24]. Authors in [25] stated that the model is important in IS success measurement because it provides a classification of all the evaluation measures that have been reported in IS success measurement in the literature. The model identifies potential stakeholders and suggests the way the different constructs interact with each other. The model proposes six dimensions: System Quality, Information Quality, Use, User Satisfaction, Individual Impact and Organizational Impact. The original model is shown in [26] (Figure 1), which depicts that system quality and information quality have direct effect on system usage and user satisfaction. This means that if the users are comfortable with the system functionality and quality of service they are likely to use the system. However, the proposed ERP success model is based on different IS theories in the context of ERP implementation in HEIs. The model is selected after extensive literature review on IS success models [27]. Many empirical studies have been reported regarding IS success measurement based on the D&M model [25,28].

B. IS Success Variable Categories
An IS is defined as a set of components that are integrated and interconnected to collect, process, store, and distribute information to support decision making and supervision [29,30]. ISs have six dimensions of success, ERP System Quality, ERP Information Quality, ERP Service Quality, ERP Use, ERP User Satisfaction, and ERP Net Benefits [26].

1) System Quality
In the proposed model, System Quality focuses on the ERP's characteristics System Quality is defined as a set of constructs related to the IS that determine its quality. These characteristics include: system reliability, ease to learn, ease of use, productivity, flexibility, and integration with other systems [26,29,31,32].

2) Information Quality
Information Quality is the degree to which information presents the required benefits. The information quality is measured as the output of the IS [30]. These constructs are refined from the literature and include, availability, timelines, relevant, accuracy, appropriateness, concise representation, and interpretability [26,29,31,32].

3) Service Quality
Service Quality is a set of characteristics related to services submitted by the IS to the customer which include service reliability, support, assurance, empathy, and security. Service Quality is considered as the overall support offered by the provider of the IS by ensuring that it can be applied in the proper way [26,29,31,32].

4) Use
Is the extent to which the end-users use the result presented by the ISs. The measures of the Use are defined from the literature and include the degree of system use, responsiveness, adaptability, and effectiveness [26,29,31,32].

5) User Satisfaction
User Satisfaction is the response of the recipient after using the IS [30]. It is associated with attitudes toward IS which include system availability, robustness, task achievement, productivity and efficiency [26,29,31,32].

6) Net Benefits
Net Benefits are the benefits obtained after using the IS. They may be individual or collective. They are a measure for the positive and negative impacts of the IS on all persons and groups effected with IS. Net benefits can be identified with a set of constructs that includes cost saving, expanded markets, incremental additional sale, reduced search cost and time saving [26,29,31,32].
The current study will look into the impact of the ERP systems and their components as suggested above, to provide support in academic processes and management in decisionmaking. Thus, the following nine hypotheses will be tested in the ERP context: • H1: There is a positive relationship between Information Quality and System Use.
• H2: There is a positive relationship between System Quality and System Use.
• H3: There is a positive relationship between Service Quality and System Use.
• H4: There is a positive relationship between Information Quality and User Satisfaction.
• H5: There is a positive relationship between System Quality and User Satisfaction.
• H6: There is a positive relationship between Service Quality and User Satisfaction.
• H7: There is a positive relationship between System Use and User Satisfaction.
• H8: There is a positive relationship between System Use and Perceived Net Benefits.
• H9: There is a positive relationship between User Satisfaction and Perceived Net Benefits. III. DATA COLLECTION A survey questionnaire was used to collect the quantitative data from the target population. The targeted population for the collection of data was students, faculty members, and the heads of the academic departments. The majority of the respondents were already ICT literate, so they were easily accessible through email correspondence. The questionnaires were designed using the online tool Survey Monkey and were sent to the respondents through an invitation email. Invitations were sent to 500 respondents (300 students and 200 faculty members). In response to that, 232 students and 141 faculty members filled the questionnaires giving a response rate of 74.6%. After data cleanup, 230 questionnaires were retained for data analysis.
IV. DATA ANALYSIS Structural equation modeling (SEM) was used to test the established relationships. For that, this study used the partial least squares (PLS) method by using Smart PLS 3.2.7 software [33]. This method utilizes the bootstrapping processes proposed in [34,35] to underline the level of significance for loadings and path coefficients for the established relationships. The PLS path modeling method is carried out in two steps (measurement model and structural model).

A. Demographic Information of the Respondents
The respondent's demographic information can be seen in Table I.

B. Measurement Model
Before testing the established relationships, the suitability of the measurement model was assessed based on reliability, convergent validity, and discriminant validity. The reliability was inspected with Composite Reliability (CR) values. Table II indicates that all scores relating to CR range from 0.809 to 0.885, which are higher than the criterion of 0.7, fulfilling the required level [34].  Moreover, using item loadings tested convergent validity and the AVE criteria. The results shown in Table II imply that the loading values of all the items are higher than the 0.5 threshold, indicating that the study has achieved scale validity. The values of AVE of two variables (Service Quality and Use) exceeded the recommended value (0.5) suggesting that convergent validity is at acceptance level. However, AVE values of three constructs (System Quality, Information Quality and Net Benefit) ranged from 0.46 to 0.483, which are near the threshold value of 0.5, whereas the AVE value of one construct (User Satisfaction) is 0.378. Table II shows the result of discriminant validity. It was examined using the square root of the average variance extracted criteria suggested in [36], according to which, the square root of the AVE should be greater than cross-correlations within construct and correlation with other constructs in the model. All the values of square root of AVE of all constructs except one (Net Benefit) shown in Table III surpassed the inter-construct correlations coefficient, suggesting that discriminant validity is acceptable.  With the achievement of adequate validity and reliability for the measurement model of the study, the proposed structural model with established relationships was estimated by applying 500 iterations of the bootstrapping technique. Table IV and Figure 2 show the results of the hypotheses testing. The results indicate that Information Quality had not significant impact on use (H1 rejected, ߚ=0.141, ܲ 0.1ሻ but had significant influence on User Satisfaction (H4 supported, ߚ=0.22, ܲ ൏ 0.05). Service Quality exerted significant impact on both Use and User Satisfaction at ܲ ൏ 0.01. Therefore, H3 and H6 were accepted ( ߚ =0. 36 Table V indicates that Information Quality, System Quality, Service Quality, Use, and User Satisfaction show significant total effects on the Net Benefits being all significant at P<0.01. Furthermore, the total effect of User Satisfaction on Net Benefits is stronger than Use. Likewise, among all quality related constructs, Service Quality shows strongest total effect (0.335) on Net Benefits. V. CONCLUSION AND FUTURE WORK The current study presents and validates a model of ERP system success based on an updated D&M IS success model [25] with six success measures that are Information Quality, System Quality, Service Quality, Use, User Satisfaction and Net Benefits. Except for the link between IS and Use and a link between System Quality and User Satisfaction all other hypothesized relationships between the six ERP success variables were significantly or marginally supported by the data [29]. The study recommends more research in this area and finding more variables that have impact on System Use and User Satisfaction. The current study has a few limitations that could be addressed in the future. First, measuring the impact of ERP systems in higher education with D&M IS success model is a new research area. The IS success model can be respecified with more variables related with the ERP context. Second, the empirical measurement sample size and the number of universities may be increased to make the findings more generalized. Third, future research may be done with longitudinal evidence that could enhance our understanding to replicate the current study with respect to time and long-term usage.

Section-B
Please rate to what extent you agree or disagree with each statement below by CLICKING the most appropriate number of the scales provided. Please click ONLY ONE option for each statement given below.

System Quality
Information quality is measured in terms of ease-of-use, functionality, reliability, flexibility, data quality, portability, etc.

2.
The CMS is effective in providing instant access to information required.

3.
The CMS System streamlines the best academic practices in the University.

4.
The CMS supports me in completing my academic related tasks efficiently.

5.
The CMS is effective in providing accurate information about student attendance and grades.

6.
The CMS provides complete attendance automation by effectively sending SMS and Email alerts to students.

Information Quality
Information quality is measured in terms of accuracy, timeliness, completeness, relevance, and consistency. 7.
The CMS system effectively provides me with relevant information.

8.
The information required is easily accessible in the CMS system.

9.
The CMS system is easy to learn.
10. The CMS user interface can easily be configured to my personal requirements.
11. I think that my data in the CMS is safe from unauthorized access.
12. The CMS effectively generates text and email alerts if someone tries to enter my account unauthorized.

Service Quality
Service quality is used to measures reliability, responsiveness, assurance, and empathy.
12. The CMS system provides reliable information.
13. The information in CMS is easily searchable.
14. The CMS provides a smooth flow of information regarding the academic calendar, assignments and announcements of results. 15. The CMS provides access to grades, attendance, fee status, exam schedule and announcements of results related to students.

Impact of ERP
16. 17. The CMS improves the academic processes in the University.
18. The CMS system effectively combines the student data from different departments of the University.
17. 19. The CMS provides me single login ID that improves the information integration between different systems of the University.

User Satisfaction
20. The CMS helps in sharing the scholarship related data of students with potential donors. 21. The CMS helps me in getting updated information about my attendance and grades on regular basis. 22. The CMS gives students a chance to provide course evaluations in each semester.
23. The feedback given by students in the course evaluations has helped to improve classroom teaching.
24. The CMS/ERP self-service automates many processes related with students (visibility of semester results, course registration and financial matters).
25. I received adequate CMS related trainings during my studies to perform my academic tasks on system effectively. 26. It is easy to get required support from IT support team whenever required.

Net Benefits
27. It is useful that the CMS manages classroom information and analytical reports of the student grades and attendance.
28. The CMS helps me to improve my academic performance by receiving SMS alerts on attendance and grades. 29. The CMS provides a chance to students to provide feedback on the courses offered in a semester. 30. The CMS improves the internal communication within the university (students, faculty and administration).

31.
The CMS effectively provides chance to register or drop in different courses in a semester through self-service module as compared from what we had before. 32. The CMS provides a chance to students to provide feedback at the end of a semester about their overall experience regarding classroom teaching/learning for courses offered in a semester. 33. The CMS provides grades related information, which may help students to monitor their progress during the semester. 34. Are there any additional comments about your experiences with CMS/ERP?