Evaluation of the Success of the Academic Information System (SIAMIK) with the DeLone and McLean Model

ABSTRACT


INTRODUCTION
The success of an organization is undeniable because of the use of information technology that helps work and business processes that run in decision making. Information technology has become the main choice in creating an information system for an organization that is strong and capable of giving birth to competitive advantages during increasingly fierce competition today [1]. The utilization of information systems has been carried out in various fields such as economics [2], management [3], e-commerce [4], health [5], government [6], and of course education [7]- [9]. An increase in the use of information systems increases the performance and quality of the information system. The quality of information systems is defined as perceived ease of use which is the level of how much computer technology is felt to be relatively easy to understand and use.
The quality of information systems shows that if information system users feel that using the system is easy, users do not require much effort to use it, so they will have more time to do other things that are likely to improve their overall performance. [10]. With the information system that has been used, of course, it is necessary to have an assessment and evaluation that becomes a measuring tool for the success of using information systems and also improvements.
Evaluation of information systems is one of the things that must be done to find out that the information system has been successfully implemented [11]. Evaluation of information systems is needed to determine the quality of system use. By managing the evaluation, it can be observed whether the system is successful enough from the user's point of view or whether the system needs to be improved to perform better, to fulfill the desired benefits with the implementation of the system. [12], [13].
Of course, an evaluation of the quality of the system that has been implemented is needed. Therefore, it is necessary to measure the success of an information system, one of which is the model provided by William H. DeLone and Ephraim R. McLean, which consists of six factors measuring the success of information systems [14]. This theory was then developed in 2003 by adding system quality, information quality, service quality, usage, user satisfaction and combining individual impact and organizational impact into net benefits [15].
University of Atma Jaya Makassar is one of the universities located in Makassar City which has implemented the use of the academic information system (SIAMIK) since 2004. In addition, it is following the Atma Jaya Makassar University Research Strategic Plan, one of which is the development of content-based information technology so that it very supports the development and improvement of information systems. SIAMIK is an academic information system designed to meet the minimum requirements needed in processing academic information. The functions of SIAMIK include filling out study plan cards (KRS) and issuing study cards, transcripts, and payment transactions. The users of this system are students, lecturers, the academic division, and the finance division [16].
SIAMIK is expected to be better, with the supervision and development of performance but until now there has never been an evaluation of the success of using SIAMIK at Atma Jaya University Makassar [17]. There is no information on whether the system that has been used is running effectively and efficiently for the required needs. Therefore, to determine the quality and increase the success of using SIAMIK, this study wants to analyze the level of success of using SIAMIK as measured by user benefits and satisfaction. The model used is the DeLone and McLean model because it is considered capable of explaining the systematic evaluation of the six measurement factors. The results of the evaluation are expected to be used as a guide to improve or optimize the SIAMIK function.

RESEARCH METHOD
In this study, we will evaluate the success of the Academic Information System (SIAMIK) with the DeLone and McLean model at Atma Jaya University Makassar which consists of six factors measuring the success of information systems by measuring system quality, information quality, service quality, usage, user satisfaction and combining individual impacts and organizational impacts into net benefits. The steps of this research are as follows.

Preparation
The preparation stage is the initial stage that has been planned to conduct this research. At this stage will conduct observations and monitoring related to system quality, information quality, service quality, usage, user satisfaction, and benefits. Furthermore, from the observations, the preparation and determination of the data to be used and collected are carried out. The results of observations and monitoring are adjusted to the hypotheses that have been previously made to be tested in this study [18], namely: H1: System quality has a positive influence on use H2: System quality has a positive influence on user satisfaction   H3: Information quality has a positive influence on use H4: Information quality has a positive influence on user satisfaction H5: Service quality has a positive influence on use H6: Service quality has a positive influence on user satisfaction H7: Use has a positive influence on user satisfaction H8: User satisfaction has a positive influence on use H9: Use has a positive effect on net benefits H10: User satisfaction has a positive effect on net benefits.

Data Collection
Data collection in this study was carried out in several ways such as observation, literature study, and questionnaires. Observations will of course be carried out at the Atma Jaya Makassar University, especially for SIAMIK users. The literature study will be obtained from related studies. As for the questionnaire, it will be distributed to SIAMIK users, namely students at Atma Jaya University Makassar. The student population is students who are active in Class 2018, 2019, 2020, and 2021 from a total of ± 1500 students. From the previously determined DeLone and McLean 2003 model, 26 indicators or research variables were used. These variables are further divided into two, namely the independent variable (independent) and the dependent variable (dependent). The three independent variables are information quality, system quality, and service quality. The three dependent or dependent variables are Use, User Satisfaction, and Net Benefits.

Data Analysis
At this stage, testing will be carried out on the results of the questionnaires that have been collected with validity and reliability tests. The analysis aims to determine how far the results of the SIAMIK evaluation have worked so far. After the validation test and reliability test, SEM (Structural Equation Modeling) analysis was performed. The conclusion of the research as a whole is based on the results of the analysis. The research tools used are SPSS and AMOS software that use SEM models. The steps for the stages of SEM analysis are as follows: 1. Development of a model based on theory, the path diagram for the success of SIAMIK is shown in Figure 3 Figure 3 above shows the initial model of the flow of each indicator that influences the criteria to be measured on system quality, information quality, service quality, usage, user satisfaction, and benefits. Each criterion has a relationship with one another as shown in Figure 2. The symbols e1 -e26 indicate the indicators obtained from the questions made in the questionnaire. The designed model is used to obtain Goodness of Fit Index results.

Data Validity Test Results
From the questionnaires collected as many as 1562 pieces. The basis for decisionmaking in the validity test is: 1. If the value of r count > r table, then the question items or questions in the questionnaire are significantly correlated with the total score (valid). 2. If the value of r count < r table, then the question items or questions in the questionnaire are not significantly correlated with the total score (invalid). In this study, a construct validity test was conducted using SPSS. Each variable is measured per indicator and compared with the r-table score. The value of r table for the significance of 0.05 (5%) and N = 1560 is 0.050.

Data Reliability Test Results
High reliability is indicated by the rxx (coefisien reability) value close to 1. The reliability test is considered satisfactory if 0.700. Each indicator will calculate its Cronbach alpha value which is then compared with the alpha value in a population of 1560 respondents, which is 0.050. In this study, all indicators are declared reliable because the value of the reliability test results is above the r table, which is 0.050, and is between 0.80 and 1.00 which states that the relationship between measuring instruments is very close.

Structural Equation Modeling (SEM) Analysis
To test the pre-determined hypothesis, a statistical test was performed with the calculation of Structural Equation Modeling (SEM) as quantitative analysis. Before conducting hypothesis testing analysis using SEM, several prerequisites must be met so that the SEM model can meet the goodness of fit criteria. The minimum sample is 5 times the number of manifest variables (indicators). In this study, the minimum number of samples was 26 (the number of indicators) multiplied by 5 so that the minimum sample was 130 people. The amount of data collected is 1560 data samples so it is declared sufficient.

Normality Test
In this test, the value of skewness and kurtosis of the data used is observed, if the value of CR on skewness and CR on kurtosis is between -2.58 and 2.58, then the research data used can be said to be normal.

Model Suitability Analysis (Goodness-of-fit)
To be able to proceed to the SEM stage, it is necessary to test the goodness of fit with good results. The first thing to do is to test the goodness of fit on the unmodified model. Based on the results of the goodness of fit index test shown in table 3, it is known that most of them fit the existing criteria, so it can be concluded that the model is in good condition (fit). However, there are still results that do not meet the criteria, including chisquare, probability, and CMIN/DF. Therefore, the goodness of fit test was carried out using a covariance-based SEM model.
Covariance analysis is a combination of regression analysis with analysis of variance. Covariance is used because it can show a valid comparison between different populations or different samples. The covariance method is used because it meets the methodological assumptions where the standard error shows a more accurate number [19]. The results of the goodness of fit test with the covariance-based SEM model can be seen in table 3.

Path Coefficient Analysis Path Analysis Structural Equation Modeling (SEM)
The model testing in SEM is carried out with two tests, namely the model suitability test and the causality significance test through the regression coefficient test. The path diagram used in this causality significance test is a covariance-based model. The use of this model is used without changing the basis of the research model and remains following the research framework. Hypothesis testing is done by analyzing the regression value. After fulfilling the requirements, the next test is the regression weight test [20]. This test is carried out in the same way as the t-test on the regression weight/factor loading coefficient of the model. The results of the regression test are shown in table 4. Hypothesis testing is done by analyzing the Critical Ratio (CR) and Probability (P) values generated from calculations by AMOS, compared with the required statistical limits, namely CR > 1.96 and P < 0.05 for the P value. If the data meet the requirements, then the hypothesis can be accepted [21]. By looking at the results in table 4, it can be concluded which hypotheses are rejected and which are accepted.
The results of the analysis show that the first hypothesis (H1) is that there is no effect of system quality on the use of SIAMIK. These results prove that the quality of the system is vital but does not become something important when used. The use of SIAMIK, which is mandatory for data management, has so far only been following the needs, so that the quality of the system does not directly play a role or affect the users, namely students. Hypothesis 2 (H2) describes that there is no effect of system quality on user satisfaction in using SIAMIK due to system reliability, and ease of learning, which is provided according to the needs needed.
Hypothesis 3 (H3) and hypothesis 4 (H4) indicate that the quality of information does not have a significant effect on use or user satisfaction, so both these hypotheses are rejected. The lack of quality information produced by SIAMIK is part of the relevance of inaccurate information, language that is not easy to understand, and presentation of information that is less attractive [22].
Meanwhile, in terms of service, hypothesis 5 (H5) makes service quality influence usage so this hypothesis is accepted. Hypothesis 6 (H6) describes the reliability of the service, the service does not affect user satisfaction in using SIAMIK. The reason for the quality of service does not significantly affect user satisfaction because students as users so far have not questioned the quality of service. After all, the SIAMIK used is still at a certain intensity.
Next are Hypothesis 7 (H7) and Hypothesis 8 (H8) which are proposed to prove the influence between usage and user satisfaction [23]. In a process, usage must come before user satisfaction. However, in a causal relationship, a positive user experience will increase user satisfaction, and increased user satisfaction will also increase the intensity of use, therefore both hypotheses are rejected.
Hypothesis 9 (H9) is that there is a positive effect on the use of benefits received due to the use of a system that requires KRS management and student study card checking through SIAMIK. The results of the analysis of hypothesis 9 are directly proportional to the results of the analysis of hypothesis 10. Hypothesis 10 (H10) is that there is a positive effect of user satisfaction on the benefits received. The reason that user satisfaction has a significant effect on benefits is that students as users feel the benefits of using SIAMIK online which greatly helps the academic process [24].
In his research, an information system is said to be successful if the information system is effective in the process and the results obtained are efficient or according to what users need, and in its development, the information system must be designed to increase user satisfaction. Based on the results of the analysis of the three hypotheses, it can be concluded that if an information system is developed and optimized, it can increase user satisfaction. With increasing user satisfaction, student interest in using information systems from time to time will increase. However, the user's bad experience when using the system will reduce user satisfaction. Therefore, for the future development of SIAMIK, it is necessary to revamp the system so that users get a positive experience. This research only focuses on how to measure the performance of the academic system (SIAMIK) at Atma Jaya University Makassar as a basis for future system development.

CONCLUSION
The evaluation of the success of the academic information system (SIAMIK) using the success of the DeLone and McLean information system tested 10 hypotheses that had been formed. From the results of the SEM analysis model on proving the hypothesis, there are 3 accepted hypotheses and 7 rejected hypotheses. Therefore, it can be concluded that the use of SIAMIK within the scope of Atma Jaya Makassar University has not been successful. The relationship between system quality and information quality greatly