Performance Evaluation IT Governance on Universities: COBIT 2019 Approach with Measurement Capability Levels

Abstract

is difficult to download data on MyUMN; downloading data takes a long time, causing delays in staff work when using the data, and the fourth problem is that it is challenging to customize MyUMN; the intended customization is to change and add features to support business processes as in the first problem. With the several issues mentioned, the MyUMN website needs to be evaluated, developed, and updated to give any issues a final solution and appropriately resolved. When a problem occurs, it will usually be conveyed to the IT team through the help desk so that the issue can be resolved. Still, when there is a problem in the CRM software, the marketing division does not communicate the problem to the IT division, causing the CRM software implementation to be imperfect because issues with the system are not immediately fixed and appropriately displayed.
If this problem is not fixed immediately, it will interfere with business processes and teaching and learning processes at Universitas Multimedia Nusantara. Therefore, UMN IT needs to be evaluated to measure the capability level using the COBIT 2019 framework. This capability level measurement aims to measure whether the solutions to each problem, change, and acceptance of IT in the UMN environment have been implemented and appropriately documented.

B. Research Method
This research will use descriptive quantitative methods to objectively analyze and describe a situation based on data in the form of numbers obtained during data collection through distributing questionnaires to the IT division of Universitas Multimedia Nusantara to measure the capability level of this institution's IT governance. Conduct literature studies by analyzing books, journals, and similar studies that have been done before. This research will use the COBIT 2019 framework to measure the capability level of IT governance at Universitas Multimedia Nusantara and provide recommendations for improvement from the research results [13]. Figure 1 represents the framework utilized for measuring governance capability based on a case study of a company using the COBIT 2019 approach.

Figure1. Research Method
1. Problem Identification: After determining the research object, identifies the problems at Universitas Multimedia Nusantara by conducting interviews, questions and answers, and discussions via email with UMN's IT division and UMN's marketing division. 2. Data Collection: The data collection process involves gathering data from various relevant sources for the research through interviews and literature [14]. The interview and questionnaire phases will be distributed to UMN's IT and marketing divisions. 3. COBIT 2019 Analysis Process: The subsequent step involves mapping the COBIT 2019 processes, which is accomplished by completing the COBIT 2019 design toolkit to determine which processes will be the focus of the research. 4. Data Analysis: Next, the data will be analyzed by distributing questionnaires and filling them out using a rating scale from 0% to 100% based on the current conditions within the institution. 5. Capability Level Measurement Stage: Subsequently, the process moves to measuring the level of capability. At this stage, the level of Information Technology governance capability is assessed based on the results of the COBIT 2019 design factor conducted in the previous step. The completed questionnaire responses will be processed into a final outcome, represented as a numerical value indicating the current institutional capability level [15]. 6. Gap Analysis: Following this gap analysis, After this gap analysis, calculate the level of capability that can be achieved by the UMN and the target capability from the COBIT 2019 design toolkit. Once the attained capability level (as-is) is determined, a comparison will be made with the expected capability level (to-be) to generate a gap analysis. 7. Recommendations and Improvements: Upon obtaining the gap analysis results, recommendations and improvements will be formulated to minimize the gap in capability levels. 8. Conclusions and Suggestions: The final step involves drawing conclusions and providing suggestions that can be considered for this research.

Capability Level Analysis
The analysis of the capability level or level of capability in this study was based on the interviewees' answers regarding the evaluation given to informants during interviews for all selected COBIT-19 functions. This assessment will consider the scale used to determine whether the COBIT-19 process stops or continues to the next level. The following is the scale used [16][17][18]: 1. N: Not Achieved (0 to 15%) We found little or no evidence-gaining scale related to the computed process attributes.
2. P: Partially Achieved (> 15% to 50%) There is some evidence of the scale of the estimated process attribute gain. Some attribute gains may be unpredictable. 3. L: Largely Achieved (> 50% to 85%) There was evidence of a systematic approach to scale and significant achievement of the calculated process attributes. Some of the weaknesses related to this attribute are that it is contained in the calculated process. 4. F: Fully Achieved (> 85% to 100%) Found complete evidence of a systematic approach scale and full achievement of the calculated process attributes.

C. Result and Discussion COBIT 2019 Process Objective Mapping
In carrying out measurements of IT governance, COBIT 2019 is used to measure each governance process at Universitas Multimedia Nusantara and then find out the parts that need performance improvement. COBIT 2019 plays a role in assisting organizations in developing business and implementing governance strategies in the company by providing suggestions and recommendations for following up on a problem faced. Then with the capability level in COBIT 2019, the company can find out the level of IT governance that is running. In the first step, an interview was conducted with the IT Manager to provide deeper information about IT governance. This interview aims to understand the company's goals and find out what problems are happening at the university related to information and technology based on the design criteria provided by COBIT 2019. The next step is analyzing design factors to determine the governance system (Design Factors 1-4) and improving the governance system design (Design Factors 5-11) per the COBIT 2019 design guide. The final step involves identifying potential improvements to the initial governance system. This process combines input from the previous phases to produce conclusions regarding the governance system design. The result produced at this stage is an aggregate score for each process on a scale of -100 to 100. The scoring is done using the COBIT 2019 Design Toolkit provided by COBIT. In COBIT 2019, all processes are assessed, but not all are considered critical [12] [19]. For this reason, a higher target capability level (level 4) has been set for processes considered very important, with a governance/management objective score of 75 or more. Governance/management objectives with a score of 50 or more require a level 3 capability level, while objectives with a score of 25 or more require a level 2 capability level. This study considers processes that score 75 or higher very important to the company. Following this procedure, companies can implement a governance system that suits their needs and characteristics.

Figure 2. Design Factor
After analyzing by determining objectives using the Design Factor (DF1-DF11), the final results of the process will be further evaluated. Multimedia Nusantara University gets three process objectives with an importance value of 75 or more, namely in BAI03 Managed Solutions Identification and Build, BAI06 Managed IT Changes, and BAI07 Managed IT Change Acceptance and Transitioning. These three processes have a target capability level of 4 because they have an importance of more than 75.

Capability Level Analysis and Measurement BAI03 Managed Solutions Identification and Build
The average calculation results for BAI03 are as follows: Tabel 1 shows the results of the calculation of all level 2 objective processes in BAI03. The calculation results show that the average result of BAI03 level 2 gets a value of 71.4%. With this result, UMN cannot continue at level 3 because it cannot reach 85%. This is because UMN is implementing a redundant system, UMN has not evenly updated the operational manual, and the test plan in UMN has not been implemented evenly. Rating in color Tabel 2 shows the results of Capability level BAI03 can only reach level 2. Thus, UMN cannot continue at level 3 because it cannot reach 85%. This is because UMN implements a redundant system, UMN has not updated the operational manual evenly, and the test plan in UMN has not been implemented evenly.

BAI06 Managed IT Changes
The average calculation results for BAI06 are as follows: Rating in color Tabel 4 shows the results of Capability level BAI06 can only reach level 2. Thus, UMN cannot continue at level 3 because it cannot reach 85%. This is because UMN has not prioritized documentation on the system that has been created.

BAI07 Managed IT Change Acceptance and Transitioning
The average calculation results for BAI07 are as follows:  Rating in color Tabel 6 shows the results of Capability level BAI07can only reach level 2. Thus, UMN cannot continue at level 3 because it cannot reach 85%. This is because UMN lacks communication between the IT department and the department concerned in converting data in a system, causing bugs or input output errors that cause the final result not to match the expected results.

Gap Analysis
The gap analysis will be carried out after processing the data results from the Questionnaire and getting the final score. The gap analysis will be carried out by comparing the results of the scores that the company can achieve at this time with the targets obtained from the COBIT 2019 design toolkit. Radar chart that displays 16 management practices to compare the score that the university can achieve with the target score.

Figure2. Radar Chart Score Capability.
Based on Figure 2, the gap is obtained in each management practice assessed by the Questionnaire. With this gap comparison, there is a need for recommendations for universities to increase the capability level according to the initial target.
The table below describes the COBIT 2019 process that has been assessed. Explained is related to the score results and the current level that has gone through the calculation process. Tabel 7 shows the results of the COBIT 2019 process that has been assessed, in the table there are score results along with the current level that has gone through the calculation process. The three processes can only reach level 2, with a score below 85% so they cannot continue to the next level.
After all selected processes are calculated, a comparison will then be made between the measurement results and the expected target capability level. In the mapping results using the 2019 COBIT design toolkit, the results of BAI03, BAI06, and BAI07 have an importance value of >75, so they require capability level 4. Tabel 8 shows that the results of the gap analysis. As-is is the capability level that UMN can currently achieve, To-be is UMN's target capability level. Gap is the result of the difference between To-be minus As-is. Based on the selected process, it appears that there are two gaps in each process to reach the target capability level of 4. Currently, the selected process can only reach level 2.

Recommendation for Improvement
Recommendations for improvement in table   Table 9. Level 3 Recommendations for BAI03

Process Management Practice
BAI03.02 Align the fulfillment of inputs and outputs required for implementation as solution criteria before moving to the production phase.
BAI03.05 Create documentation of major changes so that UMN IT has better and more systematic documentation of changes.

BAI03.07
Ensure that the timing of testing is different from that of other departments. Ensure that they do not overlap and hinder each other. Table 9 presents recommendations for enhancing the capability level 3 of objective BAI03. The recommendations encompass optimizing project execution and establishing a well-defined set of criteria to ensure alignment of inputs and outputs for successful implementation before proceeding to the production phase.

BAI03.02
Ensure that all implementations are evenly distributed by having redundancy, recovery, and backup systems developed in each work unit.

BAI03.05
Establish a documentation team and prioritize updating manual documentation. If needed, create a dedicated team or hire a new team to focus on business process and operations manual documentation.

BAI03.07
Ensure the testing process runs smoothly and is completed on time without interfering with other projects. Table 10 presents recommendations for enhancing the capability level 4 of objective BAI03. The provided recommendations include optimizing the implementation process and considering uniform implementation across all operational units by incorporating redundancy, recovery, and backup systems in each unit. -Confirm changes to users as a result of the implementation of change request projects. Table 11 presents recommendations for enhancing the capability level 3 of objective BAI06. The recommendations include ensuring consistency and effective testing, as well as prioritizing the creation of testing plans following established templates. This aids in standardizing the testing process, reducing errors, and improving the overall quality of testing procedures. Regularly review and update Standard Operating Procedures (SOPs) for application testing to align with best practices and industry standards. This ongoing evaluation ensures that the testing process remains efficient and up-to-date. Prior to implementing a test plan, schedule a special meeting with all UMN departments to discuss business continuity and disaster recovery and determine the proper documentation format in the event of an emergency. Form a team or hire staff responsible for maintaining internal IT SOPs, documenting application development, and creating and managing IT policies. Table 12 illustrates recommendations for enhancing the capability level 4 of objective BAI06. The recommendations encompass ensuring comprehensive readiness, and the UMN should adopt a proactive approach. Prior to executing the testing plan, convene specialized meetings involving all departments to address business continuity and disaster recovery strategies, collectively determining the standardized documentation format for emergency scenarios. -Make sure that before any changes or updates are made, that training or testing must be carried out first. -Provide confirmation to the user about the changes that will occur during the implementation of the change request project. -Regularly review the results of changes after they have been made to ensure they are stable and performing as expected.

BAI07.04
To ensure that the testing process runs smoothly, perform data cleansing to eliminate incomplete, irrelevant, and inaccurate data. Table 13 displays recommendations for enhancing the capability level 3 of objective BAI07. The recommendations include optimizing the change management process, and it is crucial to establish a systematic approach. Prior to implementing changes or updates, it is advised to conduct comprehensive training or testing to identify potential issues beforehand. Prior to any data conversion, UMN IT and the affected team should have a final meeting to align understanding and objectives to ensure each formula, entry, and exit definition is correct and will produce the expected result.

BAI07.04
Before creating a database for testing purposes, UMN IT and the team concerned should have a final meeting to unify understanding and goals so that each definition of formulas, inputs, and outputs can be ensured to be correct and can get the expected results. Table 14 presents recommendations for enhancing the capability level 4 of objective BAI07. For the smooth data conversion process, it is recommended that UMN IT collaborates closely with the relevant team by holding meetings prior to the conversion. These meetings should serve as a platform to align understanding and harmonize goals to ensure accuracy and anticipate outcomes of each formula, input, and output definition.

D. Conclusion
The measurement of IT governance at Universitas Multimedia Nusantara using the COBIT 2019 framework leads to the conclusion that three processes became the main focus of this research for evaluation, namely BAI03, BAI06, and BAI07. The calculated capability level results for Universitas Multimedia Nusantara are at level 2, whereas the target capability level based on the COBIT 2019 design toolkit is at level 4. Therefore, there exists a gap between the achievable capability level of the university and the 2-level target capability. These three processes can only attain a level 2 in terms of capability measurement. Consequently, the recommendations that can be offered are prioritizing the documentation of application development, ensuring that all change requests are periodically reviewed to prevent recurring requests or issues, and communicating with relevant users about change requests to facilitate a smooth implementation and improvement process that yields the desired outcomes.

E. Acknowledgment
The research has been carried out smoothly thanks to the help and support of the Universitas Multimedia Nusantara. Thank you the help and support that has been given during the writing of this article.