The application of Usability Testing Method for Evaluating the New Student Acceptance (NSA) System

The new student acceptance (NSA) system is a design system to automate the selection of new student admissions, starting from the registration process, and the selection process to announce the results of online elections. It executes usability evaluation to determine the level of effectiveness, efficiency, and user satisfaction. This study focuses on testing the usability of the NSA system using usability testing methods on aspects of effectiveness, efficiency, and user satisfaction at SMAN 1 Pringgarata. This study is quantitative with a descriptive research method approach. The population in this study was 40 people and the number of samples in this study was eight people. Performance measurements, retrospective think-aloud, and questionnaire techniques are techniques used for data collection. The task scenario and System Usability Scale (SUS) questionnaire with a Likert scale is the instrument used. Data analysis used descriptive statistics, user success rates and Mann Whitney U-tests. The results showed that, (1) the NSA system is effective with value 98.5%, (2) it system is efficient with the value 6 over 0, (3) after implementing the NSA system, it satisfies users (80 >68). The recommendation given is the addition of images and icons on the main page of verification so that the pages on the system are more interesting and varied.


Introduction
The emergence of technology in the world forces its users to keep abreast of developments so they are always up-to-date. The current technological developments are so swift that it requires users to be more active in following the development of the information. The development of technology makes information not only through offline media but also through online media, such as information systems. It uses the information system in the online media in the form of a website to convey information widely. Information systems have a major influence in the development of technology, one of which is in the world of education.
The development of a system that provides electronic services to users organized by the Ministry of Education and Culture, namely the new student admission system (NSA) is proof of technological developments in the world of education. The NSA system is a design system to automate the selection of new student admissions, starting from the registration process, and the selection process to announce the results of online elections. The NSA system has been implemented in some educational institutions, including at SMAN 1 Pringgarata. Since implementing NSA system at SMAN 1 Pringgarata, everything done in school management has changed a lot from the data collection of new students to academic information. This makes it easy for the institution to conduct the selection process of new students, but in its use there are still problems namely errors in inputting data such as the discovery of some same data, the results of acceptance are not under applicable regulations. To improve the quality, profitability and service of the product (system), there needs to be action to ensure that the user will accept the website, it is important for the website to have better [1].
Based on these, it is necessary to have effectiveness, profitability, quality and satisfaction, usability evaluation which can assess and improve the products' usability, and also usability evaluation is an important element of systems development and software development [2]. Usability is a main concept in human-computer interaction (HCI). The ISO/IEC 25000 series of standards was developed to replace and extend ISO/IEC 9126and ISO/IEC 14598. The main goal of this update is to organize, enhance and unify concepts relevant to two main processes: software quality requirements, specification and systems and software quality evaluation [3]. Usability is typically defined as the "capability of being used", other words, and the capability of an entity to be used. Standard ISO 9241-11 has been successful in providing an internationally accepted definition about what is usability and its application in several velds [4]. Usability evaluation helps improve the predictability of user interactions with products, leading to greater productivity with fewer user errors and savings in development time and cost [1]. Usability is one of the important quality characteristics of software systems and products [5].
Several methods used to evaluate systems or software consist of: Model/Metrics Based, Inspection, Testing, and Inquiry [6]. In this study used testing and inquiry methods. Usability testing is one of the most used methods to define the level of usability of a software product [7]. Testing methods are used for the effectiveness and efficiency of the software or system [8]. In the testing method, researchers used the Performance Measurement technique to measure the effectiveness and efficiency of the NSA system. Meanwhile the inquiry model uses a questionnaire technique to measure user satisfaction in using a product or system that has been applied. Performance measurement is a technique used to obtain quantitative data about the performance of users who perform tasks during usability testing and then compare the processing time of the task to see the efficiency of the product or software that has been applied. Furthermore, the questionnaire technique is a technique used to measure the level of user satisfaction with the software or product used [6], [9]. Meanwhile, the level of effectiveness of the product is seen from the user's success rate or completion rate in completing the tasks given during the usability testing process [10]. Several researchers have previously carried out usability evaluations to determine the level of efficiency and effectiveness system or software [11], meanwhile, researchers who have conducted research to determine the level of user [9], [16].
Based on this, several researchers had previously evaluated the usability of systems or software that were applied or even developed in various fields of industry and education using several methods and techniques in various fields. Meanwhile, this research focuses on evaluating NSA systems using performance measurement techniques and technical questionnaires to see the level of efficiency, effectiveness, and user satisfaction.

Method
The type of this study is quantitative research using descriptive methods that focuses or is aimed at describing the phenomena that exist, which took place at the present or in the past. The location of this study was at SMAN 1 Pringgarata with a population of 40 people, while the number of samples in this study was eighth people. According to [17], in identifying problems that are carried out by more than five users will only repeat the same problem. Therefore, researchers used eight samples consisting of four novice users and four expert users to conduct the NSA system.
The technique of data collection is performance measurement and questionnaire. Performance measurement technique used to obtain quantitative data about the performance of users performing tasks during usability testing to see the efficiency and effectiveness of the system that has been implemented. The task scenario used to measure the performance of this system can be seen in table 1. Meanwhile, the questioner used to see user satisfaction with the software or product that has been applied using the System Usability Scale (SUS) [18]. In this study used the SUS questionnaire which amounts to 10 statements using Likert scale. Furthermore, data analysis techniques in this study used descriptive statistic techniques, user success rate and Mann Whitney U-test. Log-in User log-in with the username and password that have been provided.

2
Data Read The user reads the data displayed.

3
Data Verification Users read data details. 4 Export Data The user receives data

Effectiveness
The results of the analysis of the effectiveness of the NSA system using performance measurement techniques and measured by looking at the number of successes (success rates) that occur when respondents complete the task scenario during the usability testing process. These results show that the number of successes conducted by respondents in the novice group averaged 100%, even though there was 1 person who failed in completing the task, namely in task 3 (87.5%). Meanwhile, in the expert group, all users have been successful (100%) in completing all tasks in using or accessing this system, so there is no error when using the system. After an analysis by looking at the success rate gained by 98.5%, so the NSA system has been effective (see table 2). These results show that the adept group successfully completed the task with no errors. In assignments 1, 2, and 4, novice and expert respondents logged in, read data and export data. While in task 3 namely data verification, 1 respondent from the beginner group made a mistake. Based on the results of retrospective think aloud conducted through interviewing respondents in the beginner group who made mistakes on task 3, stated that there are two buttons on the verification page that are still not understood the usefulness of each button so that respondent 4 is mistaken and presses the verification button. Respondents in the advanced group and beginners who successfully complete the task are of the opinion that the NSA system has buttons that are simple and easy to use so that respondents state they do not need help from others in doing the assignment.

Efficiency
The following findings result from research conducted to usability evaluation of the new student acceptance system. We present the following table result from a study of system efficiency using performance measurement techniques by comparing the results of working time between expert users and novice users. Data analysis techniques used to determine the efficiency of the NSA system are descriptive statistics and the Mann Whitney U-Test. Descriptive statistic will describe the results of the average time difference between users when doing usability testing, both novice and expert. Meanwhile, Mann Whitney U Test is used to see the results of the hypothesis, if the value of Whitney U-test is more than (>) man Whitney table value, and then there is no time difference in working between the expert group and the novice group.  Table 3 shows the results of getting time working on task scenario tasks using the new student admission system. These results show that the novice group on user 1 completes the task in 12.1 seconds, user 2 completes the task in 11, 9 seconds, user 3 completes the task in 11.4, and user 4 completes the task in 12, 3 seconds. In the expert group, user 1 completes the task in 11.5 seconds, user 2 in 11.3, user 3 in 11.9, and user 4 completes the task in 12.1. The results show that the average time spent in working on both the novice and expert groups is the same, despite the differences, but not too significant. We analyzed the results of the task scenario scores using Mann Whitney U-Test statistics. The results got show that the value of the Whitney U-test is more than (>) the value of the Mann Whitney table, so there is no time difference in working between the advanced group and the novice group (6> 0) (see table 4). Therefore, it can the conclusion that the time spent working on assignments by beginner groups and advanced groups is not different, even though one respondent in the beginner group did not complete task 3, namely data verification. This happens because the respondent feels that there are two buttons on the verification page that are still not understood for use so that the respondent mistakenly and incorrectly presses the verification button which results in the individual time spent by the respondent 4 being longer than the other respondents.

Satisfaction
Data got from the usability evaluation process with questionnaire techniques is data in the form of subjective statements of participants about the level of user satisfaction in using the new student acceptance system. It gave the results of the scores got after to the respondents were an average of 80 (see table 5). The score got is below the standard SUS score. As stated by [11] if the value of the SUS score is 68 or more, the product considered being included in the normal category or eligible for use. Based on these results, we can conclude it that users feel satisfied with the NSA system. In addition, 73.80% respondents state that use this system application frequently; 69.98% respondents stated that they were very confident in using the system application; 48.78% respondents state that needed to learn to using system application; 69.79% respondents state that most people would learn to use this system application quickly; 45.09% respondents state that found the system application unnecessarily complex; 76.27% respondents stated that the website application is easy to use; 66.80% respondents stated that the system application is well integrated; 47.55% respondents stated that there is inconsistency on the system application; 72.12% respondents stated that the system application was very cumbersome; and 61.54% respondents state that need the support of a technical person to using system application (see figure.1).

Recommendation
Recommended improvements to the interface design of the NSA system using the Retrospective Think Aloud (RTA) technique combined with the SUS questionnaire. It carries this technique out to respondents after carrying out the performance measurement process by working on the task scenarios in the instrument. Based on the results of performance measurement, recommendations for improving the interface design on this system are necessary to clarify the coloring on the MS and TMS buttons, so that the data verification by the user no longer makes mistakes (errors). In addition, the need to add a cancel button on the main page of verification, and images and icons so that the user interface is more user friendly so that respondents feel satisfied and comfortable in using this NSA system.

Conclusion
Based on the results of research conducted on usability evaluation using performance measurement techniques and questionnaires on the new student acceptance system, the conclusion of this research is that there is a significant time difference in accessing the NSA system between novice user and expert user, where these results show that the value of the Mann Whitney U-test is more than (>) the value of the Mann Whitney table, then there is no time difference in working between the advanced group and the novice group (6> 0). So, they can say it that the application of the NSA system has been efficient. The results of testing the NSA system with the number of errors in the system got a success rate of 98.5%, so this system was effective in its use. Meanwhile, the SUS questionnaire results got a value of 80 which means more than a score of 68, so they can say it that users feel satisfied in using this system.