Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages

The aim of this study was to investigate respondents’ satisfaction with smart device-based testing (SBT), as well as its convenience and advantages, in order to improve its implementation. The survey was conducted among 108 junior medical students at Kyungpook National University School of Medicine, Korea, who took a practice licensing examination using SBT in September 2015. The survey contained 28 items scored using a 5-point Likert scale. The items were divided into the following three categories: satisfaction with SBT administration, convenience of SBT features, and advantages of SBT compared to paper-and-pencil testing or computer-based testing. The reliability of the survey was 0.95. Of the three categories, the convenience of the SBT features received the highest mean (M) score (M= 3.75, standard deviation [SD]= 0.69), while the category of satisfaction with SBT received the lowest (M= 3.13, SD= 1.07). No statistically significant differences across these categories with respect to sex, age, or experience were observed. These results indicate that SBT was practical and effective to take and to administer.


Statistics
The results of the evaluation of SBT were summarized using descriptive statistics in SPSS for Windows ver. 23.0 (IBM Corp., Armonk, NY, USA). The responses regarding satisfaction, convenience, and advantages were compared according to sex, age, experience with CBT, and experience with SBT. Two-way repeated-measures analysis of variance test was conducted for a comparative analysis using DB-STAT ver. 5.0 (DBSTAT Co., Chuncheon, Korea).

Ethical approval
Informed consent was obtained from the subjects. The study design was approved by the institutional review board of Hallym University (HIRB-2015-092). Table 3 shows the descriptive statistics of the three categories and the two subcategories regarding the advantages of SBT. The category of the convenience of SBT received the highest mean (M) score (M= 3.75, standard deviation [SD] = 0.69). The next highest mean was that of the category assessing the advantages of SBT compared to pa-per-and-pencil testing (3.43). The lowest mean was that of the category evaluating satisfaction with SBT (M = 3.13, SD = 1.07). The means among the three categories were significantly different (P < 0.0001). Supplementary files of the coding books and integrated data for each category are presented in Supplement 1. Appendix 1 presents the results of all the items. The range of the 28 items was 2.83 to 4.23. Satisfaction with SBT was assessed using two items: satisfaction with using a tablet PC as a testing tool (M= 2.92, SD= 1.24) and lack of difficulty in carrying out the testing according to the advice for test-takers presented on the tablet PC (M = 3.13, SD = 1.26). The percentage of negative responses (strongly disagree and disagree) was higher than 30% for the former item; this implies that further improvement in SBT administration is necessary.
The category evaluating the convenience of SBT features was composed of 13 items. The range of the items in this category was 2.83 to 4.23. The highest mean score (M = 4.23, SD = 0.74) was found for the item assessing whether the function of identifying items that were not marked before submitting the answers was convenient, and the next highest mean score was found for the following item: 'it was convenient to have a 'check' function for later review' (M= 4.18, SD= 0.77). This indicates that students recognized that the features that enabled them to review and correct their answers were the most convenient. The lowest mean was for the item assessing whether it was convenient to use the functions of zoom in and zoom out when viewing an image or replaying a video file (M = 2.83, SD = 1.29). The next lowest mean was of the following item: 'the loading time was adequate before the start of video playback' (M = 3.25, SD = 1.29). Therefore, our results indicate that the features related to video need  to be improved.
The results for the advantages of SBT compared to paper-and-pencil testing were as follows: The range of mean scores was 2.91 to 3.89, indicating that the advantages of SBT were positively evaluated by the students. The highest mean was for the item assessing whether the absence of separate answer-card marking was helpful in managing the testing time (M= 3.89, SD= 1.06). The next highest mean was for the following item: 'it was convenient to select the correct answer on the tablet PC screen rather than marking an optical mark reader (OMR) answer card' (M= 3.85, SD= 1.06). The lowest mean was for the following item: 'tablet PC-based testing was more convenient than paper-and-pencil testing' (M= 2.91, SD= 1.09). The next lowest mean was for the following item: 'it was good to be able to see materials such as photographs, sounds, and video clips' (M= 3.07, SD= 1.09). The range of the advantages of SBT compared to CBT was 2.94 to 3.34. The highest mean was for the following item: 'there was no noise or heat from the desktop computer, so the examination environment was comfortable' (M = 3.34, SD = 1.02). The lowest mean was for the following item: 'the testing instrument was smart, and it helped me to focus more on the testing' (M= 2.94, SD= 1.14).
No statistically significant differences were observed in the mean scores for satisfaction with SBT, its convenience, and its advantages according to sex (P= 0.2979), age (P= 0.7126), experience with CBT (P= 0.5024), or experience with SBT (P= 0.2640) (Fig. 1).
SBT has also been administered in several other medical schools in Korea, where it has been known as ubiquitous-based testing (UBT). UBT was established to be a satisfactory and convenient testing tool in one of these medical schools [2]. In another medical school, students had a positive attitude towards the UBT interface. The factors of having experience with CBT and being male were associated with favorable responses towards UBT [3]. In this study, students thought that SBT could improve the effectiveness of taking a test, reduce extra work such as marking answers on an OMR sheet, and make the testing environment more pleasant than that in the computer lab. As the means of the items showed positive patterns for the satisfaction and convenience of SBT, most items had a high rate for 'agree' and 'strongly agree.' In conclusion, the results of the survey indicate a positive attitude toward SBT. However, the technical features of SBT, such as the zoom in and zoom out functions when viewing an image and playing video clips, could be improved further.