Varying levels of difficulty index of skills-test items randomly selected by examinees on the Korean emergency medical technician licensing examination

Purpose: The goal of this study was to characterize the difficulty index of the items in the skills test components of the class I and II Korean emergency medical technician licensing examination (KEMTLE), which requires examinees to select items randomly. Methods: The results of 1,309 class I KEMTLE examinations and 1,801 class II KEMTLE examinations in 2013 were subjected to analysis. Items from the basic and advanced skills test sections of the KEMTLE were compared to determine whether some were significantly more difficult than others. Results: In the class I KEMTLE, all 4 of the items on the basic skills test showed significant variation in difficulty index (P<0.01), as well as 4 of the 5 items on the advanced skills test (P<0.05). In the class II KEMTLE, 4 of the 5 items on the basic skills test showed significantly different difficulty index (P<0.01), as well as all 3 of the advanced skills test items (P<0.01). Conclusion: In the skills test components of the class I and II KEMTLE, the procedure in which examinees randomly select questions should be revised to require examinees to respond to a set of fixed items in order to improve the reliability of the national licensing examination.


Introduction
In Korea, there are 2 kinds of emergency medical technicians: class I and II. Class I emergency medical technicians are comparable to paramedics, while class II includes basic emergency medical technicians. Examinees of the class I Korean emergency medicine technician licensing examination (KE-MTLE) should be college or university graduates who have completed a 3-or 4-year emergency medicine technician course.
Graduates of the fire service academies, which have a six-month training period, are eligible to take the class II KEMTLE. When a class II emergency medical technician has acquired more than 3 years of experience in firefighting, he or she is eligible to take the class I KEMTLE. The KEMTLE is supervised by the National Health Personnel Licensing Examination Board (NHPLEB) of Korea. Both the class I and II KEMTLE are composed of written and skill-based tests. The skills tests of both examinations are composed of a basic skills test, an advanced skills test, and physical test items. Examinees randomly select 1 item each from the basic and advanced skills test item pools. The skills test for each examinee is scored by a grader according to a checklist. Table 1 presents the composition of the KEMTLE and the topics of the skills tests. Conhttp://jeehp.org J Educ Eval Health Prof 2016, 13: 5 • http://dx.doi.org/10.3352/jeehp.2016 cerns have been raised regarding the reliability of item selection by examinees. Administering the same test items to all examinees is ideal from the point of view of test reliability; however, this approach has not been applied to the KEMTLE due to limited time, human resources, and budget. This study aimed to determine whether the skills test items that examinees randomly select differ in difficulty, by analyzing examination data from the 2013 class I and II KEMTLE examinations provided by the NHPLEB.

Study design
This study involved the cross-sectional analysis of examination data without any personal identifiers.

Materials and subjects
Skills test data from 1,309 class I and 1,801 class II examinees from the 19th KEMTLE in 2013 were obtained from the NHPLEB. These data were used to compare the difficulty index of each item in the basic and advanced skills tests. The dif-ficulty index of each item was calculated as each examinee's scores divided by the maximum score; therefore, a high difficulty index means that examinees performed well on the question.

Statistical analysis
The difficulty index of items was compared using SPSS version 18.0 (SPSS Inc., Chicago, IL, USA). One-way analysis of variance and the t-test were used to compare the means. The cutoff for statistical significance was set at P < 0.05. Items that exhibited significant intergroup differences were further analyzed using the post-hoc Scheffe test.

Ethical approval
This study was conducted after receiving individual consent from all subjects, each of whom had previously received an explanation of the study's objectives.

Class I KEMTLE
The average difficulty index of the 4 basic skills test items was 0.87 (Table 2). Item D (the assessment of internal medicine patients) showed the highest difficulty index (0.91), while item A (endotracheal intubation) showed the lowest (0.85). The 4 items of the basic skills test significantly varied in terms of difficulty index (P < 0.01). Post-hoc testing indicated that the difficulty index of item D was different from that of all other items: A, B (automated external defibrillator therapy), and C (intravenous injection) (P < 0.01). The average difficulty index of 4 advanced skills items was 0.55. Item H (laryngeal tube intubation) showed the highest average score (0.56). Item I (electrocardiogram check and reading) showed the lowest item difficulty index (0.53). The 4 items of the advanced skills test showed significant differences in difficulty index (P < 0.05) ( Table 2). Post-hoc testing indicated that the difficulty index of item I was different than that of items E (infant airway ob-struction therapy), F (primary assessment of trauma), and G (traction splint application) (P < 0.01).

Class II KEMTLE
The average difficulty of the 5 items of the basic skills test was 0.77 (Table 3). Item J (automated external defibrillator therapy) was the least difficult (0.85), while item K (primary assessment of trauma) was the most difficult (0.73). The 4 items in the basic skills test showed significantly different levels of difficulty index (P < 0.01). Post-hoc testing indicated significant differences between items J and K, J and L (traction splint application), J and M (infant airway obstruction therapy), J and N (laryngeal tube intubation), and K and M in the basic skills test items (P < 0.05). The average difficulty index of the 3 advanced skills test items was 0.76. Item G (suction and    oxygen therapy) was the least difficult (0.84), while item O (infant cardiopulmonary resuscitation) was the most difficult (0.68). The 3 advanced skills test items significantly varied in difficulty index (P < 0.01). Post-hoc testing indicated significant differences among items O, P (suction and oxygen therapy), and Q (vacuum splint application) in the advanced skills test (P < 0.01) (Table 3).

Discussion
The KEMTLE permits examinees to randomly choose test items, which they then solve in front of the graders. This study demonstrated that the random selection of items by examinees requires reform, since differences in item difficulty index among items can affect examinees' total scores on the licensing examination. In our comparison of the difficulty index of skills test items, we made the assumption that the ability of the examinees who selected each item was evenly distributed. If an examinee selects a skills test item with a high item difficulty, he or she can achieve a better score. Ideally, implementing fixed items would solve this problem, as is already done on the national certification examination for emergency medicine technician-paramedics in the United States [2]. Out of the 25 national health personnel licensing examinations supervised by the NHPLEB of the Republic of Korea, the random selection of skills test items by examinees is present in two: the KE-MTLE and the Korean dental hygienist licensing examination. If a skills test center is established for the health personnel licensing examination, the present performance evaluation system allowing examinees to randomly select skills test items could easily be reformed. We believe that the Korean government should support the establishment of skills test centers that can be used for skills tests in a variety of health personnel licensing examinations. Fortunately, the city of Daegu announced that a health skills test training center would be constructed in the Daegu Innovation City Cluster by 2020, since the plans of this center passed the preliminary validity evaluation by the Ministry of Strategy and Finance of the Korean government (http://www.daegu.go.kr/Contents/Content.aspx?cid = 109:17 [cited 2016 Jan 1]). We hope that the construction of this center is completed in the near future and that it enters into active use. Improvement of the content of the KEMTLE is also required, as the current examination technique is simple and does not evaluate the problem-solving and comprehensive thinking skills that emergency medicine technicians must possess in emergency medical situations [3]. Previous research has also argued for the inclusion of integrated management questions in skills tests [1]. Measures for developing sophisticated yet simple administration formats for skills tests should therefore be studied. Recently, the importance of on-site skills test has been increasingly emphasized. Assessment methods using simulations have proven effective for performance-based evaluations [4]. Similarly, simulation-and scenario-based skills test items should be developed and used in national licensing examinations in order to adequately reflect the changing environment of emergency medical technician services.
In conclusion, a fixed set of items should be administered in order to overcome the aforementioned limitations posed by varying levels of difficulty index in the test items that are randomly selected by examinees.