An assessment blueprint for the Advanced Medical Life Support two-day prehospital emergency medical services training program in the United States

Purpose: Traditional approaches to blueprint creation may focus on fine-grained detail at the expense of important foundational concepts. The purpose of this study was to develop a method for constructing an assessment blueprint to guide the creation of a new post-test for a two-day prehospital emergency medical services training program. Methods: In order to create the blueprint, we first determined the proportions of the total classroom and home-study minutes associated with the lower- and higher-order cognitive objectives of each chapter of the textbook and the two-day classroom activities during training courses conducted from January to April 2015. These proportions were then applied to a 50-question test structure in order to calculate the number of desired questions by chapter and content type. Results: Our blueprint called for the test to contain an almost even split of lower- and higher-order cognitive questions. One-best-answer multiple choice items and extended matching-type items were written to assess lower- and higher-order cognitive content, respectively. Conclusion: We report the first known application of an assessment blueprint to a prehospital professional development education program. Our approach to blueprint creation is computationally straightforward and could be easily adopted by a group of instructors with a basic understanding of lower- and higher-order cognitive constructs. By blueprinting at the chapter level, as we have done, item-writers should be more inclined to construct questions that focus on important central themes or procedures.


INTRODUCTION
The National Association of Emergency Medical Technicians launched its Advanced Medical Life Support (AMLS) education program in 1999. The AMLS program is now offered in 24 countries in Europe, Central and South America, and the Middle and Far East. The course emphasizes the use of scene size-up, history taking, interactive group discussion about potential treatment strategies, and physical examinations to systematically rule out and consider possibilities and probabilities in treating medical crises. This program employs an initial assessment-based approach that progresses to a diagnostic-based approach in order to quickly develop the best treatment plan for a given scenario. Learners must successfully complete a 50-question multiple-choice written examination as well as a scenario-based patient care exercise emphasizing patient assessment and the process of differential diagnosis. The written examination draws on content from both the textbook and scenario-based slide presentations [1]. All questions contained within the current 50-item set are structured as Type A items, which have one best answer. In order to expand the scope of the written examination to more fully focus upon the application of knowledge, we developed an assessment blueprint for the AMLS 16-hour two-day program and proposed revisions to the current written examination. Assessment blueprinting is a process of identifying and mapping test content against learning objectives in an effort to produce a valid examination [2]. The current written post-test examination for the AMLS program is several years old and has not undergone recent revision. The purpose of this project was to create an assessment blueprint to guide the creation of a new post-test for the AMLS program, thus ensuring an adequate sampling of course content. During this analytical process, new extended matching questions (EMQs) were designed to raise the level of cognitive domain assessment achieved by the post-test [5].

Materials and subjects
The source materials for this program consisted of the AMLS first edition textbook [1] and the associated instructor elec-tronic resource, which primarily consists of a series of scenarios used to support the practical stations. Table 1 summarizes how much time was allocated to the lectures and practical stations over the two-day course. This project was carried out from January 2015 to April 2015.

Development of an assessment blueprint
Based upon a review of the literature, we developed an assessment blueprint derived from previous research [3]. Both the tables in the aforementioned study and our blueprint struck a balance between computational complexity and specificity of purpose. Our method for categorizing cognitive objectives into lower-and higher-order objectives was derived from that of Bridge et al., who described objectives at the recall, interpretation, and problem-solving levels [4]. Our lower-order cognitive objectives corresponded to the recall level, while the higher-order cognitive objectives corresponded to a merged category consisting of the interpretation and problem-solving   levels. This structure is well-suited to the AMLS program content. For each chapter, the following calculations were performed: first, calculation of the total program minutes devoted to each chapter by source (text vs. classroom vs. practical exercises) and cognitive level (lower vs. higher) ( Table 2); second, calculation of the total and percent-estimated lower-order and higher-order program minutes for each chapter by cognitive level; and third, calculation of the number of lowerand higher-order cognitive items by chapter, based upon a 50item written examination.

Test item design
Existing multiple choice questions were reviewed for structural consistency with the criteria described by Case and Swanson [5], and EMQs were written to address higher-order cognitive objectives. Table 2 displays the structure of the blueprint. For example, in Chapter 9, out of the 30 total practice minutes, 1.5 minutes were estimated to have addressed lower-order cognitive information, while 28.5 minutes were estimated to have addressed higher-order cognitive information. The practice time predominantly addressed algorithmic diagnosis, which is considered higher-order reasoning by definition, making the estimate of 1.5 minutes devoted to lower-order cognitive reasoning reasonable. Since the testing format is dictated by the 16hour duration of the class and by practical and logistical considerations, a 50-item examination would be overwhelmed by the 111 course objectives. As described in the methods section, the blueprint was constructed based on the distribution of classroom time devoted to each chapter and a hypothetical estimate of time devoted to student preparation. The following estimations were made. First, with the exception of the items in Chapter 1, which were weighted at 30 minutes for lower-order items versus 30 minutes for higher-order items, the items in each chapter of the textbook were weighted as 12 minutes for lower-order items versus 48 minutes for higherorder items. Second, Chapter 1 was not specifically reviewed in the lectures or classroom practice. The direct study of Chapter 1 was limited to reading the text. Hence, classroom lecture and practice minutes were not allotted for this chapter. Third, in the classroom practical exercises, where the algorithmic diagnosis of medical conditions was emphasized, higher-order cognitive level items were allocated 156.5 minutes, while lower-order cognitive level items were allocated 13.5 minutes.

RESULTS
This blueprint calculation resulted in a distribution of 53 items. The three additional items resulted from rounding error. This is not an entirely unwelcome outcome. One approach to utilizing the 53 items is to consider them a test bank, which allows the instructor some slight discretion in adjusting the mix of questions to suit local preferences. While the calculated total of approximately 50 items was predetermined by the method of calculation, the nearly even mix of lower-and higherorder cognitive level items was an unpredicted outcome. One explanation is that the proportion of 80% lower-order cognitive level to 20% higher-order cognitive level items utilized to calculate the question mix in the textbook counterbalanced the strong emphasis on algorithmic processes present in the classroom materials. Our results indicated that the items for an appropriately constructed written examination should be approximately evenly split between the lower-order and higher-order cognitive domains. An example of an EMQ to address course content reflecting higher-order cognitive domains can be found in Appendix 1.

DISCUSSION
A significant finding of this study was that after the calculation of the assessment blueprint, the AMLS post-course exam questions needed to be enhanced by creating EMQs to assess the higher-order cognitive level. This article describes the first known published account of the application of an assessment blueprint for the Advanced Medical Life Support (AMLS) twoday prehospital emergency medical services training program. It also complements accounts of the introduction of blueprinting in other assessment settings [6]. While prehospital and medical board organizations administering high-stakes examinations utilize educational specialists or psychometricians to ensure the validity of their examinations, many smaller certifying bodies may lack the resources to do so, making this an important contribution. Roberts et al. noted that adequate resources are not always available even at the university level to ensure the valid and reliable assessment of clinical competence in the United Kingdom [7]. Building upon earlier approaches, we proposed a methodology especially suited to programs constrained by program duration relative to a large number of objectives and seemingly endless numbers of possible test questions [3,8]. Based upon the authors' experience, the 50-item multiple-choice exam is a fairly common fixture in prehospital professional development examinations. A 50-item exam is reasonably sized in light of the warnings of Notar et al. [9] regarding examination length, student attention span, and fatigue. However, the 50-item ceiling can be easily overwhelmed by a large number of learning objectives, potentially resulting in an examination constructed of a set of excessively fine-grained questions that ignore important central themes. Developing the blueprint at the chapter level, as we have done, should lead item writers to be more inclined to construct questions http://jeehp.org that focus on important central themes or procedures. Assessment blueprinting also offers advantages beyond its reported enhancement of content validity in examinations [2]. Ahmad et al. recently reported that their blueprinting methods, which were somewhat similar to our approach, led to improved student outcomes as measured by test scores and self-reported student satisfaction measures [8]. Furthermore, in light of the potential variability in the construction of assessment blueprints, we concur with these authors and recommend that each testing entity should consider the educational complexity and duration of their programs and courses and choose a blueprinting method best suited to ensuring content validity for each instance of testing.
In conclusion, our approach to blueprint creation, which included EMQs, is computationally straightforward and could be easily adopted by a group of instructors with a basic understanding of lower-and higher-order cognitive constructs. This approach should therefore make item writers more inclined to construct questions that focus on important central themes or procedures.

CONFLICT OF INTEREST
No potential conflict of interest relevant to this article was reported.

Item 1.
A 48-year-old female presents with increasing 4 of 10 dull ache in mid-chest for two days. She denies radiation, nausea, vomiting, and shortness of breath. The pain worsens when supine. Skin is warm and dry, breath sounds are equal and clear with patient complaint of some increased discomfort upon inspiration. Heart sounds are normal, there is no jugular vein distention, peripheral edema is absent and patient denies pain upon palpation of chest wall and abdomen. Twelve-lead EKG (Fig. 1) was recorded during exam. BP was 142/80, pulse 110, R-18 & non-labored, O2 saturation 94% on room air, weight 55 kg. She denies medical history. States has had a bad cold with cough for 5 days. She has been taking acetaminophen for discomfort and Over-the-counter (OTC) cold medication.

Item 2.
A 42-year-old male complains of 5 of 10 chest pain at a local restaurant. Pain continues for 20 minutes while eating. He denies radiation, nausea, vomiting and shortness of breath. The pain does not change with breathing or position. Skin is warm and dry, breath sounds are equal. Heart sounds are normal, there is no jugular vein distension, and peripheral edema is absent. He denies pain upon palpation of chest wall and abdomen. Daily meds are aspirin 81 mg, cimetidine 200 mg and s\Simvastatin 20 mg. History includes hypertension and cholesterol with GERD. The pain subsides after administration of 0.4 mg nitroglycerin SL × 2 doses. Twelve-lead EKG (Fig. 2) was recorded during exam. BP 174/88, pulse 75, R-14 & non-labored, O2 saturation 96% on room air, weight 55 kg. Item 3. A 62-year-old male is found sitting in a bedroom in a tripod position complaining of 8 of 10 shortness of breath (SOB). He was awakened from sleep with SOB, denies pain and radiation. SOB worsens when he attempts to lay supine. Skin is cool, pale and diaphoretic. Breathing is labored with accessory muscle use. Breath sounds are diminished in the bases with crackles and mild wheezing is heard to mid lobes and apices. Jugular vein distension is +. Pedal edema is 2+. His single daily med is hydrochlorothiazide. He states he has high blood pressure and was diagnosed with myocardial infarction three years ago. Twelve-lead EKG (Fig. 3) was recorded during exam. BP 190/110, pulse 110, R-36 & non-labored, O2 saturation 88% on room air, weight 85 kg. Answer: D -Congestive heart failure.