Using Professional Certification Criteria to Assess Occupational Safety Curricula in Degree Programs Investigating Accreditation

This paper demonstrates a novel assessment method developed to determine if the curriculum from two separate safety degree programs provided sufficient opportunity for students to obtain the knowledge required for professional practice in occupational safety. The method relies on the Board of Certified Safety Professionals (BCSP) examination blueprints. In the graduate program case study, over 88% of the BCSP criteria were met through an explicit means and up to 64% through assignments or better. Aggregating criteria into respective subject areas showed that the curriculum covered anywhere from 58% to 100% of the items within each BCSP topic. In the undergraduate case study, over 96% of the BCSP criteria through an explicit means, and 82.8% of knowledge items were assessed in assignments, exams or better. Aggregating criteria into respective subject areas showed that the curriculum covered anywhere from 75% to 100% of the items within each BCSP topic. Once briefed on the results, all faculty/instructors agreed that the approach helped identify strengths and weaknesses in their current curriculum. Most importantly, presentation of results acted as a catalyst for curricular discussions amongst the faculty that resulted in improvement priorities and a better understanding of student learning potential in course assignments.


Introduction
Since 2010, the American Society of Safety Engineers (ASSE), in cooperation with the Board of Certified Safety Professionals (BCSP), has increased their promotion of ABET-ASAC accreditation of occupation safety, safety and health, and environmental safety and health degree programs. Some advantages of increasing the number of accredited safety degree programs include, but are not limited to: more current curricula and curricula improved through data analysis, required faculty development, evidence for more labs and lab equipment, better educated graduates, and the graduate safety profession (GSP) credential granted to graduates. However, the ABET-ASAC accreditation requires a considerable financial investment by the host-university and time and resources from the degree program faculty. Unfortunately, due to the depressed U.S. economy and reduced state budgets, universities are being required to reduce costs, postpone hiring, and simply "get by" with what they currently have. Due to the higher unemployment, college graduates are having difficulty gaining employment in their chosen field and national news outlets are questioning the value of a 4-year baccalaureate or terminal master's degree.
Although the occupational safety and health field employment outlook is rather promising (NIOSH 2011), degree programs across the country to reflect on the quality, value, and efficiency of their degrees in the face of budget issues. Safety degree programs across the U.S. are taking efforts to maintain the quality of instruction while reducing costs included efforts to more efficiently utilize instructional resources (e.g., cross listing of courses, sharing electives between programs, restructuring curriculums to reduce dependence on outside instructors) and develop consistency in admissions standards and program prerequisites across graduate programs. Whether considering curriculum-and program-level changes or committing to the accreditation process, faculty members in safety degree programs are forced to ask themselves: Are we in a position to invest our limited time and the universities' limited resources to pursue program accreditation?

Background
Faced with increasing pressure to demonstrate degree value and justify the investment into pursuing degree program accreditation, department faculty from two separate safety degree programs (at two separate times) decided that an objective, externally based set of criteria was needed to validate the curriculum and provide evidence to support or argue against proposed curricular changes. The first program was a one-totwo year terminal master degree in environmental safety and health, located at a university in northern Minnesota. The other program offered both undergraduate and graduate degrees in occupational safety, located at a university in southern Wisconsin. The one commonality in these two program case studies was a single faculty member who worked at conducted the evaluations at both sites but at different times (approximately two years apart). ABET accreditation seemed a logical first step, especially given its critical role in the Graduate Safety Professional certification developed by the Board of Certified Safety Professionals for students in BCSP-approved graduate safety programs. But, as noted in the next section, ABET criteria for safety programs are also quite vague (ABET, 2008) and provide little specific guidance on curriculum content. Instead, the faculty needed to find some other set of criteria if they were to conduct an in-depth curriculum assessment. 34 T. W. Loushine and R. G. Feyen It should be noted that this assessment was not focused on an outcomes assessment. Although the curriculum assessment results would eventually be used to help assess how well its students were learning the subject matter being presented in the curriculum, the faculty was more interested in the fundamental questions of curriculum assessment mentioned above. After all, a key assumption underlying outcomes assessment is that a good outcome measure indicates effective learning that, in turn, positively correlates with the professional quality and competence of a program's graduates. However, what if a student learns a topic well, but the topic is irrelevant to practice? Or, what if a topic relevant to practice is only mentioned in the curriculum-or worse, not presented at all? Without proper curriculum assessment, outcomes assessment may reliably measure student learning, but runs the risk of being an invalid tool for assessing their professional quality and competency.
In terms of assessing student learning in a higher education setting, the general assessment cycle appears to follow these steps (e.g., Allen, 2004;Diamond 1998;Maki 2004): 1. Establish new or review existing learning goals for the program 2. Provide opportunities to achieve these goals 3. Assess if students are learning from these opportunities to meet the goals 4. Use information from assessments to adjust program (and thus competency) 5. Repeat As inferred from above, the researchers were interested primarily in step 2 of this sequence -yet the literature search yielded little specific information on how to select and assess these "opportunities." Rather, student learner outcomes were emphasized heavily while curricular content assessment received far less attention.
In this context, learning outcomes can be considered the foundation for driving programmatic changes, but at least for STEM-based programs, most outcomes are adapted directly from ABET criteria for accreditation and are accordingly vague (e.g., "an ability to communicate effectively"). Worth noting as well, although the authors would general use the terms 'outcomes' and 'objectives' interchangeably, ABET clearly defines the terms for assessment (ABET, 2008): Program educational objectives [emphasis added] are broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve.
Program outcomes [emphasis added] are narrower statements that describe what students are expected to know and be able to do by the time of graduation. These www.hlrcjournal.com Open Access Using professional certification criteria to assess occupational safety curricula… 35 relate to the skills, knowledge, and behaviors that students acquire in their matriculation through the program. (p. 1) Once again, the "skills, knowledge, and behaviors" are not defined in any way. In fact, it is the authors' understanding that most accreditation and certification boards leave it to program faculty to decide what skills, knowledge, and behaviors should be included in their program in order to meet their stated outcomes. The typical argument behind this is similar to the one expressed by the National Council of Examiners for Engineering and Surveying (NCEES) when discussing the use of their Fundamentals of Engineering licensure exam for outcomes assessment, "[t]he exam should not, for example, be used to determine the curricular content of any program. Its purpose is to test competency for licensure; it is not intended to force programs to be similar" (Barrett et al., 2010, p.2).
Specific to this program, the Educational Standards Committee of the American Society of Safety Engineers (ASSE) had worked with ABET in the mid-2000s to specify specific program criteria required to be in place if a program wanted ABET accreditation. Subsequently, this committee published guidelines for the broad topics to be included in a safety curriculum, but with a caveat similar to that expressed by the NCEES (ASSE, 2011): The committee did not want to provide a long list of required courses or topics areas that were common in previous safety curriculum criteria by the [Board of Certified Safety Professionals] and ABET. The committee believes strongly that programs should be provided flexibility […].
So, how do programs decide what skills, knowledge, and behaviors are needed? Anecdotally, most don't assess their curricula at that level of detail. But, in programs that conduct detailed evaluations, curriculum mapping appears to be the most common tool used to make this decision (e.g., Soulsby, 2006). This method requires identifying what students do in their courses and what the faculty expects them to learn (the skills, knowledge and behaviors) and then clarifying the relationship between the two, or "mapping the curriculum." This process reveals if a student's learning opportunities are linked or consistent with faculty expectations. Inconsistencies suggest places for curriculum improvement that bridge the gap between the two and, in turn, increase the likelihood of meeting program objectives. However, to identify the skills, knowledge, and behaviors needed by a student, common practice is to glean information from a program's stakeholders (e.g., faculty, administration, alumni, employers, funding agencies, peer programs, and professional societies), which can easily suffer from the same issue described earlier: everybody seems to provide a different opinion about what (they believe) students need to know when they graduate.
Under the advice of each case study's department faculty, the researchers went back to the ASSE Educational Standards Committee caveat mentioned prior. Although the ABET criteria had not proven to be helpful, the "long list" attributed to the Board of T. W. Loushine and R. G. Feyen Certified Safety Professionals (BCSP) turned out to be a promising lead. The BCSP has a primary mission similar to the NCEES: assess the professional competency of safety professionals via the Associate Safety Professional (ASP) and Certified Safety Professional (CSP) exams. Notably, certification and licensure agencies utilize recognized methodologies (e.g., ISO/IEC 17024 -Conformity Assessment) to ensure that their examinations test people on the activities, knowledge, and skills required in their profession. The key step in this process involves a job analysis of current practitioners. However, unlike the NCEES and many other licensure and certification agencies, the BCSP was very transparent in its exam development process, publishing highly-detailed "blueprints" outlining the skills and knowledge expected of a safety professional and from which the ASP and CSP exams were developed.
The exam blueprints were derived from a three-stage job analysis study of current safety professionals, targeting 1500 survey subjects, with respect to the skills and knowledge needed to perform the safety job in a professional, competent manner (BCSP, 2008b(BCSP, , 2008c. BCSP then categorized the resulting 249 knowledge items as either "foundation" (relevant to the ASP exam) or "advanced" (relevant to the CSP exam) and listed the 249 knowledge (and an additional 298 skill items) under a hierarchy of domains (e.g., risk management) and tasks (e.g., design effective methods to reduce or eliminate risk). Relevant to this initiative, the BCSP also undertook a generalized curriculum mapping effort, linking the skills and knowledge items with 15 "subject matter" topics typically taught in a safety program (BCSP, 2008d) -but provided no guidance on how to adapt this generalized curriculum map to a specific program. However, in a separate publication, one of the individuals involved in the original job analysis study did provide some guidance. Brauer (2005) not only described the job analysis survey and, in turn, suggested several ideas for using its results to assess a safety curriculum. With these two sources of information in hand, the program faculty now had an objectively derived set of skills, knowledge, and behaviors, as well as some ideas on how to assess the curriculum. All in all, the research efforts of BCSP created a comprehensive list of subject matter categories, (foundation and advanced) knowledge items, and skill items to use as a reference for the development of certification exams. The same certification exams that deem whether a person is a "certified safety professional" (CSP) or not. And it is this basic delineation that provided the basis for evaluating degree program curriculum based on evidence of course coverage and/or assignments.

Study Objectives
The overall goal of this study was to create and test a novel curriculum assessment method (based on 2004 BCSP Blueprints). Two types of safety degree programs (graduate and undergraduate), at different universities in the Upper-Midwest, were used in this study. The specific study objectives are: 1. What percent or count of knowledge items is covered in course lectures? www.hlrcjournal.com Open Access Using professional certification criteria to assess occupational safety curricula… 37 2. What percent or count of knowledge items is covered in course assignments? 3. Which knowledge items are covered most frequently? Or not covered at all? 4. Which topic areas need attention for faculty discussion of curricular changes?

Methods
Two Case Studies -Two Unique Safety Degree Programs in the Upper-Midwest The assessment tool derived from BCSP's 2008 Blueprints was first applied to a terminal master's degree program in 2009. The master's program had three core-faculty and four part-time or adjunct instructors, and generated approximately 10 graduates per year, all of whom earned full-time employment in safety after their capstone internship project. Based on lessons learned from the master's degree program assessment, an updated version of the assessment technique was applied to an undergraduate degree program in 2011 (improvements/changes noted below). The baccalaureate program had six core-faculty and six part-time or adjunct instructors, and generated approximately 40 graduates per year, most of whom gained full-time employment in safety after their capstone internship.
In the graduate program assessment, both recent graduates and faculty were involved in the selection of topics and knowledge items rated within their fourteen courses (ten required courses and four elective courses). In the undergraduate program assessment, only faculty were involved in the selection of topics/knowledge items rated within their 22 courses (11 required courses and 11 elective courses). In the undergraduate study, faculty were also asked these questions: 1. Do you dedicate a lecture (or portion of a lecture) to a discussion on ETHICS? 2. Do you assign at least one (individually graded) writing assignment (paper) of 10 pages or more? 3. Do you assign (on average) 20 or more pages of reading per week? 4. Do you assign group or team projects (either graded as a team or individually)? 5. Do you assign student presentations (either graded as a team or individually)? 6. Do you assign technical problem solving assignments (i.e. math-based problems)?
Preparation for course assessment interviews. In both case studies, the capstone internship was not included in the assessment, because each student's experience in that course was different. In addition, due to the sheer number of skill items and the difficulties in teaching and assessing skills in a traditional academic setting (most skill development occurs during actual practice, such as in an internship or after graduation, although lab experiences mitigate this to some degree), the faculty decided to exclude the 298 skill items identified by the BCSP from the assessment and focus exclusively on the 249 knowledge items. T. W. Loushine and R. G. Feyen Faculty recognized that evaluating each course against the full 249-item list would take too much time and effort, contributing to half-hearted responses or unwillingness to participate throughout the entire project. Therefore, the first step was to narrow down the list by deciding which knowledge items from the BCSP blueprints were relevant to each course, so only those relevant items would be rated for coverage and/or application in an assignment. The faculty also recognized, however, that the course instructors and students may differ in whether or not they perceive a knowledge item to be relevant for a course. In this context, a knowledge item's perceived relevance to a course could fall into one of three categories: "definite" (all respondents reported the item as relevant to the course), "likely" (at least half, but not all, of the respondents reported the item as relevant), and "possible" (less than half, but at least one, of the respondents reported the item as relevant). This suggests that, if all respondents report an item as relevant ("definitely"), a reasonable course expectation is that the item would be covered in some depth; but, if only one respondent lists the item as relevant ("possible"), minimal coverage of the item would be sufficient or even unnecessary. Accordingly, the perceived relevance of each knowledge item played a key role in the aggregate data analysis and in selecting recommendations for improving course coverage of a specific subject area or knowledge item.

Creation of Data Collection Instruments
The resulting methodology required two separate rounds of data collection, each with its own data collection instruments: one for selecting topic areas or knowledge items to be rated within each course and another for the customized interview with instructors to rate knowledge items based on evidence of coverage and/or student learning (i.e. exams, projects, papers, etc.).
The first round of data collection consisted of a cover letter describing the study and instructions for users to select the knowledge items (sorted by subject area) relevant to the specified courses based on the original presentation of the BCSP blueprints. The cover letter and Excel spreadsheet (containing the course description in one tab and all the BCSP knowledge items in a second tab) were sent to a group of students who recently completed the courses, the course instructor, and were also completed by the student research assistant (whom also recently completed all the courses).
The second round of data collection consisted of an interview with each course instructor using a customized datasheet based on the results of the first round of data collection. The customized interview datasheet had three sections: knowledge items with "definite" relevance to a course, knowledge items "likely" to be relevant to the course, and knowledge items "possibly" relevant to a course. Within each section, items were grouped by subject area and each item was preceded by a text box in which the item's coverage rating could be entered. www.hlrcjournal.com Open Access Using professional certification criteria to assess occupational safety curricula… 39 Data collection. The first round of data collection was driven by personalized email messages, with the cover letter language in the body of the e-mail and a selfadministered survey instrument as an Excel attachment. After the initial submission, several reminder e-mails were sent to all respondents encouraging participation (this was primarily for the student volunteers as all faculty had already agreed to participate and did so immediately). After a week, the student research assistant tallied the responses, and created customized interview spreadsheets for each course with the relevant knowledge items, grouped by subject area.
In the second round of data collection, the student research assistant helped the instructors rate the course coverage of each knowledge item listed for the course, regardless of relevance level. This entailed face-to-face interviews, conducted individually with each course instructor who had been instructed to be inquisitive and acquire anecdotal evidence supporting each knowledge item rating. Based on course instructor evidence of coverage and/or assignment (exam, project, paper, etc.), each knowledge item was rated on a scale of 0 through 5 (in the graduate case) or 0 through 7 (in the undergraduate case). The ratings were entered into a single master spreadsheet, with courses across the column headings by item priority area (four columns per course, three priority and "no rating") and the 249 knowledge items horizontally (by subject area) by row. Thoroughly covered in lecture. Projects, presentations, quizzes, tests or other tangible products were utilized to assess knowledge item. 4 Discussed extensively in lecture. Material related to the item was included in homework assignments or quizzes to assess the level of knowledge acquired.
3 Item was covered in the course and included in notes, slides, handouts, activities, etc.
However, students were neither tested nor asked to demonstrate their understanding of the item. 2 The knowledge item may not have been covered or discussed in lecture, but was included in assigned reading material.
1 Although possibly relevant, the item was not covered in any way in the course.

0
The knowledge item is not relevant to this course. Data analysis. After considerable trial and error, the following data analysis approach was selected as optimal for extracting reliable results from the master spreadsheet. The first step was to calculate the total number of course ratings given to each knowledge item, and then sorting the ratings by their perceived relevance (rating). Within each level of relevance, the ratings equal to 3 or higher (per knowledge item) were tallied. From the resulting table of rating frequencies, the total percent of knowledge items by coverage rating and relevance level were calculated to provide an estimate for the quality of knowledge item coverage across the program curriculum. The percentages for each subject matter area were then calculated (percent coverage and ratings of knowledge items within subject area).
The second step consisted of transferring the data from each subject area worksheet into a new table to evaluate the aggregated ratings for the associated knowledge items and identify subject area coverage by course. The calculation of total counts and percentages was similar to the methods used on the entire data set. For each subject area, a bar chart illustrated the percent of knowledge items covered adequately (rating of 3 or greater) and inadequately (rating of 2 or less). Because items not receiving any ratings ("not rated") were excluded from this analysis, the sum of the adequate and inadequate percentages within a course fell between zero and one-hundred percent. This allowed the knowledge items with low ratings to be easily identified visually and noted for later consideration. In addition to the bar charts, two matrices were created containing each course's percentage of adequate coverage and inadequate coverage for each knowledge item. Within each matrix, courses were listed on the horizontal (top) and the subject areas were listed on the vertical (left). The matrices were then populated with the respective coverage ratings and color-coded to indicate areas receiving either above or below a specified threshold. This helped identify which courses either covered a topic well or should be considered for material additions or structural changes to improve subject area coverage and/or type of assignment to potentially assess (in the future) student learning. The following sections describe in more detail how the resulting data was interpreted and pilot-tested for curricular improvements.

Case Study 1: Graduate Program -Individual Item Coverage Within Courses
Each individual course was rated based on items selected by faculty/alumni: 60 to 124 knowledge items out of a total of 249 items (Figure 1). Results of the entire curriculum assessment indicated that an individual course might cover anywhere from 5 to 26 "definitely" relevant topics, another 9 to 51 topics "likely" to be relevant to the course, and anywhere from 10 to 89 additional topics "possibly" relevant to the course.
Using professional certification criteria to assess occupational safety curricula… 41

Graduate Program -Individual Item Coverage Across the Curriculum
Of the total 249 knowledge items, only three items (business management software, Poisson distributions, and agricultural/food supply safety) were not identified for course rating in any of the 14 courses evaluated. On the other hand, 11 items were covered at some level in at least 10 of the 14 classes (including education and training methods, several types of administrative hazard controls, facility safety principles, and hazard identification); no items appeared in all 14 courses. For example, 13 courses covered some aspect of "Administrative controls: Written plans, procedures, and work practices." Results of coverage rating and analysis indicated this topic was considered definitely relevant to two courses, and those two courses covered the topic through a written assignment (coverage rating = 5), whereas the topic was considered likely relevant to six other courses and possibly relevant to five others. Of the six courses for which the topic was likely to be relevant, two were reported as having covered the topic with a paper assignment. Of the remaining five courses for which the item was possibly relevant, one course was reported as having covered the topic to the highest rating-but, exemplifying the pattern reported the end of the prior section, the instructor reported this while the students did not consider the topic as relevant.
Overall, further analysis (Table 3) suggests that the program delivered up to 88% of the BCSP knowledge items through an explicit means, such as lecture or other in-class activity, indicated by a quality of coverage rating of 3 or higher. Two-thirds of all items (64%) were thoroughly covered in the courses (rating = 5), indicating that these topics T. W. Loushine and R. G. Feyen were not only covered in class, but also through projects, tests, and other methods to help students demonstrate learning.

Graduate Program -Items That Need Faculty Attention
Aggregating the results into the respective subject matter areas showed that, within the each of the 15 subject matter areas, anywhere from 58% to 100% of the relevant knowledge items were explicitly covered in a course (i.e., quality of coverage rating equal to 3 or better). For example, the curriculum explicitly covered all the knowledge items in four subject areas (ergonomics, measurement/monitoring, organizational/behavioral sciences, and risk assessment/ management) but failed to adequately cover between 20-42% of the items in another four subject areas (business management principles, general sciences, EHS management and auditing systems, and security sciences; Table 4). Each of these latter subject areas was investigated further by Using professional certification criteria to assess occupational safety curricula… 43 identifying coverage (or lack of it) within individual courses for each knowledge item within a subject area. At this point, the analysis could also have explored knowledge items with coverage limited to lecture or assigned reading (rated 3). Although these items are covered, no evidence of student learning exists for them, so it is in the best interest of faculty to discuss these items (Table 5).  Each individual course was rated based on knowledge items within subject areas selected by the most recent instructor for that course: 3 to 15 subject areas or 11 to 186 knowledge items out of a total of 244 items (Figure 2). The lower number of knowledge items per course was the results of choosing subject areas and not individual knowledge items for rating a course. This made the pre-selection process quicker and easier, and reduced some confusion during the interview and in analysis. Another difference from case study 1 is that the rating scale went from 0 to 7 (instead of 0-5). Although the scale is not a truly validated scale, a rating of 6 or 7 would be considered consistently better than a 4 or 5 because when a student produces a paper or written project, the instructor can assess several forms of student learning. The fluctuation in number of items rated per course was the result of course instructor perception and the unique nature of certain elective courses (such as Safety 472 -Advanced Ergonomics). Undergraduate Program -Individual item coverage across the curriculum. Another improvement over the graduate case study was that all items were rated several times due to instructors choosing topic areas instead of individual knowledge items. Indepth analysis (Table 6) suggests that the program delivered up to 96.3% of the BCSP knowledge items through an explicit means, such as lecture or other in-class activity, indicated by a quality of coverage rating of 3 or higher. Over four-fifths of all items (84.8%) were thoroughly covered in the courses (rating = 4 to 7), indicating that these T. W. Loushine and R. G. Feyen topics were not only covered in class, but also through projects, tests, and other methods to help students demonstrate learning. *It was discovered during interviews that some items appeared to be very similar, so they were combined.
Undergraduate Program -Items that need faculty attention. Aggregating the results into the respective subject matter areas showed that, within the each of the 15 subject matter areas, anywhere from 75% to 100% of the relevant knowledge items were explicitly covered in a course (i.e., quality of coverage rating equal to 3 or better). The results were sorted down to individual knowledge item ratings to identify which items were rated below 3 (not covered in a lecture or reading; Table 7) or those that received no higher rating than 3 (only covered in lecture; Table 8). The topic areas under 100% and the individually identified knowledge items were identified and prioritized for discussion with faculty about what could be done to improve coverage and/or develop course assignments to evaluate student learning. www.hlrcjournal.com Open Access Using professional certification criteria to assess occupational safety curricula… 47

Discussion
As the results suggest, both degree programs demonstrated that their curriculum covered most of the BCSP knowledge items, with fairly well-defined areas for improvement. The analysis revealed and prioritized subject areas and knowledge items for faculty discussion about improving evidence of student learning. The discussion about individual course content and how student learning is assessed through assignments actually began in the interview process, because instructors had to provide evidence and explanation to support their rating of knowledge items. The overall results show "overlap" and "gaps" in the overall degree program that can be further analyzed and discussed by faculty to determine the best means to improve curriculum to conserve resources and time.
In the case of the graduate program, the internal auditing procedure revealed a possible tool for outcomes assessment, even though that was not the focus of the assessment. Consider the occasional disconnect reported between the instructor and the student respondents in terms of knowledge item relevance. Two possible reasons may explain this: one, perhaps an instructor may have planned to present on an item, yet for whatever reason, did not do so during the most recent semester. Or perhaps, students didn't learn the item sufficiently to understand its relevance to the course. Either way, instructors could assess the effectiveness of knowledge item coverage by comparing their perceptions of item relevance with student perceptions. Instructors could also use the information over multiple semesters to explore how the use of different topics or modes of presentation might fill gaps between their intentions and the students' actual exposure to the material.
With regards to the curriculum assessment and the information gleaned from the results, the program faculty felt strongly that this approach using professional certification criteria is promising, but could still use some fine-tuning. The biggest disadvantage is the time and effort required. On average, the surveys and interviews took about one to two hours per course, and each participant had to take their task seriously to provide accurate information.
One other major difficulty encountered was that the terminology used in the BCSP documentation (e.g., blueprints, curriculum mapping, etc.) was sometimes unclear and redundant. BCSP personnel were helpful, but often were not able to provide sufficient clarification on what was meant by a specific skill or knowledge item. Nonetheless, the consensus definitions may represent a concept differing from the knowledge item originally captured in the original BCSP job analysis study.
Despite these challenges, the approach clearly helped the program answer the questions raised earlier in terms of whether or not program graduates are exposed to the material they should know in order to practice as EHS professionals. This curriculum assessment methodology provided answers at several levels by providing baseline Using professional certification criteria to assess occupational safety curricula… 49 measurements of knowledge item coverage both within individual courses and in the overall program. Within courses, as hinted earlier, the level of item coverage-particularly for items identified as definitely or likely to be relevant to the course-can be easily examined and modified if necessary.
At the program level, the approach allows faculty to more effectively explore the impact of both temporary and permanent changes to the program curriculum-particularly in light of recent economic constraints. Using this approach provides program faculty with an opportunity to objectively evaluate the impact of adding or dropping classes from the curriculum, changing the frequency of course offerings, and whether a class should be required or can be offered as an elective. Further, in terms of program objectives, the faculty was able to seriously consider examining knowledge item coverage in terms of "foundation" and 'advanced" items to distinguish how well the program prepares graduates not only for professional practice, but either the ASP or the CSP certification exam.
Perhaps more importantly, the BCSP foundation provides a significant degree of objectivity to curriculum assessment. Rather than rely on feedback from numerous stakeholders in the program, each with different agendas and conflicting opinions, the knowledge items used in this approach are derived from a profession-wide job analysis study conducted in compliance with an accepted international standard (ISO/IEC 17024) and utilizing data collected in three stages, that targeted 1500 survey subjects from practicing EHS professionals (BCSP, 2008a). Regardless of academic institution, the vast majority of faculty would not be unable to perform a study of this depth for their program. This externally validated set of skills and knowledge items removes the discussion about the value or necessity of certain topics (e.g., between college administration and program faculty) from the realm of opinion.
Because BCSP has been open with publishing the skills and knowledge items sets derived from their job analysis studies, this approach can be readily adapted to any EHS program. In a move that should simplify this assessment methodology and help address the concern regarding interpretation of terminology within the knowledge items, the BCSP has since simplified its blueprints, reducing the number of domains and more clearly detailing the knowledge and skills areas within those domains (BCSP, 2009). Interestingly, the approach could also be adapted to other degree programs with a certification or licensing body such as in the engineering disciplines. For example, the NCEES regularly conducts PAK (professional activities and knowledge) studies for the various engineering disciplines as part of its exam development process (e.g., NCEES, 2011). However, NCEES does not make these studies readily available and, should an engineering program wish to utilize this approach, they would have to obtain access from NCEES to the latest PAK data for their discipline. T. W. Loushine and R. G. Feyen

Conclusions
Overall, reflection on this curriculum assessment process identified opportunities for improvement, including streamlining the data collection and analysis process and clarifying the meaning of individual BSCP knowledge and skill items. But, ultimately, by using the certification agency's job analysis data to indicate the knowledge needed by graduates of a safety program and developing a combined rankings and ratings methodology to assess coverage of this knowledge, the faculty was able to satisfactorily answer the initial question: Are we in a position to invest our limited time and the universities limited resources to pursue program accreditation? The authors believe that for both case studies, although changes need to be made and a decent amount of effort is required, that pursuit of ABET-ASAC accreditation be warranted. Other EHS-related programs could employ this new approach, and perhaps even other disciplines, to answer the same question and make fine-tuning improvements to the methods.

Limitations
The use of the BCSP blueprints assumed that subject areas and items encompassed the breadth of knowledge required of students graduating from an accredited degree program. This may not be the case, but an argument can be made that curricula that covers near 100% of material planned to be on a professional certification exam means that students of that degree program are at an advantage to passing the exam. And that those students, if they kept good lecture notes and/or created detailed course binders, would not have to invest in costly study materials or attend study review sessions. The rating scales used to assess how well a knowledge item was either covered in a course or potential to assess student learning was not validated. From a logical standpoint, the greater the time and intensity used to study or develop written projects, the more likely a student would both understand (and possibly apply) and remember that information. However, this is not true 100% of the time, and again, is only a theoretical assumption for these case studies. All in all, the faculty received an inexpensive review of their program curricula with recommendations on what they can do to improve topic coverage in courses and identify critical assignments in courses that demonstrate student learning outcomes, which are a key metric in ABET-ASAC accreditation reviews.