Development, Delivery and Evaluation of a Safety Training Program on Demolition for Blight Reduction ()
1. Introduction
1.1. Background
Blighted facilities showing visible signs of deterioration and decay are frequently encountered in the urban US landscape, raising concerns to residents and businesses in the neighborhoods, and challenging public officials to develop viable and sustainable solutions. Challenges associated with blight may grow and worsen over time due to neglect and disinvestment, leading to high rates of vacancy and poverty; economic, social and health disparities; and high crime rates and illicit drug activity, threatening public safety and welfare. In addition, blight is unsightly, and discarded materials from dilapidated buildings cause environmental concerns due to air, water and groundwater pollution. Furthermore, failure to effectively address and manage blight diminishes property values and thwarts business development, resulting in reduced tax revenues, directly limiting the municipalities’ ability to afford crucial public services. The ultimate outcome is urban sprawl, as people prefer to build new properties elsewhere rather than reinvest in the existing ones for repairs and rehabilitation.
According to Mallach [1] , empty houses and other abandoned industrial and commercial properties are disproportionately concentrated in many older industrial cities that have lost much of their population and employment base over time. Demolition of large numbers of such structures is the basic strategy adopted by these distressed cities to realize economic recovery and revitalization. Razing the blighted structures avails new land for investment and redevelopment that can return safety, stability, and improved quality of life to the neighborhoods. However, scarcity of resources has caused serious struggles.
With support from federal agencies (e.g., Department of Housing and Urban Development), philanthropic foundations and private industry, as well as in-kind assistance from community organizations and other sources, many cities have established blight reduction programs to bolster urban revival [2] . Demolishing the vacant blighted properties that are beyond renovation and/or repair is a central element of such programs. Among the many American cities, including Baltimore, Philadelphia, Cleveland, New Orleans and others, which have faced blight management challenges over the past few decades; the most publicized case has perhaps been the city of Detroit which has received national and international attention. Specifically, while the city was going through a major fiscal crisis with pending bankruptcy between 2011 and 2013, the federal government partnered with the mayor’s office, state agencies and community stakeholders to deliver a program of assistance by securing the necessary resources [3] . The community-based Blight Task Force that was constituted identified a large number of vacant properties that presented safety and environmental risks and estimated the abatement costs for the entire stock at the time to be in the vicinity of $2B.
Considered the largest blight abatement program in the world, a Demolition Department has been established within the city organization [4] to set guidelines and manage related contracts, while partnering with the Detroit Land Bank Authority which is involved in maintaining, selling, or demolishing the targeted properties. The program aims to run through 2025 and emphasizes rehabilitating as many salvageable properties as possible. Average demolition project costs have run around $20,000; however, the costs can vary depending on the size of the building and complexity of the job.
In terms of tangible benefits, there is some evidence that investing in blight reduction programs can pay off. Walkotten [5] reports that Detroit’s demolition efforts have resulted in increased home sale prices, and according to Larson et al. [6] who studied the violent, drug and property crime statistics for the same locations, total crime rates have come down. Similar information is reported by Annis et al. [7] on the favorable impacts of addressing vacant and blighted properties through focused demolition efforts in Ohio, where homes close to Cuyahoga Land Bank-owned properties sold at higher prices than comparable ones in the general vicinity. Several other case studies reported in the same publication reveal that improved economic development opportunities and higher property values have transpired from the blight reduction programs.
1.2. Problem Statement, Rationale and Approach
Although local governments have established ordinances and guidelines for implementation of demolition projects as part of blight abatement programs, it is fair to suggest that the main thrust of such programs has been ensuring public safety, health and welfare, rather than alleviating occupational safety and health hazards on actual demolition jobsites. Therefore, occupational safety and health training opportunities to specifically address the needs of the workers and other site personnel are largely absent within the blight reduction programs at the present time. However, gaining the necessary knowledge and skills through training on hazard awareness, and maintaining a safe job environment for employees are required by federal and state laws. A new program, such as the one described here, that is dedicated to the demolition of blighted properties was undertaken to facilitate regulatory compliance and bridge the existing gap. The one-year effort to realize this goal was funded by Michigan Occupational Safety and Health Administration (MIOSHA).
Underpinning this initiative was the fact that during the past few years, the compliance officers of the agency had been visiting blighted sites in Michigan, particularly around Detroit, where unsafe demolition activity was cited. Starting with an initial focus on asbestos abatement work, the scope of investigations was later extended into all demolition related violations [8] [9] .
The objective and scope of the grant effort reported herein was to develop a customized safety training program that can be incorporated in the implementation of blight-related demolition projects. The following questions were examined under this undertaking.
· What are the elements, processes and safety and health requirements of a typical blight removal project, as enforced by municipal codes, and mandated by the national and local standards?
· What are the demographic profiles of the demolition contractors and workers?
· What are the factors affecting demolition safety exposures, and how has this knowledge base been transferred into field applications and training?
· What would be a practical learning effectiveness evaluation framework for the training program?
· Specific research questions within the evaluation component were:
· What is the demographic profile of the trainless?
· Does the training given under the program improve participants’ overall safety knowledge on demolition of blighted properties?
· Are there statistically significant associations between the knowledge level afforded by this training and the trainees’ experience or union status?
· How can the content and delivery of the program be improved?
The approach taken to generate answers to the above questions included a comprehensive review of pertinent publications and online sources; conducting interviews with knowledgeable industry representatives (local practitioners, trainers, and inspectors); creating curricular objectives and content; designing training delivery tools and methods; devising evaluation instruments to assess learning effectiveness. In addition, feedback was continuously gathered from the stakeholders (sponsor, advisors, trainees) at all stages of the program and used to improve content and delivery. The information generated from these efforts, and the results of the evaluation component of this study are described and discussed in the following sections of this paper.
2. Review of Pertinent Literature
2.1. Municipal Requirements for Demolition
Demolition, in general terms, is defined as pulling down, razing, destroying or wrecking an entire building, or a portion of it, by pre-planned and controlled methods. A related term is deconstruction, which entails tearing a structure top down and salvaging materials such as lumber, doors, windows, light fixtures, sinks, copper pipes, and other items for reuse. Demolition can be done mechanically in a fast and cost-effective manner by hammers, excavators, bulldozers, and wrecking balls. In many cases, mechanical demolition and deconstruction are combined; however, this approach is costlier due to the additional time and labor involved in manual dismantling [10] .
A review of various municipal ordinance codes and procedural manuals [11] [12] [13] confirms that the main concern with respect to the demolition of derelict structures is the protection of the health, safety, and welfare of the general public. Unsafe structures are identified in the codes as those that have inadequate sanitation, light and air inside and consequently unfit for human use. In addition, such structures are likely to undergo total or partial collapse due to weakening by fire, flood, wind, or foundation settlement damage.
Typical demolition requirements are found in the Michigan NSP2 Consortium Policy and Procedures Manual [14] , covering the required permits, checking utility connections, site fills and compaction, asbestos, dust and debris control, safe disposal of demolition waste, finish grading and site seeding. The importance of site inspections is highlighted to ensure that demolition work gets properly carried out and all requirements are met.
Building codes of most states mandate permits be obtained before starting the demolition activity. Preceding the permit application, a professional inspector must perform checks on asbestos, lead paint, mold and other hazardous materials [10] . If the building contains such hazards, proper remedial action is needed. The process usually starts with the issuance of a blight violation, prompting a hearing with the property owner to determine whether the house is salvageable and repairable. If it is not, the next step is notification of utilities (electric, gas, water and sewer) to perform disconnects in compliance with municipal criteria. Once these tasks are completed, demolition work can proceed according to applicable specifications for walls, basement partitions, steps, paved areas and all other above-ground items. Another specification requirement is that basements and cellars and other areas below grade are filled and compacted with solid fill using approved soil, gravel, and demolition rubble. Any recovered salvageable items are recycled, or sold/donated for reuse, while discardable demolition waste is removed from the lot. A graphical illustration of the processes involved in the demolition of blighted structures is presented in Figure 1.
A literature search for blight related-demolition specifications and information showed that any job-specific safety and health rules or requirements are
![]()
Figure 1. Processes involved in the demolition of blighted structures.
lacking in such documents. There are all encompassing statements such as “demolition shall be done in strict accordance with all applicable laws, ordinances, regulations and codes of the jurisdiction in which work is to be performed”, presumably referring to compliance with OSHA or state branch regulations (e.g., MIOSHA). Among other pertinent statements encountered in the specifications, a typical example was: “the Contractor shall use all proper precautions to protect persons from injury”, which may be applicable to the protection of the demolition workers as well as the public [15] .
Based on this review, it is prudent to state that any reference to occupational safety and health requirements for demolition of blighted properties is quite incomplete and weak.
2.2. OSHA and MIOSHA Standards
Generally speaking, demolition encircles just about all of the hazards of general construction, while creating additional exposures due to unknown factors, such as materials hidden within structural members, or regulated hazardous substance containing materials like asbestos. In order to address these hazards and safety concerns, OSHA has promulgated the standard 29 CFR Part 1926-Subpart T, which covers preparatory operations; stairs, passageways, ladders; chutes and removal of materials through floor openings; removal of walls, masonry sections and chimneys; and ending with removal of steel construction, mechanical demolition, and selective demolition by explosives. Note that there are several safety and health standards not specifically covered in Subpart T but are referenced by OSHA to be additionally applicable to demolition. Some examples of such standards are 1926 Subpart CC—Cranes and Derricks in Construction; 1926 Subpart L—Scaffolds; 1926 Subpart X-Stairways and Ladders; 1926 Subpart P—Excavations; 1926.501—Fall Protection; 1926.1101—Asbestos; 1926.62—Lead; 1926.1153— Respirable Crystalline Silica; and 1926.350—Gas Welding and Cutting.
OSHA and MIOSHA standards are in most part similar; however, MIOSHA Part 20 features extra and more detailed requirements than OSHA Subpart T, which is always the case for the state plans. For general requirements, MIOSHA specifies a written engineering survey of the structure and equipment to be conducted by a Qualified Person to gather information on the condition of the foundation, roof, walls, and floors. Determining the potential effects of demolition on adjacent structures and utility services entering the building is also included in this task. The survey report, which additionally documents the hazardous substances and dangerous conditions, must be filed at the field office. MIOSHA also imposes specific requirements for material chutes and drops, among several other items. The focus on health exposures on blight removal jobsites is on asbestos, lead, silica, noise, and biological hazards, while falls, electrocutions, struck-by or caught-in-between events (e.g., falling items, trench cave-in) come under physical hazards. Both health and physical exposures are considered for confined spaces.
Although not covered by the above-mentioned regulations, workplace violence can be viewed as a physical hazard, as it poses a safety threat to workers as well as the public in and around blighted properties. Currently, two state plans under OSHA, New Mexico, and Washington, have issued violence-related safety standards; but they are primarily applicable to commercial establishments [16] [17] . A limited number of publications do exist on jobsite violence relevant to construction. For example, the key issues and recommended countermeasures are covered by Sabitoni [18] .
2.3. Profile of Demolition Contractors and Workers
Safety and health statistics specific to demolition contractors and workers are not published by the Bureau of Labor Statistics (BLS); demolition is embedded in site preparation work and placed under the NAISC code 23891 as a specialty contracting category. Statistics published by BLS for 2022 [19] for site preparation contractors indicate that there were 87 fatalities in 2022, and the total recordable incident rate was 1.9 per 100 full-time workers, which is nearly double the industry average. These figures included safety and health exposures related to falls and trips; harmful substances or environments; and contacts with objects and equipment, as well as fire and explosions. It is reasonable to think that all of these hazards can occur in demolition work.
BLS has not specifically defined the work responsibilities of a demolition worker. However, it offers information on construction laborers and helpers, stating that they perform many physically demanding tasks on construction sites, often working full-time. It is also mentioned that some of the construction laborers work at great heights or outdoors in all weather conditions, and they have one of the highest injury and illness rates of all occupations. Skilled demolition workers may take on more specific responsibilities, such as welding and cutting. According to the reviews of job applicant resumes performed by Zippia Recruiters [20] , these individuals’ self-reported qualifications may include knowledge of safety codes, prior work experience in demolishing and dismantling damaged property, removing hazardous materials, and disposal of debris and trash. Some of the applicant workers cited experience with using hand tools such as wrecking bars, sledgehammers, axes, and shovels for miscellaneous tasks. Those who have demolished smaller brick or wooden structures mentioned manual work using their hands.
While the laborers may frequently acquire their skills on the job, usually with little or no formal training, operating engineers, who run motorized heavy equipment for hoisting, lifting and transporting loads, as well as performing site grading and trenching for utility installation, acquire their skills through organized classroom and field training besides the on-the-job experience. Tasks assigned to operating engineers on demolition jobs may include breaking up concrete, masonry, and asphalt, and picking up and dumping debris on hauling trucks [21] .
2.4. Factors Affecting Demolition Safety Exposures
A literature survey was included in this study to capture information from research studies on demolition safety to gain an understanding of the nature of the risk elements, and insights into the causes of accidents. Safety measures needed for injury prevention and control were also examined. The results of the Google Scholar search conducted for this purpose are summarized in the following paragraphs.
Taking a broad perspective, Zahuriddin et al. [22] have referred to four stakeholders with their collective roles and responsibilities for avoidance of demolition accidents and their adverse consequences: workers, demolition companies/contractors, the construction industry, and government agencies. Discussing the roles and responsibilities of each of these parties, the investigators recommend that all of these stakeholders work together to improve the safety and health environment in demolition projects. Continuing, based on fatality and injury data from the UK, the authors observed that building collapse and falls from elevation lead all other causes of demolition mishaps. Further, the authors noted that while demolition work constitutes only a small fraction (3%) of all construction, it exhibits a very high rate of accident occurrence, yielding a ratio of 2.9 accidents per project. This places demolition workers at one of the highest risk levels in the construction industry.
Arthur-Aidoo et al. [23] studied the injury risk factors associated with demolition operations in terms of their frequency and impact and assessed the degree of hazards for different types of demolition. Responses to questionnaires prepared by the researchers were collected from 40 workers and professionals, and the data was statistically analysed. It was found that manual demolition by hand was the most dangerous method, and the fact that a majority of the workers had not received safety training prior to engaging in demolition elevated the risk.
A detailed analysis of demolition accidents was presented by Ertas and Erdogan [24] who studied 653 accident reports complied by OSHA between 1984 and 2012. Research results revealed that the most frequent cause of demolition fatalities was collapse of buildings (31%), followed by falling from heights (28%), falling objects, or flying debris (19%), construction equipment accidents (11%), electrocution (4%) and others (7%). In enumerating the precautionary measures, the authors highlighted the importance of worker qualifications acquired through proper training and emphasized that only workers with experience and training relevant to the type of demolition work should be operating demolition equipment.
Lastly, Bhuvaneswari et al. [25] came up with a detailed list of demolition risk events, including structural collapse, site congestion, hazardous materials, and noise. The investigators reiterated that demolition workers and equipment operators must go through proper job safety training to gain awareness of the potential hazards. Overall, the totality of the research findings condensed here converge on a strong consensus that safety training is a critical part of injury prevention efforts in demolition projects.
2.5. Construction Safety and Health Training Resources
Opportunities abound across the US (and abroad) on construction safety and health training offered by universities and colleges, trade associations, unions and apprenticeship schools, as well as private consultants. In addition, larger engineering and construction companies provide in-house safety training to their employees. On the government side, the OSHA Training Institute Education Centers (https://www.osha.gov/otiec/) are a well-known outlet for outreach training. Two of the more popular outreach training programs are OSHA-10 and OSHA-30, the latter being more extensive in coverage and taking longer to complete. Participants have to pay for these training programs. Although demolition can be included in these programs in the elective category, it would not be tailored to blight abatement projects. Susan Harwood training grants (https://www.osha.gov/harwoodgrants), on the other hand, can be customized for specific audiences, and are free to the trainees and their employers. Only one recent grant on demolition safety was identified under the Susan Harwood programs; however, it had no emphasis on blight remediation. Similarly, none of the demolition focused training presentations and documents encountered on the Internet were found to be to be directly applicable to blighted structures. It is therefore reasonable to think that while the existing resource materials covering general demolition safety are useful, they are not specifically suited for covering worker safety and health exposures in blight abatement projects.
The literature survey also showed that several references are available on best practices for developing and delivering effective safety training. These resources are very helpful in understanding the success factors for training and identifying the most viable training methods and techniques. Moreover, the importance of training to ensure a safe and healthy workplace for all employees is manifested in the existence of particular requirements written into many OSHA standards. These regulations not only mandate that safety training be provided as part of a sound safety program, but they also stipulate additional criteria relative to training content, method, and quality [26] .
The American National Standards Institute and American Society of Safety Engineers have jointly issued a training standard applicable to safety, health and environmental programs [27] . This standard covers all facets of training, including training development and delivery, as well as evaluation and management of training programs. In parallel, the Susan Harwood program has produced a detailed publication that provides information and recommendations on the best practices for developing and implementing safety and health training programs [28] . According to this source, the material presented must be clear, and understood by the intended audiences, which might necessitate delivery in multiple languages and using appropriate vocabulary. Competency of the trainer and the suitability of training venue, equipment and media are essential for program success, along with setting of learning objectives that are specific, measurable, achievable within a realistic time frame, and relevant to desired outcomes. Effectiveness evaluation of a training session should be based on trainees’ reactions to the content and method of the presentation. Obtaining meaningful feedback from the trainees on instructor performance and course quality, and suggestions about areas needing improvement are essential.
3. Training Program Development and Delivery
The work to be performed under this grant comprised of three components; design and production of the curriculum and training materials, development of training delivery methods, and creating a framework for learning effectiveness evaluation. Details are presented in the following paragraphs.
3.1. Design of Curriculum and Training Materials
This task started with a literature survey to establish the state-of-the-art on demolition principles and practices. Concurrently, a group of local safety and training experts was assembled to constitute a Grant Advisory Committee (GAC) with the goal of providing guidance and assistance to the project team. The input received from this group was integrated with the information extracted from the literature to create a curriculum that would be relevant to the needs of the industry. The training module outline presented in Table 1 was formed through iterative discussions with the advisors, after which a draft.
PowerPoint training module was prepared and circulated back for final review. In addition, a set of test questions was prepared to gauge the trainees’ knowledge gain as a result of training, and an opinion survey was added to capture the trainees’ reaction to the efficacy of our program. The entire set of these materials was reviewed by GAC before their submission to MIOSHA for further feedback and approval.
The learning objectives of the training module were grouped into four categories, namely, background information, physical hazards, health exposures, and hazard prevention and mitigation using the hierarchy of controls principle. The background information highlighted the issues involved in blight removal programs for urban areas and the central role played by demolition in blight abatement. Gaining familiarity with demolition work activities and acquiring knowledge of the essential requirements for safe operations were also a part of this endeavor.
The specific learning objectives for physical hazards were formulated as: a) attaining knowledge on the elements of engineering surveys performed in the planning and pre-demolition stages to prevent structural collapse and identify hazardous materials; b) gaining awareness of the physical hazards, including falls/falling objects, struck-by, caught-in-and-between, and electrocution events; c) learning about the safety aspects of hand and power tools, excavation and trenching, fire prevention, mobile construction equipment and traffic control,
plus good housekeeping; and d) being informed about workplace violence.
The learning objectives regarding health exposures were: a) familiarizing with the essential elements of asbestos, lead and silica hazard awareness and management; b) becoming knowledgeable about hazardous atmospheres with a focus of safe entry to confined spaces; c) understanding the harmful effects of excessive construction noise and exposure controls; and d) obtaining information on avoiding biohazards (pathogens) on jobsites, that are traceable to infectious viruses, bacteria, insects, animals, and plants, as well as trash, human waste, needles and other sharp objects on site.
Developing a good understanding of the hierarchy of controls principle and its application to hazard control was the final learning objective. The pyramid model was introduced for this purpose (https://www.osha.gov/safety-management/hazard-prevention) and the hazard control steps and processes were explained with examples. According to this model, the most desirable control method is physical elimination or removal of the hazard (e.g., enclosure of noisy equipment). Next comes substitution of a critical hazard with a less critical one (e.g., using mechanical instead of manual demolition). Hazard isolation, exemplified by machine guarding or welding curtain, is another well-known engineering control that has proven to be effective. Administrative controls like changing how the people work (e.g., rotating work locations) also provide protection against certain hazards, such as exposure to extreme temperatures. Finally, the hierarchy of controls principle considers worker protection by wearing personal protective equipment (PPE), while realizing that it is a last resort; this means relying on PPE only if the adopted engineering and administrative controls fail.
It should be noted that the targeted topics embedded in the learning objectives have considerable conformity with the fatality exposure data presented by Ertas and Erdogan [24] covering building collapse, falls and falling objects, machine/ equipment accidents, and electrocutions. As a matter of fact, these causal events also demonstrate substantial overlap with OSHA’s Focus Four hazards [29] , which account for a very large fraction of construction fatalities on an annual basis. Clearly, the physical hazards included in our training module also concentrate on the same four fatal exposures. Further, hazardous material related violations in demolition projects (e.g. asbestos) are frequently cited by OSHA and are a topic of strong focus in our training. It must be understood, however, that tracking the effects of hazardous materials on related illnesses and fatalities is difficult because these effects are latent in nature, often becoming detectable after a long period.
3.2. Training Delivery and Evaluation
The content of the PowerPoint training module was driven by the learning objectives. The module employed multimedia presentation, with text, photos and graphics. The training was delivered both in traditional lecture format, or online via an instructor-led webinar. An independent self-paced mode of delivery was also available through an online portal designed and developed for this purpose. The flow diagram of a typical training session illustrated in Figure 2 is applicable to all three scenarios.
As shown in the figure, the session starts with sign-up. Next, trainees take a multiple-choice pre-test covering the entire subject matter. The purpose of this test is to establish a baseline for the trainee’s level of knowledge prior to taking the training. The PowerPoint lecture follows with presentation of the training content. In the process, one or more post-tests are given, typically one at approximately the mid-point break, and another at the end of the lecture. These tests ascertain how well the learning objectives are satisfied. Out of the 20 questions included in the pretest, twelve questions are on general demolition operational matters and standards, while eight questions are on specific hazards and their mitigation. The post-test questions are the same as those comprising the pre-test. For multiple post-tests, the results from each are combined into a single
![]()
Figure 2. Flow diagram of training session.
total score. Comparisons between the post-test and pre-test scores are used to quantify knowledge gain. Hard copies of the tests were used in the traditional instructor-led sessions, while the Google Forms software (forms.google.com) were employed when providing online delivery. After all the tests are graded, each trainee receives a certificate of completion if the final score is 70 percent or higher; if the grade is below 70, a certificate of participation is awarded. While the certificates are mailed to the participants in hard copy after in-class sessions, they can be downloaded when training is taken online.
The opinion survey designed to provide the trainees an opportunity for offering feedback is the final step. The survey questions are listed in Table 2, along with trainee response summaries. Open-ended comments are also solicited from the participants to obtain their critique and suggestions on training program effectiveness in terms of content and delivery.
In finishing out the session, the instructor goes through the correct answers to the post-test questions, answers the trainees’ questions, and adjourns the session. Unfortunately, this step is not available for self-paced training, which is considered a disadvantage. On the other hand, grading is automated in this delivery format, and the trainees can download their own certificates using their devices before signing out.
3.3. Evaluation Results
A total of 118 people were trained through our program over the grant year, through a combination of webinar and self-paced training. No traditional lecture format could be implemented for this program because of the COVID-19 pandemic restrictions. In any event, traditional lecture was not required by the grant agreement.
Demographic information was acquired from the trainees through the sign-up process.
a. 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree.
According to this data, 60 percent of our trainees work for small companies which employ 10 or fewer people. The next group came from companies with 11 to 50 employees (17 percent), and about 9 percent represented mid-size companies employing 51 to 200 people. The remaining 14 percent were hired by larger companies with 200 or more employees. The experience level of a great majority (81 percent) of the trainees was 5 years or less, with 12 percent having an experience in the 5-to-20-year range. Only 7 percent of the trainees had 20 plus years of experience.
Job classifications included operating engineers (20 percent), demolition workers (25 percent), abatement personnel (2 percent), inspector (16 percent), and others (37 percent) which were contractors, construction professionals and managers. The data also showed that 78 percent of the trainees were non-union while 22 percent were unionized. Based on these findings, it would be reasonable to state that this group trainees in large part consisted of less experienced people working for smaller companies that were predominately non-union. This suggests that the training provided would normally not be readily available to them, especially at no cost to the employer or employee, as is the case with ours. In addition, the job classifications revealed by the survey results are very much in line with those targeted by the grant.
Learning and training effectiveness evaluations were performed in accordance with the Kirkpatrick model, which refers to four levels of assessment [30] . Level 1 assessment is the reaction of the trainee to the perceived usefulness of the training received. Level 2 is the learning increment evaluated by comparisons of pre-test (baseline) and post-test scores before and after training. Level 3 assesses the retention of knowledge and skill gained in initial training as determined by evaluating the trainees’ safety behavior after a certain period has elapsed following the initial training. Level 4 is the actual result/impact (in terms of site/company safety improvements) over a period of time using a set of leading and lagging indicators of safety as assessment metrics. Only the first two levels were incorporated in this study because of the relatively short duration of the grant period and the limitations imposed by the budget. The opinion survey essentially corresponded to Level 1, and the computation of the ratio of post-test and pre-test scores served as the metric for knowledge gain.
Survey comments from the trainees were relatively few in number, and most of them suggested cosmetic changes on the PowerPoint (e.g. adding more graphics). Long-term impacts of the training provided could only be assessed through the trainee’s perceptions on how useful it would be to their future work. The quality of the overall program was vetted by MIOSHA and GAC early in the grant year in terms of the accuracy, credibility, clarity, and practicality of the information presented, as well as the efficacy of implementation of the delivery systems, after the in initial preparation of the training package. The pros and cons of the different delivery systems employed in our grant were discussed with the advisors, who shared their opinions based on their observations on and experience with other safety training programs in Michigan and elsewhere in the US.
The advisors collectively opined that the traditional lecture delivery, supplemented by hard-copy handouts and demonstration of safety equipment, lends itself to face-to-face interactions with the trainees, which reinforces their learning experience. The training team acknowledged this view, and added that when the webinar format is implemented, using platforms such as Microsoft Teams or Zoom to remotely deliver a virtual lecture, trainees can be asked to use the chat feature to field questions or make comments to get responses from the instructor. Group discussions may also take place online, facilitated by the team; however, due to time limitations, this feature did not gain much traction in our particular delivery.
The consensus of the entire group was that the self-paced training format would work best for those trainees who need or prefer flexibility with respect to the time and place of training. Although the process is similar to that of a webinar, there is no opportunity to interact with the instructional team during the session, which is a disadvantage. Trainees, opting for this format, log on to our training portal and complete the training going through the sequential steps of sign-up, pre-test, module presentation, post-test, and opinion survey, and are able to download their certificates online. The learning system for this delivery method is designed such that training is continued until the trainees identify and select the correct answer in the post-test; however, grading is based on the first multiple choice answer selected.
The trainee responses based on a 5-point Likert scale to four opinion survey questions are summarized in Table 2. A significant majority (89.8 percent) of the respondents agreed or strongly agreed that the training objectives were clear, while a similarly large portion (90.7 percent) of the trainees agreed or strongly agreed on their knowledge of the subject improving as a result of this training. The assessments expressed in the responses for questions 3 and 4 were quite similar; 94.1 percent confirmed their agreement or strong agreement on the relevance of the training topic to their career, and 89.8 percent thought the training received would be useful to them in their work. Written comments complemented and strongly reinforced the Likert-scale responses.
Turning our attention to learning effectiveness, Table 3 displays the number of program participants, together with the mean values of pre-test and post-test scores above and below the 70 percent cut-off.
Standard deviations and coefficients of variation are also included in the table to observe variability and dispersion of the results. In comparing the trainee numbers receiving passing scores in the post-test (equal to or greater than 70) with the number sperforming unsuccessfully (scores less than 70) it is seen that 44 individuals (37 percent of the participants) elevated their knowledge from unsatisfactory to satisfactory level. Only 20 trainees (17 percent) were unable to receive satisfactory scores. Not only did this group have the lowest mean score in
![]()
Table 3. Pretest and posttest scores.
the post-test, but their scores also showed the widest range of variability as indicated by the coefficient of variation. In general, the lower the mean score for any category, the higher the variability.
The overall impact of this training is best analyzed by comparing the post-test and pre-test scores for the entire group. Whereas the mean score of the pre-test taken by all 118 trainees is 64.5, the mean value of the post-test scores came out to be 79.4, yielding an improvement of almost 15 points. The average knowledge gain is 8.3 points in the case of trainees scoring 70 or better in both pre-test and post-test. Interestingly, there is actually a drop of 5.6 points in the mean post-test score relative to pre-test for the subgroup having scores below 70 in both tests. Additional analysis is warranted to accurately explain this anomaly. Overall, a jump of 14.5 points is realized from pre-test to post-test, signifying substantial gains in knowledge for the total group of trainees.
Knowledge improvement ratio (KIR), which is obtained by dividing the post-test score by the pre-test score, was used to further analyze learning effectiveness evaluation results. The KIR metric quantifies the knowledge gain by a trainee as a result of receiving training on a particular topic. By definition, KIR denotes a neutral value if it is equal to one; it connotes an improvement of knowledge if it is greater than one; and a KIR value below one implies a lower post-test score than the pre-test score. Table 4 displays the trainee numbers for two mutually exclusive KIR scenarios, with the mean, standard deviation, and coefficient of variation values of the post-test scores of 70 or more, or lower than 70. Coefficient of variation, again, is a normalized measure of the variability in the results.
The values shown in the table reveal that 97 individuals achieved knowledge gains; only one person did not, having a KIR value below one, although the accompanying post-test score is above 70.
On the other hand, 7 individuals appear to have improved their knowledge even though they were not able to meet the post-test success criterion of 70 percent. The remaining 13 participants failed to improve their knowledge, while also falling short of earning an acceptable score. In total, a sizeable increase (31 points) was realized in the mean post-test score of the trainees achieving 70
![]()
Table 4. Knowledge improvement analysis using KIR.
percent or better over the lower achieving group with a sub-70 mean score. In the final analysis, it is prudent to state that the most successful group of trainees is the one which showed improvement of knowledge while passing the post-test, based on an acceptable level of knowledge retention.
As a last step, layered cross tabulation with Chi-square analyses was performed on the mean post-test scores and trainee demographic data. Based on 2-sided asymptotic significance (p) values computed to be greater than 0.5, it was observed that there was no significant association between the test performance after taking our training and the experience level of the trainees. Likewise, no statistically significant association could be found between the test performance of the trainees and their union status (union vs. non-union).
4. Conclusions and Recommendations
Comparisons between the pre-test and post-test scores obtained by the trainees by following the contents of the instructional module demonstrated noticeable knowledge gains as a result of the training received, pointing to the success of the training program. The post-test scores and KIR values employed for quantifying respectively, the knowledge level and knowledge improvement of trainees, proved to be beneficial for program effectiveness assessment. However. it was not possible to find any statistically significant associations between the trainee post-test performance and either the experience level or the union status of the trainees. On the other hand, the overwhelmingly favorable opinion survey responses by the program participants confirmed that the knowledge improvements realized through the program described here were relevant to and useful for their work.
Further research is recommended on training and effectiveness assessment approaches. It would be highly beneficial to enhance training effectiveness analyses by adding Kirkpatrick level 3 (knowledge retention) and level 4 (impact) evaluations.
The integration of topical presentations with demonstrations involving emerging technologies such as virtual reality, augmented reality, and mixed reality (e.g., using HoloLens) would be another worthwhile effort. It is also desirable that advanced techniques such as cross tabulation and regression be more extensively incorporated in data analysis with hypothesis testing to gain additional insights; a larger sample size is recommended for these purposes. Frontiers of the fast-developing artificial intelligence applications to predictive modelling of training effectiveness should also be considered.
Acknowledgements
The authors would like to acknowledge the grant support provided by the MIOSHA Consultation Education Training (CET) Program to realize this study. Sincere thanks are extended to the members of the Grants Advisory Committee for their valuable input and feedback throughout this project. Credit is also due for the assistance provided by graduate students RanjaniNainala and Brook Kiros for the literature review and data analysis.