Evaluating the Effectiveness of Integrative STEM Education : Teacher and Administrator Professional Development

The integration of science, technology, engineering, and mathematics (STEM) education, also referred to as integrative STEM education, is a relatively new interdisciplinary teaching technique that incorporates an engineering design-based learning approach with mathematics, science, technology, and engineering education (Sanders, 2010, 2012, 2013; Wells, 2010, 2013). Over the past 11 years, 475 teachers and administrators, representing kindergarten through eighth grade teachers and elementary school administrators from 7 school districts in South Carolina, have participated in an Integrative STEM Education Institute. In this Institute, participants developed knowledge and skills to create and implement integrative STEM education activities for use in their classrooms. Participants learned how to incorporate problem-based and project-based learning that helps students work in groups to develop crosscurriculum skills. The purpose of this article was to evaluate the immediate and long-term effectiveness of the Institute. Quantitative survey data from pre–post surveys immediately revealed a statistically significant increase in self-efficacy regarding the Institute’s learning objectives. In addition, a survey was sent to alumni from the 2012–2015 Institutes. The results from this survey revealed that a significant number of alumni felt empowered through the Institute to implement integrative STEM education in their classrooms and build sustainable integrative STEM education programs at their schools following attendance at the Institute.


Introduction and Background
The world that we live in is complex and integrated; however, dating back to 1894 with the Harvard Committee of Ten, the very roots of K-12 curriculum in the United States have emphasized discrete disciplinary subject instruction (Honey, Pearson, & Schweingruber, 2014).Though education looks different today, the same sentiments of instruction being conducted in siloes still exists 120 years later.These siloes are especially prevalent with regards to the science, technology, engineering, and mathematics fields, known by the acronym STEM.Instruction in mathematics "has been a regular part of K-12 education in the Integrated STEM Education to research pedagogies that may result in positive goals of an integrated STEM instructional model.The committee crafted and debated a variety of instructional models and definitions, never coming to a unified consensus.However, the committee did highlight one approach, integrative STEM education, which serves as the theoretical focus of this research article (Honey et al., 2014).Integrative STEM education places the engineering and technological design at the center of instruction, which facilitates connections being made across science and mathematics concepts (Sanders, 2009).Within this model, educators intentionally teach science and mathematics material through the seamless application of technology and engineering design-based teaching and learning in real-world problem-, project-, and design-based tasks (Honey et al., 2014;International Technology and Engineering Educators Association, 2018;Sanders, 2010Sanders, , 2012Sanders, , 2013;;Wells, 2010Wells, , 2013)).
"'Integrative STEM education may be enhanced through further integration with other school subjects, such as language arts, social studies, art, etc.' (Sanders & Wells, 2010)" (Sanders, 2013, p. 6).Integrative STEM education provides children with opportunities for educational engagement and achievement.This approach to education involves problem-based and projectbased learning that allows students the opportunity to explore real-world problems simultaneously developing cross-curriculum skills while working in small, collaborative groups.Children now expect real-world connections to what they are learning, or else they may completely disengage.
Some scholars and educators argue that an integrative STEM education approach will better prepare all students for the global market in meaningful ways (Chute, 2009;Daugherty, 2013;Sanders, 2012).Research studies have discovered that integrating mathematics and science instruction leads to individually higher achievement scores in those disciplinary assessments (Hurley, 2001;Lehrer & Schauble, 2006).In research on the introduction of engineering instruction, the effect on mathematics and science achievement has been promising (Katehi, Pearson, & Feder, 2009), but results have been mixed on achievement improvement (Tran & Nathan, 2010).As a result, the National Academy of Engineering and the National Research Council have concluded that the reason for the inconclusive evidence may not be associated with engineering instruction itself but how engineering instruction is integrated together with established instructional methods in mathematics and science (Honey et al., 2014).Greater emphasis should be placed on how teachers are trained to weave together STEM concepts in their classrooms.

What We Did
Shifting to an integrative STEM education approach cannot occur overnight and cannot occur without training for current and future teachers.The question at hand, however, was how effective is a professional development (PD) program in equipping teachers with the pedagogical tools necessary for the successful implementation of integrative STEM education concepts?The results of this study provide evidence to support the argument that a seven school district Integrative STEM Education Institute PD program, which has served 475 educators over 11 years, has been an effective experience for empowering teachers and school administrators to implement integrative STEM education.
Results from this study of the Institute showed that participants felt greater confidence in their understanding of integrative STEM education teaching techniques and the methods necessary to implement those teaching techniques into their schools and districts.Results also showed that a significant number of Institute alumni were able to utilize and sustain integrative STEM education in their classrooms in the academic years following the Institutes.
Using a standards-based approach to integrate teaching and learning in STEM concepts established by the International Technology Education Association (2007), the Institute was designed specifically for K-5 teachers and administrators.The main purpose of this Institute was to give teachers and administrators the tools and confidence to teach integrative STEM.Through content coaches, the Institute assisted school teams with developing an integrative STEM curriculum that solved real-world problems in their schools.Teams left the Institute with an action plan for school improvement around a STEM conceptual framework.The learning objectives for this Institute were for participants to exhibit increased confidence in teaching integrative STEM education content and to understand:  the role and purpose of integrative STEM education,  how to use STEM as a curricula organizer,  how content standards can be delivered using an interdisciplinary teaching approach,  how heuristics are used as a conceptual tool in delivering project-or problem-based learning,  how integrative STEM lessons are developed and delivered in the classroom,  how the narrative curricular approach is used to launch STEM learning, and  how standards are integrated into the learning experiences delivered through STEM curricula.This program evaluation was centered around the following questions: (1) How effective was the integrative STEM education PD program at teaching the core principles associated with that pedagogy, and (2) what were the immediate and long-term outcomes of the PD program on participants?
To address our program evaluation questions, we employed two different survey instruments.The first survey was a pre-post survey used to determine the immediate outcomes and effectiveness of the PD program instruction and the immediate self-confidence (self-efficacy) that participants gained regarding integrative STEM education pedagogy.This survey was distributed to participants in the 2015 and 2016 Integrative STEM Education Institutes.The second survey provided evidence of the long-term outcomes of the Integrative STEM Education Institute.The evaluation team defined long-term outcomes as alumni members' ability to implement and sustain the pedagogy taught within the Institute.This survey was distributed to alumni of the program who had completed their cohort experience during the 2012-2015 Institutes.More detail about each of these data collection methods follows below.

Immediate Outcomes of the Institutes: Pre-Post Surveys
As mentioned above, pre-post surveys measuring immediate outcomes data were gathered regarding the 2015 and 2016 summer Integrative STEM Education Institutes.All Institute participants (N = 42) participated in the presurvey prior to the opening session of the Institutes.In 2015, all but one of the attendees (n = 18) completed the post-survey at the conclusion of the last session.Because of a logistical change with the 2016 program schedule, only 47.83% of the participants in that cohort (n = 11) completed the post-survey from a Qualtrics ™ distributed online survey after the Institute.Even though we had a smaller yield with the online post-survey than with the previous year's paper method, the data was large enough to be analyzed collectively when both cohorts of post-surveys were aggregated.Through all pre-post survey responses, participants reported their proficiency in the Institute's eight major learning outcomes through responses to a 5-point Likert scale survey.Knowledge options ranged from not much (1) to a great deal (5) of expertise in each area.The remainder of the survey afforded participants the opportunity to share demographic information about their teaching appointment and district placement for the academic year immediately following the Institute.Also, participants had the opportunity to share their perspectives on the overall quality of what they learned and the degree to which they would implement the teaching techniques in their classrooms and schools.
To determine participant perspectives about the Institute and its short-term influence on the educators' use of integrative STEM education pedagogy in the schools, our evaluation team calculated frequency statistics on various survey questions.To determine participant demographic information, we used frequency statistics.

Immediate Outcomes of the Institute: Participants
Over the two Institute cohorts involved in this evaluation, 42 teachers and school administrators participated in the Integrative STEM Education Institute and agreed to participate in this program evaluation.Because the cohorts were very similar in demographic makeup, the following is an aggregate description.Table 1 provides a detailed summary of participants' roles within the building, subject-matter instruction, and background characteristics.The majority of attendees, 78.57% (n = 33), were classroom teachers, either grade-specific, instructional coaches, or distributed multi-grade-level subject teachers.Of the remaining participants, 8 (19%) were school administrators, and 1 (2.38%) was an instructional technology (IT) staff member.Considering the instructional teachers and coaches present at the Institutes, 12 of those teachers (9%) taught one specific subject.Ten teachers (30.30%) were responsible for teaching science, technology, engineering, and mathematics subjects; however, that did not guarantee these individuals were utilizing an integrative STEM education pedagogy model prior to participating in the Institute.The remainder of the participants, 19 teachers (57.58%), were responsible for teaching more than one subject but not all four STEM disciplines.Within the two cohorts involved in the study's evaluation, six of the seven school districts sent teachers and school administrators to the Institute.The evaluation team created a survey to measure the long-term outcomes of the Institute.Our operational definition of the Institute's long-term impact was the degree to which alumni were capable of implementing integrative STEM education teaching pedagogy in their classrooms and buildings as well as the degree to which they could sustain those integrative STEM programs at their schools.The survey included seven binary questions asking participants to answer whether or not they were able to implement integrative STEM education.In addition, there were two 5-point Likert scale questions asking alumni to share their opinions of the impact of the Institute and their long-term confidence in implementing integrative STEM education.

Long-Term Outcomes of the Institute: Alumni Participants
During the fall 2015 semester, we surveyed 96 previous Integrative STEM Education Institute attendees (from the 2012-2015 Institute sessions).Forty-two alumni (43.75%) completed the survey.Frequency statistics through an online Qualtrics™ survey revealed the self-confidence and self-efficacy of alumni following the Institute in implementing the pedagogies involved in the Institute.The survey results also reported the degree to which alumni were able to implement the Institute principles and continue to sustain integrative STEM education in the years after their participation in the Institute.Survey participants also shared how valuable the Institute was as a PD program in comparison to their other continuing education activities.
Table 2 provides a detailed summary of the Institute alumni demographics at the time of the survey, with specific information about their roles within the building, subject matter instruction, and background characteristics.To summarize, all but 10 (n = 32) of the Institute alumni provided demographic information.Five alumni (15.63%) indicated at the time of the survey that they taught Kindergarten or Grade 1, 18 alumni (56.25%) taught elementary Grades 2 through 5, four alumni (12.50%) taught middle school Grades 6 through 8, and three alumni (9.38%) taught high school.The remaining two alumni (6.25%) served as district or building-level administrators or teachers of high school grades.Eleven (34.40%) of the participants indicated that they would teach mathematics in the upcoming academic year, 11 (34.04%) would teach science, four (12.50%) would teach a technology subject, and six (18.80%) would teach an engineering subject.Seventy-five percent of alumni (n = 24) had 10 or more years of experience teaching in K-12 schools, and 93.80% (n = 30) taught or worked in the same county or district as they had at the time of participating in the Institute.

2015-2016 Participant Perspectives about the Institute
At the conclusion of each Institute, all 2015-2016 participants surveyed indicated that they found the program to be a worthwhile PD opportunity.The data analyzed for classroom-related outcomes included only grade-specific teachers, instructional coaches, and distributed-grade-level discipline teachers.Administrators were not considered in the frequency calculations because many may not have had classroom-based responsibilities in the academic year following the Institute.In addition, 95.83% (n = 23) of classroom-based teachers who participated expected to work independently to implement integrative STEM education activities in their individual classrooms, and 96.4% (n = 27) of attendees indicated that they expected to work with their instructional teams to implement integrative STEM education activities throughout their schools.
Furthermore, 89.3% (n = 25) of participants indicated that they had the ability and would seek out additional resources or opportunities to learn more about integrative STEM education activities.Sixty-eight percent (n = 17) of those participants demonstrated an understanding of what those resources or opportunities could be, noting that they would refer to sources introduced to them during the Institute.These sources could include peer elementary STEM teachers and administrators whose schools were utilizing this instructional model.

Long-Term Outcomes: Ability to Implement and Sustain Integrative STEM Education
According to the results of the alumni survey, 73.80% (n = 31) of alumni indicated that they were able to introduce integrative STEM education pedagogy in their classrooms.Of those 31 participants, 61.90% (n = 26) indicated that they were able to work with other faculty in their disciplinary teaching teams to introduce integrative STEM education activities at their schools.All but five of those participants, who either taught in a K-12 setting or served as a K-12 administrator, reported that they had found a way to build a sustainable integrative STEM education program at the time of the alumni survey.See Table 5 for the descriptive statistics for alumni regarding implementation of integrative STEM education.Were able to work independently to implement integrative STEM education activities in their classrooms immediately following the institute.

(73.8%)
Were unable to work independently to implement integrative STEM education activities in their classrooms immediately following the institute.

(26.2%)
Worked with others to implement integrative STEM education at their schools immediately following the institute.

(61.9%)
Did not work with others to implement integrative STEM education at their schools immediately following the institute.

(38.1%)
Were still using integrative STEM education activities in their classroom or schools at the time of the survey.

(76.2%)
Were no longer or never did use integrative STEM education activities in their classroom or schools at the time of the survey.

(23.8%)
Were able to seek out additional resources or opportunities to help them learn about integrative STEM education activities.

(61.9%)
Additionally, alumni were surveyed regarding their opinion on the impact of the Institute and their long-term confidence in implementing integrative STEM education.See Table 6 for survey data.

What the Findings Mean
Even though research on integrative STEM education is still relatively in its infancy, evidence does exist that these teaching techniques can make a positive difference in K-12 learning environments (Hurley, 2001;Lehrer & Schauble, 2006).There is a gap in the literature on how the field of education equips teaching professionals with the skills to teach integrative STEM education.This gap is evident when we consider how midcareer and seasoned educators learn integrative STEM education principles, especially when they were previously trained to understand and operate under a different teaching model.Our report helps to fill this gap in the literature by highlighting the successes in an integrative STEM education teacher PD program.
The results of our PD evaluation indicate that the Integrative STEM Education Institutes under investigation provided building blocks that teachers needed to build successful, sustainable, integrative STEM programs.The key objectives of the Institute were that participants would be equipped with a greater level of understanding of the principles and theoretical framework of integrative STEM education.With that knowledge, educators and administrators would have the self-efficacy, skills, and networks necessary to implement this pedagogy and continue to do so for many years.Those outcomes served as the foundational concepts for the evaluation questions.
Our pre-post surveys of the 2015-2016 cohorts provided evidence that Institute participants completed the program with greater knowledge of integrative STEM education concepts than they had when they began the Institute.This is evidence that facilitators and instructors of the program did, in fact, effectively teach the core principles associated with this integrative pedagogy.Tables 3 and 4 illustrate how participants oftentimes started the program with a relatively limited understanding of the Institute learning objectives, so the survey results suggest that Institute facilitators conveyed information in an effective manner to change participant confidence levels in a short period of time.
The survey of Institute alumni provided evidence to further support the immediate and long-term outcomes of the Integrative STEM Education Institute.This statement is evidenced by the significant percentage of alumni who implemented integrative STEM education pedagogy in their classrooms and had been able to sustain that for years after the Institute, as reported through the survey.Furthermore, a number of alumni were able to introduce other faculty and administrators to the integrative STEM education model.

Conclusion
Findings of this report provide evidence that there is a need to educate teachers of all experience levels about integrative STEM education.This scholarly conversation would also benefit from an expanded, replicated study with a larger sample size that either incorporates more years of a single program's cohort evaluations or examines multiple cohorts that teach the same integrative STEM education principles.By doing so, researchers may find different learning and concept implementation success rates across teachers with different discipline specializations and grade-levels.The results of these studies could have implications for PD planning teams as they seek to convey information to varied audiences.
Moreover, further study is needed to better understand how andragogy can be used to teach integrative STEM education concepts to educators and administrators.Better meeting the learning needs of education professionals could further support providing quality integrative STEM education PD, a statement which the National Academy of Engineering and the National Research Council have also echoed (Honey et al., 2014).
The results of this program evaluation study provide some preliminary evidence the 2-day Integrative STEM Education Institute can serve as a model PD program for integrative STEM education.We found that a key to success in implementing integrative STEM education is providing opportunities for stakeholders, including teachers and administrators, to develop a shared passion for preparing students in meaningful ways to solve real-world challenges (Chute, 2009;Daugherty, 2013;Havice, 2015;Sanders, 2012).With this type of PD, teachers are more confident and prepared to work with students in approaching problem-solving through a multidisciplinary method (i.e.integrative STEM education; Honey et al., 2014).Ultimately, through PD opportunities, teachers and administrators can model for students what it looks like to be engaged, lifelong learners who strive to impact children and the larger community.

Table 1
2015-2016 Summer Cohorts of the Integrative STEM Institute: Demographic Details Relevant for Academic Year Immediately Following Institute

Table 3
Knowledge in the Integrative STEM Education Institute Learning Outcomes:Self-Reported Pre-Post Survey Means of 2015-2016 InstitutesNote.For all learning outcomes except for the second one, a Satterthwaite approximation was employed due to unequal group variances.Level of selfreported expertise with the learning outcomes was determined on a 5-point Likert scale, with 1 indicating not much knowledge and 5 indicating a great deal of knowledge.

Table 5
Descriptive Statistics for Alumni: Implementation of Integrative STEM Education

Table 6
Perspectives of Alumni on Integrative STEM Education Institute