Effectiveness of Small-Group Learning Pedagogies in Engineering and Technology Education : A Meta-Analysis

This study reports the results of a meta-analysis synthesizing the available literature on the effectiveness of various forms of small-group learning methods on the academic achievement of college students in undergraduate engineering and technology classrooms. The meta-analytic results showed that cooperative learning, collaborative learning, problem-based learning, and peer-led team learning pedagogies were studied in college technology and engineering classrooms. The results also revealed that most of the primary studies supported the effectiveness of the small-group learning methods in improving students’ academic achievement with an overall positive weighted average effect size of 0.45 in standard deviation units favoring small-group learning methods. The findings might help engineering and technology instructors and educators by providing guidance in identifying the conditions under which various forms of innovative small-group pedagogies are more effective than the traditional lecture-based teaching and individualized instruction.

For the last three decades, there have been numerous and consistent calls for instructional reforms and innovations in science, technology, engineering, and mathematics (STEM) education by national and federal agencies as well as national organizations such as the Accreditation Board for Engineering and Technology (Engineering Accreditation Commission of the Accreditation Board for Engineering and Technology, 1997), the American Association for the Advancement of Science (1989Science ( , 2004)), the National Science Board (2003,2010,2015), the National Science Foundation (1996), the National Academy of Engineering (2004Engineering ( , 2005)), the National Research Council (1996), and the Domestic Policy Council and Office of Science and Technology Policy (2006).
In their publications and recommendations, they have emphasized the need to examine and explore the teaching practices and student-learning processes that require various forms of innovative small-group pedagogies in STEM college classrooms.In addition, these calls have stressed the requirement for graduates from the various STEM disciplines and programs to have the ability to communicate effectively, think reflectively and critically, and function effectively in cooperative and collaborative multidisciplinary diverse teambased educational and workplace settings (Engineering Accreditation

Identification of the Relevant Studies
We used extensive library search procedures to identify published and unpublished primary studies that focused on the effectiveness of small group learning instruction compared to lecture-based and individualized instruction in technology and engineering college classrooms.Library searches were conducted through (a) searching electronic databases, such as the ProQuest Dissertations and Theses database, searching electronic technology and engineering journals, such as the Journal of Technology Education and the Journal of Engineering Education, and (b) examining the references of these studies to identify other potential relevant primary studies in engineering and technology.
The keywords used in this study included: "cooperative learning," "collaborative learning," "problem-based learning," "small-group learning," "peer-led group learning," "peer learning," and "team-based learning."These keywords represented the key small-group learning pedagogies and were combined with "technology" or "engineering" subject matter descriptors in "college" and "university" settings.

Inclusion or Exclusion Criteria of the Primary Studies
Stringent inclusion criteria were established and used to determine whether a primary study was qualified to be included in the present meta-analytic review.A study was included in the meta-analysis if it met the following criteria: (a) used two-group research designs (experimental, quasiexperimental, or comparative) that focused on comparing one of the various forms of small-group learning pedagogies to the traditional lecture-based and individualized instruction on college students' achievement, (b) involved undergraduate technology and engineering college classes, and (c) reported the necessary descriptive summary statistics such as the means and standard deviations of the achievement scores for the two comparison groups.With these preset criteria, we identified 18 technology and engineering primary studies.

Coding of Study Features
Based on a careful review of the collected literature, a coding instrument was constructed to cover the methodological and substantive features of each of the 18 primary studies.The coding of the study features was based on the reported information in the primary studies.Publication year, publication type, and instructional duration are examples of the coded characteristics of the primary studies.

Estimating and Calculating the Effect Sizes
In this meta-analytic study, 26 independent effect sizes, based on independent samples of students, were extracted from 18 primary studies.The 26 independent effect sizes (standardized mean differences) were calculated to measure the effectiveness of each of the various forms of small group learning instruction compared to either a lecture-based instruction or an individualized instruction in evaluating students' achievement scores in technology and engineering college courses.The effect-size index for each primary study was calculated by taking the difference between the means of achievement scores of the students who were instructed by the small-groups methods and the lecturebased groups and dividing the difference by the two groups' pooled standard deviation, known as Hedges's g (Hedges & Olkin, 1985;Kalaian & Kasim, 2014).To obtain the weighted average effect size, which is referred to in this study as d, each of the effect sizes was weighted by its inverse of the combined sampling and random errors.

Integrating and Modeling Effect Sizes
The meta-analytic results of this study were obtained by using the Comprehensive Meta-Analysis (Version 2.0) software package.The randomeffects approach for meta-analysis was used to synthesize and integrate the accumulated technology and engineering literature on the effectiveness of the various forms of small-group learning pedagogies on college students' achievement in technology and engineering college classrooms.Moderator analyses involving the categorical and continuous coded characteristics of the primary studies (e.g., publication year, instructional duration) were also performed to investigate the conditions under which the various forms of smallgroup learning methods may have different effects.

Results
The results of this study are organized into three main sections.The first section lists and describes the characteristics of the primary studies, the weighted effect sizes.The second section reports the results of the subgroup analysis for the major subgroup characteristics of the primary studies (study design characteristics, instructional characteristics, and student grouping characteristics) and includes the categories of the moderator variables.Finally, the third section reports the results of the metaregression analysis to explain the variations among the effect sizes using the coded continuous variables of the primary studies as moderators (predictors) in the regression model.

Overall Meta-Analysis Results
Figure 1 shows the forest plot of the effect sizes, which includes the first author's name, publication year, the effect size (Hedges's g), p-value, and the weighted average effect size of the 26 independent effect sizes using the random-effects model.As shown in Figure 1, the primary studies were published between 1995 and 2010.The 26 independent effect sizes, which were extracted from the 18 primary studies, ranged in value from -0.28 to +1.40.Out of the 26 effect sizes, 22 had positive effects in favor of small-group learning, whereas the remaining four had negative effects in favor of lecture-based and individualized instruction.The overall homogeneity test results revealed that the 26 effect sizes of the primary studies were heterogeneous (Q = 115.81,p < 0.00), indicating that there was significant heterogeneity among the effect sizes of the primary studies and that these differences may be explained by the coded characteristics of the studies.Also, the results of the random-effects model revealed that the overall weighted average of the 26 effect sizes was positive and significant (d = 0.45, p < 0.00).

Subgroup Analysis of the Categorical Moderator Variables
Based on the homogeneity test results, which indicated that the 26 effect sizes were significantly heterogeneous, we conducted subgroup analyses for the coded categorical variables using the random-effects methods.This analysis was performed in order to identify both the source of variability among the effect sizes and the differences among the subgroups.The following are the results of the subgroup analyses for the three major subgroup characteristics: study design characteristics, instructional characteristics, and student grouping characteristics.
Subgroup results of the study design characteristics.The subgroup results of the random-effects categorical analysis for the categories of the major coded study characteristics of the primary studies are shown in Table 1.As shown in Table 1, seven effect sizes were extracted from the primary studies that had been published in 2000 or earlier with a d-value of 0.32.The remaining 18 effect sizes were extracted from the primary studies that had been published in 2001 or later with a d-value of 0.49, which was much larger than the average effect size of the primary studies that were published in 2000 or earlier.
The results also show that 18 effect sizes were extracted from articles published in peer-reviewed journals with a significant and positive d-value of 0.58.This average effect size was much larger than the average effect size of the remaining eight effect sizes that were extracted from PhD dissertations with a nonsignificant d-value of 0.05.Regarding the research design that is used in the primary studies, the quasiexperimental studies (two-group pre-post research design) produced much larger effect sizes with a significant d-value of 0.52 than the nonexperimental (comparative post-only) studies with a significant d-value of 0.40.
In addition, the results show that the engineering primary studies in this review produced larger effect sizes (18 effect sizes with a d-value of 0.48) than the technology primary studies (eight effect sizes with a d-value of 0.38).In regards to college classroom level, the effect sizes for first-year freshmen classrooms were much larger (d = 0.84) than the effect sizes of the higher level college classrooms (sophomores, juniors, and seniors; d = 0.38).These results indicate that the various small-group learning methods were much more effective in the freshmen level of college than the higher levels.
Furthermore, the results show that the majority of the primary studies were conducted at universities and colleges in the United States and 24 effect sizes were extracted from these primary studies.The d-value of these 24 effect sizes was positive and statistically significant.The remaining two effect sizes were extracted from the primary studies that had been conducted in China and Turkey and had much larger effect sizes with a significant positive d-value of 0.90.Subgroup results of the instructional characteristics.The results of the subgroup analyses that are related to the instructional characteristics of the engineering and technology primary studies are shown in Table 2.The results show that during the last 3 decades, four different methods of small-group learning (cooperative, collaborative, problem-based, or peer-led team learning pedagogies) have been used and evaluated in the technology and engineering college classrooms.The results also show that both cooperative and collaborative learning methods promoted larger effects in increasing students' achievement with d-values of about 0.51 and 0.46, respectively, and followed by problem-based learning with a d-value of 0.36.Only one primary study implemented peer-led team learning and had the lowest effect size of 0.20.About 25% of the effect sizes that were extracted from six primary studies had much shorter instructional durations (20 hours or less) with a d-value of 0.41.The other 75% of the effect sizes had instructional durations of 30 hours or more (d = 0.45), which is almost similar to the shorter instructional duration.None of the primary studies in this review had instructional durations between 21 and 29 hours.
Courses that used computers as an instructional aid in the classrooms had larger effects sizes with a significant positive d-value of 0.50 than courses that did not use computer-aided instruction with a significant positive d-value of 0.42.The results also show that the primary studies had about similar effect sizes when the instructional intervention was delivered by either the researcher (author) of the primary study or another instructor who was not the researcher of the primary study with d-values of 0.44 and 0.45, respectively.
Regarding the ethnic diversity of the students: None of the studies had predominately minority students in the classrooms.Five effect sizes were extracted from the primary studies that were predominately White (more than 60%) with a d-value of 0.26, and the majority of the primary studies did not report the ethnic diversity of the classrooms.These studies had a much higher effect sizes (d = 0.49) than the studies that reported the ethnic diversity of the students.
Similar to the ethnic diversity of the students in the classrooms, the results show that the 13 effect sizes were extracted from the primary studies that did not report the gender diversity of the students in the engineering and technology classrooms and had a d-value of 0.53.Ten effect sizes were extracted from the primary studies with male dominated classrooms (more than 60%) and had a dvalue of 0.42.The remaining three effect sizes were extracted from the primary studies with female-dominated classrooms (more than 60%) and had much lower d-value of 0.23 than the studies that did not report the gender composition of the classrooms.
Finally, our results show that the majority of the primary studies had used teacher-made tests to assess students' achievement in engineering and technology classrooms with a significant positive d-value of 0.44.The remaining two studies used standardized tests and produced a significant positive d-value of 0.65.

Subgroup Results of the Student Grouping Characteristics
As shown in Table 3, the primary studies that placed the students into small groups by students' selecting their own groups produced much higher effect sizes (d = 0.51) than the studies that placed students in the groups by random selection (d = 0.46) and much less than the studies that placed the students in groups based on abilities such as their Grade Point Average (GPA) and Scholastic Aptitude Test (SAT) scores (d = 0.21).The results also show that the primary studies with small groups of three to five students produced larger effect sizes (d = 0.49) than the studies with groups of two students (d = 0.31).

Random-Effects Regression Analysis Results of the Continuous Predictors
Based on the homogeneity test results, which indicated that the 26 effect sizes were heterogeneous, we conducted random-effects metaregression analyses for each of the coded continuous moderator variables.The metaregression analyses were performed to: (a) determine the ways in which the coded continuous predictor variables impacted the effect sizes and (b) explain some of the variability among the effect sizes.The two coded continuous moderator variables were publication year and instructional duration (in hours) of each of the primary studies in this review.Table 4 shows that the regression coefficients (slopes) for the publication year and instructional duration of the primary studies were 0.03 and -0.004, respectively.These regression coefficients were close to not significant.The results showed that, collectively, the small-group learning methods were more effective on average than the traditional lecture-based instruction with a significant positive overall d-value of 0.45 in standard deviation units favoring small-group learning methods.This means that using small-group learning in technology and engineering classrooms could positively affect student achievement, moving the students' scores from the 50th percentile, which is the percentile score of the students in the lecture-based and individualized instructed classrooms, to the 69th percentile in the small-group classrooms.In other words, instead of scoring better than 50% of the students in a lecture-based class, the same student would score better than 69% of the students in a small-group classroom.The results also showed that during the last 3 decades, four different methods of small-group learning (cooperative, collaborative, problem-based, or peer-led team learning pedagogies) have been used and evaluated in the technology and engineering college classrooms.
In addition to exploring the scope and magnitude of the effects of the various forms of small-group learning methods, this study examined the subgroups for whom small-group learning methods are effective.Although the innovative and reform-based small-group learning methods produced positive and significant effects across the subgroup categories, educators and policy makers should note that the various small-group learning interventions appeared to be significantly more effective for freshmen students, students in countries other than the United States, students in groups of three to five, students who chose their own groups, and engineering students.The small-group learning interventions also appeared to be significantly more effective in recently published studies.
The results of this quantitative meta-analytic study are based on 18 technology and engineering primary studies that were conducted since 1997 and met the established inclusion or exclusion criteria.Based on these results, we believe that pedagogical research in engineering and technology education is limited and that there is a need to conduct more primary studies to examine the effectiveness of small-group learning methods in college engineering and technology classrooms.Also, there is a need for better reporting of the smallgroup instructional processes, activities, and the results of the effects of the various forms of small-group learning research.
In conclusion, this meta-analytic study had shed some light on the accumulated pedagogical literature of the effectiveness of the various methods of small-group learning compared to lecture-based and individualized instruction in college engineering and technology classes.Also, the results of this study added to already converging evidence from other domains such as STEM (Springer, Stanne, & Donovan, 1999), statistics (Kalaian & Kasim, 2014), and computer science (Kalaian & Kasim, 2015) that each form of the small-group learning pedagogies appear to be a promising mechanism for promoting academic success.In other words, we learned that if college students who are taking college engineering and technology classes are placed in an environment in which they have an opportunity to experience peer-supported collaborative and cooperative scientific inquiry, the academic achievement of these students will be improved and accelerated.The evidence-based findings that emerged from this quantitative review can contribute significantly to the current pedagogical knowledge concerning technology and engineering education.The findings also have significant institutional policy implications in undergraduate technology and engineering education as well as being of great interest to instructors and educators who are interested in the pedagogical knowledge to improve students' success, motivation, and persistence in the colleges of technology and engineering throughout the nation and worldwide.

Figure 1 .
Figure 1.Forest plot of the effect sizes of the primary studies.

Table 1
Subgroup Analysis of the Study Characteristics

Table 2
Subgroup Analysis of the Instructional Characteristics of the Studies

Table 3
Subgroup Analysis of the Grouping Characteristics of the Studies

Table 4
Random Effects Meta-Regression Analysis of the PredictorsConclusionThis meta-analytic study aimed to survey the engineering and technology literature and investigate the effectiveness of the various forms of small-group learning methods in comparison to the traditional lecture-based and individualized instruction in maximizing college students' achievement scores in undergraduate engineering and technology classes.As far as we know, this is the only comprehensive meta-analysis of the undergraduate technology and engineering education literature.