Evidence of the Impact of Engineering Design on Technology and Engineering Education Students

Within the technology education classroom, engineering design has been targeted as key to improving learning, enhancing interest in STEM careers, and positively impacting students. The purpose of this research review was to determine whether the research evidence bears these claims. Four scholarly journals that focus on technology and engineering education research were reviewed resulting in the identification of 25 empirical research studies from the past decade. Across all of the studies, data had been collected from a total of 6,397 technology and engineering education students to analyze: (a) how students design, (b) student learning outcomes, and (c) student interests and perceptions. Just over half of the studies used qualitative methods to explore how small samples of students engage in engineering design. Although the overall research evidence of the impact of engineering design on technology and engineering students is sparse, there are some important descriptive findings relating to how engineering design can impact student learning and how students allocate their time and access information while designing.

Several science, technology, engineering, and mathematics (STEM) reports, mostly supported by the National Research Council and the National Academies, have focused on the inclusion of engineering at the K-12 level.For example, in the report Engineering in K-12 Education: Understanding the Status and Improving the Prospects, the committee argued that "K-12 engineering education may improve student learning and achievement in science and mathematics; increase awareness of engineering and the work of engineers; boost youth interest in pursuing engineering as a career; and increase the technological literacy of all students" (Katehi, Pearson, & Feder, 2009, p. 1).In 2010, the National Academy of Engineering's Committee on Standards for K-12 Engineering Education explored the need for K-12 engineering standards.The following year, another committee outlined criteria for identifying effective STEM schools and programs (National Research Council, 2011).In 2014, yet another committee explored "integrated STEM education," finding that "far from being a single, well-defined experience, integrated STEM education includes a range of different experiences that involve some degree of connection" (Honey, Pearson, & Schweingruber, 2014, p. 2).
The concerns-and, some might argue, rhetoric-articulated in these reports tend to center on the need for the United States to remain globally competitive by producing future innovative thinkers and designers.STEM education, it is argued, is the avenue by which students will gain the knowledge needed for the global economy.The reports argue that the current educational system is lacking in rigor, particularly in mathematics and science, in preparing students for STEM-based careers.For example, the reports point to lagging test scores, such as the National Assessment of Educational Progress results, to make their case.Although much of the emphasis in these reports is on mathematics and science, technology and engineering have been offered as opportunities for improving these areas by providing authentic contexts, making the learning more relevant to students.This is seen most recently with the inclusion of engineering concepts and practices in the Next Generation Science Standards (NGSS Lead States, 2013).Integrating STEM, it is argued, "can enhance motivation for learning and improve student interest, achievement, and persistence" (Honey et al., 2014, p. 1).It is believed that these outcomes will create better prepared students for college and the workplace.
Within the technology education classroom over the past decade or so, engineering education in general and, more specifically, engineering design have been offered as keys to improving teaching and learning (Daugherty & Custer, 2012;Denson & Lammi, 2014;Lewis, 2005;Wicklein, 2006;Wilhelmsen & Dixon, 2016).This is reflected by both the Standards for Technological Literacy (STL) including engineering design in its standards (International Technology Education Association [ITEA], 2007) and the International Technology and Engineering Education Association, the professional association for the discipline, including engineering in its name.Pinelli and Haynie (2010) outlined three reasons for including engineering in the K-12 curriculum: (a) "to support the engineering pipeline" by getting more students interested in engineering careers (p.60), (b) "to enhance and enrich the teaching and learning of STEM" (p.61), and (c) "to create a technologically literate citizenry and society" (p.62).
With the numerous claims and hopes offered, what evidence exists to support these claims?Does the research support these assertions?In particular, is engineering design as impactful in the technology and engineering education classroom as the rhetoric suggests?The purpose of this study was to examine the research evidence on the impact of engineering design on technology and engineering education students.In order to address this purpose, the research questions for this study were as follows.
1. What research has been published in academic, peer-reviewed journals that provides empirical data on the impact of engineering design on technology and engineering education students in the United States? 2. What research topics exploring the impact of engineering design on technology and engineering education students have been published?3. How is engineering design impacting student learning?

Engineering Design
Although K-12 engineering education is broader than engineering design, the focus of this study is on understanding the research evidence measuring the impact of engineering design.However, defining engineering design is as difficult as defining technology or engineering because there is no single, agreed upon definition.Often engineering design is described as a problem-solving process with specific steps identified.For example, Gomez, Oakes, and Leone (2012), in an engineering textbook, described engineering design as a problemsolving process.Perhaps closest to a definitive definition is the one offered by the Accreditation Board for Engineering and Technology's Engineering Accreditation Commission (2015) in their 2016-2017 criteria for accrediting programs, which defined engineering design as the process of devising a system, component, or process to meet desired needs.It is a decision-making process (often iterative), in which the basic sciences, mathematics, and the engineering sciences are applied to convert resources optimally to meet these stated needs.(p.4) Within the K-12 engineering and technology education literature, a few definitions of engineering design exist.Within the STL (ITEA, 2007) and in Engineering in K-12 Education: Understanding the Status and Improving the Prospects (Katehi et al., 2009), engineering design is described as an approach to solving technologically related problems.Wicklein and Thompson (2008) offered that "engineering design is an orderly, structured, problem-solving activity or process through which changes can lead toward a required result" (p.58).And Gattie and Wicklein (2007) defined engineering design by citing Ullman's (2003) definition, as a process that centers around four (4) representations used to describe technological problems or solutions: (1) Semanticverbal or textual explanation of the problem, (2) Graphicaltechnical drawing of an object, (3) Analyticalmathematical equations utilized in predicting solutions to technological problems, (4) Physicalconstructing technological artifacts or physical models for testing and analyzing (International Technology Education Association, 2000;Ulman [sic], 2003).(Gattie & Wicklein, 2007, p. 10).
Across these descriptions, the consistent element of engineering design is that it is a systematic process for solving problems.However, as Lewis (2005) pointed out, although there is some agreement about the cognitive activity of engineering design, "there are nuances in how it is conceptualized" (p.40) in the technology education classroom.Unfortunately, researchers often fail to articulate these nuances or operationalize the definition of engineering design being explored in their studies.As Flowers (2010) pointed out, "too often, our literature discusses the model or the process were there was no initial introduction of a model or a process" (p.16).The assumption being that there is one engineering design process in K-12 technology education.This is not a safe assumption because of the variety of engineering design models and approaches identified in the literature and because the inclusion of or emphasis on certain steps or stages of the process vary.This is important to note when considering the research exploring the impact of engineering design on engineering and technology education students.

Method
A search and content analysis of empirical studies from four peer-reviewed academic research journals that publish research on K-12 technology and engineering education was conducted.These journals were: the Journal of Technology Education (JTE), the Journal of Technology Studies (JOTS), the Journal of Pre-College Engineering Education Research (JPEER), and the Journal of Engineering Education (JEE).Although there are other journals and conferences where researchers share their work in this domain, these four journals were believed to be the more prominent journals that would contain studies that examine the impact of engineering design on U.S. students because of their focus and audience.
As discussed previously, engineering design's prominence in technology and engineering education is reflected in the inclusion of engineering design in four STL standards.The most recent edition of the STL, published in 2007, was assumed to have spurred research on engineering design; thus, volumes from 2007-2017 of the selected journals were analyzed. 1The title and abstract (if available) of each article published in the volumes of the journals were reviewed, and if an abstract was not included, the article itself was analyzed for inclusion in the review.Articles were included in the review if they:  included empirical data collected quantitatively, qualitatively, or with mixed methods;  focused on technology and engineering education students;  were situated in the United States; and  examined the impact of engineering design.
Although there were studies exploring engineering design in other contexts (i.e., science classrooms or other countries), focusing on elementary age students, or investigating topics related to engineering (i.e., visualization or computer-aided drafting), such studies were excluded because they stray from the purpose of this study, which was to determine the research evidence concerning the impact of engineering design on technology and engineering students.The studies had to include an explicit examination of "engineering design" and a focus on "technology and engineering education" students or "STEM" students.The studies that met the identified criteria were analyzed based on their method for data collection, number of participants, grade level, and findings.Although participant data is important because it enables researchers and readers to appropriately interpret the data, draw conclusions, and determine implications, several of the studies did not specify the demographic characteristics of the participants.
A content analysis was conducted for each of the studies that met the above criteria."Content analysis is a detailed and systematic examination of the contents of a particular body of material for the purpose of identifying patterns, themes, or biases" (Leedy & Ormrod, 2016, p. 275).For the purpose of this study, the research studies were reviewed, and the method of data collection and number of research participants were documented for each study.Then, the studies were examined to identify consistent research topics on which they focused.The topics that emerged were used to categorize the studies according to their focus.For consistency, one of the researchers of this study categorized all of the studies.To establish interrater reliability, each of the two coresearchers checked a different 10% of the categories.Thus, 20% of the codes were checked and aligned with the established categorizing scheme.

Research Question 1: Research on the Impact of Engineering Design
Based on the method established for this study, 25 research studies were identified in the four journals.JPEER had the most studies with 10, JTE had eight studies, JEE contained five studies, and JOTS had two studies.Across the 25 studies, 17 focused on high school level students, six included middle school students, one included both middle and high school students, and one did not report grade level.One study collected data from first-semester college students asking them information about their high school experiences and was thus coded as including high school students.
Across 24 of the 25 research studies (one did not report the number of research participants), data were collected from 6,397 students.One of the studies collected test score data from 2,530 students and another from 1,835 students, skewing the total number of students studied across the articles.For example, 76% of the studies had less than 200 students in their samples, and 56% of the studies had less than 50 students.Because very few reported the students' ethnicity or other student demographics, this cannot be reported.In lieu of reporting the research participants' demographics, researchers often shared the school district demographics as an apparent proxy for this data.Some of the studies reported the gender of the participants, and a few others provided percentages of males and females for control and experimental groups but did not report the number within these subgroups, so the number of female and male participants could not be determined.
In terms of the research methods used in the 25 studies, there was not uniformity in describing the research design.Some researchers identified both the research design and the data collection methods, whereas others chose to only identify the methods used.For example, in some of the studies that analyzed and reported qualitative data, the researchers identified the method for data collection (i.e., interviews) but did not describe a particular qualitative research design (i.e., case study), or the researchers used inconsistent terms (i.e., verbal protocol analysis).The authors for this study first used the terminology used by the researcher or researchers and then grouped similar methods together to determine which method (quantitative, qualitative, or mixed methods) was used most frequently in the research.
The most frequently used research methods were qualitative.Twelve studies relied on qualitative data; some identified a specific qualitative research design, and others simply described qualitative data collection methods and analysis.For example, researchers in one study described their design as collecting ethnographic student reflections, and in another, researchers identified their study as a focus group study.Although identified differently within the studies, 11 of the qualitative studies collected verbal data either by video or audio (or both) of small samples of students engaged in design.In five of these studies, researchers described their approach as verbal protocol analysis or think-aloud protocol analysis.Researchers in one study described collecting video data of small group discussions.In one study, researchers described conducting discourse analysis, and in another study, researchers described conducting collaborative video analysis using grounded theory.Two studies described the method used as exploratory triangulation mixed methods using function-behavior-structure ontology collecting verbal data.In addition to individual interviews and observations of group meetings, verbal protocols were also collected in an ethnographic study.
Researchers in seven of the studies identified mixed methods as the research design.Two of the studies used a quasiexperimental or "educational design experiment" that included a pretest and posttest, no control group, and follow-up interviews.Two studies described the method used as exploratory triangulation mixed methods using function-behavior-structure ontology.One study's researchers described the method as a combined quantitative pre-/post-test and interviews; another study used an embedded design mixed-methods framework with a two-group posttest; and another study's researchers described using surveys, questionnaires, and focus groups to collect data.
Of the 25 studies examining the impact of engineering design, six of the studies used quantitative research designs or methods.Four of the studies described specific statistical methods, including longitudinal multilevel modeling, multiple linear regression, multilevel statistical modeling, and regression and mediation analyses to analyze the quantitative data collected.In one study, researchers reported descriptive statistical results from a survey, and in another study, researchers reported descriptive statistical results from a questionnaire.These six studies accounted for the larger sample sizes, ranging from 41 to 2,530 students included in the sample sizes.

Research Question 2: Research Topics
The research topic categories emerged from the stated purpose of each of the studies.After reviewing each of the studies thoroughly, consistent research topics were identified within the larger goal of examining the impact of engineering design on technology and engineering education students.The topics that emerged were: (a) how students design, (b) student learning outcomes, and (c) student interests and perceptions.The studies classified under the first topic, how students design, investigated student design strategies, typically through think-aloud or verbal protocol analysis.For the second topic, student learning outcomes, these studies examined the impact of engineering design on student learning outcomes measured primarily using test scores.Studies classified under the last topic, student interests and perceptions, largely captured students' interests in engineering careers or their perceptions about specific engineering programs or their engineering design experiences.
Table 1 includes the number of studies by research topic within each of the journals.Almost half of the studies (12) explored how students design.These studies aimed to capture technology and engineering students' thought processes as they engaged in engineering design.Seven of the studies focused on measuring student learning outcomes as a result of an engineering design curricular program or experience.Six of the studies explored the impact of engineering design experiences on students' interests and perceptions-often, either their perceptions of engineering as a career or of the engineering program itself.

Research Question 3: Impact of Engineering Design on Students
The third research question under investigation in this study focused on how engineering design was impacting students.The studies were analyzed to better understand the research evidence concerning the impact of engineering design on technology and engineering education students.Below are the combined findings from the studies that were grouped together by the three categories that emerged based on the research topics of the studies: (a) how students design, (b) student learning outcomes, and (c) student interests and perceptions.
How students design.Most of the 12 studies that explored how students engage in design examined student cognition during engineering design activities to inform curriculum and instruction.Student engagement in design can be tied to student learning; however, most of the studies examined small samples of high school students outside the context of the classroom environment as they interact with the process of engineering design and with each other.Many of the studies make reference to the intention of identifying the gaps between novice and expert engineering designers and equipping students with the knowledge, skills, and abilities needed to advance toward expertise.
Mentzer is a researcher or coresearcher on five of the 12 studies; thus, his work greatly impacts this area of research.Three of his studies focused on information access and time allocation while students are engaged in the design process.In terms of information access, Pieper and Mentzer (2013) analyzed videos from 12 high school students engaged in an engineering design problem who had different information sources available.They found that "on average, participants spent 38.8% of their total time accessing information" (p.86), primarily from Internet-based sources, which was significantly more than college-level engineering students and expert engineers in previous studies.Another Mentzer (2014a) study compared two groups of high school students (30 students in each group) engaged in "a design problem in a three-hour design experience" in which "one group has access to the internet while the other does not" (p.31).Mentzer found that "the most commonly requested piece of information related to cost of materials" (p.31).He also found that "students with access to the internet spent substantially more time in the design process," although "most of the difference in design session duration was explained by the additional time allocated to gathering information" (p.39).In another study, Mentzer and Fosmire (2015) used verbal protocol analysis of video and audio recordings to measure "the information gathering behaviors of [19] high school students who had taken engineering design courses as they solved a design problem" (p.22).As in the previous study, Mentzer and Fosmire found that students spent the most time searching for material costs; however, they also spent time searching for information concerning construction techniques or processes and information related to the solution being considered."The high school students understood the need for information, . . .but their skill in locating high-quality information was relatively poor" (p.22).
Two of the other studies that Mentzer was involved in focused on how students allocate their time when engaged in engineering design.Mentzer's (2014b) study examined 17 design teams comprised of 47 high school students as they engaged in an engineering design activity.In comparison to experts, the high school teams spent less time working on the problem and modeling but spent more time communicating.Finally, Mentzer, Becker, and Sutton's (2015) study compared the design processes of high school students, "freshmen who have taken one engineering course and seniors who have taken a series of engineering courses" (p.417), to expert engineers.Using verbal protocol analysis, 59 "students from four states were asked to think aloud in a three-hour design task that was audio and video recorded" (p.417).The researchers found that the "students and experts alike spent a large portion of their time modeling" (p.417)-unlike the students in Mentzer's (2014b) study.However, the "students spent significantly less time in the process of information gathering" and thinking about the problem from the client's perspective than experts (Mentzer, Becker, & Sutton, 2015, p. 417).Also, "freshmen spent significantly less time in the idea generation process than seniors and experts" (p.417).
The comparison of different groups of students was also an element of Kelley's (2008) verbal protocol study that compared the impact of Project Lead the Way (PLTW) and the National Center for Engineering and Technology Education's (NCETE) engineering-design focused instruction on how seven (three PLTW and four NCETE) students engaged in an ill-defined problemsolving activity.Kelley found that both groups of students used similar strategies but spent varying time developing solutions.The study published by Wells et al. (2016) also used verbal protocol analysis to compare the design cognition of high school students who have had a pre-engineering course experience, high school students who have not, and undergraduate engineering students as part of a larger longitudinal study; they found no significant differences in the design cognition of these different groups.
The following two studies sought to understand student design cognition in groups using verbal data without a comparison group.Lammi and Becker (2013), for example, used verbal protocol analysis and the Function-Behavior-Structure framework, as well as interviews and the artifacts of the design process, to examine 12 high school students' cognitive processes while working in pairs on an engineering design challenge.They found that structure was the most prevalent element and that the lowest was function.Valtorta and Berland's (2015) study used discourse analysis of video data that captured 31 high school students working on a unit in an engineering course over 15 class sessions.They sought to determine if students applied STEM concepts when engaged in engineering design.Valtorta and Berland found that students successfully applied math and science concepts to their engineering design work without teacher prompting when the concepts were familiar.However, explicit teacher prompting and instruction regarding the integration of less familiar concepts did not seem to facilitate student use of those concepts.(p.15) Two other studies used verbal data to understand the role of culture on how students engage in engineering design.Wilson-Lopez, Mejia, Hasbún, and Kasun's (2016) ethnographic study relied on verbal data, interviews, and observations of seven groups of 25 Latina/o high school students as they engaged in engineering design.Wilson-Lopez et al. found that the students' "familial, community, and recreational funds of knowledge" were connected to their understanding and approach to engineering design (p.278).Schnittka and Schnittka's (2016) study explored "how cultural gender norms are navigated within informal K-12 engineering contexts . . . .[using discourse analysis to analyze] video of single-and mixed-gender collaborative groups participating in . . .a design-based, environmentally themed afterschool program" (p. 1).
Discrepancies were found regarding functional and cultural characteristics of groups based on gender composition.Single-gender groups adhered more closely to social gender norms . . . .In contrast, characteristics of interactional styles within mixed gender groups strayed from social gender norms.(Schnittka & Schnittka, 2016, p. 1) The only research study exploring how students design that did not analyze verbal data was Menekse, Higashi, Schunn, and Baehr's (2017) study of "366 youths on 61 K-8 robotics teams that participated in a FIRST LEGO League Championship.Regression and mediation analyses were conducted to explore the relation between effective team collaboration and team performance" (p. 1).They found that "Collaboration Quality was a good predictor of robotics team performance across all measures" (p. 1).In other words, they found that how students collaborated during design impacts how they performed in the competition.
Student learning outcomes.Of the seven studies published exploring the impact of engineering design on student learning outcomes, four used mixed methods, and three were quantitative.The majority of the studies found that engineering design via a unit, course, program, or curriculum positively impacted some facet of student learning, but the results were mixed.For example, Merrill, Custer, Daugherty, Westrick, and Zeng's (2008) study was designed to measure student learning of engineering concepts via a unit of instruction with an engineering design challenge.Using a quasiexperimental design, 114 high school students engaged in the unit of instruction designed to teach three engineering concepts: constraints, optimization, and predictive analysis (COPA).Using a pre-/post-test design, the researchers found statistically significant gain scores.Although, "mean score gains . . .were modest, they did indicate significant improvement in understanding of COPA concepts" (p.62).Berland et al.'s (2013) mixed-methods study sought to determine the impact of an engineering design course in seven high schools with 106 students.The researchers found mixed results in that students' understanding of engineering increased, but it did so inconsistently and without much detail.And Svarovsky's (2011) mixed-methods study investigated the impact of a 60-hour program called Digital Zoo on 10 middle school female students using a pre-/post-test and interviews.The researcher found that the program enabled students "to develop each of the five epistemic frame elements-engineering skills, knowledge, identity, values, and epistemology" (p.19).
Two of the studies explored the impact of PLTW curriculum on student learning.Tran and Nathan (2010) used multilevel statistical modeling to explore the relationship between PLTW course enrollment and student achievement on the state math and science standardized test scores of 140 high school students with a matched comparison group of 70 students.The results indicated that While students gained in math and science achievement overall from eighth to tenth grade, students enrolled in PLTW foundation courses showed significantly smaller math assessment gains than those in a matched group that did not enroll, and no measurable advantages on science assessments, when controlling for prior achievement and teacher experience.(p.143) Dixon and Brown (2012) also investigated the impact of PLTW on student learning, finding mixed results.Their study was designed to compare PLTW students with students who have not taken PLTW courses in terms of their ability to "transfer mathematics, science, and design concepts from one situation to another" (p.3).They "found significant relationships between the number of PLTW courses students took and students' performance in design score and total score.Also, there was no significant difference in mathematics and science performance between PLTW and non-PLTW students" (p.10).
The last two studies in this category were a bit different than the others but still examined the impact of engineering design on student learning by correlating data.Mentzer and Becker (2010) investigated the possible correlation between the prior academic achievement of 41 high school students, as determined by student GPA in science, mathematics, communication courses, and "their experience during an engineering design challenge, as measured by an achievement test" (p.27).They found that "student achievement was significantly correlated to science GPA, but not significantly [correlated] to mathematics or communication GPA" (p.37).The study by Crotty et al. (2017) correlated "different approaches to integrating engineering practices in science, technology, engineering, and mathematics (STEM) curriculum units . . .with student outcomes on engineering assessment items" (p. 1).They found that when and how engineering design was placed in the curriculum impacted students' performance on the assessments.
Including engineering at the beginning of a STEM unit to frame the learning and provide context for the unit with engineering being revisited and used as a project at the end produced stronger engineering understandings for students compared to when engineering was used solely as a culminating project.(p.9) Student interests and perceptions.The last topic of research examining the impact of engineering design on technology and engineering education students included six studies that explored students' interests and perceptions.Two of the studies sought to understand students' perceptions of engineering design in general.Four of the studies captured the students' perceptions after experiencing a specific engineering program or experience: one using surveys, one using surveys and a focus group, on using only a focus group, and one using an ethnographic approach.The studies were designed to capture the students' perceptions of the program or experience itself as well as its impact on their interests in engineering or STEM.Sirinterlikci, Zane, and Sirinterlikci (2009) described the results of a survey administered to elementary and middle students involved in the TOYchallenge competition, finding that "some of the student survey responses reflected positive attitudes toward the engineering process, albeit their lack of interest in pursuing the field as an adult" (p.20).Using mixed methods, Blanchard et al. (2015) surveyed nearly 2,000 middle school students and conducted a focus group of 19 students who had participated in Beyond Blackboards, "an inquirycentered, after-school program designed to enhance middle school students' engagement with engineering through design-based experiences" (p. 1).
Students reported that as a result of their participation, their interest in engineering careers and their interest in pursuing a 4-year degree increased.Denson, Lammi, White, and Bottomley (2015) convened a focus group "to further understand the student experience and ascertain the perceived value of an informal learning environment for students engaged in an engineering design challenge" during a summer camp (p.40).The eight high school students who participated in the study reported that they perceived "the benefits of the summer camp to include the use of mathematical modeling (application of math and science), a field experience, and teamwork" (p.43).Carroll (2014) reported on an ethnographic study that involved 4 months of data collection in an urban afterschool program in which university students worked with 36 middle school students "engaged in design thinking and STEM activities" (p.17).The researcher concluded that design thinking permeated the experience for both the university and middle school students, informing how the students approached mentoring, "how to create user-centered learning experiences, and how to share their experiences" (p.29).
Although Ing, Aschbacher, and Tsai's (2014) longitudinal study sought to examine the possible gender differences in students' interests in careers in engineering and science.They surveyed 482 students over 3 years (Grades 7-9) "to explore gender differences in engineering and science career preferences" (p. 1).The findings indicated that "females were far more likely to express interest in a science career (31%) than an engineering career (13%), while the reverse was true for males (58% in engineering, 39% in science)" (p. 1).Additionally, "females were less interested in designing and inventing, solving problems, and using technology" than males (p. 1).
Seeing self-efficacy as an important indicator of students electing to major in STEM subjects, Fantz, Siller, and DeMiranda (2011) surveyed 332 firstsemester college students about their precollegiate experiences, including "preengineering classes, multi-day programs, engineering hobbies, working in an engineering environment, extra-curricular engineering programs, and single-day field trips" that included exposure to engineering design (p.604).The results indicated that that there were "significant differences in self-efficacy . . .between groups of students who had pre-engineering classes and engineering hobbies versus students who did not have these experiences" (p.604).

Discussion
Overall, the research evidence of the impact of engineering design on technology and engineering students is sparse.In over a decade of time, only 25 studies in four journals with a total of only 6,397 students has been published.In addition, the majority of the studies used qualitative or mixed methods to collect data from purposively selected small samples, mostly of high school students.This prevents the generalization of findings about how students design, the impact of engineering design on student learning, and its impact on their interests and perceptions.Despite the limitations of this body of research, there are some descriptive findings explored in these studies that are worth further discussion.
With almost half of the studies seeking to understand how students engage in engineering design, this is a prominent topic in the research, and verbal protocol analysis is a prominent method used.Although these studies seek to explore student cognition, the studies exploring how students engage in the design process were often conducted outside of the classroom learning environment.However, by understanding how students design (i.e., how they allocate time, apply STEM concepts, or collaborate), the intent of these studies is to inform and improve engineering design-based curriculum and instruction in the classroom.This also appears to be the case for those studies exploring how particular curricula (e.g., PLTW vs. NCETE), access to information, and cultural and gender norms impact students' ability to design.How and to what extent these findings are informing curriculum and instruction is an important question.
In terms of student learning outcomes, a few of the seven studies reported some positive impacts, but several documented minimal or mixed results.Nevertheless, it is challenging to identify any consistent findings across the studies because the research contexts, designs, and outcomes measured varied greatly.The context of engineering design varied across the studies, whether it was embedded in a unit of instruction, a course, or an entire curriculum.The study designs also varied from using pre-and post-test data in a quasiexperiment to correlating variables to determine possible relationships between them.Variables such as exposure to a type of curriculum, academic history, and standardized test scores were used to determine possible correlations.The outcomes being measured across the seven studies also varied from measuring the impact of engineering design on students' understanding of engineering, student achievement on state mathematics and science standardized test scores, and students' ability to transfer mathematics, science, and design concepts.It appears that the targeted outcome of engineering design on student learning includes several dimensions or aspects of learning and that the evidence of impact is scant to nonexistent.
As several of the researchers noted, it is important to understand how students' exposure to engineering design impacts their perceptions because their self-efficacy and interest levels can impact their future engagement in engineering.The majority of these studies were more evaluative in nature, collecting student perception data as a result of their involvement in an engineering design-oriented program or experience.The other two sought to explore students' future interests in STEM and possible gender differences or as a result of exposure to engineering education.These types of studies are particularly important for engineering and technology education because it is largely an elective in the K-12 classroom.Staying attuned to students' interests and perceptions is key to orienting the curriculum to draw the most number of students possible.In addition, one of the primary motivators for focusing on STEM education is to motivate students to major in and pursue careers in STEM.The role of engineering design in accomplishing this goal is important to study, but clarity in what is being measured and what is being reported is crucial to draw broader conclusions.

Conclusion
The purpose of this study was to understand the research evidence regarding the impact of engineering design on technology and engineering education students.Admittedly, the research footprint is not very extensive because it has only been a decade or so since the field has been actively engaged in researching the impact of engineering design in the technology and engineering education classroom.In terms of many scholarly endeavors, this area of research is in its infancy.Further, the limits of the design of this study, including the identification of studies from only four journals in the past decade, further narrows the scope of analysis.Publications from other research journals, proceedings from conferences such as the American Society for Engineering Education annual conference, and dissertations might contain further research on the impact of engineering design on students.
Another potential limitation of this research review, and perhaps in the framing of the purpose of this study, is the assumption that the research community and practitioners (e.g., teachers, curriculum developers, and professional development providers) are all approaching engineering design in a similar way; that there is an "engineering design process" in technology and engineering education.As discussed above and indicated by the variety of curricular approaches and experiences in the studies reviewed, perhaps there is not one (and should not be one) engineering design process.Flowers (2010) cautioned against the dogmatic use of the definite article in phrases such as the engineering design process and suggested that "one solution to the problems mentioned concerning definite article usage and the bigger issue of dogma is to question our assumptions, even at the expense of our comfort" (p.18).If the points of comparison are to be fair, a more thorough review of how researchers, teachers, and students are defining or approaching engineering design would help.In other words, a more nuanced understanding of how engineering design is being implemented in classrooms, how students are experiencing engineering design, and the outcomes of those experiences is needed.
There are certainly lessons to be learned in terms of how engineering design can impact student learning, students' perceptions of engineering and STEM careers, and how students approach the design process.Obviously, there is considerably more work that needs to be done to provide the kind of evidence needed to be able to determine the impact of engineering design experiences on dimensions such as learning, interest, and creativity.As Katehi, Pearson, and Feder (2009) stated, Meaningful improvements in the learning and teaching of engineering-and movement toward integrated STEM education-will not come easily or quickly.Progress will be measured in decades, rather than months or years.The necessary changes will only happen with a sustained commitment of financial resources, the support of policy makers and other leaders, and the efforts of many individuals in and outside K-12 schools.(p.14) The lack of strong research evidence on the impact of engineering design in technology and engineering education points to the need for more concentrated efforts in this regard.The National Academy of Engineering and National Research Council reports identified in this study offer guidance for next steps and point to needed areas of research that would help inform the collective efforts of engineering and technology education.Given that the STL were published over a decade ago and that technology and engineering education has charted the course toward engineering design, it would seem like an opportune time to develop a focused and strategic research agenda that would help inform the collective efforts of researchers and scholars to be able to better answer questions concerning the evidence of impact.Expanding the number of student research participants, diligently reporting the demographics of those students, following rigorous research design methods, clearly describing those methods and the engineering design approaches and experiences that students are engaged in, and documenting the outcomes (whether on learning, interest, or some other dimension) are crucial steps forward.

Table 1
Research Topics of the Studies