Quality and Characteristics of Recent Research in Technology Education

The focus of research in technology education has evolved throughout its history as the field changed from industrial arts to technology education (Spencer & Rogers, 2006). With the move to technology education, the field has begun to broaden its focus to better understand the teaching, learning, curriculum, and policy implications of preparing the next generation of technological thinkers. Although a complete “paradigm shift” may not have occurred completely within all technology-oriented programs (Sanders, 2001), the current emphasis on engineering within technology education indicates a need to examine and assess the status of technology education research over the past ten years to identify strengths and areas that need to be addressed in order to guide the field into the future.

would focus its financial efforts to expand the number of programs and projects that use rigorous scientifically based research methods. "The definition of scientifically based research in section 9201(37) of NCLB includes other research designs in addition to the random assignment and quasi -experimental designs that are the subject of this priority. However, the Secretary considers random assignment and quasi -experimental designs to be the most rigorous methods to address the question of project effectiveness" (Scientifically Based Evaluation Methods, 2005, p. 3586).
Recently, the U.S. Department of Education's Report of the Academic Competitiveness Council (2007) was released to address concerns within the science, technology, engineering, and mathematics (STEM) disciplines. Focusing largely on research methods, the report highlighted the "critical pathway for the development of successful educational interventions and activities, starting generally with small-scale studies to test new ideas and generate hypotheses, leading to increasingly larger and more rigorous studies to test the effect of a given intervention or activity on a variety of studies and in a variety of settings" (p. 13). The American Competitiveness Council adopted a methodological framework that displayed a hierarchy of methods for evaluating the quality of current and future STEM research. At the top of the hierarchy are experimental methods such as randomized control trials (RCTs), which "enable one to determine with a high degree of confidence if the intervention alone caused observed outcomes" (p. 15). Following experimental methods, the next level of research methods is quasi -experimental approaches that include comparisongroups that are closely matched on key characteristics (e.g., prior educational achievement, demographics, etc.). At the base of their framework were other designs including pre-and post-test studies and designs that may not have careful matching of comparison groups. The hierarchy does not include other methods (e.g., qualitative, survey), however, the Council acknowledges that other research methods are a key part of the research agenda needed to improve STEM education and that these methods can be "rigorous" in their own context. Many within education have reacted either favorably or critically to these calls for education reform. For example, Borman (2002) agreed with the NRC's call for more rigorous research, stating that in order to "develop and improve programs and practices in U.S. schools and classrooms, research methods must separate fact from advocacy, provide the most believable results, and inform with great confidence the question, 'What works?'" (p. 10). Borman argued that the best answers come from experimental studies because they ensure that the intervention caused the different outcomes in the treatment and control groups. Others within the educational research community, however, have responded critically (Lather, 2004;Moss, 2005;Willinsky, 2005). In particular, Maxwell (2004) argued that the federal reports privilege quantitative methods, "treating qualitative methods as merely descriptive and supplementary to 'causal,' quantitative methods, largely ignoring the unique contributions that qualitative methods can make to causal investigation" (p. 8). In addition, St. Pierre (2006) complained that scientifically based research has "become the 'truth' in education, and that truth is being maintained and perpetuated by a whole network of discursive formations and material practices that are increasingly elaborated by a knowledge/power system that may not be in the best interests of education" (p. 243).

Purpose of the Study
The purpose of this study was to assess the quality and characteristics of the research that has been published within the field of technology education between 1997 and 2007. More specifically, this study was designed to determine the types of research conducted within the technology education field, including the research focus, methods, primary data sources, and data types. The following questions were explored to accomplish this purpose: 1. What types of research have been conducted in tec hnology education over the past 10 years? 2. What research methods have been most commonly used in technology education research over the past 10 years? 3. What types of people and data have been the focus of research in technology education over the past 10 years?
A driving motivation for this study was to explore the extent to which technology education research conforms to Shavelson and Towne's (2002) "guiding principles" of scientific inquiry, and to gauge the alignment of technology education research with the current national trend toward a "gold standard" for educational research methods. Answers to these questions provide insight into the degree to which recent technology education research aligns with the "gold standard" for educational research.

Prior Critiques of Research in Technology Education
There have been a number of prior analyses of the research quality in technology education over several decades. These analyses have been consistent in terms of the concerns that have been raised regarding the o verall quality of the research (Foster, 1992;Johnson, 1993;Lewis, 1999;McCrory, 1987;Passmore, 1987;Sanders, 1987). Most recently, Zuga (1997) examined research that was published in the main technology education journals and dissertation abstracts from 1987 through 1993. Zuga found that half of the 220 studies she reviewed were primarily descriptive and focused on curriculum. Zuga outlined four areas missing from technology education research: (a) constructivism, (b) integration, (c) inclusion of all students, and (d) cognition. Constructivist problem-based instruction, according to Zuga, is fundamental to technology education, along with the integration of other subjects, especially science and mathematics. However, she found that few of the published studies explored either of these components. What Zuga found to be most disturbing about technology education research was the lack of research that focused on students. In particular, specific groups of students such as females, ethnic minorities, or those who are physically and mentally challenged have been neglected in much of the research. In addition, few research studies explored cognition within a technology education context. Zuga concluded that the technology education research focus "on descripti ons of status and curriculum development points to researchers who are narrow, inwardly focused, and oblivious to the goals of their own field" (p. 213). Petrina (1998) conducted a similar meta-study of research published from 1989 to 1997 in the Journal of Technology Education (JTE). Utilizing metaethnography and both quantitative and qualitative analysis, Petrina performed a content and critical discourse analysis of the studies published in JTE. In terms of research, Petrina concluded that of the 96 articles, 62% involved research methods that were either conceptual or descriptive and only 35% of these involved human subjects. In his examination of "analytical units of substance" he found that few studies explored issues such as appropriate technology, c lass, ecology, gender, labor, race, and sexuality. Petrina concluded that the lack of this type of research indicated a lack of "understanding of the way inequities play out in technology and the trades" (p. 38). Citing a study by Foster (1992) and studies by Zuga (1994Zuga ( , 1995Zuga ( , 1997, he stated that those who examine research in the field have concluded it to be a "malfunctioning practice" (p. 28). Petrina's final analysis of the "state" of technology education research was that "conservative voices are favored and critical voices the exception" (p. 51). For research to be relevant, he recommends that it have "a distinct theoretical component and be cast within particular areas of research practice" (p. 48).

Analysis of Current Research in Technology Education
This study involved another, more recent, examination of the top technology education journals to provide a critique of the current status of the published research. Understanding where we are in terms of research type, method, primary data source, data type, and research focus is a necessary step in improving the quality and impact of research in the future. We asked key leaders in the field to identify what they consider the top research-focused journals in the field of technology education. The following four technology education journals were consistently mentioned by the panel of experts: (a) the International Journal of Technology and Design Education (ITDE), (b) the Journal of Industrial Teacher Education (JITE), (c) the Journal of Technology Studies (JTS), and (d) the Journal of Technology Education (JTE). This is essentially the same list of refereed journals that Zuga analyzed in her 1994 study. The only difference is that Zuga included The Technology Teacher while this study included the International Journal of Technology and Design Education.
All of the articles that were published in these four journals within the past 10 years were obtained and reviewed (see Table 1). Articles were selected for further analysis if they explored some issue directly related to technology education and were based on empirical data that was collected through either quantitative or qualitative methods. A guiding rule was that the studies to be reviewed needed to involve the collection and analysis of data. Therefore, synthesis pieces, commentaries, and opinion pieces were not included in the analysis.
The studies that met the above criteria were then reviewed and coded according to type of research, research method, primary data source, data type, and research focus. The initial codes for research type and method were generated from the classification provided in a typical educational research textbook (Gall, Gall, & Borg, 2007). The articles were screened carefully and thoroughly because some of the studies did not e xplicitly state the method that was used, while for others it was questionable whether the study held true to the method that was stated. For example, many of the studies that were referred to as experimental were actually quasi -experimental because the participants were not randomly selected. Codes for data source, data type, and focus were developed to provide a general, yet descriptive, term that could be used for generating frequency counts across all articles. The initial codes used to classify the research focus were adapted from the coding scheme used in Wankat (2004) in his analysis of Journal of Engineering Education articles. As the analysis proceeded it became clear that modifications to Wankat's coding scheme were needed to better conform to the types of research found in technology education. Changes included combining his Computer and Internet/Web codes into an Educational Technology code, expanding his Gender/Women code to Gender -Race, and adding Opinions-Attitudes and Problem Solving as new codes.
Each article was then reviewed and codes were assigned for the categories of research type, research method, primary data source, data type, and research focus. To determine coding reliability, a second coder reviewed and coded a subset of articles from JTE. This resulted in 38% of the codes being examined by another person. When coding disagreement occurred, the coders discusse d and resolved the disparity. For those codes that could not be resolved, a third coder was used to independently assign a final code followed by a discussion to achieve consensus with the original coder.

Types of Research in Technology Education
As shown in Table 2, the majority of the studies were classified as quantitative research, with fewer qualitative studies and a very limited number of studies involving mixed methods. It should be noted however that the low number of mixed methods studies is a conservative figure. Several of the articles mentioned that they utilized mixed methods, although in most cases only one research method was described in the published study and therefore the research was coded accordingly. The predominance of quantitative studies was considerably more than in the Zuga (1997) and Petrina (1998) analyses. Petrina concluded that technology educators had yet to adopt the interpretive methods used by researchers in other "practical" fields, leading to a lack of qualitative studies in technology education research. As revealed in the current analysis, qualitative research has increased within technology education. Perhaps spurred by Hoepfl's (1997) qualitative methods "primer," technology education researchers appear to be rising to the challenge of pursing research questions through a sustained, in-depth analysis. Regarding the primary research method used, the major ity of the analyzed studies was primarily descriptive in nature and relied heavily on descriptive surveys (see Table 3). This is similar to Zuga's finding that 65% of the 220 studies she classified were descriptive. Petrina reported that 25% of the studies published in JTE were descriptive in nature while Zuga noted that the descriptive research in her review relied primarily on the Delphi technique and mailed surveys. Similarly, Foster (1992) found that the majority of the graduate research in technology education relied on descriptive surveys.
Besides descriptive studies involving the administration of questionnaires, quasi-experimental, correlation, and causal comparative were the next most commonly used quantitative methods. There were no experimental studies that involved true randomization, although this is not atypical for social science research where random selection and assignment of students is often impractical. The use of quasi -experimental, correlation, and causal comparative methods differed dramatically from the analysis reported by Zuga and Petrina, who found that very few studies used these methods. In terms of qualitative methods, interpretive research and case study were the most used, while few studies relied on naturalistic or cognitive methods such as ethnography and protocol analysis.

Types of Data in Technology Education Research
Students or teachers were the primary population groups for the r eviewed studies (see Table 4). When students were the focus of the study, they ranged from preschool students to adult learners. This is a vast change from Zuga's findings (1997) from a decade earlier and Petrina's more recent findings in 2004. Petrina had concluded that "relatively little time has been spent investigating the practice of technology at the local, school -based level" (p. 35). This, however, no longer seems to be the case. The majority of the technology education studies sampled from secondary education populations (n = 54, 49.5%) followed by college students (n = 36, 33.0%) and primary students (n = 18, 16.5%). Almost one-fourth of the technology education studies involved teachers ( n = 40, 20.0%). In the majority of the studies, the teacher population was not defined beyond a general technology education category (n = 22, 55.0%). The number of teachers explicitly specified at the secondary level was low ( n = 6, 15.0%), however it is reasonable to conclude that most of the studies classified a s "general technology education" would encompass the secondary school category. A few studies focused on pre-service teachers (n = 3, 7.5%) and preschool/primary school (n = 9, 22.5%) level.
Zuga noted that most technology education studies seemed to rely on a "closed circle of people" (1994, p. 209) that comprise technology educators and industrialists. This narrow scope appears to be widening somewhat within technology education research. The increase in the number of studies focused on students and the inclusion of administrators, parents, and the general public as population groups, while still a small percentage of the total (n = 9, 4.5%), may indicate a discipline that is beginning to extend its research base and perhaps its influence.

Focus of Research in Technology Education
As noted by both Zuga (1997) and Petrina (1998), technology education research tends to rely heavily on perceptions and self -reports rather than observable or verifiable data. As shown in Table 5, this continues to be the case with the majority of the technology education studies relying on subjective data such as perceptions (25.9%) and self-reports (33.0%). Only 16.8% of the studies relied on observable behavior and very few studies relied on objective or verifiable data such as the analysis of test scores (15.7%) review of existing documents (5.6%), verbal protocols (2.0%), and archival data (1.0%). Regarding the primary focus of the research in technology education (T able 6), most studies addressed issues related to teaching ( n = 42, 21.1%), curriculum (n = 41, 20.6%), and learning (n = 41, 20.6%). This is consistent with Zuga's finding that 50% of the technology education research she reviewed dealt with curriculum, most often by assessing the beliefs of state supervisors and teacher educators. Foster (1992) also noted that the majority of graduate research in technology education focused on pedagogy, curriculum, and program evaluation. In spite of Cajas' (2000) call for more emphasis on studies of student learning, only 20.6% of the technology education research addressed this critical area.

Discussion
As indicated in the above analysis, and also in the studies by Zuga (1994, 1995), Foster (1992, and Petrina (1998), research in technology education has a long way to go before it can be considered "gold standard" research. With no studies in the past 10 years involving randomized controlled trials and relatively few comparison group studies, one would be hard pressed to defend the quality of technology education research, at least when using the U.S. Department of Education's "gold standard" cr iterion. However, given that this national standard for research is not universally accepted, and given the argument that alternative methods of research can be equally powerful in their own right, it would be difficult to criticize the quality of technology education research along these lines.
However, based on the review of published research provided through this study, it is apparent that the focus, methods, and overall quality and rigor of research in technology education needs to be improved along th e same lines as advocated by Zuga, Foster, and Petrina in previous decades. There are indications that Zuga and Petrina's calls for studies focusing on specific issues such as integration, gender, and race have begun to be answered or at least echoed by others. The study by Dyer, Reed, and Berry (2006) is an example of integration within the mathematics and technology disciplines as a viable avenue of exploration. Their study compared the end of year mathematics test scores of high school students who had completed specific technology courses and those who had not. Arguing that "technology education provides a contextual basis for reinforcing the content of the core academic areas" (p. 7), they found that students who took the courses had significantly higher math test scores.
Studies exploring issues of diversity, such as gender and race, are also emerging in the field. For example, Weber and Custer's (2005) study of genderbased preferences set out to describe middle and high school female and male students ' preferences for technology education activities, topics, and instructional methods. In addition, Fazarro and Stevens (2004) explored African-American and European-American learning style preferences to understand how these groups of individuals learn, which the authors argue is "essential to designing and implementing the shift in teaching practice so that all students benefit" (p. 5).
It is important to note that research on the teaching of technology in schools appears to be occurring with a large number of studies using students for their target populations. Studies range from exploring elementary students' ideas about concepts and skills associated with structural stability ( Gustafson, Rowell, & Rose, 1999) to examining the effects of tests on undergraduate technology education students' learning retention (Haynie, 2003). Unfortunately, many of these studies rely on descriptive methods and perceptions, which leads to a rather superficial analysis of the problems that students face when learning about technology. In order to better understand the teaching and learning process as related to technology education, research needs to provide a deeper examination of the complexities and influencing factors that ultimately impact student learning.
This deeper examination can be pursued in cognition studies, as advocated by Zuga (1997) and others. Petrina, Feng, and Kim (in press), for example, investigated research that examined and conceptualized how different age groups learn technology to better characterize cognition research in technology. They found that much of this research investigates age groups from children to adults in isolation, failing to "conceptualise either how we learn technology across the lifespan or how we might study this problem" (p. 2). Th e authors offer two broad categories of research methods to help remedy this problem including: (a) design-based research, and (b) cognitive ethnography. Design-based research is an "intervention research with an experimental connotation but its utility is more general in facilitating research in fairly controlled lab and field settings" (p. 15). Cognitive ethnography, on the other hand, "reframes ethnography through distributed cognition, cognitive psychology, and human factors" (p. 14). Cognitive ethnographers use different measures including analogies, concept mapping, audio and video recording, interviewing, observation, think-aloud, and retrospective protocols.
In particular, protocol analysis is a method that few technology education researchers have used to examine the thought processes of individuals while they complete a task or solve a problem (Atman & Bursic, 1998, Johnson & Chung, 1999. Verbal protocol analysis requires subjects to say aloud everything they think to themselves while performing a task or solving a problem. The researcher's task is to take the incomplete record provided by the protocol and infer the underlying psychological processes by which the subject performed the task (Ericsson & Simon, 1984). Such a method can provide insights and clarity of hidden processes that are only conjectures when examined through self -reports of processes and perceptions. The same can be said for the use of video recordings of students while engaged in design and problem solving activities. A careful and thoughtful analysis of their recorded conversations and actions can provide insights into patterns of behavior that would be transparent through other, more superficial, means of analysis (Crismond, 1997).
A few studies that have utilized protocol analysis include Lavonen, Meisalo, and Lattu's 2002 study. The authors examined collaborative problem solving of 8 th grade students by video recording activities and then coding the video protocols into episodes. Welch, Barlex, and Lim (2000) also utilized protocol analysis and video recordings in their study of 7 th grade student pairs as they produced a solution to a design brief. Likewise, Welch (1998) videotaped pairs of 5 th grade students as they completed a design-and-make task. Each of these studies utilized video recordings to carefully and thoughtfully analyze the conversations and interactions of the students to better understand and interpret the particular questions of interest.
There is also a need to better align the focus of research in technology education with the national movement within the field to place more emphasis on engineering, design, creativity, and problem solving. There are examples of studies that have explored these more recent trends in technology education, for example, Dugger's 1994 study of the similarities and differences in the design processes used by engineers and technology educators. Other studies have explored design thinking by comparing expert and novice design behavior (Christianns & Venselaar, 2005;. Creativity is also a line of research that is emerging within technology education as it moves to embrace engineering design as part of its content base. Lewis (2006) has been a particularly strong advocate, not only for research exploring elements of creativity, but also for creativity to serve as an overarching framework for design and problem solving in technology education. A creativity framework provides "opportunities for students to step outside of conventional reasoning processes imposed by the rest of the curriculum" (p. 36). Studies have begun to explore issues of creativity within technology education including computer simulation (Michael, 2001) and assessment (Doppelt, 2007).
Problem solving is another avenue of research that is being explored w ithin technology education. For example, Sutton's 2003 study explored problemsolving research outside the field of technology education; focusing primarily on research from cognitive science and mathematics. Sutton concluded that there are three primary areas of problem solving that are of particular interest to technology education from the problem solver's perspective: (a) the representation of the problem, (b) his or her background and experiences, and (c) his or her understanding of the problem and its structure. According to Sutton, these three areas provide a "fertile field" (p. 59) for problem solving research in technology education.
While there seems to be movement in a positive direction (i.e., a better balance of quantitative and qualitative research; more inclusive studies; and cognition studies) than in the past, the recent collection of technology education research is still dominated by descriptive studies that rely on self -reports and perceptions. As indicated by the national movement toward more scientifically