AC 2007-1732: DELIVERING CORE ENGINEERING CONCEPTS TO SECONDARY LEVEL STUDENTS

Introduction Within primary and secondary school technology education, engineering has been proposed as an avenue to bring about technological literacy (Lewis, 2005; Dearing & Daugherty, 2004). Different initiatives such as curriculum development projects (i.e., Project ProBase and Project Lead The Way) and National Science Foundation funded projects such as the National Center for Engineering and Technology Education (NCETE) have been developed to infuse engineering into primary and secondary education. For example, one key goal of the technology teacher education component of NCETE is to impact the focus and content of the technology education field at the secondary level (National Center for Engineering and Technology Education, 2005). More specifically, the goal is to facilitate students' learning relative to core engineering principles, concepts, and ideas. A number of activities have been developed by the Center to facilitate these goals, including a series of teacher professional development experiences, research designed to identify core engineering concepts, development of engineering-related activities, engagement with faculty from the STEM disciplines, and interaction with technology education pre-service teachers. Through the efforts of NCETE, three core engineering concepts within the realm of engineering design have emerged as crucial areas of need within secondary level technology education. These concepts are constraints, optimization, and predictive analysis (COPA). COPA appears to be at the core


Delivering Core Engineering Concepts to Secondary Level Students Introduction
Within primary and secondary school technology education, engineering has been proposed as an avenue to bring about technological literacy. 1,2 Different initiatives such as curriculum development projects (i.e., Project ProBase and Project Lead The Way) and National Science Foundation funded projects such as the National Center for Engineering and Technology Education (NCETE) have been developed to infuse engineering into primary and secondary education. For example, one key goal of the Technology Teacher Education component of NCETE is to impact the focus and content of the technology education field at the secondary level. 3 More specifically, the goal is to facilitate students' learning relative to core engineering principles, concepts, and ideas. A number of activities have been developed by the center to facilitate these goals, including a series of teacher professional development experiences, research designed to identify core engineering concepts, development of engineering-related activities, engagement with faculty from the STEM disciplines, and interaction with technology education pre-service teachers.
Through the efforts of NCETE, three core engineering concepts within the realm of engineering design have emerged as crucial areas of need within secondary level technology education. These concepts are constraints, optimization, and predictive analysis (COPA). COPA appears to be at the core of the conceptual knowledge needed for students to understand and be able to do engineering design.
One of NCETE's five technology teacher professional development institutions (Illinois State University) has focused exclusively on the delivery of these COPA concepts. These concepts have emerged as distinctly important. At Illinois State University, two cohorts of practicing and pre-service technology teachers have engaged in professional development workshops to become better prepared to deliver engineering concepts to their students. Through discussions about the nature of engineering and engineering design, how it differs from technology education processes, and the necessary conceptual and procedural knowledge, constraints, optimization, and predictive analysis were identified as necessary for the delivery of engineering design.
Empirical knowledge is needed to better understand how to increase student learning of COPA. This study sought to provide that knowledge using a NCETE cohort of practicing and pre-service technology teachers who designed and developed a unit of instruction to deliver these three core engineering concepts to secondary level technology education students. Using a mixed method quasi-experimental, pre-test post-test no control group design, this study explored the extent to which students understood and were able to demonstrate an understanding of constraints, optimization, and constraints. It is believed that through this strong conceptual base, a better understanding of engineering and engineering design can be achieved. the analytical stage of the design process represents an indispensable part of engineering design. 5 Eekels observed how the prediction component functions in the engineering design process noting that "if the conditional prediction sounds unfavorable, then we generally simply abstain from that action and design another action," (p. 176) which is to make the informed decision before constructing the prototype of a design. 6 Hayes observed that predictive analysis is carried out in the planning environment, not the task environment, with several distinct advantages: (a) moves made in planning the environment can be easily undone while task environment actions cannot be reversed; (b) predictive analysis is relatively inexpensive; and (c) it permits design flexibility. 7 "The process of thinking before acting" is critical if designing is to be a predictive rather than a trial-and-error process. 7 Trial-and-error remains the prevailing approach to design in technology education classrooms, where analytical mathematical tools are frequently not used to design and prototype design ideas. 8,9 Lewis argued that conceptual design is within normal purview of technology education and that science and mathematics should be taught to help students make predictions about the design through the process of analytical design. 1 The Standards for Technological Literacy: Content for the Study of Technology reinforce the systematic aspects of predictive analysis. "Because so many different designs and approaches exist to solving a problem, a designer is required to be systematic or else face the prospect of wandering endlessly in search of a solution" (p. 91). 10

Constraints
The design processes utilized in engineering and technology education are very similar with some notable exceptions. Lewis has suggested that engineering design places more emphasis on assessing constraints, trade-offs, and utilizing predictive analysis compared to technology education. 1 The importance of constraints is, however, included in the Standards for Technological Literacy: Content for the Study of Technology. 10 In Standard Eight, constraints are viewed as an integral part of an iterative process that typically requires students to consider costs, economics, feasibility, time, material, and environmental implications. Students should be able to assess and incorporate constraints into design activities.
Addressing constraints early in the problem identification stage may assist students in developing viable solutions, since this process helps reduce the size of the solution space. 11 Expert designers typically move quickly from defining the problem (problem space) to the solution space by assessing the problem constraints and searching for contextually related problems that they have solved in the past. 12,13,14 This is similar to a model revised by Middleton, where the problem space is defined as the problem state, goal state, and search state. Middleton's "search state" can be viewed as identifying the constraints; while iteratively moving between the problem state and goal state; and concurrently decomposing an ill-defined problem into well-defined subproblems. 12,15 Optimization Design optimization extends beyond simply producing a design that adheres to a defined set of constraints or criteria. The purpose of optimization is to achieve the "best" design relative to a set of prioritized criteria or constraints. These include maximizing factors such as productivity, strength, reliability, longevity, efficiency, and utilization. Engineers must make many technological and managerial decisions during the design process in order to produce a best design. The ultimate goal of all such decisions is to minimize undesirable effects, while Page 12.443.3 maximizing desirable effects, producing a "better, more efficient, less expensive solution that is in harmony with the laws of man and nature". 16 Optimization typically occurs during the formulation of a design problem. According to Arora, formulation of a problem requires approximately 50% of the total effort needed to solve it. Optimization techniques provide well-defined procedures to aid the designer in correctly formulating the problem. 17 For example, Statnikov outlined three questions that designers should be able to answer when formulating a design problem: 1. What to search for? (resulting in identifying the performance criteria.) 2. Where to search? (resulting in defining all the constraints imposed on the design, which produces a set of feasible solutions.) 3. How to search? (resulting in identifying the optimization technique that is most suited for the specific features of the problem being solved.) 18 Formulating a design problem to achieve an optimal solution often involves transcribing a verbal description of the problem into a well-defined mathematical statement. This process enables the designer to search for the optimal design according to the identified performance criteria. Optimization methods frequently use mathematical concepts such as vector and matrix algebra, and calculus to analyze and optimize variables. As Arora pointed out, "the importance of proper formulation of a design optimization problem must be clearly understood because the optimum solution will only be as good as the formulation is" (p. 21). 17

Purpose and Research Questions
The purpose of the study was to assess the effectiveness of a unit of instruction in teaching core engineering concepts to secondary level technology education students. The following research questions guided the study. 1. What type of engineering design activities and lessons will effectively deliver selected core engineering concepts to technology education students at the 10-12 grade level?
2. Is there a relationship between performance in mathematics courses taken prior to participating in the unit of instruction and post-test instructional gain?
3. Is there a relationship between performance in physical science courses taken prior to participating in the unit of instruction and post-test instructional gain?

Research Design
This study used a mixed methods quasi-experimental, pre-test post-test, no control group design, with the treatment as the independent variable and pretest-posttest as dependent variables. The participants in this study received a pre-test, treatment, and a post-test. Additionally, after the posttest, focus groups with randomly selected participants were conducted to provide qualitative data that allowed for a more in-depth analysis of the data.

Treatment
During the 2005-2006 school year, each technology education teacher that was involved in this study, including one mathematics and science teacher, completed 120 hours of professional development related to infusing engineering concepts into high school technology education. Page 12.443.4 During this professional development, the eight technology education, one mathematics, and one science teacher helped to develop the 20-class session unit of instruction and the activities that supported infusing engineering concepts into the curriculum. These teachers were chosen because they were already participating in the NCETE professional development from which the study emerged. In addition, these teachers helped to solidify the treatment fidelity because they were key researchers in the development and delivery of the unit of instruction. The unit of instruction included four lessons, with specific content and activities. Other than the first lesson and activity (see below), teachers were permitted to teach the remaining lessons in any order. The unit of instruction was scheduled to be completed in 20 class sessions. Some of the teachers completed the unit prior to the 20 days, while other teachers went beyond the targeted number of days. Each lesson had a student version and teacher version. The teacher version of each lesson included supplemental materials including PowerPoint presentation materials specifically designed to address the key concepts being delivered in the unit. Grading rubrics were provided to the teachers and students for each lesson. Below is a description of each lesson and activity.
The first lesson and activity (treatment) that the students (research participants) completed during this study to introduce COPA was called "Volume Barge." In this lesson and activity, students were challenged to design and create a barge-type artifact made from one piece of 8.5" x 11" laminated card stock capable of holding the most weight before sinking; the barge had to be a rectangular prism shape. Students used volume calculations to optimize the best design based on the constraints. In a competitive style format, students graphed, using Microsoft Excel, the entire class performance to determine the winner. This lesson and activity was deemed the favorite by most of the students from the eight schools, largely due to its competitive nature.
A second lesson and activity the students completed was related to energy efficiency. During this lesson, students used mathematical formulas and existing data to determine R and U values for insulated wall cavities. Each group of students was provided with four completed wall sections, each having a 12" x 12" opening constructed with 2" x 4" and 2" x 6" framing materials. Three of the wall cavities were filled with different insulating materials, while the fourth section was left empty. The students were challenged to calculate the efficiency versus cost in a life-cycle approach, to determine the most optimal choice for insulation based on an average daily temperature and cost per thermal unit. Each completed wall section (four in all to create a square) was covered with a sheet of plywood, and a 100 Watt incandescent lamp was placed in the center. Using an infrared heat-sensing device, the students were able to determine insulation efficiency. From the experiences learned from this activity, the students had to use predictive analysis to optimize the life cycle costs of construction and building ownership over 5, 10, 15, and 25 year periods for a 2000 square foot structure. Students used Microsoft Excel to graph their results and present their results to the class. This lesson and activity was classified as the second favorite of most of the students from the eight schools, largely because it involved a hands-on experience.
The third lesson and activity the students completed utilized a pre-fabricated golf ball launching device made from PVC and wood framing materials. Students used predictive analysis techniques to accurately launch a golf ball from a specific height and angle to a specific end distance. Students had to understand vectors, laws of motion, and energy to succeed. Students first predicted (non-analytic) how far the golf ball would travel and then used mathematical Page 12.443.5 formulas to analyze how and where the ball would travel. In addition, students quickly learned that there exists an efficiency factor and that no machine or mechanism is 100% efficient. Students graphed their results and presented their findings to the class. This lesson and activity was rates as the third favorite by most of the students from the eight schools involved in this study, largely because the students could see the mathematical calculations in action in observing where the ball should and would land.
The fourth lesson and activity the students completed dealt with identifying where and how mechanical energy is used and lost in their school. As an introductory activity to this lesson, students were provided with four different types of light (incandescent, fluorescent, halogen, and LED) to examine their efficiency. After classroom discussion regarding the cost, life-span, and energy used from these different sources of light, students became engaged in an activity where they had to locate four different sources of mechanical energy in their school to determine their efficiency. Students created a proposal that outlined where mechanical energy is being lost throughout their school and how they would use the wasted energy to complete productive work in other applications. Students used Newton scales, stop watches, and tape measures to determine mechanical energy (i.e., force, distance, and time). Each potential solution that students determined also included a wattage factor. During the presentation, students discussed their data collection methods, design solutions, the constraints associated with each design solution, and how each design would be optimized. Overall, students enjoyed this lesson and activity, but felt because it lacked a hands-on (building) approach, their engagement was not as high.

Instrumentation
The research team and a technology education teacher, who has an engineering degree, developed the test instrument used in this study. The development of the test instrument was guided by a review of the literature related to the engineering concepts: constraints, optimization, and predictive analysis. The thirty-item test instrument was developed to target the three concepts across three levels of Bloom's taxonomy. For the purpose of this study, Bloom's framework was reduced to three levels; comprehension, application, and analysis/synthesis. Comprehension included Bloom's Knowledge and Comprehension categories, application included Bloom's Application category, and analysis/synthesis combined Bloom's Analysis, Synthesis, and Evaluation categories. 19 For example, questions written at the comprehension level used verbs offered by Dalton and Smith, including explain, predict, or discuss.
Ten items were developed to target each of the three engineering concepts and were spread relatively evenly across the three levels of Bloom's taxonomy. At the comprehension level there were four constraint, four optimization, and four predictive analysis questions. At the application level there were three constraint, three optimization, and three predictive analysis questions. At the analysis/synthesis level there were three constraint, three optimization, and three predictive analysis questions.
The test was administered to the cohort of practicing and pre-service teachers during the summer professional development experience to ensure that the instrument was at the appropriate level of difficulty for secondary students and to identify any problematic questions. The research team and the technology education teacher then later refined potentially problematic questions. A further measure to ensure content validity was undertaken by sending the instrument to a panel of engineering and technology education professors and practitioners. These individuals completed a review of the instrument and a survey asking whether the questions measured an Page 12.443.6 understanding of the three concepts at the different levels. Based on the feedback from the expert review panel, the test was further refined by the research team. This process resulted in an instrument containing thirty items that was believed to be at the appropriate difficulty level that measured an understanding of COPA at the different levels. The reliability of the test instrument using Cronbach's coefficient alpha was r = .782.

Sampling Procedures
A purposive sampling frame was utilized for this study, so the same teachers who developed the instructional materials were able to deliver the actual instruction in their classes to the study participants. Eight of the technology education teachers who participated in the NCETE professional development sessions at ISU recruited the students from their schools. Recruitment was conducted in nine intact technology education classes, since one teacher was able to recruit participants from two separate classes. Initially, 124 high school-level technology education students agreed to participate in the study. However, as a result of attrition, only 114 (n = 114) students remained in the study at the time of the posttest. Within the final population there were 102 male and 12 female students.
In order to assess if there were significant differences between the subjects who remained in the study and those who did not, a one-way ANOVA at the a=.05 level was conducted using the pretest scores as the dependent variable. The results suggested that there were not significant differences between the two groups in terms of test scores F(1,123) = .04, p > .05.
Within twenty-four hours after completing the posttest, a focus group of systematic selection procedure was used to identify students from the eight high schools who participated in the study. Using the alphabetized course roster for each of the nine classes, every third student was selected to potentially participate in focus group. However, if that student did not wish to participate or had not submitted a consent form, that student was not selected for the sample. In a few cases, every third student resulted in a sample size of less than six, so two cycles of every third student, beginning with the last student selected, was conducted. For one school, the class size was small enough to conduct a focus group with all of the students. For two other schools, only a few students submitted consent forms or agreed to participate in the focus group, so all of those students participated in the focus group. A total of nine students from each school were selected to participate in the focus groups. The first six students selected were targeted as the primary participants, and three students were selected as alternates in case of absences or withdrawals. A total of eight schools and nine separate classes were involved with the focus group (n = 54). Of the 54 students selected for the focus groups, 47 were males, 7 were females.
Each focus group, lasting no longer than forty-five minutes, was guided by a script of fifteen questions that were divided into three categories: appeal questions, probing questions, and suggestions for improving the unit of instruction. Each focus group was conducted by two members of the research team. One of the researchers asked the questions and the other researcher acted as the scribe. In some cases, both the interviewer and scribe asked the students questions. Each focus group session was digitally recorded and saved as an MP3 file. Each of the researchers independently listened to each focus group session and developed synthesis paragraphs. Page 12.443.7

Procedures
This study was conducted in two phases. Phase One was the development of the 20-class session unit of instruction to be delivered to high school level technology education students, while Phase Two consisted of delivering and assessing the unit of instruction with the participants of this study. Phase One began during the Summer 2006 technology education session at ISU. Twelve high school level technology education teachers attended a five-day professional development session to develop the unit of instruction to integrate COPA in their technology education courses. Some of the teachers in this experience had previously attended ISU's professional development sessions the previous year. Those sessions were also focused on integrating engineering concepts into high school level technology education curricula.
During the first two days of the summer 2006 session, the teachers participated in presentations by technology education faculty members from ISU, an engineering professor from the University of Illinois at Urbana-Champaign (UIUC), NCETE doctoral fellows from UIUC; and science, mathematics, and technology education teachers from the public high school system. These presentations included a review of COPA, Wiggins and McTighe's backward design process, and activities that emphasized the COPA concepts. 20 During the third and fourth day of the workshop, the teachers working in groups of four developed the unit of instruction using the backward design process. 20 As discussed in the instruction development section, each group of teachers developed artifacts for each of the different activities. Concurrently, the NCETE doctoral fellows, in conjunction with the technology education teacher from the high school system, began developing the test instrument to assess students understanding of COPA.
On day five, the initial instrument was pilot tested with the high school teachers and graded. The results were shared with the teachers in order to obtain feedback regarding content and construct validity, and appropriate floor and ceiling height for high school level students. Additionally, the groups of teachers presented their activities to the session participants in order to obtain feedback that could be used to later enhance the activities. After the conclusion of the experience, the eight teachers who would actually be able to deliver the unit of instruction were asked to continue with refinement and final development of the activities.
Between August and October, final lessons and activities related to Phase One were completed: (a) Institutional Review Board protocol approval was obtained from ISU and the UIUC, as well as from the individual high schools, (b) the test instrument was further refined after obtaining feedback from engineering, technology education professors, and practicing teachers, and (c) the eight teachers returned to ISU on October 5, 2006 for a one-day session to deliver the finalized units of instruction to the participating teachers involved in the study to receive formative feedback. Additionally during this time, technology education pre-service teachers enrolled at ISU began constructing the artifacts that were needed for the activities and pre-assembled these into kits that were sent to the participating high schools. The use of pre-assembled kits was deemed necessary in order to maintain treatment fidelity by making certain that all sites were using identical materials. In addition to uniformity provided by the pre-assembled kits, the process reduced the total time required to implement the study. This time element was an Page 12.443.8 important factor, since teachers were injecting the research unit into their regular semester's curriculum.
Phase Two began during the first week of October 2006. Since the high school students were under 18, both parental consent and student assent were required. The technology education teachers distributed the appropriate forms to the participants; signatures were obtained and the forms were returned to their teachers who then forwarded the forms to the research team. The participants were then administered the pre-test that consisted of the 30 items related to measuring COPA at the three-levels of understanding.
The delivery of the unit instruction by the teachers began between the second and third week of October 2006. Because the teachers were working within the limitations of their existing curriculum, it was not feasible for all sites to begin delivery of the unit of instruction on the same day. The unit of instruction was delivered over the span of 20 class sessions and immediately following the conclusion of delivering the 20 class session unit of instruction; the posttest was administered to the participants and returned to the research team.

Findings and Discussion
Consistent with the mixed model research design, data analysis consisted of quantitative and qualitative components. With the quantitative component, a series of dependent groups t-tests were conducted to explore differences between the pre-test and post-test. Student performance related to selected demographic variables was also examined. Qualitative data analysis consisted of examining the transcripts of post-instruction focus groups, which were conducted by the research team to explore students' understanding of core concepts and the efficacy of the study's unit of instruction.

Quantitative Data Analysis
A series of dependent groups t-tests were conducted to compare pre-test and post test scores. The initial analysis, which was conducted on the composite test scores, was followed by separate analyses of the instrument's three dimensions (i.e., constraints, optimization, and predictive analysis). Student scores consisted of the number of items answered correctly of the instrument's 30 questions. A significant composite score gain of 3.22 was obtained between the pre-test and post-test (see Table 1). While the gain scores were statistically significant, the overall percentage of items answered correctly was somewhat disappointing. Based on focus group discussion and interaction with participating teachers, low composite test scores were attributable to several factors. Among Page 12.443.9 these factors were perceived test difficulty and voluntary participation in the study, where students were informed that test results would not be counted in their semester grades. In spite of relatively low test scores, focus group discussion, however, indicated that students clearly were able to identify the core engineering concepts selected for the study. In the aggregate, they also possessed a relatively sophisticated understanding of the interrelationship among the concepts. Focus group results suggest that the instruction may have been more effective than what the test scores indicate.
In a focus group conducted with participating teachers, there was broad consensus that the test results were negatively impacted by the structure of the test. This included its high readability and conceptual levels as well as difficulties with knowledge transfer of examples used in the instrument and activities used during instruction. In spite of efforts made to validate the instrument during the planning stages of the study, it is clear that the study's outcomes were influenced by these assessment issues. In spite of these issues, the teachers indicated that their students' understanding of COPA concepts was clearly achieved as a result of the study.
Separate dependent groups t-tests were also conducted for the instrument's sub-scores (i.e., items assessing the three core engineering concepts). Mean score gains ranging from approximately 1 -1 ½ items were obtained with the highest gain score on the predictive analysis dimension. All three gain scores were statistically significant (see Table 2). The instrument's design also included the development of items at three levels of complexity along Bloom's taxonomy of cognitive skills (comprehension, application, and analysis/synthesis). As could be anticipated, the highest net score gain occurred at the comprehension level (2.07) and the lowest score gain was obtained at the analysis/synthesis level (.53) (see Table 3).
As noted above, focus group discussion indicates that the level of students' understanding may exceed that suggested by the test score data. For example, a common observation of students was how the three engineering concept dimensions interact with one another in real world engineering design situations. Specifically, students commented on how optimized engineering Page 12.443.10 designs routinely require tradeoffs among constraints (e.g., it is not possible to optimize all constraints; constraints tend to compete with one another). The data were also analyzed to examine the possible effects of selected demographic factors on student learning. These factors included gender, ethnicity, and level of mathematics and science courses. Analysis of variance procedures, which were conducted on each of the variables, detected no statistically significant differences.
Non-significant differences on the selected demographic variables are encouraging. Given the well-documented concern about female and minority involvement in scientific and engineering careers, this study's results indicate that gender and ethnic differences may be minimized or reduced in controlled instructional situations. In other words, gender and ethnically-sensitive instructional design may facilitate learning of engineering concepts in ways that minimize demographic differences. The results of this study are encouraging, given the concern of many technology educators that the growing emphasis on engineering could reduce participation of a broad spectrum of students.

Qualitative Data Analysis
All of the focus group students commented on the amount of mathematics and science that was included within the unit of instruction and activities; predictive analysis throughout all of the activities was the least favorite aspect of the twenty-day unit of instruction. With the exception of one focus group, all of the students knew that constraints, optimization, and predictive analysis (COPA) were the key concepts being taught throughout the unit of instruction and provided examples of how and when they learned these concepts. As discussed in the treatment section, each lesson and activity targeted one or more of the key concepts. A majority of the students defined optimization as "the best solution to a problem, balancing trade-offs between competing factors." Students defined predictive analysis as the "mathematical or scientific equations that are used before the artifact or problem is completed." Students defined constraints as "the boundaries for what you can do and the parameters you have to stick to." Of the three COPA concepts, predictive analysis was the most difficult to understand for the students and constraints was the easiest. A small minority of the students who completed the Page 12.443.11 focus groups identified the COPA concepts as interconnected. Students used an analogy of the scientific method to COPA. Students were not familiar with optimization and predictive analysis before the treatment, but through their prior or existing technology education courses were familiar with constraints. Students also commented that they take technology education courses because they are fun and activity-based, not mathematics or science-based.
All of the focus group students rated the "Volume Barge" activity as their favorite. This activity focused on the concept of optimization. Furthermore, the activity challenged students to compete against their classmates for the best barge. A majority of the focus group students wanted more open design activities that were similar to the barge activity.
Almost all of the focus group sessions revealed that students wanted a launching device, similar to a pneumatic powered device instead of the golf ball launching device that was used in the unit of instruction; students wanted a "boom" effect rather than the gravity fed device provided. Students seemed to like the wall insulation activity because it was more hands-on than the other activities within the unit.
Overwhelming, the students in the focus groups commented on how mathematics and science concepts taught throughout the unit were better understood when they were connected to solving a problem or building an artifact. Students commented that they did not understand mathematics and science in their stand alone courses. Students also commented on the positive nature of including most of the formulas they would need to solve for problems within student handouts or embedded within the activities. However, students commented that the theory of mathematics and science does not always translate into a properly working artifact.
The last question to each focus group was whether or not this unit of instruction had influenced them to pursue post-secondary studies and a career in engineering or a related field. There was no indication that after completing the twenty-day unit of instruction that a positive or negative influence existed.

Conclusions and Recommendations
Based on the results of this study, some conclusions become apparent. It is clear that student learning was achieved as a result of students' participation in the engineering design unit of instruction. While mean score gains from pre-test to post test were modest, they did indicate significant improvement in understanding of COPA concepts. Given the lack of significant gender, ethnic group, and mathematics/science background performance differences, the study also indicates that engineering concepts can be successfully delivered to a broad spectrum of students. These preliminary results are important since many technology educators are concerned that an engineering curricular focus could appeal to an academically capable subset of the technology education student body.
Based on focus group discussions with students and teachers, some important factors emerged related to how engineering concepts were delivered in this study. These factors have important implications for future research, curriculum development, and professional development.
One key factor has to do with an overt shift from procedural/activity-based curriculum and Page 12.443.12 instruction, which as been typical for technology education, to an overt concept-based focus. The importance of this shift certainly extends beyond this research study or engineering curriculum. In this era of standards-based instruction, the technology education field must learn how to balance the historical appeal of engaging activities with curriculum development that is specifically designed to teach concepts (standards). While students generally indicated that they enjoyed the study's activities, they also reported that they would have preferred to have actually constructed more of the devices used in the study (rather than having them pre-constructed in order to meet the time and treatment fidelity constraints of the study).
Another significant challenge of research of this type has to do with the constraints involved with informed consent research. Focus group results indicate that student motivation to perform well in the study was eroded by their awareness that the test outcomes would not be included in their semester grades.
Another important factor that emerged from the study had to do with the challenges associated with developing high quality, authentic assessments of COPA concepts. The outcomes of the study indicate that the test instrument was capable of detecting student learning at the various levels of conceptual difficulty. The psychometric properties were also sound, with acceptable levels of reliability and validity. Teacher involvement in the development and validation of the instrument used in this study was designed to ensure its appropriateness, including appropriate level of difficulty. However, focus group feedback indicated that students found the items to be demanding both in terms of reading level and conceptual load. While the multiple choice format provided objective data, future research should include more diverse and authentic formats.
However, the findings of this study also indicated that there are specific areas of need in order to better develop these engineering concepts. For example, existing and pre-service technology education teachers need to be better equipped to develop and teach instruction focused on engineering design concepts. In particular, professional development focused on preparing technology education teachers to develop and teach instruction that is both concept-driven and activity-oriented is an area of need. Historically, technology education has focused primarily on procedural knowledge through hands-on activities that focus primarily on the artifact. In order to integrate engineering concepts within technology education, teachers need to develop pedagogical skills that include more focus on conceptual knowledge and the processes involved in engineering.
Another area of need is the development of sound curriculum, activities, and assessments that target engineering design concepts. The instruction and activities developed for this study appear to have done an adequate job relaying the concepts to the students. However, with more refinement and focus these and similar activities could be used to teach engineering concepts even more effectively beyond the twenty-session research treatment. For example, separate units on constraints, optimization, and predictive analysis may help students better understand these engineering concepts. These activities need to maintain a hands-on component, which is an area of strength for technology education, because it appears to be a key to student motivation. In addition, authentic assessments need to be developed to assess student understanding. As revealed in this study, there are limitations to using tests to assess student learning of engineering Page 12.443.13