Using Teaching Portfolios to Revise Curriculum and Explore Instructional Practices of Technology and Engineering Education Teachers

This paper reports on the use of teaching portfolios to assist in curriculum revision and the exploration of instructional practices used by middle school technology and engineering education teachers. Two new middle school technology and engineering education units were developed through the Engineering for All (EfA) project. One EfA unit focused on addressing world food shortages via the design and construction of urban vertical hydroponic farming systems, and the other focused on providing safe drinking water through the design and construction of water filtration and purification systems (modeled to reflect needs of people in a developing nation, in this case, Bangladesh). To explore the implementation of the new EfA units by teachers and to help with their revision, a new teaching portfolio instrument was developed, validated, and used. The teaching portfolios that participating EfA teachers compiled were evaluated based on a set of Design Teaching Standards that were developed for the project, and which grew out of the informed design teaching and learning model. Findings from the review of the teaching portfolios were used to (a) revise the curriculum, (b) create design-based teaching performance rubrics, and (c) develop specific materials for the professional development of prospective EfA teachers. Findings from this research project were also used to explore the strengths of middle school technology and engineering teachers and the challenges that they face when supporting students in doing engineering design in a social context.

The Engineering for All (EfA) project was a five-year-long collaboration between Hofstra University and the International Technology and Engineering Educators Association (ITEEA), and was funded by the National Science Foundation's DRK-12 program. 1 The main goal of this project was to develop middle school technology and engineering (T&E) education units that emphasize the role of engineers in solving important global and communitybased problems.The project developed two units that contain authentic design challenges appropriate for the experiences and capabilities of middle school students.The two developed units dealt with the global shortage of fresh food and clean water.Both challenges were cited as priorities for integration into technology education curriculum (Buelin, Clark, & Ernst, 2016) and related to the Grand Challenges for Engineering identified by the National Academy of Engineering (NAS, 2017).One unit, Vertical Farming: Fresh Food for Cities, is focused on the development of sustainable food sources for cities through the inclusion of urban vertical hydroponics farms.The second unit, Water: The World in Crisis, is focused on the development of water filtration systems to be used in countries afflicted with contaminated water sources.Both units incorporate the "informed design" curricular structure (Burghardt & Hacker, 2004), which first introduces students to the major design challenge, followed by a "progression of knowledge and skill builders (KSBs)" (p.7), activities that provide the students with the prerequisite experiences, knowledge, and skills to deal with the design challenge from a STEM-informed knowledge and skill base with the aim of reducing the uses of craft-based and trial-and-error approaches to solving design challenges.
The units were developed by T&E teacher teams, led by experienced curriculum developers, and were subsequently tested by 22 middle school T&E teachers and 755 students.As preparation for testing the units, the teachers participated in professional development workshops in which they learned the content and tried out the hands-on activities included in the two EfA units.During a pilot study of the new units, participating teachers constructed teaching portfolios that contained a structured weekly log and student work.In addition, the teachers participated in monthly WebEx phone conferences in which they reported on specific challenges in implementing the curriculum.Based on these conversations and data gathered during the pilot study, the units were revised and improved, making them more accessible to both teachers and students.The curriculum materials are available from ITEEA.
The following sections include descriptions of the development of the project's instruments, including: EfA Design Teaching Standards, EfA Design Teaching Portfolio, and the EfA Design Teaching Performance Rubrics.

EfA Design Teaching Standards
Knowing how to teach engineering involves quite a different knowledge and skill set than knowing how to do engineering.Like engineers, teachers using design tasks need to have content knowledge and process skills, but they also need pedagogical content knowledge (PCK), which is domain-specific and contextualized to each content area (Ball, Lubienski, & Mewborn, 2001;Magnusson, Krajcik, & Borko, 1999;Shulman, 1986) (National Research Council, 1996) in the United States, and frameworks for engineering design teaching and learning (Crismond & Adams, 2012;Cross, 2000;Hacker, 2014;Reimers, Farmer, & Klein-Gardner, 2015).
The final set of the EfA DTS (see Table 2) created for this project is organized around the following three dimensions.
 Dimension I: Design Practices-This standard describes different practices that are part of the "informed design" teaching. Dimension II: Engineering Themes-This standard identifies crosscutting themes and concepts that consistently appear in the engineering design literature (i.e., design, modeling, systems, resources, and human values). Dimension III: Classroom Instructional Practices-This standard describes essential instructional practices that are commonly considered necessary to support student learning.The DTS were examined and validated by K-12 science, engineering and technology education teachers, teacher trainers, curriculum developers, STEM education researchers, administrators, and policy makers.Some participants were asked to comment on the quality of the standards in face-to-face interviews, whereas others completed an online questionnaire.In this online survey, the educators were asked to read each standard and rate their level of agreement with the following statements.
 Survey Statement 1: "The standard is feasible for teaching in the technology classroom." Survey Statement 2: "The standard is important for teaching engineering design." Survey Statement 3: "The standard is clearly written."Statements were rated on a 5-point Likert scale from strongly disagree (1) to strongly agree (5).Table 1 summarizes the results of the online validation survey of the DTS.As can be seen in Table 1, most of the 38 survey participants agreed that the standards are instructionally feasible (93%), important (97%), and clearly written (92%).These results add validity to the EfA Design Teaching Standards.a Includes the combined responses of agree (4) and strongly agree (5).Reflect on and share with peers the design work, the processes used and decisions made.
EfA Design Teaching Portfolios Teaching portfolios have been used by educators for more than two decades.Teaching portfolios that include student work and teachers' reflections capture the complexities of the teaching practice better than written tests or classroom observations.Not only are portfolios an effective way to assess teaching quality, but they also provide teachers with unique opportunities for self-reflection and collegial interactions based on documented episodes of their own teaching.(Wolf, 1996, p. 34) Typically, teacher portfolios are used as a tool for teacher evaluation.For example, teacher portfolios were used by the Connecticut State Department of Education for 2 decades for the evaluation and support of beginning school teachers (Lomask, Pecheone, & Baron, 1995).Teaching portfolios are also used by the National Board for Professional Teaching Standards (Darling-Hammond, 1999) as a major component of their comprehensive assessment of quality teaching.Lately, many states have adopted the teaching portfolio, as in the edTPA national project, as a tool to assess the quality of preservice teacher performance (Sato, 2014).In all of these examples, the portfolios were designed to gather authentic data in order to evaluate the quality of the teachers.
During the EfA project, teaching portfolios were selected and primarily used to review and gather data about the implementation of the new curricula and about common instructional practices of the participating teachers.In addition, the portfolios were employed by the research team as a way to provide ongoing feedback to the developers of the two EfA units and as a source of materials for future professional development.For example, copies of student work and classroom videos from submitted portfolios became part of the training materials for interested teachers at the 2016 ITEEA national conference held in National Harbor, Maryland.The main entries in the EfA teaching portfolio are described in Figure 1.
EfA teaching portfolio entries and required materials: 1. Instructional logs written at the end of each KSB (5-6 entries per unit of instruction).These logs address the following issues: a. Main STEM concepts that were taught, b.Main engineering practices that were practiced by students, c.Findings about students' learning strengths and challenges during each KSB, and d.Challenges in teaching each instructional sequence within the EfA unit.
2. Student work from one male and one female student, done to complete each of the KSB's formative assessment tasks.Student work was reviewed and evaluated by each participating teacher.
3. Three unedited instructional video clips, each 5-10 minutes in length, and selected to depict the following: a. Teaching a STEM concept, b.Teaching an engineering design practice, and c.Students' oral presentations of their final project with oral feedback from the teacher.
4. Written reflection on the implementation of the curriculum and what revisions may be needed in order to improve the unit and enhance student learning.EfA Teaching Performance Rubrics The development of the teaching performance rubrics was an iterative process in which portfolios were reviewed by three professional technology and science educators in light of the evolving DTS.The rubrics were designed to provide a framework for the research about teachers' design PCK and to provide feedback to the curriculum developers.Since the rubrics were not used for formal teacher evaluation, no attempt was done to explore the reliability of the rubrics as a scoring tool.Rather, the portfolio reviewers worked collaboratively to develop the rubrics and then applied them in their review of the submitted teachers' portfolios.Tables 3, 4, and 5 describe the three EfA Design Teaching Performance Rubrics, one for each of the DTS dimensions.

Framing the Challenge
Teacher helps students grasp the design challenge and its context, as well as the criteria and constraints for a successful design solution.
Teacher describes the design challenge and its context, as well as the criteria and constraints for a successful design solution.
Teacher describes the design challenge and reviews with the students the design criteria and constraints.

b. Doing Research
Teacher requires students to conduct research and hands-on investigations to gather relevant data on the design challenge.
Teacher requires students to gather data on the design challenge, mainly through reading of relevant materials.
Teacher does not require students to gather relevant data before they start working on the design challenge.

c. Generating Alternatives
Teacher encourages students to develop several different possible solutions to the design challenge.
Teacher encourages students to develop two different possible solutions to the design challenge.
Teacher accepts one possible solution early in the design process.

d. Making Decisions
Teacher asks students to discuss the benefits and tradeoffs of the different solutions and to justify their selection of the solution they would develop into a prototype.
Teacher asks students to justify their selection of the solution they would develop into a prototype.
Teacher allows students to develop their chosen solutions into a prototype without explaining their design decisions.
e Teacher asks students to test and document the performance of the prototypes they develop.
Teacher doesn't require students to conduct tests of the performance of their prototypes.

g. Iterating and Improving
Teacher provides students with time and materials to revise their prototypes based on evidence they collected during testing.
Teacher provides students with time and materials to revise and improve their prototypes.
Teacher doesn't require students to revise and improve their prototypes.

h. Communicating and Reflecting
Teacher requires students to present their design work and provides them with formative feedback.
Teacher requires students to present their design work but doesn't provide them with formative feedback.
Teacher doesn't require students to present their design work.Teacher provides limited exposure to the use of drawings and models when designing.
in representing more complex products and systems. c.

Systems
Teacher helps students to identify subsystems and the inputs, processes, and outputs in their designed system and to distinguish between open and closed-loop systems.
Teacher helps students to identify subsystems and the inputs, processes, and outputs in their designed system.
Teacher helps students to identify the parts that work together and makeup their designed system.

d. Resources
Teacher helps students to explore the need to choose resources based on availability, appropriateness, cost, ease of use, and sustainability when making their design choices.
Teacher reviews the availability of resources and their use when making design decisions.
Teacher offers little or no support to students for considering the selection and rationales for use of particular resources. e.

Needs, Impacts, and Human Values
Teacher encourages students to explore how the designed product may impact intended users and the environment.
Teacher encourages students to pay attention to the needs of those who will use the designed product.
Teacher provides limited attention to the design context and the users of the designed product.EfA Units in Light of Participating Teachers' Feedback A total of 22 teachers participated in the EfA curriculum development study.In the study, half of the teachers taught the EfA water unit, and the other half taught the food unit.The teachers implemented the new units over an 8-10 week period of time with technology education students in Grades 6-9.After the teachers finished their teaching of the original EfA units (before the revision of the curricular materials), they were asked to express their opinions on the new curriculum by rating various aspects of their experience on a 5-point Likert scale in which 1 is strongly disagree and 5 is strongly agree.Results from this online survey are displayed in Table 6. a Includes the combined responses of agree (4) and strongly agree (5).
The results shown in Table 7 indicate that although most teachers had positive opinions about the new curriculum, they also found that the content of the unit required knowledge that students do not have (Item 24) and that it was above the ability of their students (Item 25).EfA teachers also were concerned about the readiness of the units for classroom instruction and the cost and availability of materials to implement these units on a regular basis (Items 11-13 and Items 17-20).The findings from this online survey, in addition to more detailed information that was gathered through the teachers' portfolio, were used to revise and improve the curriculum.

Findings from Review of Teachers' Portfolios and Implications for EfA
Curricular Revisions The participating EfA teachers submitted the requested portfolio logs and the three videotaped teaching vignettes while teaching the units.At the conclusion of the units, the teachers submitted student work with teacher annotations as well as personal reflections.Teachers' portfolio materials were submitted electronically via Dropbox or physically via regular mail.At the conclusion of the project all of the written materials were printed and bound and were also rendered as PDF files in order to make the portfolio review more accessible.Three trained researchers reviewed the teacher portfolios individually and then met to compare evaluations and explore patterns found in the data reviewed.Differing interpretations of teachers' performances were resolved through discussion.
The following describes findings from the review of the teaching portfolios and main curricular revisions.Findings are organized by the three dimensions of the design teaching standards and rubrics: Design Practices, Engineering Themes, and Classroom Instruction.

Dimension I: Design Practices
Review of teacher logs and videos showed that EfA teachers understood the steps of the informed design process and made references to them during instruction.Burghardt and Hacker's (2004) informed design model was introduced in the EfA's introductory materials for students and was used to structure the learning activities that students completed to address the Grand Design Challenge found at the end of the unit.Teachers noted that even though the informed design process was introduced to students at the outset of the unit, they had for the most part forgotten and had difficulties recalling and applying this model when addressing the culminating design challenge.
One difficulty that teachers encountered when implementing the first edition of the EfA materials revolved around the use of scientific inquiry during the design process.For example, lessons in which students attempted to design fair-test experiments that explored key factors influencing plant growth in the hydroponics systems that they were building (e.g., the makeup of the nutrient solution, its pH levels, and lighting conditions) required that EfA teachers be able to explain to students the notion of a controlled experiment, and independent variables -common misconceptions that students have regarding the use of control-of-variables strategy (Schwichow, Croker, Zimmerman, Höffler, & Härtig, 2016;Klahr & Nigam, 2004), and ways to measure key outcomes in an experiment effectively.Videos of several EfA teachers conducting lessons in which students designed experiments showed that several were unfamiliar with the practices related to scientific experimentation.Others, who may have known the key elements of good experiments (e.g., Harlen, 2001), did not integrate them into their teaching.Instead of engaging their learners in building a better understanding of scientific inquiry and the practices of designing fair-test experiments, some teachers gave cookbook directions for their students to follow.Thus, their students did not plan their own investigations but rather followed the directions and did the tasks that their teachers gave them.Another set of difficulties that were noted involved the ways in which teachers did or did not help students develop and evaluate several alternative design solutions before letting them move on to building their prototypes.In addition, most of the EfA teachers, perhaps to save time, did not give their students opportunities to revise their prototypes, even when prior tests had revealed flaws in those prototypes.
Based on these findings, the revised units were shortened to include fewer KSB activities prior to the main design activities so that those remaining could be done in more depth before students took on the unit's Grand Design Challenge.In addition, the scientific inquiry was connected more directly to EfA design challenges and better scaffolding, which was done to clarify the essence of the scientific experimental method for both teachers and students.

Dimension II: Understanding the Engineering Themes
The grasp and depth of teachers' portrayal of EfA engineering themes (i.e., design, models, systems, resources, needs, impacts, and human values) and of working within given constraints when developing and optimizing, were varied.Videos of instruction included in EfA teaching portfolios revealed how some teachers engaged their learners in discussing key themes and addressing students' misconceptions.Other EfA teachers did not seem to know ways to engage and elicit students' understanding.These teachers therefore rarely noted and addressed students' shortcomings.In general, teachers did not emphasize the importance of the engineering themes as crosscutting (meaning that these themes are important in multiple design challenges) but rather discussed the themes as they related to the specific design challenge at hand.Although most of the participating technology teachers had difficulties with the concepts and the themes, some teachers revealed a deeper understanding of the EfA concepts.For example, one teacher infused instruction about other types of hydroponics systems in addition to the two systems highlighted in the EfA curriculum.Another teacher added just-in-time instruction on the periodic table when students were learning about types of water chemical contamination.
Based on these findings, the units' revisions focused on clarifying the thematic focus of each subunit and providing more explanations and examples to engage students in explorations of the relevant themes.

Dimension III: Classroom Instruction
Abundant evidence was found in teachers' portfolios that EfA teachers were capable and effective in the management of their classrooms and their use of general pedagogical skills in engaging students and managing instruction.Teachers were found to use whole-class and small-group settings when presenting and implementing EfA content and activities.They excelled when teaching procedural and practical knowledge relevant to the field of T&E education, including the appropriate use of tools for making prototypes and use of a computer-aided drawing system, such as Google's Sketchup program that was highlighted in the EfA materials.
However, teachers were lacking when it came to two major components of effective design instruction: understanding essential science concepts and using assessment to support learning.The design challenges in the EfA units had strong links to relevant science concepts.For example, in the food unit, in order to design functional vertical hydroponic urban farms, students needed to understand concepts such as plant physiology and growth, the function and performances of pumps in different hydroponic systems, and concepts related to building stable and strong structures, as with wall-mounted reservoirs or hydroponics growth beds.In the water filtration unit, students needed to understand various concepts related to physical, chemical, and biological sources of water contamination as well as the operation and maintenance of filters that are designed to meet important performance objectives.Most of the participating technology teachers faced challenges in engaging their students in learning these concepts.This shortcoming may be rooted in lack of proper content preparation of the teachers or lack of time to explain the content well.
The second instructional practice that was challenging for teachers was the use of assessment.The EfA predesign activities (e.g., the KSBs) included one performance-based assessment task each.Students' work on these assessment tasks were collected and annotated with instructor comments by the teachers as part of their portfolio submission.Teachers' annotations that were written on their student work showed that the EfA teachers are not accustomed to providing meaningful formative feedback to students about the quality of their performance.Most written feedback included praises and encouragements (e.g., "I love your answer," "good work," or illustrated "smiles").Instances in which teachers overlooked student misconceptions or did not comment on a feature of a design that would not work if built (e.g., a gravity fed filtration system in which the source of water was lower in height than the filter itself) were also noted.In general, there was a high correlation between teachers' conceptual understanding and the quality of their feedback to students: Teachers who understood the units' science concepts well were able to provide appropriate feedback, and teachers who lacked familiarity with the design and science concepts provided only limited formative feedback to their students.
Based on these findings, the revised EfA units included content-based support materials for the teachers to strengthen their understanding of the relevant science.In addition, every KSB included one major performance task (e.g., drawing a model, providing explanations, reporting experimental data, or creating concept maps) and content-specific rubrics for the evaluation of student work on each task.

Use of Teaching Portfolios for Professional Development
At the start of the EfA project, the exposure of the teachers to the design teaching standards and the accompanying rubrics was limited because these materials were developed later in the life of the project; in fact, they occurred hand-in-hand with the review of the portfolios.However, the project's materials were used extensively later in the preparation of additional EfA teachers during professional development workshops.For example, during the 2017 ITEE conference, the project introduced the EfA units and used materials from submitted portfolios (e.g., video clips and annotated student work) to train prospective EfA technology teachers.
Teaching portfolios can also provide contexts for peer coaching and mentoring in which teachers analyze their own and others' classroom work via the portfolios that they create.A recent study of Harvard's Best Foot Forward program, a video-based teacher evaluation system, showed improved instruction as a result of peer review of shared videos (Quinn, Kane, Greenberg, & Thal, 2015).

Conclusions
In our EfA research, we found that the design teaching standards and rubrics were extremely useful in evaluating curricula under development and the learning opportunities they provide to students, similar to findings about the use of science content standards in evaluating published curricula in previous research (Kesidou & Roseman, 2002).
The teaching portfolios were found to be a rich and useful instrument for collecting and reviewing data about the ways in which the participating T&E teachers implemented the new EfA curriculum.In general, teachers see the portfolio development as an extra instructional load and would probably not develop portfolios on their own.However, if trained and paid for the effort, teachers will develop authentic teaching portfolios that can serve as an efficient substitute for actual classroom observations.The EfA teaching portfolios developed in this project were not intended to be evaluative measures of teacher performance; however, they provided valuable evidence and directed the necessary curriculum revision and changes.In addition, the teaching portfolios opened windows into the common instructional practices of middle school T&E teachers and increased our understanding of the needed professional development to improve current T&E instruction.
The standards and the accompanying rubrics that were developed in the project can be used for teachers' self-assessment as well as for professional development purposes.Because the standards and the rubrics are not content specific, they can be adapted and used with a wide range of engineering designbased K-12 STEM curricula.Providing teachers with teaching standards and performance rubrics can guide and improve instruction in T&E settings.
. The Design Teaching Standards (DTS) were developed to describe this elusive PCK by defining what middle school T&E teachers need to know and be able to do in order to support students' learning with design-based curriculum.Because there were no published teaching standards for the teaching of K-12 engineering design in the United States, the EfA research team had to develop these standards to guide the work of the participating teachers.The development of the Design Teaching Standards was informed by the scholarly experiences in science and technology education of the three authors.Other sources that informed the development process include the Minimum Competences for Trainees to Teach Design and Technology in Secondary Schools in the UK (Design and Technology Association, 2003), the Standards for Preparation and Professional Development for Teachers of Engineering (American Society for Engineering Education, 2014), the National Science Education Standards

Figure 1 .
Figure 1.An overview of the EfA Design Teaching Portfolio.

Table 1
Results from the Online Validation Survey of the DTS

Table 3
EfA Teaching Performance Rubrics: Design Practices

Table 4
EfA Teaching Performance Rubrics: Engineering Themes

Table 5
EfA Teaching Performance Rubrics: Classroom Instruction

Table 6
Results from the Online Teacher Survey(Prior to Curriculum Revision)