Team Based Engineering Design Thinking

The objective of this research was to explore design thinking among teams of high school students. This objective was encompassed in the research question driving the inquiry: How do teams of high school students allocate time across stages of design? Design thinking on the professional level typically occurs in a team environment. Many individuals contribute in a variety of ways to facilitate the successful development of a solution to a problem. Teachers often require students to work in teams, but little is known about how the team functions in the context of design and the potential interaction between team performance and authentic design challenges. Few research results are available to guide high school teachers in developing successful design teams and how to encourage teams in their efforts.


Objective
The objective of this research was to explore design thinking among teams of high school students.This objective is encompassed in the research question driving this inquiry: How do teams of high school students allocate time across stages of design?Design thinking on the professional level typically occurs in a team environment.Many individuals contribute in a variety of ways to facilitate the successful development of a solution to a problem.Teachers often require students to work in groups, but little is known about how the group functions in the context of design and the potential interaction between group performance and authentic design challenges.Few research results are available to guide teachers in developing successful design teams and encouraging them in their efforts.

Problem Statement
The discrepancy between our society's dependence upon technology and our ability to understand various technological issues has emerged as a serious concern for educators."Technology is the outcome of engineering; it is rare that science translates directly into technology, just as it is not true that engineering is just applied science" (National Academy of Engineering, 2004, p. 7).Specifically, "Americans are poorly equipped to recognize, let alone ponder or address, the challenges technology poses or the problems it could solve" (Pearson & Young, 2002, pp. 1-2).The relationship between understanding engineering and technological literacy is of special urgency during the high school years, since "technologically literate people should also know something about the engineering design process" (Pearson & Young, 2002, p. 18).
Design thinking is fundamental to understanding the technologically dependent nature of our society.A need for a technologically literate populace, therefore, includes an understanding of the engineering design process.The design process links technology and engineering, two elements of STEM education."Design is the central component of the practice of engineering and a key element in technology education" (Pearson & Young, 2002, p. 58).This study identified quality high school technology and engineering learning and teaching environments in a criterion based sampling strategy, the setting envisioned by Pearson and Young, where "technology teachers with a good understanding of science and the interactions between technology, science, and society will be well prepared to work with other teachers to integrate technology with other subjects" (2002, p. 108).
While design thinking is an elusive and difficult construct to define, measurements for this study included a pertinent subset of measurements consistent with previous literature, much of which was generated through work of the Center for Engineering Learning and Teaching (Atman, Chimka, Bursic, & Nachtmann, 1999;Atman, Kilgore, & McKenna, 2008;Morozov, Yasuhara, Kilgore, & Atman, 2008;Mosborg et al., 2005;Mosborg et al., 2006).This paper reports on measurements including time allocated across essential elements of the design process elements of the design process and disaggregates the data by problem type and the gender composition of the teams.

Engineering Design Problems
According to the National Center for Engineering and Technology Education Caucus Report of 2012, "There is a need for more definitive guidance about what makes quality design challenges and how they can be implemented well in existing courses" (Householder & Hailey, 2012, p. 2).Two different design problems were administered in this study.The "playground problem" was comparable to the design problem used in previous studies with individual high school students (Mentzer, Becker, & Park, 2011) and previous work with college students and experts (Atman, et al., 1999;Mosborg, et al., 2005;Mosborg, et al., 2006).The "pedestrian flow problem" administered to approximately half of the teams was a variation of the "street crossing problem" adapted from previous literature (Cardella, Atman, Turns, & Adams, 2008;Mullins, Atman, & Shuman, 1999).The "playground problem" was provided to permit comparisons between team and individual performances while the "street crossing problem" variation facilitated comparisons among types of problem structures.The street crossing problem was less structured, could be readily adapted in order to be locally relevant, and was potentially more authentic for the participants, who also had an opportunity to specify constraints and criteria for their problem.
The Playground Problem has been used in multiple studies and can be traced to Dally and Zang (1993).The original need for project driven approaches in the freshman engineering design course was to increase student performance and retention and to situate student learning of abstract concepts through real world applications in an experiential activity.In the original activity, students designed a swing set with slides and seesaw.Atman et al. (1999) revised the foundational work of Dally and Zang to create a playground design problem.In their challenge, engineering students were presented with a brief playground design task and access to background information upon request.Participants were provided with a maximum of three hours to develop a solution to the problem while thinking aloud.Mosborg et al. (2005) applied the playground design challenge using the "think aloud" research protocol with 19 practicing engineers who were identified as experts in the field.Mosborg et al. (2006) compared groups of freshman and senior engineering students with practicing engineers using data previously collected on the playground design challenge.Atman et al. (2008) analyzed data from previous studies with a focus on the language of design, its relationship to design thinking as a mediator, and relationships between the internalization of design thinking and language acquisition.The endeavor to model problem solving satisfactorily has engaged scholars across domains (Hayes, 1989;Newell & Simon, 1972;Polya, 1945;Rubenzer, 1979).The playground design task, an open-ended, realistic, accessible, and complex problem, is an effective design challenge to enable researchers to study design thinking (Mosborg et al., 2006).
In the previous studies, participants were given a one page design brief of the playground problem.The participating teams, acting as engineers, were assigned to design a playground on a donated city block.The constraints include limited budget, child safety, and compliance with zoning regulations and applicable laws.Participants were able to query the research administrator for additional specific information on the lot layout, cost of materials, neighborhood demographics, or other information.There was a two-hour time limit for completion of the design proposal, which was a modification of the original three hour limit.This modification was made because the average design time in the previous study of high school students was about 90 minutes.The two-hour limit provided more time than the average individual needed, yet it reduced the resources needed for data collection.The participants presented a written proposal describing their design.This activity engaged the participants in problem framing and the development of an initial solution.Limitations of this design task included the lack of opportunity for participants to investigate the need for a solution, since the problem was simply assigned to them.
Students did not have an opportunity to construct physical models or prototypes.Participants were aware that implementation of the design project would not occur, that their designs would not be realized.
The Street Crossing Design Problem was adapted from previous research (Cardella, et al., 2008;Mullins, et al., 1999).The National Center for Engineering and Technology Education assembled a Caucus in August 2011 to identify characteristics of engineering design challenges (Householder, 2011) The Street Crossing Design Problem was potentially more authentic and more closely aligned with the National Center's Caucus suggestions."Authentic problems currently affect real-life situations encountered by the learners, their families, and their communitiesand they do not have a generally recognized "right answer."(Householder & Hailey, 2012, p. 22).The problem was modified slightly from its original administration to more closely exemplify these characteristics.This modification presented a unique opportunity to contrast results on the street crossing design problem with results on the playground problem, which has been used extensively, but does not meet these characteristics as well for the target audience.
The street crossing problem was presented to teachers at the schools involved.Teachers were asked to think about an intersection fitting characteristics of the original problem but located at or near the school where students would immediately recognize the problem as they personally experienced it daily.After negotiating with the teachers, it was discovered that car/pedestrian traffic flow was an issue, but a more relevant and pervasive similar issue was pedestrian (student) flow in school hallways.Each school had one or more significant blockages that caused congestion, frustration and delay at passing times between classes.The problem was modified to focus on student hallway flow rather than traffic flow, making the problem more relevant and personal, as most students experience the congestion several times per day.
Each pedestrian flow problem was presented in a similar format: the school floor plan ("map") was provided to participants along with a brief narrative stating that the student team was a team of engineers contacted by the school district.The narrative introduced an area that the students immediately recognized as a congested area and requested that the team present a proposal for resolving the issue.The constraints and criteria were not specified; leaving student design teams the opportunity to discuss and negotiate their specific problem definition and determine the most appropriate solution proposal.This less structured problem is consistent with the National Center's suggestion that "Engineering design challenges are ill-structured problems that may be approached and resolved using strategies and approaches commonly considered to be engineering practices" (Householder & Hailey, 2012, p. 2).Typical office supplies were provided for the participants, a condition similar to those in the playground problem.However, participants were given ready access to the Internet in lieu of printed sheets of relevant data.The decision to provide Internet access was made to increase the sense of authenticity and relevance as students are familiar with and accustomed to having Internet access.The notion of having predetermined what information is needed for their solutions may unintentionally guide student design decisions to those based upon a finite resource pool.

Methodology
A descriptive study was conducted spanning multiple high schools in urban, rural and suburban environments.The quantitative research method design leveraged the use of data from 17 design teams comprised of 2-4 students each.Seven of the teams were comprised of males, five were comprised of females, and five teams included both male and female members.Seven of the teams received the playground design problem and eight received a locally relevant school hallway traffic flow problem.Teams were expected to develop a solution in 2 hours.The interactions of group members were video and audio recorded while they were developing the design solution.
Sample.A sample of 47 students was used in this study.The teacher grouped the students into teams according to their personal schedules and the teams were assigned the design challenge.Some team members were friends and other teams were comprised of students who did not know each other well.The 17 teams that were created came from four schools in two states.Each of the schools selected to participate had a recognized engineering program associated with an outreach effort by a university engineering program.Curricular offerings at the high schools included Project Lead the Way (PLTW), Engineering Projects in Community Service (EPICS) High, First Robotics, and locally developed engineering and/or technology courses supported by their regional University.
Teachers at the target schools permitted advertising to recruit their students for participation in the study.Students in this study were considered to be representative of experienced students who had taken most or all engineering related courses at their high school.Students were recruited who were actively engaged in the study of engineering design through a criterion sampling strategy (Creswell, 1998) using the following criteria:  The high schools had an established program of study which employs a focus on engineering in a sequence of courses developed in association with an engineering outreach effort as part of a university program. In these courses, students participated in design activities which engage their critical thinking and problem solving skills within the framework of the engineering design process. Students were selected who represent diverse backgrounds and have chosen to enroll in this sequence of courses.

Administration of the Design Challenges.
A team of graduate and undergraduate students and a university faculty member conducted the data collection.A lead researcher administered the problem and trained the student researchers through discussion, observation and direct participation.The researcher reviewed data collected and reflected with the students following the session to standardize procedures and provide oversight on consistent research administration.Student researchers' administration of the design problem was video recorded as part of the data collection and videos were reviewed for training purposes to ensure consistency during data collection.
Data included video and audio recordings of the design sessions.Video cameras were small, mounted on miniature tripods to minimize their intrusion.Audio recorders were used as a backup to the video cameras and were positioned on the work space near the student.All students were wired with a lavalier microphone to ensure high quality audio feeds.Wires were run under the team workspace to prevent tangling, however, the wires limited student mobility.Students generated documents and other artifacts with traditional office supplies provided.Artifacts typically included sketches, notes, and formal drawings.Two-dimensional works were anticipated by the research team and scanned to digital image form.Data were archived in digital format on hard drives.Drives were shared with the research team for analysis purposes.
Data Analysis.The playground problem coding scheme was congruent with the approach used in earlier studies (Atman, et al., 1999;Bursic & Atman, 1997;Mosborg, et al., 2005;Mosborg, et al., 2006).Time is a limited resource and the ways designers allocate their time among the areas of the design process has been a focus of previous work.Two measurements of time were made while the designers are at work: time allocated to elements of the design process; and total time engaged in design.The unit of analysis was the team.The coding team used NVIVO software to analyze each video.The coding scheme was similar to the approach used in prior studies (Atman, et al., 1999;Mosborg, et al., 2005;Mosborg, et al., 2006).The data were coded into the nine categories presented by Mosborg et al. (2006, p. 15): (1) Problem Definition, defining what the problem really is; (2) Gather Information, searching for and collecting information needed to solve the problem; (3) Generating Ideas, thinking up potential solutions (or parts of potential solution) to the problem; (4) Modeling, detailing how to build the solution (or parts of the solution) to the problem; (5) Feasibility Analysis, assessing and passing judgment on a possible or planned solution to the problem; (6) Evaluation, comparing and contrasting two (or more) solutions to the problem on a particular dimension (or set of dimensions) such as strength or cost; (7) Decision, selecting one idea or solution to the problem (or parts of the problem) from among those considered; (8) Communication, the participants' communicating elements of the design in writing, or with oral reports, to parties such as contractors and the community; and (9) Other, none of the above codes apply.
Data analysis began with segmenting the data sets.A team of three researchers were tasked with the responsibility of segmenting.A segment was defined as a pause bound utterance, as suggested by Atman et al. (1999).Researchers were instructed to create a new segment in the video timeline for each instance when any student on the team began a new thought, which was typically indicated by beginning to speak after a pause.In previous literature, this segmenting procedure was applied to individuals.For the current study of teams, researchers created these segments each time any member of the team made a transition.The resulting segmented data represented the composite of all team member segments.At some points in the videos, all team members were functioning as one cohesive unit and segmenting was simple and a single layer.In other times, a team of four students might naturally divide into two teams of two and the segments represent start/stop times for each sub-team.By segmenting in this fashion, a divergence in design activities could be coded with two separate codes with two different, but overlapping episodes.
Quantitative measures of inter rater reliability on segmenting were not made.The research leadership determined the segmenting would be of reasonable quality if the inter-rater reliability measures for coding were high.If segmenting were done successfully, coding could potentially result in high inter rater reliability.Coding served as a proxy for quality control of the segmenting process.As a preliminary quality control mechanism, the lead researcher reviewed segmented work and provided feedback and guidance as the research assistants progressed.
Two undergraduate students coded the data.Three phases were conducted.The first was to establish calibration of the research assistant's coding work.The second phase served to document the calibration using Kappa values as a measure of inter-rater reliability.The team of two research assistants coded 25% of each video and compared.The Kappa values averaged 0.71; details are presented in Table 1 along with the number of references used to generate the values.In the third phase, all videos were fully coded, approximately one-half by one research assistant and one-half by the other.
In the calibration phase, research assistants were provided with a conceptual overview of the coding process, structure, technique and rationale.They were presented with examples from previous work and practiced coding these data.Research assistants then coded a portion of a video and compared with each other.They would meet with a senior research team member and discuss the individual interpretations and differences seeking clarification on coding.A "Dynamic" Code Book was adopted and maintained.This was a document with very specific examples of the different codes developed by creating a description of the code and compilation examples in context.This included adding detail and clarifying the meaning of our segmenting and coding procedures and providing examples as coders did their work.The document was updated regularly and shared via network real time.As understanding and interpretation was negotiated by the coders and research team leaders, the codebook documents the increasingly specific definitions.
While the coding scheme was consistent with previous literature, the technique was slightly different.Previous work used transcription, segmenting and coding as three separate activities in the analysis process (Atman, et al., 1999).Inter-rater reliability was calculated on the coding to ensure reliability of the multiple coding analysts.Our project bypassed transcription by using NVIVO software which presented coding analysts with synchronized video and audio feed.Codes were associated with the timeline on the video/audio tracks and inter-rater reliability was computed using the Cohen's Kappa statistic.
The calibration process was iterative.Each coding session was followed by a debriefing session and the cycle started over.Kappa values began relatively low and increased gradually as the research assistants became more closely aligned in their designations.When average Kappa values for each code approached 0.70, the research team transitioned into the next phase, documentation.Some effort was focused on calibration, but most effort was allocated toward coding a random 25% of each video and documenting the comparison.In the final phase of coding, the 17 videos were divided among the two research assistants.Earlier work resulted in 25% of each video being coded already; the remaining 75% was coded.The entire video was reviewed and changes were made as needed to the coding structure in context of the newly coded 75%.

Results
Team time allocation was measured as a proxy of effort in the design process.Student teams were considered the units of analysis and the video provided data on team performance.The video data were coded by time allocated to: Problem Definition, Gathering Information, Generating Ideas, Modeling, Feasibility, Evaluation, Decision Making and Communication efforts.Activities that the team engaged in were coded.At times in the process, team members were all simultaneously engaged in one activity, but at other times, individual students would engage in different activities.When team members provided reasonable evidence that they were doing two different activities, two or more codes were applied.The total coded data exceeded 100% in all teams because, at times, the team was receiving credit for two or more codes simultaneously.Teams averaged 102 minutes in the design process as compared to individuals from previous work who finished, on average, at 92 minutes (Becker, Mentzer, & Park, 2012).Table 2 shows the average time invested by the teams in each design activity in this study and the average time invested by individuals in the previous study.With this study's small sample size (n=17) statistical analysis was not conducted.However, trends emergent in the time allocation between individuals and teams may provide a foundation for future study.Teams spent nearly twice the percentage of time engaged in gathering information.Information gathering was coded when students were actively requesting, reading, and reviewing information related to the problem or solution.Information requests could be made of the administrator.Teams working on the hallway traffic design challenge were provided with a laptop and Internet access.Student use of the Internet was generally coded as information gathering and represents a difference from the data collection protocol used with individuals and playground design teams as they did not have access to the computer.
Modeling and communication time allocations show differences between teams and individuals.The teams tended to spend less time modeling and more time communicating.Modeling was defined as detailing how to build something, including calculations, estimations, determining locations, and description of how something will be assembled or fabricated.Communicating was defined as the efforts involved in telling someone how to build the playground.Communication efforts focused on sharing the team's plan with others and could be directed toward a contractor or a board of directors considering the team's proposal.
Differentiating between modeling and communication in the abstract was simple for the undergraduate research assistants as the difference centered on purpose.If the purpose of the sketching, for example, was to understand and improve appearance, functionality or fabrication techniques and the team used this information to think through challenges and determine specifications, it was coded as modeling.If, on the other hand, team effort was directed at documenting their plans for fabrication for the purpose of telling someone how to build from the plans, it was coded as communicating.Student teams often started modeling and the work evolved into communication.This evolution made precise determination difficult.In cases where the transition was gradual and vague, the coders generally defaulted to modeling until there was evidence that the purpose was an attempt to communicate team intentions/plans.In some cases, teams were very deliberate about this transition.In other cases, the transitions occurred gradually but were clarified later.For example, what might have appeared to be modeling was later determined to be communication and codes were changed appropriately as the coding process progressed through the team's work.
An underlying assumption in generalizing previous work (Becker, et al., 2012) to a larger population of high school students is that time allocated to elements of the design process is representative of student understanding of process and not overly dependent on the nature of the problem.In this study, two different problems were used.Table 2 (above) represents both problems in the team environment and compares to individuals who were engaged in just one of those problems.Table 3 compares teams who engaged in the playground problem (n=8) and the pedestrian flow problem (n=9).These small sample sizes preclude the use of meaningful statistical analysis, but do provide a hint into a potential discovery which could serve as a foundation for future work.Teams for each design problem included single gender and mixed gender compositions.The playground problem was administered to one female team, five male teams and two mixed gender teams.The pedestrian flow problem administrations included three female teams, three male teams and three mixed gender teams.Time spent on problem definition differed between the pedestrian flow and playground problems.Teams spent more than twice the amount of time reading, considering, reflecting on and considering the problem for the playground compared to the pedestrian flow challenge.The playground problem presentation was longer and more specific.Constraints and criteria were specified as compared to the pedestrian flow in which constraints and criteria were not specified.The research team had anticipated that the lack of definition would permit students to develop their own constraints and criteria relative to their local problem, but time in the problem definition phase was actually less when constraints and criteria were not provided.
The following is an excerpt from the playground design problem as adopted from Atman (1999;2008): You estimate that most of the children who will use the playground will range from 1 to 10 years of age.Twelve children should be kept busy at any one time.There should be at least three different types of activities for the children.Any equipment you design must  be safe for the children  remain outside all year long  not cost too much  comply with the Americans with Disabilities Act The neighborhood does not have the time or money to buy ready-made pieces of equipment.Your design should use materials that are available at any hardware or lumber store.The playground must be ready for use in 2 months.In this excerpt, constraints included the target audience of users, the number of children using the playground simultaneously, the number of different types of pieces of the playground that will remain outside, and a timeline.Criteria included safety, costs, compliance with the Americans with Disabilities Act (ADA) and the use of commonly available materials.
The playground problem brief differs from the pedestrian crossing in that the pedestrian flow problem provides no guidance on constraints and criteria which allows students to identify and specify their design parameters, potentially expressing why their unique solution is optimal.An example pedestrian flow problem looked like this, though details varied across schools to situated the problem in the local context: You are a team of engineers contacted by [your school name here] School District.Often hallways are congested at passing time between classes.Hallway one, which is between the new and old portions of the school, is difficult to navigate.[your school name here] School District would like your team to develop and propose a solution.
Information gathering efforts were noticeably different across the two problems.Teams working on the pedestrian flow problem spent about twice the percentage of time searching for and digesting information relative to the problem than did the teams working on the less familiar playground problem.Examples of information gathered in the playground problem focused on identifying typical components on playgrounds such as swings, slides, monkey bars and material characteristics such as strength, durability and cost.Information across both problems included benchmarking, but on different conceptual levels.Searching for playground components was a concrete task resulting in a list of typical play things while the hallway problem yielded much more complex transfer from other schools or public places where traffic congestion was a problem.Students looked at airports as examples of moving people in short periods of time as a potential method of benchmarking and gathered these examples to spawn ideas in their scenario.The transfer from an airport or mall hallway to a school hallway was challenging for students perhaps because of the population of users were different (i.e.adults in airports vs. students; adults may be motivated to run to their next flight vs. students who may not be interested in getting to the next class).
Students in the hallway problem appeared to spend more time searching, perhaps motivated by their personal interest.In each administration of the problem, students were obviously bothered by the problem and were quick to engage as compared to the playground problem where students engaged at our request but seemed less intrinsically motivated.The hallway challenge included Internet access, which could have related to the additional search time.Our informal observations seemed to indicate that students not only accessed the computer, they also gathered information from memories of direct observations.They recalled their experiences in airports and malls with pedestrian congestion.They recalled traffic flow rates and locker placements in the school and considered the impacts of this information on their design process.
Teams on the pedestrian flow problem spent more than twice the percentage of time considering feasibility of their solutions as compared to the teams on the playground problem.Feasibility was defined as considering the practicality or viability of a solution or element of the solution.This was differentiated from evaluation in that evaluation includes comparing two or more options while feasibility is passing judgment on one potential idea.In the playground problem, feasibility typically centered around cost in addressing the question: "Would an item/component cost too much?"Also, playground design teams considered the extent to which they met the constraints.This differed in the hallway problem because students were not provided with constraints or criteria nor did teams spend time to specify either constraints or criteria for the solution of the problem.Feasibility, however, consumed a much greater percentage of time as students attempted to determine if their solutions would work.Students implicitly must have identified some constraints and criteria as they talked about feasibility, but not directly.Typical examples included students discussing the financial cost of a solution or the impact that the proposed solution might have on the problem without explicitly identifying a budget or rationale that costs should be limited or minimal.Some solutions were structural while others were behavioral and students considered their potential solutions and students' behavioral responses.This led to discussions of teachers' roles, administrators' media campaigns for pedestrian traffic patterns and the feasibility consideration: "would it work?"Feasibility seemed more relevant for students to consider in the hallway challenge as compared to the playground problem.Students were familiar with the hallway issues and the solutions had direct impact on their lives.They seemed to have capacity for understanding the complexities of hallway traffic more than complexities in the playground problem.Student design teams seldom considered the issues of safety in the playground, overlooking such facts as the difficulties that 2 year olds have in negotiating ladders.If the students had been parents of children for whom the playground was being designed, they might have considered safety and functionality with greater understanding, but as 17 and 18 year olds, they seemed to lack a sense of understanding about the functionality and dangers surrounding playground equipment design.
In both problems, evaluation and decision making activities were rarely observed.Student teams spent very little time comparing alternatives on a criterion, which was our working definition of evaluation.Students also spent very little time choosing among the alternatives.Decision making was defined in this study as a deliberate choice between two or more alternatives.Evaluation was defined as the comparison of two or more alternates using criteria.A typical decision and evaluation activity in the playground problem included material selection.Student teams would ponder using wood or metal as a construction material, discuss costs, strength and durability, then make a selection.In the hallway problem, very few evaluations or decisions were observed.This lack of evaluation and decision making may be directly related to the fact that students developed few alternative solutions during the brainstorming phase.While they did brainstorm and develop ideas, selection decisions tended to be related to the feasibility of individual components of the solution rather than a comparison of alternative solutions.Students would frequently say, "Let's put in a slide, it's cheap" with no externalized comparison of the alternatives.
Teams were comprised of one gender or mixed genders.Four female teams, seven male teams and five mixed gender teams displayed differences in design processes.The gender of individuals assigned to the teams was difficult for the research team to control and, as a result, did not split equally across design problems.Three of the four female teams were concerned with the pedestrian flow problem while only one female team was concerned with the playground problem.Five of the seven male teams received the playground problem while two male teams received the pedestrian flow problem.Two of the five mixed gender teams engaged in the playground problem while three attempted the pedestrian flow problem.
Results of team effort disaggregated by gender are presented in Table 4.Total design time varied across groups, with female teams finishing their work nearly 22 minutes before their male counterparts.Mixed groups of males and females averaged about eight minutes less than male teams.While female teams spent less time engaged in design, they spent more time on problem definition (8.4 minutes) than did the male teams (7.0 minutes) and mixed gender groups (5.2 minutes).This finding is particularly noteworthy, as most of the female teams were engaged in the pedestrian flow problem which, according to table 2, typically received relatively little attention to problem definition (4.1 minutes).Modeling and communication efforts account for most of the differences in overall design time.Females spent about 25% less time modeling than did males and mixed gender groups.The average female modeling time was 36 minutes while males and mixed groups spent 47 and 46 minutes respectively.Male student teams spent nearly twice as much time communicating (29 minutes) as did female student teams (16 minutes).

Discussion and Implications
Inter-rater reliability in modeling and feasibility were low despite extensive calibration efforts by the research team.Some of the lack of agreement could be related to researcher calibration, but the research team suspects that the lack of agreement is also related to lack of clarity by the student teams about the nature of these activities.The students were vague about how modeling was related to other aspects of the design process, particularly communication.What initially appeared to be graphical sketching as a method of developing ideas and laying out a potential solution for discussion evolved into a document for the final proposed solution.The boundaries between thinking on paper and communicating with external stakeholders blurred and presented the research team with difficulty identifying student intentions.At the end of the design challenge, students frequently presented rough sketches and messy notes, resulting in poor technical communication.If this situation is to be improved, it is important to make clear how to communicate technical information in a persuasive way to external stakeholders.Preparatory experiences might be focused on presentation skills where teams present their work to other classes or an invited audience of people who are not familiar with the daily student design experiences.This external audience would challenge teams to provide details and rationale for decisions made in context which was generally absent from participant work in this research.
Other research efforts using student observations might increase their inter-rater reliability measures with teams by identifying what papers students are using when they are writing or sketching.In previous work with individuals, reviewing the digitized artifacts and video observation data typically provided ample evidence to determine what students were writing.However, with teams of students and multiple artifacts, researchers were less able to identify which paper was being used during a particular phase of the design process.Cameras positioned from an angle overhead might allow association between papers and content of the writing.However, in this research effort, a wide angle video of four students all with papers made identifying what was written and when difficult.In addition, the research team noticed that they were able to code feasibility, for example, consistently, but they had difficulty determining exact start and stop times.One research assistant might include a background statement as leading to feasibility while another might code a narrower band of feasibility leading to general agreement between researches, but low Kappa values.
Students engaged in dialog about the feasibility of their potential solutions or elements of solutions in the hallway problem more than in the playground problem, but problem definition was considered more in the playground problem.These two activities may be correlated such that a general lack of problem definition would lead to a tendency for students to be quick to question whether a potential solution would work.In the hallway problem, student teams seldom made explicit the constraints and criteria which made determining the feasibility of an idea more difficult and time consuming.The National Center for Engineering and Technology Education suggested, "As designs are considered for viability, optimization is essential.Students should make their value structures and goals for design success explicit early in the decision process.This sense of clarity provides opportunities to select and promote designs that make the most successful balance of trade-offs" (Householder & Hailey, 2012, p. 26).In the less structured hallway problem, students would consider an idea and then ask if it would work or be too expensive without having specified the definition of success or budget.
Teachers should encourage students to identify the constraints and criteria as well as how success should be measured early in the design process.Feasibility considerations were slightly higher for team based design problems than individual design problems.Further research might test for a causal relationship, which if present, would indicate that teamwork might facilitate experience and exposure to critical thinking about solutions in the feasibility phase of the design experience.The National Center for Engineering and Technology Education caucus of 2012 suggested that "In collective team efforts, students may hold each other accountable for meeting criteria" (Householder & Hailey, 2012, p. 18).The sense of accountability may have been manifested in feasibility as students questioned each other's ideas prompting consideration of flaws and opening the door for improvements.Stakeholder interests were included in student discussions of feasibility in the hallway problem much more frequently than the playground problem.In the hallway problem, they mentioned considerations such as how students would interact with their solution, how teachers would be involved and react to students in a redesigned hallway.They considered impacts of hallway reconstruction on the neighboring rooms and how the changes impact roles of librarians, cafeteria staff, and classroom teachers.Occasionally, parents and shopkeepers were mentioned in the playground problem as stakeholders, but with far less emphasis.This may be related to the sense of relevance provided by the hallway problem, because students cared in a very personal way about the success of their design and considered a larger system of stakeholders.
Teachers may want to be sensitive to the gender composition of their student teams in view of the fact that single gender teams spent more time defining the problem than did mixed gender teams.This might suggest that encouraging and prompting students to develop an understanding of the problem is more urgent with mixed teams than with single gender teams.
The general lack of evaluation and decision making may relate to a lack of alternatives for consideration.Students tended to think about new ideas until they had a few viable options and developed those into their final design.The lack of alternatives generally reduced the need to evaluate differences between them and reduced the number of decisions (choices between alternatives) to make.Teachers should encourage students to develop a significant list of alternative ideas before evaluation and decision making.Decisions regarding materials were made on the playground problem, but in both design problems, students did not develop many alternative designs.They considered the advantages of different materials but seldom considered holistically different solutions.

Future Research
Findings from this study suggest potential trends and correlations between individual design activities and group design work as well as suggesting differences between teamwork on two different kinds of problems.This study had a sample size of 17 teams in total.Eight teams were challenged with the playgroup problem and nine teams engaged in the hallway problem.With only eight or nine teams in comparison, statistical analysis was not conducted.This work is potentially foundational to larger studies as it may allude to trends and correlations that could be tested in experimental or quasi-experimental research conditions on a larger scale.
From a methodological perspective, the inverse relationship between time spent in problem definition and time spent in feasibility might provide insight into student problem definition.Though problem definition in the hallway design task was seldom coded, future researchers could use feasibility as a method of extracting student definitions of the problem.Feasibility considerations have inherent value statements that could provide a proxy for problem definition.Therefore, by analyzing student conversation about the feasibility of an idea, a future research team may be able to identify the implicit constraints and criteria that students do not mention explicitly.For example, if a student judges a potential solution to be too expensive, we can infer that cost is a criterion even though it was not mentioned as one.
Students mentioned concerns such as slide or platform height or soccer field location in the playground problem, which relate to the constraint presented that the playground must be "safe".In this example, the students silently operationalized safety by considering a minimum distance from a nearby road to the soccer field.In addition, students' brainstorming activities may provide insight to the problem definition in that they tend to think of potential solutions and the commonalities across those solutions may be hints into their problem definitions.As an example, students who list different ways of controlling student hallway traffic such as traffic lights, teachers, or mirrors, may suggest implicitly that widening the hallway is not practical or that they feel constrained by the lack of resources to make major structural changes in the school architecture.
Further research might investigate qualitative differences in the ways and methods in which all female teams engaged in problem definition as compared to males and mixed groups.Female groups spent more time and a much higher percentage of time on problem definition than their peer groups.
Additional investigation might clarify differences between genders in terms of modeling and communication activities.Female groups spent dramatically less time modeling and communicating which accounted for most of the differences in overall design problem time.Are females modeling and communicating less or differently than their male and mixed gender teams?Would this effect be reduced if more explicit modeling and communication expectations were provided to the teams?

Summary
In this study, seventeen teams, each comprised of 2-4 high school students, were asked to complete a team based engineering design challenge.Observational protocol analysis was conducted based on a foundation of previous work, including the adoption of previous coding schemes.Differences between groups and individuals were compared.Teams of students were split in two groups; one set of teams received a playground design problem while the other received a hallway design problem.Teams worked up to two hours after school on the design problems and provided the recommendations resulting from their work at the conclusion of the session.
. Results of the discussions by this group of experts indicated that excellent design challenges should incorporate the following characteristics: