High School Students ’ Use of Paper-Based and Internet-Based Information Sources in the Engineering Design Process

Science, Technology, Engineering, and Mathematics (STEM) education continues to be a national concern. In the State of the Union address in January 2011, President Obama called for 100,000 new STEM teachers over the next decade. The Programme for International Students Assessment (PISA) investigated the academic achievement of 15-year-old students from 60 countries and five education systems in the areas of math, science, and reading. Results of the 2009 study indicated that the U.S. students were ranked 23 of 60 countries involved (Fleischman, Hopstock, Pelczar, & Shelley, 2010). Published in 1983, A Nation At Risk: The Imperative for Educational Reform called for improvement of education at the secondary levels (National Commission on Excellence in Education, 1983). The report demanded more emphasis on science and mathematics at both the primary and secondary level, creating the academic foundation for science, technology, engineering, and mathematics education. Technology and Engineering education (the T and E of STEM) have seen increased attention in recent years. The National Academy of Engineering commissioned a study titled “Engineering in K–12 Education,” which included a review of curriculum materials related to the T and E of STEM education as well as the relationship between Science, Technology, Engineering, and Mathematics education. The National Academy’s work emphasized the role of engineering in improving STEM education as it may be a “catalyst” serving to draw connections between mathematics, science, and technology education (Katehi, Pearson, & Feder, 2009). Design is essential to the disciplines of engineering and technology. Atman, Cardella, Turns, and Adams (2005) stated that “Design is a central activity to all types of engineering. Mechanical, Civil and Electrical Engineers attempt to solve very different types of problems, but they all design some solution to the problem at hand” (p. 325). Critical to differentiating technology from other fields, such as science, is “the ability to design” according to Layton (1974, p. 37). Sheppard, Macatangay, Colby, and Sullivan (2009) stated that “engineering design involves a way of thinking that is increasingly referred to as design thinking: a high level of creativity and mental discipline as the engineer tries to discover the heart of the problem and explore beyond the solutions at easy reach” (p. 100). The National Center for Engineering and Technology Education focused its efforts on infusing engineering design into high school technology education classrooms. Through a series of research studies focused on student

learning and professional development, the center refined a design process emphasizing eight essential elements appropriate for high school learners (Childress & Maurizio, 2007, p. 3): 1.
Identification of a need 2.
Definition of the problem/specifications 3. Search 4.
Test prototype and verify the solution 8.
Communication These eight steps are generally congruent with texts describing the engineering design process for engineering students (Dym & Little, 2004;Eide, Jenison, Mashaw, & Northup, 1998;Eide, Jenison, Northup, & Mickelson, 2008;Moore, Atman, Bursic, Shuman, & Gottfried, 1995).The engineering design process, as noted by Sheppard et al. (2009), "is not linear: at any phase of the process, the engineer may need to identify and define subproblems, then generate and evaluate solutions to the subproblems to integrate back into the overall process" (p.104).Sheppard et al. summarized the design process to include three broad areas of focus: defining the problem, generating candidate solutions, and evaluating and implementing candidate solutions.Sheppard et al. also added that communication, teamwork, time management, and project management are essential broader professional skills requisite to success.
The need to gather information is ubiquitous in the design process.Bursic and Atman (1997) stated that the step of gathering information is critical to create a successful solution of an engineering-based problem.Childress and Maurizio (2007) used the term search to mean exploring existing solutions and how they work.This search also includes parts of the solution or components that may already exist and can be combined in a novel way.Dym and Little suggested that gathering information was an essential element of the problem definition, conceptual design, preliminary design, detailed design, and design communication.The sources of information throughout the design process include literature on modern solutions, experts, codes, and regulations; competitive products; heuristics; models; known physical relationships; design codes; handbooks; local laws/regulations; suppliers component specifications; and feedback from clients/users (Dym & Little, 2004, pp. 24-25).Eide et al., suggested that after problem definition, "The team next acquires and assembles all pertinent information on the problem (Step 2).Internal company documents, available systems, Internet searches, and other engineers are possible sources of information" (2008, p. 44).In addition, Eide suggested, "Often customer requirements are not well defined.The design team must determine, in consultation with the customer, the expectations of the solution" (p.46).
Engineering design problems present an opportunity to contextualize the study of technology and engineering in authentic learning experiences where improvement of people's lives are the focus (Svihla, Petrosino, & Diller, 2012).Problem-based learning literature related to technology and engineering education suggest engaging the learner in a constructivist learning environment through design problems as a teaching methodology (Brodeur, Young, & Blair, 2002;Fosmire, 2011;Gijselaers, 1996).Creating an authentic learning environment requires that as students work through a design challenge, they have access to information relevant to their problem (Ekwaro-Osire, Afuh, & Orono, 2008;Fosmire, 2011;Wang, Dyehouse, Weber, & Strobel, 2012;Zimmerman & Muraski, 1995).Information access in classrooms may come from teacher generated documents or texts onsite, but to be authentic, should also come from access to the Internet (Katehi, et al., 2009).Engineers working on design problems use onsite resources but also access databases and search for information beyond the limits of their peers and local documents.
Teachers often present students with some information related to their design problems, but that information will be limited.Teachers have limited time to prepare and cannot explore all possible aspects of the problem at hand.To be authentic, students should engage in some problem definition which is illstructured and open-ended (National Center for Engineering and Technology Education, 2012).Teachers inherently have a bias toward potential solutions paths, and teacher gathered information will inherently be guided by this bias therefore steering the students and potentially limiting creativity.Due to the limits of teacher prepared information resources, providing Internet access may help to address these concerns.However, not all classrooms have convenient computer access.
Though efforts to provide all students with computer and Internet access are rapidly expanding, not all students have access and not all students with access are successfully using the technology (Penuel, 2006).Studies have shown that one-to-one computing (Lei & Zhao, 2008) provides students with opportunities to engage with communication technologies, but also raises concerns about digital literacy.Mentzer and Becker (2011) and Becker and Mentzer (2012) demonstrated that high school students engaged in engineering design problems spent more time accessing information and spent more time designing when provided with Internet access.They studied high school students engaged in an engineering design challenge.The two studies attempted to apply the same research methodology as was used in previous work by Atman to facilitate comparison between high school students and experts.The 2011 study included Internet access, but the 2012 study did not.Their work showed that with Internet access, students spent an average of 137 minutes engaged in designing a playground and students allocated 47 minutes (35%) to information access.Without Internet access, similar students from the same schools on the same design problem spent an average of 92 minutes of which, 10 minutes (10%) was dedicated to information access.
With limited computer access or limited time to enable students to access a computer in some classrooms, the research questions guiding this study are: 1. What information do high school students spend time accessing during an engineering design challenge?How much information comes from paper-based resources as compared to the Internet?2. How much time do they spend accessing information?What is the balance of time spent accessing information from paper-based sources as compared to the Internet?

Significance
Secondary education is increasingly pressured to deliver quality STEM education.Mathematics and Science education have received substantial investigation, but Technology and Engineering education are emerging as fields of inquiry related to pedagogy in K-12 environments.Little empirical research based guidance exists for teachers related to teaching engineering design in a secondary context.A variety of existing curriculums require students to engage in design thinking and specifically expect students to gather information to inform their design.Three curriculums discussed in this paper stop short of providing the teacher with details related to the information gathering effort.This investigation attempted to shed some light on student behavior related to information access, which has implications for secondary education.Answering these questions may help guide teacher and administrator decisions regarding how and when to use the Internet in design challenges by presenting information on how students are currently using the resources and for what purpose.

Treatment of Information Gathering in Curriculum Efforts
The National Academy of Engineering and the National Research Council (2009) identified 34 engineering-based curriculums or engineering resources that have been developed for implementation in the middle and high school classroom.Project Lead the Way and Engineering by Design were among the curricula reviewed and have a significant national footprint.Both curricula include a sequence of courses spanning middle and high school learning environments.Each curriculum engages the learner in a senior level capstone course, which includes a substantial focus on an engineering design problem.The smaller design problems students encounter in each course, as well as the more significant capstone design problem, present students with a need to access information.
Information gathering in these two curriculums is explicitly called for in the researching phases and is situated early in the design process, as shown in Table 1 (next page).As stated in the Standards for Technological Literacy (International Technology Education Association, 2000), "Design is regarded by many as the core problem solving process of technological development" (p.90).Within the surveyed curricula, Project Lead the Way and Engineering by Design, each offers its own approach to a design process when solving a design challenge.
Selecting an approach Develop design proposal Develop a design proposal Make a model or prototype Making a model or prototype Test and evaluate the design Testing and evaluating the design using specifications Refine design Refining the design Create or make solution Creating or making it Communicate processes and results

Communicating processes and results
The design processes proposed by Project Lead The Way and Engineering By Design suggest that the research and generate idea stage requires students to search for previously developed solutions to the problem (International Technology Education Association, 2008; Project Lead the Way, 2010), a form of information gathering.Also, in the develop design proposal stage, students are expected to gather information on what type of materials they will need to make their solution (Project Lead the Way, 2010).Student may need to search for prices, material strength, and other solution element characteristics to complete their design during all stages of design.

Foundational Research Efforts
Working with nine expert engineers, Kruger and Cross (2006) were able to identify four design strategies: problem driven, solution driven, information driven, and knowledge driven.Problem driven, solution driven, and information driven strategies rely heavily on the designer's ability to gather and use information.Knowledge driven design is situated heavily in prior experience and person's knowledge.An information driven designer defines the problem and then spends a majority of their time gathering information.The information found provides the basis for developing their final design.Information driven designs are low in creativity, have few solution ideas, and many of the activities are emphasized by the gathered data (Kruger & Cross, 2006).With Internet access, the time spent gathering information could increase and students have the potential to access an unlimited data set.
Though information gathering is an essential element of the design thinking process, Christiaans and Dorst (1992) discovered that during information gathering, students became stuck on the collection of information rather than progressing on to the development of their solution.This could be interpreted in a few different ways.Are students not finding the right information?Are they looking for other ideas rather than creating their own?Or are they spending too much time looking for information?Prensky coined the term digital natives, which is a person who has been surrounded by information technology their entire lives (Prensky, 2001).He stated that, "Our students today are all "native speakers" of the digital language of computers, video games and the internet" (2001, p. 1).Over the decade since Prensky labeled the generation as digital natives, accessibility to the Internet has only become easier and increasingly ubiquitous.Digital information access is almost instant due to the development of electronic portable devices such as smartphones, ultra-portable netbook computers, and tablet technology.
Efficient development of solutions for problems is critical in today's fastpaced economy.Though digital information is available almost instantly, the sheer volume available may be overwhelming for high school students presented with a design challenge.Given access to the Internet, they must decide what information they need to know and where to search.In engineering and technology education curriculum, paper-based and Internet-based information is often shared with students as they work through a problem.Teacher and curriculum delivered information can be focused, concise, and organized efficiently for students to quickly apply the information to their challenge.When presented with an Internet search engine in the context of a design problem, students may find the lack of structure difficult to effectively focus their efforts and the additional information access may be a hindrance to problem solving, as they might not be capable of efficiently utilizing the broad array of sources available.
In efforts to improve college-level education, previous work has focused on information gathering (Adams, Turns, & Atman, 2003;Atman et al., 2007;Atman, Chimka, Bursic, & Nachtmann, 1999;Atman, Kilgore, & McKenna, 2008;Cross, Christiaans, & Dorts, 2007).In a study by Mosoborg et al., (2005) 19 engineers with an average of 19 years of field experience were given a list of 23 words and phrases related to design activities and asked to pick which they thought were the six most important.Fifty-three percent stated that seeking information was one of the top six activities.In a similar study conducted at the University of Washington, 178 college-level engineering students were given the same list and were asked to complete the survey.Thirty percent of the students stated that seeking information was one of the top six activities of design (Morozov, Yasuhara, Kilgore, & Atman, 2008).
Several studies have been conducted in which college-level engineering students completed an engineering design task.Verbal data was transcribed and then coded using a set of eight codes related to the design process, one being information gathering (Atman, et al., 2005;Atman, et al., 1999).In one of the studies, college level engineering students were given a design task in which the main focus was to design a playground.The students that completed the design task spent 13.2% of their design time accessing information, which equates to an average of 14 minutes of the total 107 minutes spent on the design task (Atman, et al., 1999).Atman also found that after students had completed four years of engineering education, there was an increase in the amount of time spent gathering information.Freshman spent less time accessing information, while seniors and experts spent a comparable amount of time (Atman, et al., 2007).

Research Design
Methods.Students were given a design task to complete during a three hour session.The design challenge was not different from those that were used in various engineering design curriculums.The design task description can be seen in Figure 1(next page).The task was adopted from previous work implemented by research efforts put forth through the University of Washington (Adams, et al., 2003;Atman, et al., 2007;Atman, et al., 2005;Atman, et al., 1999;Atman, et al., 2008;Morozov, et al., 2008;Mosborg, et al., 2005).
Participants.The sample participants were high school students who were enrolled or had completed engineering-based classes.Although not required, the target candidate was a student who has had more than three different engineering-based classes during their academic career.Of the 12 students that volunteered to participate in the study, all were senior design students who had completed at least 4 courses related to engineering.Four of the students were female participants.All of the students who participated in the study considered themselves White or Caucasian.
Data Collected.Video data was recorded of the design performance.Students were audio recorded and asked to think out loud, consistent with verbal protocol analysis.Paper-based information requests were documented by the administrator by topic and time requested.Internet-based information requests were monitored by a software program running in the background that logged each search term and web site visited.

Figure 1
Playground Task Instructions.Adapted from Atman et al., 1999

Playground Problem Task Instructions
You live in a mid-size city.A local resident has recently donated a corner lot for a playground.Since you are an engineer who lives in a neighborhood, you have been asked by the city to design a playground.
You estimate that most of the children who will use the playground will range from 1 to 10 years of age.Twelve children should be kept busy at any one time.There should be at least three different types of activities for the children.Any equipment you design must be safe for the children remain outside all year long not cost too much comply with the Americans with Disabilities Act The neighborhood does not have the time or money to buy readymade pieces of equipment.Your design should use materials that are available at any hardware or lumber store.The playground must be ready for use in 2 months.
Please explain your solution as clearly and completely as possible.Someone should be able to build the playground from your solution without any questions.The administrator has a lot more information to help you address this problem if you need it.Be as specific as possible in your requests.
For example, if you would like a diagram of the corner lot, some information about the lot appearance, etc, you may ask for it now.If you think of any more information you need as you solve the problem, please ask for it.Remember, you have approximately 3 hours to develop a complete solution.The administrator will tell you how much time is left while you work.

Data Analysis
Information Time Measures.Using Nvivo qualitative research software, the video of the students' performance was coded when they were directly gathering information.The software allowed the video to be played and coded simultaneously.The recording of the design session was then broken down further to compare the amount of time each participant spent accessing information from the paper-based source and Internet-based sources.The overall time spent using each source was then compared for each participant.A Microsoft Excel file was compiled of each participant's time gathering information from the two sources and group's means, and standard deviations were calculated.
Information Categorization Sources.Using the output from the Internet activity tracking software and the requests documented by the administrator, a chart was created for each participant that included the information request and source type.Using a list of 29 different information types that was developed for previous research (Mosborg et al., 2006), the students' requests were placed into one of the categories.Information requests were coded in chunks.For example, if a participant asked for several pieces of the same type of information, it would be coded once instead of how many pieces of information were found within one request.This was completed by undergraduate students who were trained and calibrated.
Calibration of the undergraduate coders was iterative and began with two students working together until they came to a general consensus on how to categorize the piece of gathered information.This was done by calculating coder inter-rater reliability.Once the training was completed, each coder was given one half of the design sessions.An overlap of 25% was coded to document reliability.An acceptable Kappa value for inter-rated reliability is above 0.75 (Orwin, 1994).The calculated Kappa values for the coder were above 0.90.
Results.The collected data provided results that were used to address each research question, refer to Tables 2 and 3 (page 68).On average, participants spent 38.8% of their total time accessing information.Of that 38.8%, participants spent 26% of their time gathering information using paper-based sources while spending 74% of their time using Internet-based sources.Of the 29 information request categories, participants only request information from 20 of the categories.

Table 2 Participant Time Allocation
Each participant requested 19.8 pieces of information on average with over half of those requested coming from Internet-based sources.The most sought after piece of information was material cost, being requested from Internetbased sources 5 times on average and 4.3 times from paper-based sources.Comparing the use of Internet-based and paper-based sources, participants spent nearly triple the amount of time using the Internet-based information sources when gathering information.
Using the categories that were implemented by Mosoborg et al., (2006) the information requests of the participants was categorized.Information categories that were not requested were as follows: (a) Age, (b) Facilities, (c) Legal, (d) Occupancy, (e) Park area inside the lot, (f) Utilities, (g) Supplier, (h) Supervision, and (i) Schedule.Of the categories that participants gathered information for, information on material cost was the most prevalent.On average, material cost was requested 9.3 times per participant.Of those 9.3 times, 4.3 pieces of information on material cost were accessed from the paperbased source and 5.0 pieces of material cost information was accessed from the Internet.

Discussion
Results showed that students spent more time on average gathering information when compared to their peers who did not have Internet access.High school students spent a total of 38.8% of the time on task gathering information.Previous studies of college students and experts used similar methodology with the exception of Internet access, which was not provided.Compared to previous studies college level freshman engineers spent 12.4% of their time, senior level college engineers spent 14.1% of their time and expert engineers spent 16.3% of their time gathering information during their design time (Atman, et al., 2007).
Past research studies have shown that as engineers move from college freshman to college seniors to experts, their time on information gathering increases.When comparing results from the current study, this trend does not hold true, as high school students spent much more time gathering information.The additional time spent accessing information may be caused by the Internet access.Access to the Internet may change the ways in which students attempt to solve engineering design problems as they spend more time on design, spend more time on information access, and access different types of information from the Internet than they do from paper-based sources.
Time is a precious resource, and time spent in high school classrooms is limited.Access to the Internet increased the amount of time spent in the design process.Most of the increase in time was invested in information access, but time spent in other aspects of design increased in addition.However, with the exception of cost of materials, results indicate that students generally searched for different types of information from the Internet as compared to paper-based sources.Students typically used the Internet to investigate dimensions, typical playground activities, material specifications, images of playgrounds, and technical information at a much higher rate than they looked for the same types of information from paper-based sources.On the other hand, students tended to access ADA information, budget, material type, and neighborhood characteristics dominantly from paper-based sources.

Implications of this Research on Student Learning
Students tended to spend more time investigating the problem with Internet access, and they access more pieces of information via the Internet than they did from paper-based sources.With Internet access limited to schools that have resources to provide computers and network connections, not all students have access to this authentic source of information.Teachers may consider scheduling time in computer labs or ensuring that students share computers in classrooms where one-to-one computing is not available.The preference of students to increase the information gathered when the Internet is available may change their design solutions.
Increases in total design time and information gathering time has a cost in the classroom.The additional time spent on one design problem is less time spent elsewhere.Teachers should prioritize their objectives such that they can justify the extra time spent on design.As students use the Internet for design thinking, they may need support developing efficient information access skills.Previous studies showed that experts access 1.1 pieces of information per minute while high school students were accessing 0.38 in this study.The difference might be related to students having Internet access, but this might also be related to a lack of information literacy skills.Teachers should closely observe student Internet use to determine levels of guidance needed to improve efficient use of the resource.
Design work includes consideration of costs, but students are spending substantial amounts of time searching, and the bulk of their searching is for material cost.They spent time looking for the cost of materials through paperand Internet-based sources to the extent that one-half of the pieces of information accessed related to cost.This time might be more effectively used searching for other information or used for other elements of the design process.To minimize the time spent searching for costs, teachers might encourage students to estimate costs based on the stages of the design process.In the preliminary stages of design, where ideas are rough and developing, an estimate will permit comparisons to be made and feasibility to be assessed.Spending time searching for the exact cost provides little additional benefit over an estimation in this phase of the design work.In this study, it was common for students to ask for the cost of (for example) a wooden 2 x 4.After asking for the paper-based cost, they searched multiple vendors including Lowes, Home Depot, and even Craigslist for the cost of the same material, looking for the cheapest source for their bill of materials.In later stages of the design process, optimizing resources by minimizing costs are significant, but most student designs tended to be more conceptual.

Recommendations for Future Research
Data from this study suggested that students spend substantial amounts of time on the Internet with few information pieces accessed.Observations of student behavior by research administrators tended to suggest that students drifted from one website to another and accidently discovering information rather than purposefully searching for it.Additional research might differentiate between students' purposeful search activity and accidental information discovery.As an iconic example from data review, students would search for pictures of playgrounds.Frequently, students would view a website selling equipment with safety mentioned on the page; the student would then search for safety and notice maintenance issues.After noticing that wood would need to be maintained, they might add paint to the budget and ask about a budget for annual inspection.This string of events occurred regularly and may be triggered by the web-like interface of the Internet rather than purposeful forethought of the student.This leads the research team to consider the impact Internet use has on solution quality, as students might not have considered a variety of facets of the solution (such as maintenance in this example) essential in the design process.
Students rarely commented on the quality of the information source.The research team frequently thought about the validity information.There have been efforts to rate the validity of information, especially in direct relation to Internet-based sources (Wilson & Risk, 2002).Following the same procedures, data could be collected for the intent to determine whether or not high school engineering students considered information validity.Data were not rated for validity, but frequently students went to websites such as Wikipedia which may not be considered a valid website (Waters, 2007), and students often relied on commercial websites.
Information access has dramatically accelerated in recent years.Future pedagogical efforts may need to refine student information literacy skills to prepare students for applying available information in meaningful ways to the design problem at hand.Students in this study demonstrated frequent use of the Internet and made requests of the administrator for paper-based information.However, they spent a substantial amount of time searching for information with a relatively (as compared to previous research) low yield.Information literacy skills and educational efforts focusing student attention of critical missing pieces of information may increase efficiency of student research work.

Table 1
Engineering Design Processes as Presented by Two National Curriculums

Table 3
Information Request Categorization

Table 4
Comparison of Efficiency Rates