The Energy Conversion Playground (ECP) Design Task: Assessing How Students Think About Technical and Non-technical Considerations in Sustainable Community Development

Students in global service-learning and similar programs frequently encounter substantial social, cultural, political, and ethical differences when working with project partners in different countries and regions. Neglecting such differences can lead to project failures and/or disempowered communities. In response to these challenges, educational resources have been developed to teach students to think about how the people, social structures, and other contextual factors associated with projects can affect, and be affected by, students’ designs. Yet, there remains a scarcity of valid and reliable instruments to evaluate the effectiveness of such interventions. The purpose of this study is create a theoretically and empirically grounded instrument, the Energy Conversion Playground (ECP) design task, that is able to provide a meaningful and robust assessment of an individual’s ability to identify salient technical and non-technical considerations when approaching an engineering design task situated in a developing country context. We present the scenario and an accompanying rubric that was first developed inductively from student responses to the scenario (specifically 449 discrete items from 93 ECP design tasks submitted by students who attended a Global Engineering Design Symposium). Further development of the rubric involved deductive grounding in relevant literature. To demonstrate the sensitivity of ECP design task to changes in students’ thinking, we also performed comparative analysis of responses from a subset of the students (n=37) who completed the same instrument both before and after participating in the GEDS.


INTRODUCTION
Engineering instructors and degree programs are increasingly challenged to foster both technical and non-technical competence in pre-professional engineers [1][2][3] . Among a large and growing array of existing courses, programs, and organizations that potentially support this outcome, those et al. used a mixed-methods approach in which an initial quantitative data collection phase leveraged the National Engineering Students' Learning Outcomes Survey (NESLOS), an instrument developed to assess students' self-reported perceptions of competencies. 20 While assessing perceived gains in learning outcomes might be useful to determine students' sense of confidence in their knowledge and capabilities, such assessment is also limited to the extent that students might not interpret the learning outcomes as intended by the course instructor, and/or may experience considerable under-or overconfidence when rating themselves. Consequently, there is a need to assess learning outcomes using a wider variety of valid and reliable methods.
In order to gain insight into how students think about solving a design task without relying on self-reported competency measures, we developed the Energy Conversion Playground (ECP) design task. This scenario-based instrument is comprised of an assessment task and accompanying scoring rubric that is focused on engineering design in a less developed country context. More specifically, the instrument is designed to evaluate the main types of considerations identified and prioritized by students in relation to a realistic design scenario. While pilot results were presented in a previous study 21 , this paper reports on further development of the ECP with an emphasis on its conceptual and empirical grounding. This paper describes a version of the ECP instrument that was administered to students participating in Global Engineering Design Symposium (GEDS), a half-day event designed to make participants more aware of the full spectrum of social, cultural, and political issues that can surface when working on design projects in less developed communities. 18 As we elaborate below, we have substantially refined the rubric used in the original pilot study through inductive grounding in the data collected (449 items from 93 ECP completed design tasks), as well as further deductive grounding in relevant literature. In this paper, we also describe the extent to which students (n=37) changed their thinking due to participation in the GEDS event, namely by evaluating and quantifying the types of considerations they reported in responding to our instrument. However, the goal of this paper is not to make claims about the effectiveness of the GEDS event, but rather to show the ECP tool's sensitivity to detecting changes in student thinking (a form of pre-post test validity). Yet before elaborating on development and use of our study instrument, we first explore literature discussing the types of considerations and factors that are typically relevant in SCD and other engineering design settings, including theoretical and conceptual frameworks that inform our research. This paper is particularly intended for engineering educators and engineering education researchers who are looking for innovative ways to teach and/or assess design capabilities among students and professionals, with particular emphasis on their levels of awareness regarding both technical and non-technical design factors and considerations.

ENGINEERING DESIGN: TECHNICAL AND NON-TECHNICAL CONSIDERATIONS
Many commentators claim that engineering, and particularly design, involves integrating a wide variety of technical and non-technical elements and considerations. As Bordogna, Fromm, and Ernst argue, "the concept of integration, or synthesis … is the hallmark philosophy of the engineering profession" (p. 3). 22 More recent publications reiterate these themes. Duderstadt, for instance, describes the ability to integrate knowledge "across an increasingly broad intellectual span" (p. 45) 23 as an essential competency for engineers, while Sheppard et al. portray engineering as "integrative to its core", requiring "the purposeful and thoughtful integration of knowledge and process to create a solution to some particular problem" (p. 174). 3 While such commentators recognize the value of integrating multiple considerations in engineering design, how do engineers and engineering students actually identify, relate, and prioritize such considerations when doing design work? Some prior research has addressed this question. Adams, Turns, and Atman, for example, systematically categorized design considerations through their coding framework for the "Midwest Floods (MWF) Design Task", including in terms of both frames of reference (e.g., technical, logistical, natural, and social) and physical locations (e.g., wall, water, bank, and surroundings). 24 Building on this work, Kilgore, Atman, Yasuhara, Barker, and Morozov framed technical or logistical considerations of the wall or water as "design detail," while all other considerations (e.g., natural or social considerations, considerations related to the bank or surroundings) were regarded as part of the "design context" (p. 326). 25 Among other significant findings, studies involving the MWF design task show that women are more likely than men to attend to contextual considerations 25 and graduating seniors are more likely than first year students to identify more general design considerations. 24 Although these studies provide useful language to categorize design considerations, other studies suggest that "social" considerations are often more nuanced than what can be captured by the coding framework developed for the MWF design task. For example, Zoltowski, Oakes, and Cardella's phenomenographic study of how engineering students experience human-centered design generated seven qualitatively distinct categories that differentiate how students view the social considerations of engineering design. 26 As this work describes, an engineering student might only consider how users provide information for a technical design (Category 3), or may fully empathize with users via immersion in user experiences (Category 7). 26 Additionally, through his analysis of multiple case studies and examples of sustainable community design, Lucena discusses how engineers tend to prioritize considerations in design based on how they view the people who are connected to the design. 27 He suggests that taking social considerations into account is inevitable in sustainable engineering design, but can vary considerably according to how engineers view relevant customers, quantifiable stakeholders, users, citizens, etc. 27 While this prior work suggests wide variations in how engineers and engineering students regard social considerations of engineering design, in contrast to the MWF design task, these studies did not look first-hand at student performance on hypothetical design tasks.
Informed by this prior literature, we sought to categorize the responses of students as they articulate their design considerations with respect to a sustainable community design task, as described below. Like the coding framework of Adams et al. 24 , we sought to develop distinct categories that captured these design considerations. Aligning with Zoltowski et al. 26 and Lucena, 26 we also sought to develop a categorization scheme with a more nuanced view of nontechnical considerations, especially with respect to how students view the people connected to a design task. Leveraging both prior literature and our own empirical data, we identified four types of design considerations: technical, non-technical constraints, stakeholders, and broader considerations. In our research methods section below, we describe the inductive process of developing these categories through several iterations of coding. In the sections that immediately follow, we articulate our deductive process of aligning these categories of design considerations with extant literature. In contrast to previously discussed literature, 26,27 we do not necessarily organize these categories in a hierarchical manner. Rather, like Adams et al., 24 we conceive of these categories of design considerations as distinct from one another, but also interrelated.

Technical Considerations
Engineering design frequently includes some attention to how physical systems might be modeled scientifically and mathematically. Design is marked, at least in part, by considerations of what variables are needed for a mathematical model, how these variables relate to one another, and how they might be assigned meaningful values. We label these and related considerations as technical considerations. The role of technical considerations in design is made visible in several studies on engineering design. For example, an ethnographic study by Gainsburg, Rodriguez-Lluesma, and Bailey discusses how structural engineers regularly employ "considerable numbers of theoretical tools covering a broad spectrum of topics" (p. 206) 28 among several forms of knowledge. And while Buciarrelli's work is well-known for richly describing engineering design as a "social process", his ethnographic research also shows how practicing engineers consciously utilize knowledge from their respective technical "object worlds," or realms of instrumental thinking where abstraction, reductionism, and quantification are predominant. 29,30 Several other studies corroborate how technical considerations are manifested, and often prioritized, in realworld engineering design situations. 31 -33 While such studies depict how engineers employ technical considerations in design, other research illustrates how engineering students experience technical considerations. For example, Cardella shows how senior industrial engineering students employ extensive mathematical thinking in their design work, and are often unaware of how often they relate such mathematical thinking to design. 34 Her research is consistent with that of Downey and Lucena, who found that even when senior mechanical engineering students are prompted to integrate multiple considerations of engineering design, they tend to regard design as a mere extension of applying equations, models, and methods learned in engineering science courses. 35 While Cardella focuses on the cognitive depth of such mathematical thinking, 34 Downey and Lucena discuss how merely applying mathematical principles from prior coursework may blind students to other considerations, 35 While these studies provide different perspectives on engineering design, they both note how technical considerations are often employed by students who are regularly exposed to mathematical thinking in their coursework. Consequently, when eliciting students' considerations of a design task, we expect to find at least some considerations falling into the technical category.

Non-Technical Constraints
As noted above, several commentators and reports have emphasized the need for greater attention to non-technical considerations in engineering design. However, such considerations are often described as constraints to design. For instance, ABET calls for engineering graduates who are prepared to "design…within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability" (p.3, emphasis ours). 36 Further, in discussing their framework for their meta-analysis on engineering design literature, Mehalik and Schunn describe how engineering design involves building "normative models", or "what [designs] might look like if they were not constrained or limited" (p. 522, emphasis ours). 37 They also depict engineering design in terms of "explor[ing] constraints", "redefin[ing] constraints", and "validat[ing] assumptions and constraints" (p. 523). 37 Dym, Agogino, Eris, Frey, and Leifer define engineering design as more broadly resulting in "devices, systems, or products whose form and function achieve clients' objectives or users' needs while satisfying a specified set of constraints" (p. 104). 38 The National Academy of Engineering's Grand Challenges report also discusses how "governmental and institutional, political and economic, and personal and social barriers will repeatedly arise to impede the pursuit of solutions to problems" (p. 6). 39 This framing of non-technical considerations as constraints illustrates a dominant discourse in engineering design. We label such considerations as non-technical constraints. Considerations within this category represent a broader concern with how non-technical elements and factors may affect, limit, or generally inform the design of a technical system or artifact. In this category, the system that is designed occupies the central focus, while non-technical elements serve to inform or bound the design of this system. When non-technical elements are viewed as constraints, the information sought about such elements tends to be quantifiable. 27,28 This category can also include non-technical constraints related to the process by which a system or artifact is designed, e.g., team schedules, travel logistics, etc. Inclusion of such considerations reflects the fuzzy boundary between the design itself and the "social process" of design. 9,29,40 We include in this category those considerations that Adams et al. 24 would call logistical considerations, and perhaps some that they would regard as technical, social, or natural. Nontechnical constraints also encompass considerations regarding the best materials that would increase system effectiveness, in contrast to materials that are locally available (a broader/contextual consideration). Further, those who regard people connected to engineering design as customers or seek to express users' interaction with the system with quantifiable variables (e.g., frequency of use) might primarily regard non-technical considerations in their design as constraints. 26 Finally, those who align with Zoltowski et al.'s 26 lower categories of experiencing human-centered design (e.g., Category 3: User as Information Source Input to Linear Process) might be inclined to articulate considerations in the non-technical constraints category. Some have criticized a tendency among engineers to exclusively consider non-technical elements as constraints or quantifiable elements, 27, 30, 41 but we also recognize that engineering design does, and perhaps should, include at least some identification and articulation of nontechnical constraints 38 .
In sum, given the dominant emphasis on non-technical constraints in engineering design processes or systems, we are unsurprised when student responses reflect considerations in this broad category. Furthermore, labeling this category as non-technical constraints helps differentiate these factors from other kinds of non-technical considerations, as we elaborate in the next two sub-sections.

Stakeholders
In contrast with the previous category, the stakeholder category places the people connected to the design as central, while the technical system or solution occupies a significant but peripheral position with respect to those who affect or are affected by the designed system. Considerations in this category reflect concerns about identifying a variety of stakeholders, articulating the needs and assets of these stakeholders, and incorporating different perspectives into the system's design. It is also worth noting that our discussion of stakeholder concerns is distinct from Lucena's use of the term to describe how dominant discourses of engineering design often frame people as quantifiable elements. 27 At a more basic level, Dym et al.'s definition of engineering design cited above recognizes the role of stakeholders in the design process, where the outputs of design are meant to "achieve clients' objectives or users' needs" (p. 104). 38 In a deeper sense, this category also encompasses a focus on design that fully originates from the stakeholders, and especially those who directly interact with such a system. This conceptualization is compatible with principles of human-centered design, user-centered design, participatory design, empathic design, or co-design, all of which are discussed extensively in design literature. [42][43][44][45] While we recognize that such a central concern for stakeholders of design may not mirror the dominant rhetoric of engineering design, engineering education researchers such as Zoltowski et al. have made strong cases for the relevance of human-centered design principles in engineering practice and education. 26 Informed by this literature, we generally employ the stakeholder category to comprise design considerations that seem to place a prominent focus on the people who interact with a design or solution rather than emphasizing the technical system itself. Yet in contrast to the next category, stakeholder considerations reflect concerns about the interactions between people and designed systems or artifacts rather than more general concerns about the people themselves. Although we do not expect to find profound human-centered responses in our brief design task, we do suspect some awareness among students for the centrally important roles people play in engineered systems.

Broader Consideration
The final category reflects broader considerations that may be relevant to the design of a system, but without necessarily focusing on the design itself nor direct interactions between people and the design. For example, one might articulate a concern for the designers' interpersonal relationship with stakeholders, ethical considerations associated with the design, the stakeholders' sociocultural context, or how the design interacts with the natural environment. This category aligns with several considerations that are discussed in literature already cited. As Lucena et al. propose, for instance, design considerations in sustainable community development should encompass knowledge of culture and history. 4 Additionally, Lucena argues that if an engineering designer views stakeholders as citizens, this will help make visible core design considerations such as interpersonal relationships and the social justice of the stakeholders. 27 Finally, Zoltowski et al. demonstrate that empathic design, which involves broad immersion in the users' lived experiences, is the highest form of experiencing human-centered design (Category 7). 26 Alternatively, one might appeal to any of the concerns cited earlier in order to voice criticism of a given task. The design task scenario that we present to students, as later described, is intentionally problematic with respect to the social justice of the stakeholders. As Riley 46 and Kabo and Baillie 47 have demonstrated, core mindsets among many engineering students and professionals may blind them to the kinds of social justice issues that are often inextricably bound up with design tasks. Among these mindsets is "an uncritical acceptance of authority" (p. 42), 46 which, in the case of our study participants, would lead them to avoid questioning the design task itself. As Claris and Riley review, engineering education often fosters critical thinking "within focused elements of engineering" quite well (p. 102). 48 However, they argue that "[critical thinking] also ought to entail thinking critically about engineering" (p. 102). 48 and in the case of the present study, about engineering design challenges. The broader considerations category, thus, includes foundational critiques of given design tasks or problems.
The label of broader considerations admittedly comprises multiple, nuanced considerations. In part, we employ this label to capture a variety of considerations that do not directly focus on technical or non-technical features of a design. However, we mainly employ this category to capture considerations that generally demonstrate how the design of a technical system deeply interacts with a broad sociocultural ecosystem of actors, including those not directly connected to the system. Additionally, this ecosystem includes the natural environment, which may interact deeply with a given technical system. 49,50 Such a connection between a technical system and an interconnected web of human and non-human actors is reflected in several sociological theories of technology and society (e.g., Social Construction of Technology; 51 Affordance Theory, 52 Actor-Network Theory, 53,54 Sociotechnical Co-production 55 ). Although we would not expect most undergraduate students to be versed in this literature, it may nonetheless be the case that some students develop awareness for how a designed system might be linked to these kinds of broader considerations.

Context of study: Global Engineering Design Symposium (GEDS).
Responding to requests by students, faculty, and staff to provide more training and events for Purdue University (PU) students involved in global service learning programs and projects, a half-day Global Engineering Design Symposium (GEDS) event was created and launched. The data for this study was collected during the second annual GEDS in January of 2013. The event was scheduled early in the semester so students would likely have some initial familiarity with their service-learning projects, but still in a position to apply lessons learned from the workshop. To maximize the benefits of coordinating such an event and provide cross-program interaction and cross-fertilization for participants, invitations were sent to students and staff affiliated with two of PU's major service-learning programs, as well as students in PU's Engineers Without Borders (EWB) chapter. The event attracted more than 90 participants, although only a portion completed all activities.
Through presentations, panels, and interactive exercises, the five-hour workshop covered topics such as the moral and ethical dimensions of global service learning, cross-cultural communication strategies, and mechanisms to enhance stakeholder participation. Presentations were provided by invited speakers and PU faculty with backgrounds in engineering, engineering education, environmental science, and international development. The event also included a panel of students and recent alum involved with international development projects and international research, allowing attendees to hear first-hand about the successes and challenges of such projects and pose questions to the panelists. Additionally, interactive exercises using case studies from international development failures allowed students to reflect on factors that can contribute to project successes and failures, while challenging them to relate such lessons to their own work. More information about the GEDS event can be found in a prior conference paper, 18 while slides and videos from select presentations can be found on globalHub. It is worth emphasizing that one main goal of the workshop is to make participants more aware of the full spectrum of social, cultural, and political issues that can surface when working on design projects in developing communities. The assessment instrument described in this paper is well aligned with this objective, providing a mechanism for investigating the extent to which the workshop changed student perceptions about the salience of such issues.

Development and deployment of ECP
As documented by Jesiek and Woo, 56 a small but growing body of work has involved use of scenario-based assessment instruments to evaluate specific areas of competence among engineering student populations, with particular emphasis on design skills and abilities. Especially relevant to the present investigation are studies where respondents are asked to generate a list of specific criteria or factors that should be taken into account when addressing a realistic engineering design problem, e.g., the well-known "Midwest Floods Design Task" discussed above. 24,56 Scenario-based questions have also been used to assess adaptive expertise 58 and to study aspects of global competency, including understandings of how national differences are important in engineering work 59 and perceptions of desirable attributes for global engineers. 60,61 Scenario-based approaches to research and assessment are appealing for many reasons, including their grounding in realistic contexts of practice, and ability to more directly probe student abilities and perceptions rather than relying on indirect evidence, e.g., via selfassessments. 56 These types of questions are also readily adaptable for use as scaffolds in teaching and learning environments, and especially in tandem with case-based instructional approaches.
For the present study, we used an updated version of an instrument deployed in a previous pilot study that asked students to report five important considerations needed to address a design task in a developing country. 21 The task ( Figure I) was inspired by Pandian, who explains how playground devices such as the seesaw, merry-go-round, and swing can be used as humanpowered energy conversion systems. 62 The dominant focus of his article is on the detailed technical aspects of such a solution, and he suggests the system would be ideal for communities in developing countries. The author does not explicitly discuss the sociocultural aspects of the system and its installation, and only briefly notes that "[e]thical questions may be raised on the use of children for power generation" (p. 8). 62 Hence, we recognize this design task as a promising means to elicit the categories of considerations reviewed above. Additionally, others 63, 64 have implemented similar projects to that proposed by Pandian, suggesting that the design task is realistic and plausible.
Inspired by previous work 60 , the task's response form listed the numbers 1-5 to encourage students to report five distinct design considerations. Additionally, we asked students to circle their most important consideration. Finally, students were asked on a separate page to compare and reflect on their pre and post responses, and then discuss whether and how their responses to the design task changed.
Prior to the workshop, a variety of data was collected from prospective workshop participants via an online survey comprised of demographic questions, a Political and Social Involvement Scale (PSIS) 65 survey, and the Miville-Guzman Universality-Diversity-Scale Short form (MGUDS-S). 66,67 The assessment task was deployed at the very beginning of the GEDS event, and again at the very end, along with a program evaluation survey. Pre-event deployment of the activity was framed as part of a research project, but also as a reflective learning activity that was In developing countries, energy production is one of the most critical problems. Resources or technologies to produce energy are often not available. Thus, human power conversion systems might be used to power small appliances.
Imagine that you and your team are assigned to a design project in partnership with a Non-governmental Organization (NGO) of a developing country. The NGO needs a low-cost power system that can generate enough energy for the lights of a primary school. One of the members of your team suggests using merry-goround, seesaw, and swing to produce energy that can be converted to electricity for the lights.
In your opinion, what are the five most important things that you need to consider in order to successfully accomplish this design task? After you have written all five, please circle the consideration that you believe is the most important. relevant to the major themes and objectives of the workshop. All results were tracked using an anonymous identifier, and all data collection and analysis was carried out under PU IRB protocol no. 1212013060.

ECP scoring rubric
The scoring rubric for the design task was developed from the data prior to examining the literature previously discussed. The first version of the rubric was developed solely from data collected in our previous study 20 and was comprised of the following categories: technical, constraints, stakeholders, and culture of stakeholders. When the first two authors applied the original rubric to this study's item responses (n=449 randomized individual items from 93 completed design tasks), we achieved an inter-rater reliability score of 0.68, as assessed by Fleiss' kappa. In doing so, we recognized that the original rubric required further refinement to reduce the ambiguity of the categories and better capture the full breadth of responses. After several rounds of review and discussion, we identified a significant quantity of responses reporting ethical, political, and environmental considerations that did not belong to any of our original scoring categories. Moreover, some students also criticized the prompt itself or wrote about building relationships with stakeholders. Such considerations led us to modify the rubric by clarifying our definitions of the technical, constraints, and stakeholder categories, and renaming and redefining culture of stakeholders to become broader considerations.
The first two authors then applied the revised rubric to every randomized item response, resulting in an improved inter-rater reliability of 0.77 that is typically viewed as "substantial agreement" (0.61-0.80) 69 for this type of measure. In order to reach consensus the first two authors met several times to discuss the item codes and establish a final set of results. This process also allowed further, minor refinements to clarify the fuzzy boundaries around some of the coding categories. To assess the empirical robustness of the rubric, the third author and other researchers evaluated 100 items that the original coders agreed on and another 50 that they disagreed on. The third author and two fellow researchers were given some basic training and tips about the fuzzy boundaries among categories and were told to primarily rely on the wording of the rubric. In this final iteration, the inter-rater reliability among the ratings reached by the first two authors' consensus, as well as the ratings of the third author and the fellow researcher, was again substantial with a Fleiss' kappa score of 0.74. All raters also had a final meeting to discuss disagreements and establish final consensus. This last iteration of coding inspired a final round of minor modifications to the rubric, which appears in this paper as the final version.
The final rubric is presented in table I and includes definitions of the coding categoriestechnical (T), non-technical constraints (C), stakeholders (S), and broader considerations (BC)-and corresponding examples from selected students' responses. When applying the rubric, particular attention must be paid to the use of ambiguous terms like "sustainability". In fact, when such terms are used in many different ways by respondents they are coded using different categories. For example, responses such as "The sustainability of the designmaintenance" and "Sustainability of project. Is this a long term solution?" were coded as C, because of their explicit focus on the system. "Environmental sustainability" was instead coded as BC, as it reflected concern about the broader environmental context. When students simply wrote "sustainability", the response was simply coded as unknown (UK) due to its ambiguity. Another term at the boundary of categories is "safety." Although this term ultimately implies considering people, if people were not explicitly mentioned in the response, we coded the item as C to reflect the subject's primary concern with the technical system. Items that discussed "efficiency" were also categorized in multiple ways. When respondents wrote "Efficiency-How long will it last? Can it be repaired?", or "Economic efficiency, low budget project," we coded these items as C. However, when they wrote "Work output efficiency -serves the main purpose of producing enough electricity" or "Energy conversion efficiency," we coded them as T as these responses tended to reflect quantifiable considerations related to developing mathematical models. Focus is on technical characteristics of the system. Considerations reflect something that could be answered by a set of equations and/or variable(s) within a set of equations related to the physics of the design. This category also encompasses considerations that seek for quantifiable parameters related to the operation of the system. "Efficiency of electrical systems" "Will it produce enough energy?" "How often and how long does it need to be operated to produce enough energy?"

Non-Technical Constraints (C)
Focus is clearly on considerations about constraints to system or project and not on technical (T) considerations described above. Even if people are referenced, the primary focus is on the system/project or considerations that seek to quantify the interaction between system and users (e.g., frequency of use). This includes concerns about design process, costs/budget, best materials (rather than locally available materials), time considerations and/or other considerations of how the system/project might be limited or constrained.
"Schedule of project progression." "Frequency with which kids use these items" "What is the exact budget?" "How to maintain after implementation" "Safety of system" "Materials to use" "Climate effect on system"

Stakeholders (S)
Focus is explicitly on human beings. This includes concerns for not harming people, hearing the voices of stakeholders, communicating with stakeholders. This category also encompasses items focused on stakeholder's needs, opinions, involvement, and knowledge, how people will know how to use/maintain/repair devices or systems, education programs, and considerations of who will interact with the project. This category is distinct from the previous category (C) because items rated in this category are explicitly focused on people more than the system. "safety of students" "How do I involve the community?" "The skills of people living there" "Teach people to maintain system" "Are students willing to play on these?" "understand who will be working" "Who will be paying for this project?" Focus on considerations that go beyond solving the task as it is given in the prompt. This includes questioning the assumptions behind and appropriateness of the prompt, or criticizing the real need for such a project. This category also includes items that focus on building a relationship beyond the project, and making decisions based on broader sociocultural systems such as law, ethics, politics, culture, and environment. It also takes into account the effects the project might have on these broader systems. This category is distinct from the previous (S) as items in this category are not focused on the immediate stakeholders, but on the broader systems/contexts that include these stakeholders. This category is distinct from the category (C) as questioning of the task is not related to a design process (e.g., choosing "If design is appropriate for the region" "Is there a need for a low cost power system in the first place" "environmental effects introducing this system could cause" "Need to know the culture" "Is harnessing the power from children and dictating how or where or what they play with ethical?" "Understand country's energy system." "local natural, social, and financial International Journal for Service Learning in Engineering Vol. 9, No. 2, pp. 64-84, Fall 2014 ISSN 1555-9033 between alternative solutions), but rather the questioning reflects a concern about how the project/system is appropriate with respect to the broader systems named earlier. It could also differ from (C) because it considers locally available materials, rather than best materials. resources" "Local regulations, procedures, or standards" The considerations are too vague to be classified in the above categories.

Analysis
After we reached consensus on coding all 449 item responses, we performed two types of data analysis. First, we looked at frequency of items in each category, including overall count and counts for each student. We focused especially on changes in the number of items in the stakeholder and broader considerations categories before and after the interventions. We focused on these categories because the general goals of GEDS are to increase awareness of such design considerations. We analyzed responses from a subset of the original data, comprised of all complete, matched pre/post forms (37 students, 74 completed design tasks, 370 total responses). Likewise, we looked at pre/post-event changes of frequency for all students who identified one most important consideration (26 students, 52 total top responses). The statistical significance of changes in frequency was investigated using Wilcoxon signed-rank tests. Additionally, we compared our ratings to student reflections about pre/post-event changes in their responses. This allowed us to check whether our ratings were aligned with their own perceptions of change.

Results of item frequency counts
The first part of our analysis investigated how students' responses changed after the intervention. We counted the total number of instances for each coding category in the pre-and post-event forms, as illustrated in figure II. The results demonstrate that before the workshop the vast majority of item responses (116, 62.7%) fell in the T or C categories. The shape of the curve tends toward these considerations with a maximum on C (91, 49.2%). Analysis of the post-event responses shows a large drop of device-centered consideration in favor of responses falling in the S or BC categories (102, 55.1%), and with a maximum on S (72, 38.9%). The biggest drop in responses is in the area of technical considerations, with just one fourth the number of such items that were coded in the pre-event data. While the post-event curve is flatter than the pre-event curve, it is skewed toward S and BC, showing that after the intervention students listed more considerations focused on people and their broader sociocultural and environmental contexts.
Second, we examined the percentage of students who had relatively smaller and larger changes in the number of responses belonging to S or BC categories. Figure III reports the number of students who had a negative, null, or positive change in number of responses coded as S or BC, as well as the combined total number of non-technical (S+BC) items. Note that any given respondent could have changed two responses coded as S to one coded as S and another coded as BC. This would result in a -1 for the number of S responses (row 1 in figure III), +1 for number of BC (row 2 in figure III), and 0 for S+BC (row 3 in figure III). Consequently, the percentages in row 1 and row 2 of figure III do not add up to the percentages of row 3. The results show that only two students (5.4%) decreased the number of items belonging to the S+BC categories. Results for the combined responses from S+BC categories show that 70.5% (32.4% + 24.3% + 10.8%) of students added at least one consideration belonging to S+BC, and 35.1% (24.3% + 10.8%) of students added at least two responses belonging to such categories. Furthermore, results for the individual S and BC (row 1 and 2 in figure III) categories show a strong increase for each category, especially for responses coded as S. While less than half the students increased the number of BC considerations (18.9% + 16.2% = 35.1%), more than half increased the number of considerations belonging to S (32.4% + 21.6% + 8.1% = 62.1%). We assessed the increase in the individual and combined responses from the S or BC categories using Wilcoxon signed-rank tests. There were significant increases of responses for both the S category (p < 0.001) and the BC category (p < 0.05). Additionally, there was a significant increase in responses for the combined S+BC categories (p < 0.0001). shows that the majority of the students indicated an S or BC as the most important consideration both before and after the intervention (62% and 77%, respectively). The number of responses coded as S or BC increased after the intervention, with the majority of the students (62%) reporting an S consideration. Overall, our results show that after the event there was a slight increase of students that selected their top design consideration as S or BC considerations. However, these changes were not significant, as measured by Wilcoxon signed-rank tests.  Results of reflections A second line of inquiry compared our item ratings to student reflections on possible changes in their pre/post-event responses. Among the students who responded to this question (n=26), table  II gives examples of students who had the highest change (+3), no change, and negative change in the number of responses coded as S or BC. Generally, students whose response patterns changed acknowledged this change in their reflection. As mentioned in the methods section, Sarah's reflection also reveals that our coding can be strongly influenced by amount of detail a student reported in their responses. While Sarah recognizes that the considerations are "generally the same questions," she also recognizes that she was able to include more detail. Matt's reflection appears to be the only one not fully aligned with our rating. The student explains that in the post-test responses he was very concerned with "cultural boundaries and norms." We would have coded such consideration as BC. However, his responses lack such consideration as his post-test responses were scored as T (1 item), C (3 items), and S (1 item). Yet overall, our ratings and student responses were well-aligned except for a few exceptions where reflections were much richer in S and BC considerations as compared to what students actually reported on their post forms.

DISCUSSION
The main contribution of this study is a theoretically and empirically grounded instrument that is able to provide a meaningful and robust assessment of an individual's ability to identify salient technical and non-technical considerations when approaching an engineering design task. While the instrument and accompanying rubric allow identification of the types of technical considerations that often dominate engineering problem-solving 4 , it also captures sensitivity to relevant human stakeholders and broader socio-cultural and environmental concerns. The latter considerations have been shown to be essential for successful SCD.
The results of administrating the ECP to students participating in the GEDS suggest that our instrument is able to capture changes in students thinking (a form of pre/post-test validity). That is, participating students both identified the mathematical and physical aspects of the problem and showed awareness for the people involved in the project, as well as the greater sociocultural and economic systems surrounding them. In fact, there was a large increase in the number of responses belonging to the stakeholder and broader consideration categories, with a peak for stakeholders. Yet the distribution of responses was also more equally distributed across the four categories after the intervention than it was before. This result is noteworthy, as the intention of the GEDS was not to disparage considerations categorized as T or C, but instead to enable students to foster a broader and more differentiated way of thinking about the design "problem definition space" (p. 282), 24 while also cultivating more human-centered design skills.
Great care has been taken to validate the design task and corresponding rubric. The categories were inductively generated through consensus of several researchers and multiple iterations of categorizing a large number of response items (n = 449). Additionally, we have aligned the categories with extant literature to conceptually validate the categories. Finally, student reflections about their pre-/post-event responses are generally well-aligned with our rating system.
We invite engineering educators, especially those involved in SCD projects, to employ this design task and rubric as a way to assess how students identify and prioritize considerations while solving design problems in less developed country contexts. Assessing how students change their thinking with regard to design problems has important implications for how students will enact skills related to their design. For example, if students embody a stark prioritization of device-related considerations, the epistemology (i.e., what they know) and teleology (i.e., intended results) of their designs might disregard considerations of the people who are connected to the design. In contrast, if they embody a more expanded view of design considerations that are anchored in human and other contextual considerations, they will likely seek knowledge and make decisions that consider the stakeholders associated with a design, as well as the socio-cultural and environmental systems that encompass the technical system. 26 Thus, we recognize the salience of eliciting the types of considerations that students mentally represent when thinking about design tasks. The instrument employed in this study provides a way to elicit and categorize such considerations. This type of assessment helps make visible to design instructors how educational interventions might alter the mental representation of engineering design problems. If these design considerations were made visible at the beginning Steve -2 -2 0 I would add more of a stronger involvement from the community and project partner. Making sure to involve them, entirety of the design making process. of a design course, instructors may be able to adapt their curriculum to inculcate a broader and more differentiated view of engineering design problems, ultimately cultivating students who are better prepared to tackle real-world problems.

AREAS OF FUTURE RESEARCH
While this paper has reported the systematic inductive and deductive development of an instrument to elicit student's design considerations in SCD projects, it also presents many opportunities for further work. Future research may include administering the tool to a larger sample of students, while also collecting more information about their background to investigate what factors might influence how they approach the design task. These factors could include students' demographics, individual differences, cross-cultural abilities, degrees of community engagement, and/or previous courses taken. And while we have taken care to ground the design task and corresponding rubric in data and extant literature, recognized experts in SCD could also be enrolled as participants to further validate the design task and corresponding rubric. Additionally, we conceive of the ECP rubric as a flexible instrument that could be adapted to other specific research needs. For instance, if researchers wanted to score the ability of students to express non-technical thinking, they could assign 0 points to each consideration belonging to technical and/or constraints, 1 point to stakeholder responses, and 2 points to broader considerations. Alternatively, if researchers were interested in evaluating the ability of students to cover all categories, they could assign equal weight to each category. Thus, we invite researchers to use our rubric as a starting model, but also to expand and refine it based on other students' responses. Additionally, we acknowledge that our "broader considerations" category might be further differentiated by sub-categories such as culture, environment, ethics, etc. Finally, while we have demonstrated the effectiveness of this design task in assessing outcomes of a brief, half-day intervention, future research could assess the effects of more prolonged activities related to SCD training, coursework, and/or projects (e.g., international programs, service-learning courses, etc.).