Opening the black box: Developing metrics to assess the cognitive processes of prototyping
Section snippets
Literature review
Before we propose a set of design metrics aimed at capturing data on the cognitive processes involved in prototyping, it is first necessary to situate the current work within the context of prior theories of design cognition and mental model development. Next, the known relationships between prototyping and design knowledge are discussed. Finally, recent efforts towards quantifying the value of design knowledge are reviewed with focus on their relevance to the development of future metrics.
Synthesis of research gaps and proposed metrics
One challenge researchers face in closing this gap, is the difficulty associated with identifying and measuring knowledge generated through prototyping (B. Camburn et al., 2017). Tiong et al. (Tiong et al., 2019) developed a rubric to categorize the value of design information generated through prototyping; design information is scored on a four-point Likert-type scale, dependent upon the novelty of the design information gained. While this rubric marks an important step by researchers to
Preliminary evaluation of proposed metrics via in-situ design study
In line with Messick's theory of validity (1998), it is necessary to develop the substantive aspect of the proposed construct's validity via empirical evidence that support or undermine the theoretical foundations outlined in the content aspect of the construct domain. Traditionally qualitative data are used to compare findings with the outcomes of metrics and assess the ability of the metrics to successfully and robustly capture the construct of interest. As such, to ensure the proposed
Results
In order to demonstrate the utility of the proposed metrics, we first sought to explore the relationships between the emergence of design knowledge and prototyping tendencies of designers. To do this, Pearson's correlation tests were used to identify possible connections between designers' intent of a prototype and the outcomes of that prototype, measured using the goal and outcome metrics proposed previously. These relationships illustrate how the usage of prototypes evolve over a project. The
Discussion
The purpose of this work was to identify new metrics that could provide a more holistic assessment of the value of a prototype beyond functional ability and cost incurred. A fundamental shift in the way prototyping efforts and prototypes are valued is needed. The development of designer knowledge through prototyping should be seen as an investment over the life of a project and across future projects. The knowledge and skills developed through prototyping experiences, not only affect the
Limitations and future work
This study was performed with engineering students in a structured, academic setting. As such their perceptions and use of prototypes may be different than designers working in industry. We argue that as the purpose of the work was to evaluate the utility of the proposed metrics to capture emergent design knowledge the use of a student sample does not detract from the overall implications of the work for the field. As the process of validating any new survey involves the use of multiple sample
Conclusion
When designers gather, synthesize, communicate, and exchange emergent design knowledge their mental models of the problem and solution space evolve to more accurately represent the individual designer's understanding and perception of the design space (Dorst, 2019; Dorst & Cross, 2001). Prototypes represent a unique design artifact as they bridge internal mental models and external representations within and across individuals (Bucciarelli, 2002). The act of creating a prototype relies on
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (86)
- et al.
Analogical reasoning and mental simulation in design: Two strategies linked to uncertainty resolution
Design Studies
(2009) - et al.
Advancing an understanding of design cognition and design metacognition: Progress and prospects
Design Studies
(2019) Between thought and object in engineering design
Design Studies
(2002)The effect of a structured method on mental model accuracy in complex decision making
Procedia CIRP
(2014)The core of “design thinking” and its application
Design Studies
(2011)Co-evolution and emergence in design
Design Studies
(2019)- et al.
The psychological experience of prototyping
Design Studies
(2012) - et al.
Eliciting unknown unknowns with prototypes: Introducing prototrials and prototrial-driven cultures
Design Studies
(2017) - et al.
The role of prototypes in communication between stakeholders
Design Studies
(2020) - et al.
The benefits of model building in teaching engineering design
Design Studies
(2010)
Prototype for X (PFX): A holistic framework for structuring prototyping methods to support engineering design
Design Studies
The influence of user expertise and prototype fidelity in usability tests
Applied Ergonomics
Collaborative problem-solution co-evolution in creative design
Design Studies
Exploring the neurological basis of design cognition using brain imaging: Some preliminary results
Design Studies
Integrated product development
Sadhana
The effects of representation on idea generation and design fixation: A study comparing sketches and function trees
Design Studies
Why we prototype ! An international comparison of the linkage between embedded knowledge and objective learning why we prototype ! An international comparison of the linkage between embedded knowledge and objective learning
Engineering Education
Collaborative prototyping: Cross-fertilization of knowledge in prototype-driven problem solving
Journal of Product Innovation Management
Insights of problem-solving activities while designing a product
International Conference on Engineering, Technology and Innovation (ICE/ITMC)
A systematic method for design prototyping
Journal of Mechanical Design
Methods for prototyping strategies in conceptual phases of design: Framework and experimental assessment
Design prototyping methods: State of the art in strategies, techniques, and guidelines
Design Science
The role and impact of mental simulation in design
Applied Cognitive Psychology
Situated cognition: On human knowledge and computer representations. Series: Learning in doing
Computer-aided sketching as a tool to promote innovation in the new product development process
Computers in Industry
Designerly ways of knowing Designerly ways of knowing
Design Studies
Design cognition: Results from protocol and other empirical studies of design activity
Designerly ways of knowing: Design discipline versus design science. Design issues
Co-evolution of problem and solution space in creative design. Computational Models of Creative Design IV
Towards a holistic enterprise innovation performance measurement system
Technovation
Creativity in the design process: Co-evolution of problem-solution
Design Studies
The efficacy of prototyping under time constraints
Students as experimental participants: A defense of the “narrow data base
SSRN Electronic Journal
Hueristics-based prototyping strategy formation - development and testing of a new prototype planning tool
Representation in early stage design: An analysis of the influence of sketching and prototyping in design projects
Improving construct validity with cognitive psychology principles
Journal of Educational Measurement
Coping with changes: Causes, findings, and strategies
Systems Engineering
Prototyping: Facing uncertainty through small wins
Fixation and commitment while designing and its measurement
Journal of Creative Behavior
A framework for studying design thinking through measuring designers' minds, bodies and brains
Design Science
Managing requirements knowledge
Prototypes as inquiry, visualization and communication
To see eye to eye: The role of visual representations in building shared mental models in design teams
CoDesign
Cited by (7)
Use it early: The effect of immersion on spatial and design space aspects in team-based mechanical design reviews
2024, Advanced Engineering InformaticsVirtually Hosted Hackathons for Design Research: Lessons Learned from the International Design Engineering Annual (IDEA) Challenge 2021
2022, Proceedings of the Design SocietyAssessing Early Stage Design Sketches and Reflections on Prototyping
2022, Journal of Mechanical DesignStudent Perspectives of the Nature and Purpose of "deep Modeling" as Situated Knowing
2022, International Journal of Engineering EducationFuture-ready skills development through Experiential Learning: perceptions from students working in multidisciplinary teams
2022, CERN IdeaSquare Journal of Experimental InnovationStudy on the impact of cognitive load on performance in engineering design
2021, Proceedings of the Design Society