Elsevier

Design Studies

Volume 70, September 2020, 100964
Design Studies

Opening the black box: Developing metrics to assess the cognitive processes of prototyping

https://doi.org/10.1016/j.destud.2020.100964Get rights and content

Highlights

  • Novel metrics used assess the effect of prototypes on design knowledge across time.

  • Utility of metrics demonstrated to capture fine-grain individual and team data.

  • Results suggest intangible indicators add to our understanding of design cognition.

While past work has demonstrated the ability of prototypes to boost design project performance, influence stakeholder buy-in, and improve design team communication, prototyping remains a ‘black-box’ process. The specific mechanisms through which prototypes affect designer cognition remains unknown. To date, there has been little work to operationalize theory and enable the fine-grain measurement of cognitive processes during prototyping tasks. In short, there is a lack of appropriate metrics to establish the intangible value of prototyping tasks via the measurement of the knowledge generated. The purpose of this work is to propose metrics that more holistically assess the value of prototypes. The proposed metrics are implemented in a design project to validate their utility in studying prototyping tendencies.

Section snippets

Literature review

Before we propose a set of design metrics aimed at capturing data on the cognitive processes involved in prototyping, it is first necessary to situate the current work within the context of prior theories of design cognition and mental model development. Next, the known relationships between prototyping and design knowledge are discussed. Finally, recent efforts towards quantifying the value of design knowledge are reviewed with focus on their relevance to the development of future metrics.

Synthesis of research gaps and proposed metrics

One challenge researchers face in closing this gap, is the difficulty associated with identifying and measuring knowledge generated through prototyping (B. Camburn et al., 2017). Tiong et al. (Tiong et al., 2019) developed a rubric to categorize the value of design information generated through prototyping; design information is scored on a four-point Likert-type scale, dependent upon the novelty of the design information gained. While this rubric marks an important step by researchers to

Preliminary evaluation of proposed metrics via in-situ design study

In line with Messick's theory of validity (1998), it is necessary to develop the substantive aspect of the proposed construct's validity via empirical evidence that support or undermine the theoretical foundations outlined in the content aspect of the construct domain. Traditionally qualitative data are used to compare findings with the outcomes of metrics and assess the ability of the metrics to successfully and robustly capture the construct of interest. As such, to ensure the proposed

Results

In order to demonstrate the utility of the proposed metrics, we first sought to explore the relationships between the emergence of design knowledge and prototyping tendencies of designers. To do this, Pearson's correlation tests were used to identify possible connections between designers' intent of a prototype and the outcomes of that prototype, measured using the goal and outcome metrics proposed previously. These relationships illustrate how the usage of prototypes evolve over a project. The

Discussion

The purpose of this work was to identify new metrics that could provide a more holistic assessment of the value of a prototype beyond functional ability and cost incurred. A fundamental shift in the way prototyping efforts and prototypes are valued is needed. The development of designer knowledge through prototyping should be seen as an investment over the life of a project and across future projects. The knowledge and skills developed through prototyping experiences, not only affect the

Limitations and future work

This study was performed with engineering students in a structured, academic setting. As such their perceptions and use of prototypes may be different than designers working in industry. We argue that as the purpose of the work was to evaluate the utility of the proposed metrics to capture emergent design knowledge the use of a student sample does not detract from the overall implications of the work for the field. As the process of validating any new survey involves the use of multiple sample

Conclusion

When designers gather, synthesize, communicate, and exchange emergent design knowledge their mental models of the problem and solution space evolve to more accurately represent the individual designer's understanding and perception of the design space (Dorst, 2019; Dorst & Cross, 2001). Prototypes represent a unique design artifact as they bridge internal mental models and external representations within and across individuals (Bucciarelli, 2002). The act of creating a prototype relies on

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (86)

  • J. Menold et al.

    Prototype for X (PFX): A holistic framework for structuring prototyping methods to support engineering design

    Design Studies

    (2017)
  • J. Sauer et al.

    The influence of user expertise and prototype fidelity in usability tests

    Applied Ergonomics

    (2010)
  • S. Wiltschnig et al.

    Collaborative problem-solution co-evolution in creative design

    Design Studies

    (2013)
  • K. Alexiou et al.

    Exploring the neurological basis of design cognition using brain imaging: Some preliminary results

    Design Studies

    (2009)
  • M.M. Andreasen et al.

    Integrated product development

    Sadhana

    (1987)
  • O. Atilola et al.

    The effects of representation on idea generation and design fixation: A study comparing sketches and function trees

    Design Studies

    (2016)
  • A. Berglund et al.

    Why we prototype ! An international comparison of the linkage between embedded knowledge and objective learning why we prototype ! An international comparison of the linkage between embedded knowledge and objective learning

    Engineering Education

    (2013)
  • M. Bogers et al.

    Collaborative prototyping: Cross-fertilization of knowledge in prototype-driven problem solving

    Journal of Product Innovation Management

    (2014)
  • A.I. Böhmer et al.

    Insights of problem-solving activities while designing a product

    International Conference on Engineering, Technology and Innovation (ICE/ITMC)

    (2017)
  • B. Camburn et al.

    A systematic method for design prototyping

    Journal of Mechanical Design

    (2015)
  • B.A. Camburn et al.

    Methods for prototyping strategies in conceptual phases of design: Framework and experimental assessment

    (2013)
  • B. Camburn et al.

    Design prototyping methods: State of the art in strategies, techniques, and guidelines

    Design Science

    (2017)
  • B.T. Christensen et al.

    The role and impact of mental simulation in design

    Applied Cognitive Psychology

    (2009)
  • W.J. Clancey

    Situated cognition: On human knowledge and computer representations. Series: Learning in doing

    (1997)
  • P. Company et al.

    Computer-aided sketching as a tool to promote innovation in the new product development process

    Computers in Industry

    (2009)
  • N. Cross

    Designerly ways of knowing Designerly ways of knowing

    Design Studies

    (1982)
  • N. Cross

    Design cognition: Results from protocol and other empirical studies of design activity

  • N. Cross

    Designerly ways of knowing: Design discipline versus design science. Design issues

    (2002)
  • N. Cross et al.

    Co-evolution of problem and solution space in creative design. Computational Models of Creative Design IV

    (1999)
  • V. Dewangan et al.

    Towards a holistic enterprise innovation performance measurement system

    Technovation

    (2014)
  • K. Dorst et al.

    Creativity in the design process: Co-evolution of problem-solution

    Design Studies

    (2001)
  • S.P. Dow et al.

    The efficacy of prototyping under time constraints

  • J.N. Druckman et al.

    Students as experimental participants: A defense of the “narrow data base

    SSRN Electronic Journal

    (2011)
  • B.U. Dunlap et al.

    Hueristics-based prototyping strategy formation - development and testing of a new prototype planning tool

    (2014)
  • C. Elsen et al.

    Representation in early stage design: An analysis of the influence of sketching and prototyping in design projects

  • S. Embretson et al.

    Improving construct validity with cognitive psychology principles

    Journal of Educational Measurement

    (2001)
  • E. Fricke et al.

    Coping with changes: Causes, findings, and strategies

    Systems Engineering

    (2000)
  • E. Gerber

    Prototyping: Facing uncertainty through small wins

  • J.S. Gero

    Fixation and commitment while designing and its measurement

    Journal of Creative Behavior

    (2011)
  • J. Gero et al.

    A framework for studying design thinking through measuring designers' minds, bodies and brains

    Design Science

    (2020)
  • V. Gervasi et al.

    Managing requirements knowledge

  • C. Gill et al.

    Prototypes as inquiry, visualization and communication

  • G. Goldschmidt

    To see eye to eye: The role of visual representations in building shared mental models in design teams

    CoDesign

    (2007)
  • Cited by (7)

    View all citing articles on Scopus
    View full text