Digitizing creativity evaluation in design education: a systematic literature review

Abstract

Evaluation in Design education is subjective and generally depends upon the pedagogues’ personal perspective. Conducting subjective evaluation on a large scale is associated with multiple challenges; therefore, digitized evaluation is integral to maintain consistency in the evaluation process. This systematic literature review utilized SCOPUS, Web of Science, JSTOR, ScienceDirect (Elsevier), EBSCOhost, and Google Scholar data repositories to retrieve and analyse available literature on digitizing creativity evaluation in Design education. This review intends to provide the researcher community with multiple aspects of digitized creativity evaluation from 2008 to 2021 in Design education. This paper highlights digitized creativity evaluation in the context of Design education, factors of digitized creativity evaluation, research purposes, methods, results, findings, and limitations of this review. Significant findings indicate that most literature studies suggested factors of creativity evaluation, but hardly any studies have highlighted indicators associated with digitizing creativity evaluation. In addition, many articles focused on generalised digitization approaches; however, only a few studies highlighted integrating digitization with creativity evaluation. Moreover, few studies enlightened the difference in factors and techniques associated with evaluation in different Design educational settings, such as classrooms, design studios, mass examinations, etc. Future research may investigate factors and problem-solving techniques of digitized creativity evaluation in Design education from the aspect of multiple educational settings and self-adapting intelligent models that might migrate from one setting to another on demand.

Appendices

Appendix A

Factors of evaluating creativity highlighted in literature.

Indicators	Definition	Articles
Novelty/Originality	“Not resembling something formerly known”	(Sarkar & Chakrabarti, 2011); (Cropley et al., 2011); (Hoseinifar et al., 2011); (Cropley, 2009); (al-Rifaie et al., 2012); (Baer, 2011); (Zeng et al., 2011); (Chulvi et al., 2012); (Cropley & Kaufman, 2012); (Csikszentmihalyi & Wolfe, 2014); (Maher & Fisher, 2012); (Tan et al., 2012); (Demirkan & Afacan, 2012); (Jankowska & Karwowski, 2015); (Storme et al., 2014); (Fischer et al., 2016); (Simon Colton et al., 2011); (Siswono, 2010); (Pozzebon et al., 2011); (Ventura & Gates, 2018); (S Colton et al., 2009); (Simon Colton et al., 2015); (Stojcic et al., 2018); (Kim & Horii, 2015); (Glaveanu, 2012); (Sawle et al., 2011); (Surkova, 2012); (Boden, 2009); (Kaufman & Beghetto, 2009); (Pearce, 2010); (Boden, 2013); (Bila-Deroussy et al., 2015); (Gabora, 2017); (Srinivasan & Chakrabarti, 2008); (Hay et al., 2019); (Kaufman et al., 2012); (Lou Maher, 2010); (Alvarado & Wiggins, 2018); (Diedrich et al., 2015); (Brown, 2012); (Taura & Nagai, 2017); (Jagtap, 2019); (Laske & Schröder, 2017); (Runco & Acar, 2012)
Usefulness	“Able to be used for practical purpose or in several ways, or having a beneficial use, or being of practical use”	(Sarkar & Chakrabarti, 2011); (Chulvi et al., 2012); (Pozzebon et al., 2011); (S Colton et al., 2009); (Stojcic et al., 2018); (Glaveanu, 2012); (Sawle et al., 2011); (Kaufman & Beghetto, 2009); (Bila-Deroussy et al., 2015); (Gabora, 2017); (Kaufman et al., 2012); (Diedrich et al., 2015); (Jagtap, 2019); (Runco & Acar, 2012)
Resolution	“The solution is valuable, logical, useful, and understandable”	(Cropley & Kaufman, 2012); (Storme et al., 2014)
Relevance	“The solution accurately reflects conventional knowledge and/or techniques and the solution fits within task constraints”	(Cropley et al., 2011); (Cropley & Kaufman, 2012); (Storme et al., 2014); (Pearce, 2010)
Effectiveness	“The solution is easy to use, safe to use, and reasonably strong	(Cropley et al., 2011); (Cropley & Kaufman, 2012); (Pearce, 2010); (Runco & Acar, 2012);
Elegance	“The solution is neat, well-finished, well-proportionate, nicely formed”	(Cropley et al., 2011); (Cropley & Kaufman, 2012); (Storme et al., 2014); (M. Boden, 2009)
Genesis	“The solution provides a new way to look at existing problems, draws attention to previously unnoticed problems”	(Cropley et al., 2011); (Cropley & Kaufman, 2012); (Storme et al., 2014)
Fluency	“Number of ideas or thoughts generated”	(Hoseinifar et al., 2011); (Baer, 2011); (Zeng et al., 2011); (Tan et al., 2012); (Siswono, 2010); (Runco & Acar, 2012)
Flexibility	“Number of domains or fields or different perspectives from which ideas are generated”	(Hoseinifar et al., 2011); (Cropley, 2009); (Baer, 2011); (Zeng et al., 2011); (Tan et al., 2012); (Siswono, 2010); (Runco & Acar, 2012)
Elaboration	“The degree of stylishness or detailing of a finished product”	(Hoseinifar et al., 2011); (Baer, 2011); (Zeng et al., 2011); (Tan et al., 2012); (Storme et al., 2014); (Bresciani, 2019); (Dippo & Kudrowitz, 2015); (Vasconcelos et al., 2018); (Runco & Acar, 2012)
Appropriate	“The solution fits within task constraints. The solution does what it is supposed to do”	(Zeng et al., 2011); (Cropley & Kaufman, 2012); (Runco & Acar, 2012)
Value	“It is a measure of how the potentially creative artefact compares to other artifacts in its class in utility, performance, or attractiveness”	(Cropley & Kaufman, 2012); (Csikszentmihalyi & Wolfe, 2014); (Maher & Fisher, 2012); (Ventura & Gates, 2018); (Glaveanu, 2012); (Surkova, 2012); (M. Boden, 2009); (Lou Maher, 2010); (Alvarado & Wiggins, 2018); (Brown, 2012); (Runco & Acar, 2012)
Surprise	“The measurement for unexpectedness has to do with the recent past and how we develop expectations for the next new artefact in a class”	(Cropley & Kaufman, 2012); (Maher & Fisher, 2012); (S Colton et al., 2009); (Boden, 2009); (Pearce, 2010); (Lou Maher, 2010); (Brown, 2012)
Synthesis	“The degree to which a product combines unlike elements into a refined, developed, coherent whole”	(Cropley & Kaufman, 2012); (Taura & Nagai, 2017); (Noel et al., 2015)
Vividness	“The degree of clarity in a solution”	(Jankowska & Karwowski, 2015); (Palmiero et al., 2011)
Transformative abilities	“Necessary change leading to creativity”	(Jankowska & Karwowski, 2015); (Boden, 2009)
Variety	“It can be achieved when certain characteristics of a solution are changed”	(Fischer et al., 2016); (Jagtap, 2019)
Quality	“The solution is well thought and meet the standards associated with an embodiment”	(Fischer et al., 2016); (Simon Colton et al., 2015); (Jagtap, 2019); (Laske & Schröder, 2017);(Runco & Acar, 2012); (Paek & Runco, 2017)
Divergent Thinking	“Ability to provide relevant alternatives leading to multiple solutions”	(Kaufman et al., 2012); (Diedrich et al., 2015); (Runco & Acar, 2012)
Quantity	“Number of creative ideas or divergent thoughts”	(Fischer et al., 2016); (Laske & Schröder, 2017); (Runco & Acar, 2012); (Kachelmeier et al., 2008); (Paek & Runco, 2017)

Appendix B

Application areas and parameters associated with digitized creativity evaluation in Design education associated with multiple application areas.

Investigation	Application areas	Parameters mentioned
(Chaudhuri et al., 2020)	Educational settings	Grammatical syntax, misspellings, narration or coherence between sentences, relevance between question and answer, and relative uniqueness of a solution
(Yazici, 2020)	Architectural sites	Modular design system, folding design system, biomimetic design system, form, visual, function, structure, material, fabrication tools, risks, architectural geometry, environmental needs, transportability, iterations, repetition, gradual transformation, precision, component number, component size, component geometry such as circle, rectangle, square, triangle and hexagon
(Chien & Chu, 2018)	Educational settings	Novel, useful, functionality, aesthetics, critical thinking, cooperative learning, disciplines such as science, technology, engineering, arts, and mathematics, physics, aerodynamics, design manufacture, branding, graphics, sponsorship, marketing, leadership, teamwork, media skills, financial strategy, practical, imaginative, and competitive
(Krause et al., 2017)	Educational settings	Quality, context, motivation, knowledge, sensitivity, perceived helpfulness, complexity, rarity, specificity, feedback length, text complexity, justification, actionable, sentiment, and subjective
(Casakin & Georgiev, 2021)	Educational settings	Originality, usability, feasibility, overall value, overall creativity, information content, semantic similarity, abstraction, polysemy, novel, complex, unique, and non-routine
(Britain et al., 2020)	Educational settings	Text fluency, image fluency, word count, image count, element count, originality, and open-ended
(Salvador et al., 2015)	Educational settings	Substitute, combine, adapt, modify, put to other uses, eliminate, and rearrange
(Knorn & Varagnolo, 2020)	Educational settings	Not mentioned
(Pelliccia et al., 2021)	Industrial workspace	Time, cost-effective, usability including ease of use, ease of learning, object-control, overall capability, functionality, final matching grade, and task concentration
(Georgiev & Georgiev, 2018)	Educational settings	Novel, unexpected, surprising, useful, efficient, valuable, convergent thinking, divergent thinking, participant roles, successfulness of ideas, first feedback, first evaluation, semantic measures, psychological attributes such as cognition, knowledge, and noesis,
(Chiarello et al., 2021)	Industrial workspace	Temporal, behavioural
(Gomez-Laich et al., 2019)	Educational settings	Rhetorical features involving academic language, negativity, reasoning, and citation, symbols, visual hierarchy, useful, useable, desirable, forcefulness, first person, confidence, contingency, description, facilitation, future, information, inquiry, interactivity, meta discourse, characters narrative, positivity, public communication, reasoning strategy, and uncertainty
(Afacan & Demirkan, 2011)	Architectural sites	Usefulness, clarity, efficiency, support, and satisfaction
(Xu et al., 2021)	Architectural sites	Functional spaces, decoration, development process, project stakeholder communication, building management, construction load, health, construction record, architecture regulations, technical resources including maintenance, electrical appliances, life protection, piping, eco-friendly, device preparation, defect sensitivity, optimization of resources, representation, picture presenting, art of design, and picture refinement
(Prieto et al., 2015)	Educational settings	Pragmatism, constraints, empowerment, control, management, visibility, awareness, monitoring, flexibility, adaptation, minimalism, teacher-centrism, load sharing, designing for preparation, appropriation, enactment, multi-level integration, synergy, synthesis including tangibility, flexibility, immersive experience with visual, audio, and other multimedia, learning process, enabling complex communication preventing cognitive or visual load, bridging gap between physical and virtual to enhance visual or spatial ability, and management of documents in an effective way
(Bartholomew et al., 2018)	Educational settings	Collaboration, process of thinking, and product creation, and ease of use

Appendix C

Charting the data.

Authors	Year of Publication	Name of Journal/Conference	Title	Publisher	Country of Publication	Discipline
Chaudhuri et al.	2020	Journal: International Journal of Technology and Design Education	A computational model for subjective evaluation of novelty in descriptive aptitude	Springer Nature	Netherlands	Engineering, Social Sciences
Yazici	2020	Journal: International Journal of Technology and Design Education	Rule-based rationalization of form: learning by computational making	Springer Nature	Netherlands	Engineering, Social Sciences
Chien & Chu	2018	Journal: International Journal of Science and Mathematics Education	The Different Learning Outcomes of High School and College Students on a 3D-Printing STEAM Engineering Design Curriculum	Springer Nature	Netherlands	Mathematics, Social Sciences
Krause et al.	2017	Conference: Conference on Human Factors in Computing Systems (CHI)	Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model	ACM	USA	Computer Science
Casakin & Georgiev	2021	Journal: International Journal of Design Creativity and Innovation	International Journal of Design Creativity and Innovation	Taylor & Francis	United Kingdom	Arts and Humanities, Engineering, Neuroscience, Psychology, Social Sciences
Britain et al.	2020	Conference: Conference on Human Factors in Computing Systems (CHI)	Design is (A)live: An Environment Integrating Ideation and Assessment	ACM	USA	Computer Science
Salvador et al.	2015	Conference: IEEE Frontiers in Education Conference (FIE) Proceedings	Evaluation of a Distributed Collaborative Workspace as a Creativity Tool in the Context of Design Education	IEEE	Europe	Computer Science, Social Sciences
Knorn & Varagnolo	2020	Conference: IFAC-PapersOnLine	Automatic control: the natural approach for a quantitative-based personalized education	Elsevier	Austria	Engineering
Pelliccia et al.	2021	Conference: CIRP Conference on Intelligent Computation in Manufacturing Engineering	Applicability of 3D-factory simulation software for computer-aided participatory design industrial workspaces and processes	Elsevier	Netherlands	Engineering
Georgiev & Georgiev	2018	Journal: Knowledge-Based Systems	Enhancing user creativity: Semantic measures for idea generation	Elsevier	Netherlands	Business, Management and Accounting, Computer Science, Decision Sciences
Chiarello et al.	2021	Journal: Computers in Industry	Data science for engineering design: State of the art and future directions	Elsevier	Netherlands	Computer science, Engineering
Pia et al.	2019	Journal: Linguistics and Education	Scaffolding analytical argumentative writing in a design class: A corpus analysis of student writing	Elsevier	United Kingdom	Arts and Humanities, Social Sciences
Afacan & Demirkan	2011	Journal: Knowledge-Based Systems	An ontology-based universal design knowledge support system	Elsevier	Netherlands	Business, Management and Accounting, Computer Science, Decision Sciences
Xu et al.	2021	Journal: Environmental Impact Assessment Review	Application of ecological ideas in indoor environmental art design based on hybrid conformal prediction algorithm framework	Elsevier	United States	Environment Science, Social Sciences
Prieto et al.	2015	Journal: Educational technology & society	Review of Augmented Paper Systems in Education: An Orchestration Perspective	National Taiwan Normal University	Taiwan	Engineering, Social Sciences
Bartholomew et al.	2018	Journal: The Journal of Technology Studies	Examining the Potential of Adaptive Comparative Judgment for Elementary STEM Design Assessment	Epsilon Pi Tau, Inc	United States	Engineering, Social Sciences

Appendix D

Limitations of the selected publications.

Investigation	Limitations
(Chaudhuri et al., 2020)	Small dataset; Proposed model is associated with Design education and is not adaptable to any other domains
(Yazici, 2020)	It is difficult for students to use differentiate scales. It is also difficult to apply rule-base, if there are large number of rules. Finding a realistic material is an issue during the migration from small-scale model to large-scale mock-up
(Chien & Chu, 2018)	Less time together data from non-science courses. Research and material collection for engineering courses were time-consuming
(Krause et al., 2017)	More accurate language models might provide better predictions
(Casakin & Georgiev, 2021)	Small sample size
(Britain et al., 2020)	Not mentioned
(Salvador et al., 2015)	Not mentioned
(Knorn & Varagnolo, 2020)	Not mentioned
(Pelliccia et al., 2021)	Small sample size
(Georgiev & Georgiev, 2018)	Inclusion of more lexicon categories in semantic analysis might optimize temporal resolution decomposing verbalizations into smaller chunks of text
(Chiarello et al., 2021)	Student generated data might produce scientific replicability problem in experiments; Data science systems are context-specific and are not adaptable for other problems
(Gomez-Laich et al., 2019)	Generally, the dictionaries are not exhaustive, therefore many instances may not be caught on dictionary-based studies
(Afacan & Demirkan, 2011)	Low user acceptance score obtained for support/help tool in the system
(Xu et al., 2021)	Not mentioned
(Prieto et al., 2015)	Affordances of self-learning system are less explored; Teachers perception and system compliance are less focused in classroom constraints
(Bartholomew et al., 2018)	Small sample size; In some cases, system and teacher’s evaluation did not match; System lacked stable ranking