A pattern-based approach for developing creativity applications supported by surface computing
Introduction
The work of creativity has to do with the generation of new ideas or new associations between ideas. Many techniques have been devised to facilitate the participation of individuals and groups in the work of creativity (de Bono, 1999, Osborn, 1957, Rohrbach, 1969, Whiting, 1958). We focus on the group dynamics that occur when participants share their ideas and select the one that is most appropriate to solve a problem, as in the case of the brainstorming method. Most of these techniques are carried out with the support of different tools and artifacts (such as pens, markers, paper, dice, or timers). More recently, some creativity techniques for idea generation have been supported by multi-touch surfaces (MTS).
MTS are interfaces between humans and computers that provide many technological advantages. They are capable of identifying two or more points of contact between the user and the screen in different positions, offer direct-touch input and bimanual interaction in a natural way, enable simultaneous interaction between multiple users and have a higher level of input flexibility than a mouse. Thus, given their characteristics, MTS show great potential to support the work of creativity (Ardito et al., 2015).
Generally speaking, applications that are based on MTS are appealing for a wide range of users, mainly because of the ease and intuitiveness of the interaction style they promote. Users have found it natural to select or move objects, or to zoom in or out in order to obtain more detail or a better perspective, by just tapping, pressing or sliding their fingers on the display. But so far, MTS have been used mostly for navigation and visualization purposes. Authoring on MTS still is generally limited to entering text using virtual keyboards, or sketching through relatively simple palettes.
The success of software applications for MTS depends on how well they support human activities. Collaborative work of creativity involves generating a large number of idea representations in the form of verbal utterances, handwritten text, sketches, idea associations, diagrams and more elaborated narratives. If MTS are intended to effectively support groups involved in the work of creativity, they must enable the diverse set of interactions and behaviors that take place in collaborative settings. Platforms for supporting a wide range of methodologies for fostering creativity are needed. Unfortunately, developing new MTS-based applications still requires significant effort in using low-level primitives that facilitate interaction with tactile surfaces and building complex components on top of them in order to implement natural user interfaces needed by creative workers.
The aim of this paper is not only to motivate the use of MTS, but also to offer a new model for supporting the development of MTS-based creativity applications. To this end, it is necessary to thoroughly analyze and understand how mechanisms that foster creativity are performed. Among the existing approaches for supporting work analysis and design, we can mention Requirements Engineering (RE) (Nuseibeh and Easterbrook, 2000) in software engineering, as well as User-Centered Design (UCD) and Activity-Centered Design (ACD) (Williams, 2009) in human-computer interaction.
While RE and UCD focus on the user (the process of designing the system that users want and considering goals and tasks in so doing), ACD focuses on the activity (activities, actions and operations performed by users to achieve a specific goal). RE asks for the purpose, by identifying stakeholders and their needs (Nuseibeh and Easterbrook, 2000). UCD considers who the users are and their level of knowledge, their context of use, their reasons for use, their performance patterns and their preferences. ACD ``emphasizes the design of computer-mediated environments to support and structure the interactions and interdependencies of an activity system'' (Gifford and Enyedy, 1999).
Since our goal is to support the activities that promote creativity, with special emphasis on the process of idea generation, we decided to rely on ACD. We used ACD to find patterns in interactions between people who generate new ideas with MTS. We perform direct observation and video analysis of creative sessions using MTS applications. We paid special attention to the main activities and sub activities carried out during these creative sessions. Then we focused on specific interactions of participants with the applications. In this way we obtained the main activities, actions and operations of users during creativity sessions using these interfaces.
The basic reason for focusing on activities is to determine which tasks or activities must be enabled by a multi-touch application built for creativity and to determine how a multi-touch device would be operated. We need to help application developers in their goal to develop software applications designed for creativity using multi-touch interfaces. To achieve this goal, we need to learn about the activities, actions and operations that end users perform in the processes involved in creativity.
Activity-Centered Design is an approach that divides activities into various levels of sub-activities. The objective of ACD is to understand users as participants in activities. ACD is a human-centered approach for large and heterogeneous populations (Norman, 2013). ACD has its theoretical basis in Activity Theory (AT), which defines activities and actions taken by users to achieve a specific goal. AT is a broad framework that provides a cultural, social, developmental, and tool-centric perspective on people engaged in activities (Leontiev, 1978).
We posit that if ACD is applied properly in surface computing, through an analysis of the structure of activities and requirements, a helpful classification of user interactions can be obtained for the support of the development of interfaces for innovation and synchronous collocated collaboration. Such interactions should exhibit patterns, which in our context refer to actions or operations that are performed in a regular and repeated way when interacting with MTS to achieve the specific goal of a given task. Patterns can thus be regarded as components of a model that describes interactions in detail – close to reality – including the essential elements of the process of creativity that are the basis for the optimization of the mechanism that support creativity at its most elementary level – where creativity is carried out – at the user level.
This paper presents an analysis (based on ACD) of how software developers apply various techniques to create applications for creativity. This paper also presents a model for the process of creativity that relies on MTS, derived from this analysis. The paper is structured as follows: Section 2 offers a brief overview of models that support the process of creativity, followed by a more detailed review of works related to the support of creativity by MTS. We analyze salient activity models and widget toolkits for developing applications that support creativity using MTS in Section 3. Then, Section 4 presents the ISCALI model. Section 5 presents the architecture that we are proposing for applications that use MTS. Section 6 shows our toolkit for applications development. The paper closes by presenting conclusions and future work.
Section snippets
Related work
The work of creativity is the manifestation of creative effort as in the formation of concepts, artwork, literature, music, painting and design (Osborn, 1957, Sawyer, 2006). One of the basic strategies used by the mind to solve problems is the formation of concepts. The work of creativity can be supported by multiple methods, ranging from highly structured methods such as TRIZ (Altshuller, 1999) to less structured methods such as ``analysis'', ``association'', ``synthesis'', ``lateral
Support for the process of creativity using MTS: the need for patterns
The assumption that human activities have discoverable requirements (the process of identifying, analyzing, modeling, verifying and managing activities) has motivated considerable work reported in the literature (Williams, 2009). Examples include conceptual tools such as Requirements Engineering (Nuseibeh and Easterbrook, 2000), Participatory Design (Spinuzzi, 2005), Contextual Design (Holtzblatt and Beyer, 2014), Activity Centered Design (Norman, 2005) and Collaboration Engineering (Briggs
ISCALI: innovation solutions centered on activities for large-sized interfaces
Based on our analysis presented in Section 2 we propose ISCALI, a model for developing applications to support the process of creativity using MTS. ISCALI can be used for implementing various creativity methodologies. ISCALI comprises three main activities: idea generation, organization and evaluation, and their subsequent actions. Each activity consists of a set of actions performed by groups that stimulate creativity as shown in Fig. 2.
An effective interaction design using MTS responds to
An architecture for creativity-building applications using MTS
In order to demonstrate the model's applicability, as well as how its elements could be incorporated into a real system, we designed an open architecture to be extended with new applications and new tools that use MTS based on interaction patterns. Our architecture consists of three layers: physical access, services and presentation. Fig. 12 shows a simplified, high-level representation of these layers and their relationships with external services that are consumed by the applications.
The
A toolkit for application development
Based on our architecture (see Fig. 12), we used TOKAs in the development of software applications designed for creativity. The development of TOKAs consisted of all three activities proposed in the model. This was illustrated in Fig. 2. Regarding actions, we selected only some of those proposed in ISCALI, in this case representing, filtering, grouping and evaluating ideas. From the set of operations to achieve each action proposed in the model, we have selected, developed and implemented a
Validating the model through the use of TOKAs and the development of applications designed for creativity
For demonstrative purposes, four applications were developed for the methods known as Six Thinking Hats, Lotus Blossom, SCAMPER, and Circle of opportunity as instances which validate ISCALI's ability to represent different mechanisms for promoting creativity. Of these four applications, we developed the ones for the Six Thinking Hats and Lotus Blossom methodologies using the model elements and our building blocks. We invited some software developers to develop software applications using the
Future work
The ISCALI platform can be regarded as a testbed for investigating the role of multi-touch surfaces in the work of creativity. In addition to addressing the limitations discussed in the previous section, work is needed to take advantage of the availability of the TOKAs building blocks to develop applications that implement a wider variety of methodologies to support creativity. These implementations can be the basis for formally and thoroughly investigating the role multi-touch user interfaces
Conclusions
We have presented ISCALI, a model that describes the main activities, actions and operations that are carried out by groups of innovators who use multi-touch surfaces as a part of the creativity process. The model has been generated through the application of Activity Centered Design, and comprises the three major activities that are carried out in various techniques to support creativity: generation, organization and evaluation of ideas. ISCALI is capable of expressing the main activities
Acknowledgments
This work was partially supported by the Consejo Nacional de Ciencia y Tecnología (CONACYT) [grant number 290649].
Yazmín Magallanes is a staff member of the Laboratory of Interactive and Cooperative Technologies at Universidad de las Américas Puebla (UDLAP), México. She conducts research in the areas of human-computer interaction, multi-touch surfaces, innovation, and techniques to foster creativity. Yazmín has participated in several research projects and has published her results in many scientific conferences. Currently she is working towards a PhD in Computer Science at UDLAP, where she also obtained a
References (46)
- et al.
A Pattern Language: Towns, Buildings, Construction
(1977) The Innovation Algorithm: TRIZ, Systematic Innovation and Technical Creativity
(1999)- Ardito, C., Buono, P., Costabile, M.F., Desolda, G., 2015. Interaction with large displays: a survey. ACM Comput. Surv....
- et al.
Sparse tangibles: collaborative exploration of gene networks using active tangibles and interactive tabletops
- et al.
Collaboration engineering with thinklets to pursue sustained success with group support systems
J. Manage. Inf. Syst.
(2003) - et al.
Computer-supported creativity: evaluation of a tabletop mind-map application
- et al.
Affinity+: Semi-Structured Brainstorming On Large Displays (No. PNNL-SA-93014)
(2013) - et al.
The effect of aesthetic on the usability of data visualization
- et al.
SynFlo: an interactive installation introducing synthetic biology concepts
- et al.
ScriptStorm: scripting to enhance tabletop brainstorming
Pers. Ubiquitous Comput.
(2014)
Six Thinking Hats
Lateral thinking: a Textbook of Creativity
Collaboration engineering: foundations and opportunities: editorial to the special issue on the journal of the association of information systems
J. Assoc. Inf. Syst.
A tabletop application environment for generic creativity techniques
Int. J. Comput. Inf. Syst. Ind. Manag. Appl. IJCISIM
Designing reality-based interfaces for creative group work
Activity centered design: towards a theoretical framework for CSCL
The NiCE discussion room: integrating paper and digital media to support co-located group meetings
Designing for collaborative creative problem solving
Pen + Touch = New Tools
Contextual Design
Speak up your mind: using speech to capture innovative ideas on interactive surfaces
MemTable: an integrated system for capture and recall of shared histories in group workspaces
Cited by (3)
Evaluating an information system to provide creative guidance about health-and-safety in manufacturing
2021, Behaviour and Information TechnologyUser-Centered Design Approaches to Integrating Intellectual Property Information into Early Design Processes with a Design Patent Retrieval Application
2020, International Journal of Human-Computer InteractionEffect of Crowd Voting on Participation in Crowdsourcing Contests
2020, Journal of Management Information Systems
Yazmín Magallanes is a staff member of the Laboratory of Interactive and Cooperative Technologies at Universidad de las Américas Puebla (UDLAP), México. She conducts research in the areas of human-computer interaction, multi-touch surfaces, innovation, and techniques to foster creativity. Yazmín has participated in several research projects and has published her results in many scientific conferences. Currently she is working towards a PhD in Computer Science at UDLAP, where she also obtained a master's degree in the same area. Before coming to UDLAP, Yazmín studied Computer Systems Engineering at Instituto Tecnológico Superior Zacatecas Sur.
Alfredo Sánchez is a professor of computer science and director of the Laboratory of Interactive and Cooperative Technologies at Universidad de las Américas Puebla (UDLAP), Mexico. He holds MSc and PhD degrees in Computer Science from Texas A&M University. His research interests lie in the areas of Human-Computer Interaction, Natural User Interfaces and Information Visualization. He has been a visiting professor at the University of Waikato, New Zealand, a visiting scientist at the Center for Botanical Informatics of the Missouri Botanical Garden, and a visiting scholar at Shanghai University's Institute of Smart City.
Ofelia Cervantes is a professor of computer science and director of the Innovation for Development (INNOVA4D) research group at Universidad de las Américas Puebla (UDLAP), Mexico. She holds an MSc degree in Computer Systems from École Supérieure en Informatique in Grenoble, France, and a PhD degree from Institut National Polytechnique de Grenoble, France. Her research interests lie in the areas of Data Science, Natural Language Processing and Human-Computer Interaction. She has been a visiting professor at the Université Joseph Fourier in Grenoble France and the Université du Québec à Trois Rivières, Québec, Canada.
Wanggen Wan received his PhD degree from Xidian University, China. He has been a visiting scholar at the Minsk Radio Engineering Institute, former USSR, and a postdoctoral research fellow at Xi'an Jiao-Tong University, China. Since 2004, he has been with the School of Communication and Information Engineering, Shanghai University, where he is currently a Full Professor, Dean of International Affairs, Director of Institute of Smart City, and Program Leader of Circuits and Systems. His research interests include computer graphics, signal processing and data mining. He is a coauthor of approximately 200 academic papers.