Using linkography to compare creative methods for group ideation

This paper compares a new creative design method, based on the principles of drama improvisation, with brainstorming. The evolution and development of the Design Improv method is reviewed, and the results of an evaluation of both methods in controlled team ideation sessions are documented. The creative characteristics relevant to their quantitative and visual interpretation are analysed using linkography, identifying the most prominent variations in performance with respect to inter-connectedness, parallel thinking and idea diversity. We describe an adapted and expanded process for robust linkograph development and reﬂect on the value, challenges and limitations of both the linkograph creation and the perceptive insights they can provide.

D esign ideation research is continually evolving. A number of studies have fuelled debate over the influences of different factors and interventions such as individual or group work; working environment; creative stimuli; and different processes for generating and recording ideas, for example (Faure, 2004;Howard, Culley, & Dekoninck, 2011;Knight & Baer, 2014;McMahon, Ruggeri, K€ ammer, & Katsikopoulos, 2016;Vidal, Mulet, & G omez-Senent, 2004). Studies often involve a comparison of two or more ideation methods, with measurements of their creative outputs as the key discussion points. Some use a simple quantity and/or originality measurement as an evaluation while others use more detailed creativity metrics such as those outlined by Shah, Smith, and Vargas-Hernandez (2003).
However, measuring creative output does not provide the full story. For example, some have argued that while studies show group ideation to be less productive than individual, there are additional benefits that should not be dismissed such as improved organisational memory, building and pooling of designer knowledge and skills; and creating an attitude of support (Kohn, Paulus, & Choi, 2011;McMahon et al., 2016;Sutton & Hargadon, 1996). Furthermore, the outputs of ideation approaches intended for use in the early phase of the design process may not contain the level of embodiment and detail required to adequately utilise more in-depth creativity metrics. These require quantitative inputs and weightings of multiple functional attributes, which is simply not possible for the brief and discrete (yet potentially significant) ideas typically expressed during an early phase group ideation.
This phase therefore requires other ways to determine whether a particular ideation method is fostering creativity. By looking at the ideation process itself, we can gain insights into the style and quality of creative thinking, and in the case of group ideation, the style and quality of collaboration.
'Linkography' has become an established method for visualising and analysing design processes (Roozenburg, 2016). Originally developed by Gabriela Goldschmidt to evaluate the creative processes of individual designers (Goldschmidt, 1995). Further developments such as those by van der Lugt (2000) have enabled linkography to be effectively applied to the evaluation of group ideation in a product design and design engineering context through links that emerge as ideas are developed. While several accounts of this method exist in the literature, published accounts do not usually share the level of detail that would enable others to replicate the full link coding process efficiently and effectively.
Using data from a recent study on the application of improvisation techniques, developed for drama and comedy performance, to design ideation (Hatcher et al., 2018) this paper demonstrates the use of linkography to trace the development of ideas in a group ideation process, while also revealing the analytical power of this method when used to compare two alternative ideation methods. The new Design Improv method draws on the referents (rules) of improvised comedy aiming for collaborative divergence to overcome barriers that often prevent the full potential of group idea generation being realised. We present a detailed account and provide guidance on our adapted and expanded process for producing a robust set of linkographs for both methods, and discuss the various new and refined insights they provide when analysed both numerically and visually. These highlight the key differences between the ideation methods. We then reflect on the value, challenges and limitations of both the linkography creation process and its perceptive insights.

Design Improv: its evolution and characteristics
The new 'Design Improv' approach was initially developed through workshops with practitioners, students and professional improvisers (Hatcher et al., 2018) before being tested under controlled conditions. Following a 128 Design Studies Vol 58 No. C September 2018 warm-up and brief discussion of the design challenge, teams build up a 'base reality' for a concept using the 'Yes And' technique e where the premise offered by others is accepted and built upon to create a shared narrative. A single idea that stands out as being most surprising or unusual is then heightened using the 'If Then' technique to embellish it further, allowing that single idea to be pushed to its limits (see Figure 1).

Linkography
Linkography is a protocol analysis method developed by Goldschmidt (1990) that codes and visualises the links between the 'design moves' expressed during an ideation session. A linkograph has several common features, as shown in the annotated example in Figure 2. Firstly, each design move (a) identified in the data is numbered in sequential order along the horizontal axis (1e49 in Figure 2). Goldschmidt (2014) describes a design move as 'a step, an act or an operation in the process that changes the situation'. van der Lugt (2000) goes further by stating that a design move 'requires communication with an indication that the idea was transmitted to and received by the group' and 'must be related to the task at hand and providing some kind of solution'. 'Nodes' (b) denote a link between two design moves assessed through a process of judging whether each move is linked to any other move that precedes it (Goldschmidt, 1995). For instance, the highlighted link node in Figure 2 represents a link between design moves 1 and 10. Goldschmidt's approach to link-coding relies on 'common sense', and clarifies that 'a link between two moves is established when the two moves pertain to the same, or closely related, subject matter(s), such as a particular component of the designed entity, its properties and functions, a concept or a design strategy' (Goldschmidt & Weil, 1998). The density of these nodes alone can provide insights into the inter-connectivity of the ideation process. Too sparse, and the process could be considered sporadic and poorly structured. Conversely, a very dense linkograph may imply design fixation (El-Khouly & Penn, 2014).
During a productive ideation session, the linkograph may show structural patterns that are geometrically discernible. Goldschmidt describes linkograph patterns in terms of 'chunks', 'webs' and 'sawtooths' (Goldschmidt, 2014). 'Chunks' (c) show distinct, triangular clusters of nodes, indicating a high rate of inter-connectivity between a particular range of design moves. They show cycles of thought, or the examination on a sub-problem, and suggest efficient design thinking. 'Webs' (d) are smaller, denser clusters of nodes, indicating a more intense burst of idea-building. Linkographs with chunk and web patterns have been found to be associated with more productive creative processes (Goldschmidt, 1992). 'Sawtooths' (e), on the other hand, are chains of nodes that also show ideas building but in a way that may not widen or deepen the exploration of the design problem.
Using linkography to compare creative methods 129 'Critical moves' (f) are design moves with a high number of backlinks (links to preceding moves) or forelinks (links to subsequent moves). A high number of backlinks means the design move drew upon and/or consolidated lots of previous moves, while a high number of forelinks indicates that the move inspired lots of future design moves, and was therefore instrumental in the progression of the ideation session. A linkograph with a high proportion of backlink critical moves suggests a gathering together of ideas as convergent thinking, whereas forelink critical moves suggest the generative spinning out of divergent thinking (Goldschmidt, 2014).

Adaptations and applications of linkography
As linkography gains recognition and popularity, it is increasingly being used in a variety of research contexts, from cybersecurity (Mitchell, Fisher, Watson, & Jarocki, 2017) to the evaluation of comics and animation (H.-W. Chou, Chen, & Chou, 2014;S.-B. Chou, Chou, & Chen, 2013). It was initially developed for analysing the cognitive creative processes of individual designers, and there are several examples in the literature of this being put into practice, for example (El-Khouly & Penn, 2014;Kan & Gero, 2008;Salman, 2014). Linkography has also been used more extensively in the study of design engineering teams and processes. van der Lugt (2002) used the aforementioned adapted approach to compare two different ideation methods: 'brainsketching' and brainstorming. By analysing the link density and link types, it was observed that while brainstorming resulted in a higher quantity of ideas, brainsketching led to more connections between ideas and more incremental ideas-building. From this it could be concluded that brainsketching had certain advantages over brainstorming, an observation that would not have been made by assessing quantity of ideas alone. Vidal et al. (2004) adopted a similar approach in a comparison of brainstorming through verbal communication, sketching and quick prototyping. However, they condensed the protocol further by only coding the links between 'global ideas'. This of course is only possible if all the ideas within a global idea are expressed sequentially. By analysing linkographs they were able to reveal that although verbal ideation produced the highest quantity of ideas, quick prototyping showed greatest inter-connectivity. Lee (2014) used both link density and entropy to determine the rate of creative collaboration between multi-disciplinary design groups. The study found that teamwork was enhanced by shared terminologies, a diverse range of experience and strong leadership. Furthermore, linkographs have been used to identify frequent shifts of divergent (equated to forelinking of moves) and convergent thinking (equated to backlinking of moves) through the ideation process (Goldschmidt, 2016), suggesting this characteristic should be considered as a measure or indicator in analysis of creative thinking.
To gain additional insights and to meet specific project needs, further adaptations and analysis techniques have been added in recent years to the linkography literature.
Some key examples are summarized in Table 1.

Use in our context
Goldschmidt's original linkography method was adapted by van der Lugt (2000) for the purpose of comparing group ideation methods used by industrial designers. The most significant adaptations included: Using linkography to compare creative methods Investigating the links between ideas only, because according to van der Lugt, in early divergent phase it is difficult to assess the 'relative state of the design situation', and therefore identify other design moves.
Improved guidelines for determining the presence of a link between two ideas, including verbalizations, looking actions and gestures.
A method for coding the type of links present: supplementary, modify and tangential.
This approach has been adopted for analysis of the Design Improv method, with further adaptations made as outlined below.

Method: benchmarking against brainstorming
Brainstorming represents the dominant method paradigm for group ideation. It is the most explicitly used ideation method in industry (Geschka, 1996), the most studied group creativity technique (Williams, Runco, & Berlow, 2016) and is a generic descriptor for group idea generation. The original rules continue to be promoted as the base technique; 1. Aim for quantity, 2. Avoid criticism, 3. Build on ideas, and 4. Wild ideas are welcome (van Boeijen, Daalhuizen, Zijlstra, & van der Schoor, 2014). Studies of the fundamental technique have given rise to the development of descriptive socio cognitive models (Gl aveanu, 2011) which explain that group activities can extend concentration on idea generation beyond the period of time typically spent by an individual. However, this potential benefit is often not evidenced as expected with an increased quantity of ideas. Participants' fear of judgement, unequal contribution, premature rejection of ideas, idea fixation and production  Vidal et al. (2004) Created a more condensed linkograph by grouping discrete ideas into 'global ideas'dgroupings of one or more ideas that refer to the same solution. Kan and Gero (2005) Followed Shannon's information theory to calculate entropy or 'unpredictability' in a linkograph. High rates of entropy indicate a richer ideation process, with greater potential for surprise. This adaptation addresses the issue that a simple link density calculation may favour overly saturated linkographs. Kan and Gero (2008) Cluster analysis: observing density and distance of links along an X and Y axis to identify periods of intense activity using standard deviation. Kan and Gero (2009) Coded design moves using FBS (function, behaviour, structure) ontology, to show the distribution of solution focus throughout the ideation process. Cai, Do, and Zimring (2010) Included a Y axis for nodes to distinguish between lateral (large, tangential) and vertical (elaborative) moves. Pourmohamadi and Gero (2011) Development of LINKODER software for generating visual linkographs from spreadsheet inputs, and calculating entropy.

Cash and
Storga (2015) A complimentary and more detailed approach to understanding ideation processes through the creation of link networks. Jiang and Gero (2017) Developed a 'quick and dirty' approach to linkography that codes links between conversational turns rather than individual design moves.
132 Design Studies Vol 58 No. C September 2018 blocking appear to induce terminal velocity for the method and group work. The enduring nature of brainstorming along with the sense that the true potential of group creativity has yet to be realised, means that it is the natural choice of comparative datum for a new ideation method.
The development of Design Improv was initiated through proposals that improvisation approaches from performance art forms (theatre, comedy, music and dance) could be applied in ideation for heightened group cohesion and creativity. Theories of improv align with brainstorming where participants follow a 'referent' set of rules, but the referent emphasis is on reducing cognitive loading and facilitating shared mental models between participants towards a collaborative divergence (Magerko et al., 2009;Stevens & Leach, 2015).
To undertake benchmarking of the 2 methods, ten experimental workshops were conducted with groups of 3e5 participants, all of whom were either senior undergraduate or postgraduate design students with experience of working in creative teams. Five groups used the Design Improv method developed as part of the research, while the other five used the brainstorming method, following the rules 'aim for quantity, defer judgement, build on ideas; and wild ideas are welcome'. The workshops lasted approximately 1 h, and began with a short warm-up period. This incorporated of activities tailored towards the assigned design method (word-association improv activities or a short informal brainstorm), followed by a consistent introduction to the design challenge by the facilitator. Design Improv groups would spend up to 5 min having an informal discussion about the challenge, as this is highlighted as a key aspect of preparation for effective improvisation. Each group was then required to use their assigned ideation method for 20 min to address a design challengedeffective delivery of packages by drone at the point of customer interaction. The groups were encouraged to consider drop-off mechanisms, ergonomics, security and safety. Apart from the method differences, as many steps as possible were taken to ensure variables across the ten workshops remained consistent.

Linkography production process
This section outlines the process that was followed to produce a robust linkograph suitable for further analysis here and in similar creative design scenarios. Our approach to generating a linkograph from an ideation session was founded upon the guidelines outlined in Goldschmidt (2014) and van der Lugt (2000), but has been adapted and extended to be more consistent and discriminating in the documentation of move criteria and link type. The software used in their construction was LiNKODER (www.linkoder.com).
The experimental workshop was conducted by a researcher who had previously developed the Design Improv method through exploratory and iterative workshops. The researcher also participated in coding the linkographs. According to Goldschmidt, linkographs should be produced by at least three judges, and arbitrated by at least two (Goldschmidt, 2014). Therefore, a second researcher was recruited for the linkography process who was not previously involved in the research; and a third researcher independently produced four linkographs from the same set of design moves following a period training. Figure 3 illustrates the judging process.

Develop guidelines through pilot study
The linkography approach detailed in this section was first completed using the two pilot study workshops. These enabled Researchers 1 and 2 to gain an understanding of the approach and to develop guidelines that would ensure consistency in the subsequent analysis. These guidelines were informed by previous literature on the linkography process and updated based on common discrepancies noted during researcher comparison and discussion. The guidelines informed the transcription of the workshops (i.e. determining what should be considered a 'design move'), the coding of links between design moves and the coding of link types.

Workshop transcription
Before the linkographs could be produced, the audio recordings of the workshops had to be transcribed. As per the guidelines outlined by Goldschmidt (2014), these were not word-for-word transcriptions; rather each line of the protocol represented a design move. Researchers 1 and 2 each transcribed five sessions, which were then checked by the other researcher to improve accuracy and agreed through discussion where necessary. An excerpt of a linkography transcript is shown in Table 2. In this study, design moves included all ideas but also included other contributions that moved (or had the potential to move) the ideation session forward. The decision to include these arose from an interest in whether such 'supplementary' design moves had any influence on the creative process and outcomes. The original linkography transcripts included segments that were coded as one of the following types: Idea (I): a new solution or partial solution relating to the design challenge, e.g. 'a windowsill delivery box'. No judgement was made on the quality of these ideas.
Specification (S): statements concerning design requirements or perceived requirements e.g. 'the propellers shouldn't cut people's hands'.
Question (Q): questions relating to the design challenge that could inspire ideas or steer the ideation in a particular direction e.g. 'how would that work in the rain?' 134 Design Studies Vol 58 No. C September 2018 Knowledge (K): when a participant shared knowledge that could inspire ideas or steer the ideation in a particular direction, for example information about existing solutions, or relevant personal experiences e.g. 'I've seen similar services in supermarkets'.
Comment (C): any other statements that contained new information that had the potential to inspire ideas e.g. 'that idea could also help people with mobility issues'.

Figure 3 Linkography generation process
Using linkography to compare creative methods 135 Instances where the discussion did not move the ideation session forward were not included, for example basic repetition of ideas, statements of general agreement or facilitation-related discussion. Within the transcript, each design move was numbered sequentially and coded according to its type. Segments were also coded with the participant who contributed them, to enable contribution and collaboration to be analysed later in the process.

Code nodes linking design moves
Researchers 1 and 2 both independently coded links for all 10 workshops, using the guidelines that were developed from the pilot study data. These guidelines were based upon those found in the linkography literature, with additions and elaborations based on the experience of coding and consolidating the pilot study linkographs. A link coding excerpt from one of the brainstorming workshops is shown in Table 3.
Links were recorded when one or more of the following criteria were met: Participants relate directly to earlier ideas when verbalising their own ideas (van der Lugt, 2000). E.g. 5 links to 4 as the discussion focuses on scanning barcodes.
There are visible indications such as hand gestures (van der Lugt, 2000). There are functional, behavioural or structural similarities (van der Lugt, 2000). E.g. 6 links to 2 as both discuss landing places at residential locations.
Design moves occur sequentially and are within the same chain of thought, building an image of a single concept or solution. E.g. 4 links to 3 as the conversation is focussed on how the drone will land and deposit the parcel.
The same basic idea is applied in a different context. E.g. 11 links to 4 as one refers to the drone using unique codes, the other refers to the user having a unique code.
Nevertheless, the link coding process is subjective. To help improve consistency across the ten linkographs additional guidelines were produced during the pilot study phase to help determine whether two design moves should be coded as linked or not: Repeated words alone do not necessarily mean there is a link. E.g. 15 and 1 do not link, despite the common word 'notify'. 1 discusses notifications when the drone in on route to a specified delivery location, while 15 notifies the customer when the parcel is ready to be delivered so a destination can be set.
Links between design moves that have a large number of intervening moves can be coded if the related idea has not occurred between these moves. E.g. if idea 70 discussed the use of landing plinths, it will be linked to idea 3, despite a large gap between the re-occurrence of the idea within the protocol. If idea 40 also discussed landing plinths, idea 70 would be linked to 40.
Where elements of an idea reoccurred multiple times within the transcript, segments were only linked back to the first time the idea was presented, unless a new element was introduced in a subsequent move that would produce a new link. E.g. if the use of codes to release parcels re-occurs later in the transcript, the later idea would be linked back to 4 rather than 11. The exception would be if QR scanners specifically were discussed, in this case the link would be to 11.
If there is uncertainty about non-sequential links being connected, put the design moves together without intervening moves. E.g. 9 links to 7 as it addresses the issue raised of what happens when the drone arrives and no one is home. Both 9 and 7 link to each other if idea 8 is removed, therefore a link is present.
Both researchers 1 and 2 followed these guidelines when independently coding links for each workshop, then their consistency was established by calculating the inter-rater agreement for each set of linkographs. A confusion matrix (van der Lugt, 2001) was used to record the ratio of agreement and disagreementdthe number of instances where both researchers agreed there was a link, where only one researcher found a link, and where neither researcher Instead of a post-box you have a landing place at each house I 1 3 Like a landing plinth I 2 4 With a unique barcode that the drone reads from the bottom I 3 2 5 There could be issues e.g. scan not working properly C 4 6 Individual landing spaces for each flat that's outdoors but covered I 2 7 The biggest issue is people being in -deliveries happen during the day C 6 1 8 How does a drone buzz you? Q 7 1 9 The app lets you know when its outside the door I 8 7 1 10 A big communal box that opens then secures the parcel I 6 2 11 With a QR scanner, you get a unique code once parcel has been delivered I 10 4 12 Would be difficult to have a partitioned space for every single flat C 10 6 13 It can track your phone location and deliver the parcel straight to you I 7 14 There's data protection issues -it could track you C 13 15 Notifies you when it's ready to deliver and asks where you want it right now I 14 13 9 found a link. Fleiss' Kappa (Fleiss, Levin, & Paik, 2003) was then calculated. Inter-rater agreement was calculated as 'high' to 'excellent', ranging from 0.7 to 0.82. Where disagreements occurred, the final linkographs were determined through a discussion between the two researchers, where each explained their reasoning and came to a mutual decision.

External judge
As recommended by Goldschmidt (2014), a third researcher joined the process to enhance confidence in the linkographs. Following a two-day training period, the third researcher coded links for four randomly selected workshops (two Design Improv, two brainstorming). Four was considered sufficient, given the limited time available, to establish whether the link coding could be considered consistent and reliable. As before, inter-rated agreement was calculated between Research 3 and Researcher 1 and 2's consolidated linkograph. The agreement for all four was rated as 'fair', between 0.4 and 0.54. A lower rate of agreement was expected as Researcher 3 had spent less time with both the project and the linkography method.

Link type coding
It was of interest to categorise the types of design links to indicate the type of thinking taking place. van der Lugt (2000) suggests three categories where all design moves are 'ideas': supplementary, modification and tangential. van der Lugt states that 'This division into link types is still somewhat ambiguous. As the three link types are segments of the same adaptor-innovator scale, there are bound to be difficulties at the borders of the types .' (p. 74). Through the pilot studies these difficulties were experienced, but clear distinctions were noted between links that might have been classified together as 'modification' and 'supplementary'. 'As the protocols in this study contained design moves other than ideas (a key differentiator between brainstorming and Design Improv), it was decided that additional categories were necessary to appropriately code all the link types. The link type coding used within this research is outlined in Figure 4. At this stage it was only necessary to code links associated with design moves classified as ideas; it was only of interest to understand the type of thinking that was taking place that led to the emergence of new solutions, and furthermore the link type categories did not lend themselves to the coding of the design moves that do not link to ideas.

Results
Following the process detailed above, two linkographs were created for each workshope one containing all design moves and a second containing ideas only. This allowed observation of the influence of comments, questions, specifications and knowledge-sharing, as it was predicted that Design Improv would generate far fewer of these 'supplementary' design moves. However,

138
Design Studies Vol 58 No. C September 2018 ideas-only linkographs enabled a more accurate reflection of ideas-building across each workshop. Table 4 illustrates the measurements that can be drawn from linkographs, and the mean results and standard deviations for the Design Improv and brainstorming workshops, which will be discussed in more detail throughout this section.
Coding the types of links between ideas creates a picture of the prominent thinking styles used by designers to generate new solutions. For example, a high level of tangential thinking signifies a process that is rich in novel combinations (van der Lugt, 2000). This may be desired in the early ideation stages, when novel, innovative solutions are required. In the later stages, when concepts are being developed, a high level of tangential thinking may indicate a lack of focus, while alternative and incremental thinking indicate progressive development towards a solution.
Each link associated with an idea was coded using the parallel, incremental, new, alternative and tangential (PINAT) guidelines outlined in Table 4. The index for each link type (LTI) is calculated by dividing the number of links of the selected type by the total number of design moves.
There was not a significant difference (p > 0.005) in the mean number of ideas produced by each method. Number of ideas is often a quick initial measure of the performance of an ideation session, but it would be misleading to propose that Design Improv, having the higher mean score, does create more ideas; it is not a significant result (p > 0.05), the values are close between methods and there is a higher SD across the 5 improv sessions than for brainstorming  suggesting the higher value is unlikely a pattern. However, a significantly (p < 0.005) higher proportion of the design moves in Design Improv workshops were ideas.
Entropy calculations reflect the link pattern's 'unpredictability', i.e. it's ability to generate surprising outcomes (Kan & Gero, 2005). Forelink and backlink entropy, as well as horizonlink entropy (the lateral space between links) was calculated using the LINKODER software (Pourmohamadi & Gero, 2011). Although there were not great differences between the scores, Design Improv produced higher mean results on all three counts, with forelink entropy significantly so (p < 0.05).
We have identified four critical areas where there were differences: interconnectedness, progressiveness, diversity and productivity. These are examined in more detail with the use of linkography below.

Discussion
We have embodied the linkography method to form a comprehensive comparison between the creative processes of two group ideation methods. In the following sections we will reflect upon the process that was followed to produce the linkographs, and the value (and limitations) of using linkography to analyse the creativity of design methods.

Inter-connectedness: linkography structures
The link density index (LDI) indicates the interconnectedness between ideas, providing greater insight into the quality of Design Improv as a creative process. LDI is calculated by dividing the total number of links by the total number of design moves (Goldschmidt, 2014). We calculated LDI for all design moves and for ideas only. When all design moves were considered, the two methods produced similar link densities, indicating similarly inter-connected creative processes. However, Design Improv produced a significantly (p < 0.005) higher mean LDI when only ideas were considered; non-idea moves are removed. This suggests that Design Improv resulted in a more inter-connected idea-building process. Standard deviation for LDI shows greater variation between brainstorming teams.
Observing the linkographs themselves provides further insight into the ideation process of each method, as illustrated by the two 'all design moves' examples shown in Figure 5 from Design Improv workshop 1b and brainstorming workshop 3b. The Design Improv example shows a number of 'chunks' that were noted to be typical of the method. These shapes indicated areas of divergent thinking where ideas were being built upon. The brainstorming linkograph shows at least 1 less chunk than Design Improv (the first potential Using linkography to compare creative methods brainstorming chunk has a low link node count for its first move), indicating a less structured approach, despite similar link index results.
Self-linking is another means to interpret collaboration: it occurs when a participant builds upon their own design move rather than responding to the ideas of others. Self-links were recorded during the link coding phase, using the participant labelling of each design move to note whether a participant was building on their own idea. It was of most interest to record self-linking between ideas only. The self-link index (SLI) is calculated by dividing the number of self-links by the total number of ideas (van der Lugt, 2000). Some selflinking is always expected in a group ideation session. For example, in a group of four, we may expect a SLI of around 0.25, while a group of five would expect a SLI of around 0.2. Higher self-linking indicates a more individual approach to problem solving, while lower SLIs indicate more collaborative ideation. In our study the mean SLI for Design Improv was significantly (p < 0.005) lower than that of brainstorming.
Observing the linkographs provides a more in-depth understanding of this phenomenon. Self-linking has been highlighted in the Design Improv linkograph excerpt shown in Figure 6. Self-links often show instances where individuals have 'championed' their own ideas, often at the expense of input from less dominant participants. However, it should be noted that the linkographs also show areas where one participant contributed an idea that was then built on collectively by the whole group. In these cases, self-linking was in the context of group development, rather than the fixation of one group member on their own idea. This style of self-linking was most common in the Design Improv linkographs, again indicating a more collaborative approach.

Progressiveness: critical moves
It was found that Design Improv resulted in more tangential (p < 0.05) and incremental (p < 0.005) ideas. This can be related to the number of critical moves which were identified in the linkographs. Critical moves provide insights into the ideation process, with a higher number of critical moves showing a more productive and coherent process. The number and ratio of backlink critical moves to forelink critical moves can also provide insights into a method's levels of divergent and convergent thinking. The criteria for what constitutes a critical move varies depending on the study, but should typically amount to no more than 10e12% of the total number of moves (Goldschmidt, 2014). Based on these guidelines, critical moves in our study were established as having at least four backlinks or forelinks. They were recorded using ideas-only linkographs ensure an accurate reflection of the divergent or convergent ideation process. For example, in the 'all moves' linkographs multiple responses to a question appeared to denote a critical move. Table 4 starkly reveals that brainstorming sessions rarely contained moves with more than 5 forelink connections suggesting that ideas were less often developed by the brainstorming groups. In light of this, it is interesting to reflect on the influence of the explicit brainstorming rule that encourages 'building on the ideas of others'. While this information is useful in highlighting particularly significant issues, or turning points in the conversation, these 'critical moves' do not represent divergent or convergent idea-building. Design Improv on average created significantly more forelink critical moves (p < 0.05), in addition to a greater number of links within those critical moves. Neither method produced significant numbers of backlink critical moves, indicating that both are largely divergent ideation approaches; the ultimate aim of ideation is divergence and the results align with findings of van der Lugt (2000). Goldschmidt (2016) proposes that a creative design process should demonstrate a balance of forelinks and backlinks, the backlinks potentially signifying that 'the designer reserves some attention to evaluative activity and affirm (or question) what has already been done.' (p. 120). There then might be some concern about the absence of backlinks in sessions that aim for creativity. However, at least the sessions in our study are intended to reflect approaches suitable for early design process phases and are facilitated with method referents that aim to focus all attention on new idea generation.
The linkograph provides a visual identification and further insights of critical moves (Figure 7). In the Design Improv example, it is clear that critical moves result in distinct 'chunks', indicative of a more productive ideation process. There is less structure visible within the brainstorming linkograph, as the forelink critical moves do not generally result in obvious clustering of ideas.

Diversity: link types
The results showed that the brainstorming mean resulted in more alternativestyle thinking, although this wasn't significant (p > 0.05). Brainstorming also showed more parallel (p < 0.05) and new (p < 0.0005) idea link types e this is to be expected as the brainstorm sessions produced higher content but fewer design ideas compared to the improv method, indicating that more ideas in the brainstorming sessions came as a result of questions or comments within the discussion. Figure 8 shows 'all design move' linkographs with the link types highlighted. Note that it was only necessary to code links associated with ideas, as we are interested in the thinking styles that lead to new solutions. However it is of interest to code nodes that backlink to other design moves, as this helps create a picture of the style if ideation thinking that questions, comments and so on encourage. Visualising link types in this way may help identify clusters or patterns of thinking styles. Brainstorming sessions were judged to contain more parallel links (p < 0.05) than improv with 3 of the 5 improv sessions having no parallel link types. So it seems that the improv method referent may make it less likely for a participant to transfer an idea into a new context, whereas the more abundant tangential links might suggest that, where participants must agree and develop the ideas and statements of others, themes or contexts do persist. Ideas develop within the themes but it seems less likely that these ideas are transferred to new themes.

144
Design Studies Vol 58 No. C September 2018 5.4 Productivity: design moves Figure 9 shows linkographs for each ideation approach highlighting the difference between linkographs based on all design moves, and for ideas only included. This was useful to help understand the productivity and efficiency of the two methods. The linkograph for Design Improv shows little variation between the two linkographs, as 75 of 79 design moves were ideas. Brainstorming however shows clear differences, with 90 design moves reduced to just 46 when only ideas were included. The same move coding introduced in the transcription section and exemplified in Table 2, is used again in Figure 9 to give an example of the distribution of non-idea moves within a session. On average, brainstorming linkographs reduced in size by 43% when other design moves were removed, compared to 15% for Design Improv linkographs. There is therefore valuable insight to be gained from considering all design moves, as not doing so would result in the loss of half the content of the brainstorming ideation session. By including these design moves, we were able to make observations regarding the influence that specifications, questions and comments have on the solution-finding process. On the other hand, including all design moves may not always paint a true picture of the fluency and efficiency of a method. Looking at the 'all design moves' linkographs in Figure 7, one might assume the two methods produce a similar level of productivity, when this is clearly not the case. Including all design moves may also make ideas-building look more divergent than it really is, for example when critical moves emerge from multiple responses to a question, or a debate surrounding a comment, rather than the development or elaboration of an idea.

Insights for the linkography process
Linkography not only created a visualisation of the structure of each ideation process, it also facilitated quantitative measures of qualitative transcript data that provided insights into process efficiency, inter-connectivity of ideas, group collaboration, and ideation thinking styles. These results could then be explored in more detail by revisiting the linkographs, observing structural shapes and patterns to draw further insights into the nuances of each method. In addition to the findings we have made on the creative processes through the linkogrpahy analysis, there are a range of reflections on its practical application that are of use for those interested in conducting linkography in other research contexts.

Transcription
In our study, all relevant design moves that progressed the ideation were included in the linkography transcript. This included ideas, specifications, questions, comments, and participant knowledge, which is in contrast to previous studies (van der Lugt, 2000;Vidal et al., 2004), where only ideas were considered. While the majority of the content of the Design Improv workshops was ideas, a large proportion of the brainstorming workshops were made up of supplementary discussion surrounding the design problem. Coding the links between these moves showed that much of this discussion helped to progress the group towards exploring new issues and solution spaces. It was therefore relevant to include such content in the linkographs to create a more representative comparison of how new ideas were being formed with each method.
We found it beneficial to create linkographs for both conditions and to note the differences as part of the method productivity measures. There were, however, challenges. The transcribing and coding phases of linkography are by their nature highly subjective, based on what the researcher defines as a design move or link. We found that for brainstorming workshops in particular, it was challenging to determine what content should be kept and what should be removed, as these workshops included more discussion around the design problem scattered throughout. This additional discussion made it harder to determine what comments, questions, etc. either progressed or had the potential to progress the ideation. For a comparison to be made, the most important factor at this stage was consistency across the transcribing of all workshops. Having two researchers agree upon the transcripts improved reliability in this respect.

Link identification
Identification of links is a subjective activity. Utilising multiple judges is said to increase the reliability of the technique (Goldschmidt, 2014), however, we noted that disagreements could occur not only between researchers, but also between decisions made by the same researcher over time, depending on what perspective was adopted when assessing the design move. We found that while it was beneficial to review links one or two times prior to comparison, it was also important to accept the consolidated linkograph as final and not be tempted to continually alter the links during analysis. Working closely together throughout the linkography process created a shared understanding that was reflected in the high-excellent inter-rater agreement. It was also noted that it was more difficult to accurately identify design moves that were not ideas. While still rated as high, agreement for brainstorming workshops was typically lower (around 0.7e0.75), while Design Improv (which mostly consisted of ideas) showed slightly more consistency between researchers with reliability agreement typically around 0.75e0.82. It was found to be easier to determine clear links between design ideas, where elements from one idea could be traced back to previous ideas. When reviewing brainstorming workshops, it proved more challenging to link comments, questions and specifications as core elements were not as easily defined.

Judges
The identification of links between design moves forms the foundation for analysis of the linkography approach. Ideally 3 independent judges would be employed to coding design sessions, but this can be a challenging to arrange within many projects. A third researcher was recruited and trained before creating four sets of link codes, for which the inter-rater agreement was calculated as 'fair' (between 0.4 and 0.54). This lower level of agreement is most likely due to the more limited time for the third researcher to familiarise themselves with the project and the linkography process. Additionally English as a second language may have made it more difficult for the third researcher to link isolated segments of text within the protocol. Being new to the research, the third researcher provided a more neutral perspective in determining links. However, the result also highlights issues with this approach. As discussed in 148 Design Studies Vol 58 No. C September 2018 the previous section, link coding is subjective and context-dependent, and an interpretivist viewpoint suggests that there is no 'correct' linkograph, only reflections of each researcher's understanding of the creative process. Therefore, a consolidation of linkographs from completely independent judges is unlikely to be any more accurate or representative than the linkograph on one single researcher. So while there were clear benefits to having multiple minds develop a shared understanding of the transcript and how it should be coded, the value of 'external' judges in linkography is more limited.

Link type coding
Coding the link types provides another level of insight into the ideation process, specifically how new ideas were being formed. We adapted van der Lugt's (2000) coding guidelines to accommodate links between ideas and other design moves, as well as repeated ideas that had been applied to a new context. This enabled assessment of the importance of discussion in an ideation session. The Design Improv method was very ideas-focussed whereas the brainstorming workshops allowed more room for questions, comments and knowledge-sharing throughout. In the brainstorming workshops, 'N' links (new ideas in response to a question or comment) were the second most common after 'I' (incremental), suggesting these additional design moves had a significant influence on the direction of the ideation.

Conclusion
In this paper we have shown in detail how 2 ideation methods may meaningfully differ beyond measures of the quantity of ideas produced. Brainstorming is the familiar datum against which the new Design Improv method is compared. Design Improv had a higher link density and less self-linking which suggesting an interconnected and collaborative ideation process. Design Improv also produced considerably higher numbers of critical moves interpreted as showing higher levels of divergence and, together with more incremental and tangential idea link types, we propose indicative of a progressiveness. Brainstorming displayed a higher number of design moves, but more of Design Improv's moves were ideas. We related this difference to productivity and efficiency of the idea methods. With brainstorming having more design moves, that were not ideas, participants were more likely to generate what were coded as 'new idea links'; idea moves inspired by non-idea moves (questions, statements etc). Brainstorming linkographs also contained more parallel links which was considered to indicate that participants were more likely to restate ideas in new contexts. In contrast Design Improv had considerably more tangential links types which was indicative of participants maintaining themes and contexts, and developing ideas within these.
We have also illustrated how linkography can be used to compare the creative processes of two design ideation methods, including an adapted and expanded Using linkography to compare creative methods process for robust linkograph development. We have provided a detailed account of the steps followed to produce linkographs ready for analysis, and reflected on both the value and challenges of developing guidelines, producing transcripts, coding links and link types and involving external judges in the process. We have also reflected on the interpretation of linkographs, contrasting the insights provided by including all design moves and ideas only, and by analysing the visual structure of linkographs and quantitative measurements drawn from them.
We did not utilise the full suite of measurements and notation styles discussed; we selected those which provided a comprehensive comparison of the two design methods within the resources and factors of interest of our study.
One of the benefits of linkography is its versatility, although there is a need for better guidance on which methods and measurements to select and how to carry these out most effectively. This paper is intended to contribute toward the further development of improved linkography guidelines.