Biosensing and geography: A mixed methods approach

Biosensing measures of physiological (somatic) response offer a potentially powerful tool for capturing people's subconscious reactions to environmental stimuli. Combining biosensing with other techniques allows insights to be generated not only about the intensity of somatic response but also, crucially, the underlying causes of that response. Despite pioneering work in this area, a rigorous, empirically-led analysis of biosensing measures in mixed methods research has hitherto been lacking. We address this lacuna through a case study of urban walking, comparing a field-based study (30 participants) with a virtual exercise undertaken in a lab (25 participants). Combining biosensing with data on environmental stimuli (recorded using video/GPS) and interviews, three analytical modes are examined: biosensing-led; environment-led; and thematic-led. The analysis shows how each dataset can add contextualizing information to significant phenomena observed in the others. We demonstrate, however, that biosensing measures become considerably more difficult to interpret beyond the controlled environment of the lab. The paper concludes that biosensing should be seen as a valuable measure in field studies, but one which requires careful interpretation through other datasets, being of limited usefulness and reliability taken


Introduction
Much research has been undertaken in cultural geography examining questions of the body, although very little of this work has sought to quantify physiological (somatic) response. In recent years biosensing has been seen as offering the potential to explore participants' reactions at an embodied level, beyond the subjectivity of self-reporting (Spinney, 2015). Despite excitement about the potential for using biosensing equipment, geographers have thus far failed to undertake a rigorous, empirically informed analysis of whether and how these devices should be deployed in field research.
Biosensing is an umbrella term for a variety of different somatic measures including electrodermal activity (EDA), heart rate (HR), blood volume pulse (BVP) and electroencephalograms (EEG). Devices recording somatic response have become cheaper and more widely available in recent years and some measures can be used to make inferences about the emotional state of the wearer. Unless deployed within a mixed methods research design, however, the context for these somatic responses is missing; in essence, biosensing can capture the what but not the why.
There are, therefore, two critical issues that we address in this paper: the usability of biosensing data recorded in a field context; and how a mixed methods approach can contextualize the somatic responses captured by biosensing devices. This analysis is undertaken through a case study of urban walking in which we contrast data recorded in a lab using a virtual proxy with materials recorded by participants in the field. The comparison highlights the difficulty of interpreting biosensing data recorded in the kinds of field settings that tend to be of interest to geographers, where physiological response and sensory stimuli are considerably less controlled than in a lab context. We demonstrate how biosensing, environmental and narrative datasets need to be combined to overcome the shortcomings of biosensing technologies. The approach we outline can be used to enhance our understandings of how people interact with and respond to their environments, offering cultural geographers and others a significant new technique for examining questions around embodiment.

Working with biosensing data
Christian Nold's (2009) Biomapping project pioneered a mixed methods approach combining biosensing data with spatial position and qualitative data. Nold built a device to measure galvanic skin response (GSR) and asked participants to walk around their neighbourhood. He combined these GSR measurements with a GPS track to produce maps. The GSR served as a measure of somatic arousal and participants were asked to talk through their maps, explaining the cause of observed peaks in arousal. Short quotes from these interviews were subsequently pinned to the maps to create a kind of qualitative GIS (Cope & Elwood, 2009) which gave narrative contextualization to the biosensing data.
Nold's highly innovative work was not without its limitations. His biosensor only captured GSR; devices available today record a range of measures that can be examined in tandem to give more nuanced insights beyond a simple measure of arousal. Nold's participants were also presented with their biosensing data and asked to explain it, running the risk that they might ascribe false meaning to the data in order to make sense of it themselves.
The Biomapping project had a strong element of artistic practice and was undertaken as a commission for several municipalities. Nold did not, however, undertake a detailed evaluation of the strengths and weaknesses of combining biosensing and qualitative data. The need for such an evaluation has become more pressing in the years since Nold devised his biomapping technique in 2004. Wearable biosensors have now become smaller, more reliable and readily available. Today there is a large range of equipment available with differing degrees of accuracy (and cost), recording a variety of somatic measures (Table 1).
Research in affective science and affective computing has been using biosensing equipment for many years to explore how somatic response measured by these devices might be used to infer the emotional state of the subject (Evans, 2001;Picard, 1997). The circumplex model of affect (see Fig. 1) used within these disciplines (Posner, Russell, & Peterson, 2005) for categorizing different states of arousal offers a useful framework for undertaking this analysis. With careful interpretation, biosensing data of different kinds can be used as a baseline from which to infer emotional response to stimuli. These data cannot, however, be unambiguously correlated with a person's actual emotion nor the context in which it arose (Resch, Summa, Sagl, Zeile, & Exner, 2014) since emotions are profoundly subjective.

Exploring worlds with biosensors
There has been some interesting work using biosensors to examine the reactions of lab-based participants to virtual worlds, particularly through video games (Sykes & Brown, 2003). Within these virtual spaces, game designers purposefully manipulate the somatic responses of players to keep them engaged (Ash, 2010) and thus there is an obvious appeal to researching this engagement using biosensing techniques. McMahan, Parberry, and Parsons (2015), for example, employed the EMOTIV Epoc to measure the EEG of participants playing platforming game 'Super Meat Boy'. Engagement and arousal levels were found to significantly increase during death events compared to regular gameplay.
Using biosensing equipment outside the controlled environs of a lab, however, brings significant challenges. Measuring EEG, for example, works best when there is little or no movement of the head, making data generated in the field significantly more difficult to interpret with precision. Readings of electrodermal activity (EDA e a similar measure to the GSR used by Nold) can be disrupted where a participant engages in physical activity that stimulates sweating. Moreover, urban environments generate complex and sometimes overwhelming multisensory stimulation (Edensor, 2007;Jones, 2012). As a result, it is difficult to isolate what stimulus (or combination of stimuli) is producing a given somatic response.
Regardless, geographers are beginning to explore the possibilities afforded by using biosensors in a mobile context (Spinney, 2015), particularly within subdisciplines concerned with health and urban design. Aspinall, Mavros, Coyne, and Roe (2015), for example, used measures of EEG, to examine a participant's somatic (and by inference, emotional) response to walking around Edinburgh, UK. Using the EMOTIV Epoc and a GPS, they mapped embodied responses experienced in different spaces of the city. Crucially, however, there was no context given for what triggered those somatic responses.
The Urban Emotions project (Resch et al., 2014;Zeile, Resch, Exner, & Sagl, 2015) attempted to develop techniques for extracting georeferenced data about people's emotional state in real time to inform spatial planning processes. The team sought to 'ground truth' the biosensing data they had collected using a smartphone application and volunteered geographic information. There were, however, limitations to the ways in which this project attempted to contextualize the biosensing data. In the smartphone application, for example, self-reporting is limited to one word (for example 'traffic') providing a rather superficial account of the individual's experience.
In a similar vein, Gravenhorst et al. (2012) used Self-Assessment Manikin's surveys to quantify their participants' valence and arousal rather than engaging with qualitative approaches such as interviews, thereby missing the opportunity to collect more indepth reflections from participants. In a study closest to what we report below, Ruskamp (2016) used bodyworn video in combination with EDA, which was used as a measure of stress. Ruskamp's study is pioneering, but limited by being restricted to a fixed route in a homogeneous suburban area and, as he acknowledges, raises more questions than it answers in terms of the number of variables that could influence participants' EDA response.

Methods
This paper is based on two related projects that combined biosensing data with video and qualitative interviewing to examine urban walking. The case studies were based, respectively, in a physical and a virtual environment. Sitting playing a videogame is, clearly, an imperfect proxy for urban walking, but allows participants to control 'their' apparent movements and thus feel in control and interact with a space in a way that is not possible when, for example, simply watching a video of a walk. The purpose of the comparative study was to examine the limitations of the biosensing data when taken into an uncontrolled outdoor environment. The virtual study creates a baseline, with a much simplified sensory environment (visual and auditory) to take account of when analyzing the biosensing data. The virtual study also removes the physical factors (movement and sweating) that can confound biosensing measures. Thus, by undertaking a comparative study the challenges of collecting these measures in a field setting can be evaluated against the more controlled (lab) environment in which they were originally intended to be used. In both cases, participants were set an exploration task during which biosensing data was recorded using an Empatica E4 (Fig. 2). The E4 is a compact, discreet wristband that measures EDA, BVP, HR and skin temperature. We concentrated on using EDA, BVP and skin temperature measures. This combination allows an analysis on both axes of the circumplex model, going beyond Nold's work which looked purely at arousal. Where EDA response indicates arousal, a small rise in skin temperature (flushing) suggests a positive emotional response, whereas a small drop in temperature (cold sweat) can indicate a negative response (Calderon & Thompson, 2004;Harris, 2001). Because of variations in the physiology of individuals and the prevailing environmental conditions, absolute numerical comparison between the data recorded would reveal little. Instead relative variation in arousal was visually interpreted for individual participants, with significant events and broad patterns identified.
The physical environment study examined participants' emotional connection to heritage conservation areas using two sites in Birmingham, UK. Thirty participants completed the task: twenty in Bournville (average age 43, ranging from early twenties to late sixties) and ten in the Jewellery Quarter (average age 34, ranging from early twenties to late fifties). Participants who either lived or worked in the area were asked to undertake an unaccompanied walk around their neighbourhood for approximately 45 min wearing the Empatica E4 wristband, a GPS and a chest mounted GoPro camera. Following the walk, participants watched the video footage while undertaking a qualitative interview about their emotional responses to the different spaces passed through.
The virtual environment study used the recreation of midnineteenth century London presented in the game Assassin's Creed: Syndicate (Ubisoft, 2015). Twenty five participants (age range mid-20s to mid-50s, 20 non-gamers, 12 females) were recruited and instructed to navigate their way around the virtual city in a completed and fully unlocked version of the game. Participants were only shown how to operate the controls for walking, running, and looking around and thus did not have to engage with the more complex parkour and combat elements of the game (although some chose to do so), leaving them free to explore the virtual city in a more touristic fashion. Each participant wore the Empatica E4 for a 20-min period while playing the game, with a camcorder trained at the screen to record gameplay. Participants subsequently undertook an interview whilst watching the gameplay video to discuss their emotional responses to the navigation exercise. Reviewing the footage, it was clear that although a small number of participants took longer than the others to get to grips with the control mechanism, all managed to undertake quite long 'walks' exploring different areas of the virtual city.
Both the virtual and physical studies employed a combination of three datasets: biosensing, used to examine arousal and elements of valence; qualitative interviews, to provide self-reported material on valence and causal triggers; and video (plus GPS on the heritage project), to provide spatial and environmental context. We analyze each of these datasets in turn, examining how the other two can  add context and interpretative power to the third. The biosensingled approach looks for points of fluctuation (i.e. arousal and deactivation) in the biosensing data which is then contextualized by examining the video/GPS and interview data to explore triggers and valence. The environment-led approach starts by examining the spatial and environmental context shown in the video/GPS data, looking for significant events or general trends and examining whether these environmental variations were reflected in the biosensing and interview datasets. The thematic-led approach starts with key themes discussed by participants, identifying and exploring whether and how these align with the video/GPS and biosensing data.

Biosensing-led
The biosensing readings in both the physical and virtual studies were subject to considerable variations between participants. Using EDA as a proxy for arousal, one can identify three types of response: i) very little variation in arousal (virtual study n ¼ 6/25, field study n ¼ 2/30) ii) broad patterns, such as a slow rise or fall over time (virtual study n ¼ 9/25, field study n ¼ 17/30) iii) rapid response generating distinct peaks in the dataset (virtual study n ¼ 10/25, field study n ¼ 11/30).
A small but significant number of recordings within the sample fall into the first category, showing little or no arousal response. Clearly then, biosensing data cannot be relied upon to consistently serve as an objective measure of arousal, given that the success of measuring variations in stimulation appears to be highly dependent on participants' physiology. Indeed, some recordings in the field study showing slow variation can be interpreted as simply reflecting participants' generally increasing perspiration levels as they engaged in physical exercise. Exercise-led sweating is an issue with the usability of EDA as a measure of arousal, particularly where participants are engaged in moving around a field setting. Eight recordings of biosensing data on the physical environment project failed to show anything more than movement-generated sweating with a general climb in EDA levels as the walk progressed.
Notwithstanding this problem, for other participants this second category of slower trend-like responses within the EDA measures could be interpreted as having been driven by emotional response when examined in conjunction with the video and interview data. Participant V25 in the virtual city study, for example, initially had a high level of arousal before a slow, steady decline in EDA response was recorded. In the interview, this participant revealed that in the early part of the virtual walk she experienced significant difficulty and stress in getting to grips with the controls. As the session went on she became more confident in manipulating the walking experience, which would in itself partially explain the lowering of EDA response. More than this, however, examining the video and interview data reveals that she chose a highly conservative strategy for navigating the virtual city e walking very slowly around a park and being exceedingly careful not to bump into any pedestrians. In the interview, about halfway through watching her gameplay she commented that: I think that at the moment I was a bit bored. I didn't know what to do. (interview with Participant V25, 11 November 2016) This boredom e which she reflected on several times in the remainder of the interview e can be understood as another factor underlying the somatic deactivation that we see in her biosensing data as she completed the virtual walk.
Another example of trend-like response can be seen in participant HB104 where there is a period of increased simulation, which peaks around 13 min into the walk, followed by a gradual decline. The skin temperature declines sharply during the stimulation (Fig. 3), which could indicate the cooling effect of sweating during what was an uphill walk. The sudden decline of skin temperature around 45 min, however, does not occur alongside a rise in EDA, thus ruling out bodily exertion (sweating) as a factor. In addition, this decline in skin temperature occurred whilst walking around the suburban streets of Bournville in the height of summer (Fig. 4). It is therefore unlikely that an environmental factor could cause such a significant lowering of skin temperature. The interview data, however, gives context for this decline: When my wife and I moved here from town, it had been a difficult year, we had just moved house, which was very stressful, and we were pregnant with our second child. I was walking around getting to know the area because I needed to get down to Cotteridge. I had never been down that road before at time and I was listening to Mike Oldfields's Tubular Bells on my iPod. There's this bit where it builds up and because it had been a stressful time I started looking at all these houses and suddenly being faced with all these terraces and all these different realms. I just got overloaded and I just had to sit down and take a breath. It made me think back to that today. (interview with Participant HB104, 8 August 2016) The decline in skin temperature represents a physiological response which, following Evans (2001), we can infer was brought on by a powerful and negative emotional reaction. It is clear, however, that the explanation for this physiological response could not be worked out from the GPS/video data alone. This emphasizes the importance of the qualitative interview material in understanding what is being seen in the biosensing data e in this case the space prompting recall of a stressful time in his life.
The third category of responses within the biosensing data where rapid peaks are observed is perhaps the most tantalizing because it seems to offer the potential to identify individual events within people's interactions with different environments. As an example, a significant spike can be seen at one point in the EDA readings of participant HB120. Although this was not explained by the interview data, the video footage shows that the participant passes a member of the public at the time of the spike (Fig. 5). This was, however, a relatively unusual example among the field-based participants and our comparative study indicates that event-type peaks are significantly more intense and easier to read in biosensing data collected within the controlled environmental conditions of a lab. In the virtual city project, there were some good examples of individuals having quite dramatic variations in their EDA record. This was the case with Participant V1 (Fig. 6): Reflecting on the last of these incidents, the participant commented that: I was trying really hard to cross the road, sensibly, without getting run over or whatever. So he [carriage driver] stops, so I went round him, then I stopped for the next one. The large variations in arousal can be easily mapped onto a series of events within the video (see Table 2); here the interview material merely reinforces the fact that the participant was finding it rather challenging to explore a busy, unfamiliar city, crowded   3. The initial decline in skin temperature seen here can be interpreted as an artefact of response to physical exertion. The drop from around 45 min is not accompanied by a rise in EDA, suggesting that exertion is not the driving factor. Interview material links this response to a personal narrative about a stressful event previously having taken place in that location.
with pedestrians and slow moving vehicles.
In some cases, however, it is more difficult to interpret an apparent event-type arousal from the video data alone. Participant V21, for example, sees a sharp upward trend in EDA levels at around the 12 min mark. The gameplay footage shows nothing remarkable e the character is simply running along the long straight road of Embankment heading toward the Houses of Parliament. At this point in the interview, however, the interviewee commented: I was conscious that at this point time was beginning to run out, and you'd be knocking on the door soon. I thought "I want to see these sites before, before I get … to the end of the kind of time period". (Interview with Participant V21, 27 October 2016)  Given that this participant talked extensively about planning, building, and architecture, it was clear that seeing key sites held a great deal of personal importance, hence the self-reported worrying about potentially missing out on seeing them. The slow downward trajectory of EDA levels from about 1 min after the initial peak can then be reinterpreted through the video footage, showing the point where Big Ben appears on the horizon, followed by a few minutes of running around a series of sites in Whitehall which were familiar to the participant from his real life.

Environment-led
The physical environment project recorded the location of participants' walks using GPS, meaning that maps of the biosensing data could be produced indicating relative arousal in different locations. This mode of analysis allowed us to identify a potential effect of blue and green space on arousal levels among some participants. As an example, Participant HB107's walk (Fig. 8) shows a high level of arousal whilst walking downhill along a very narrow, busy road (Oak Tree Lane). Upon entering the green space at the Valley Parkway and walking around a small lake, EDA levels declined markedly: Interviewer: What was your favorite place on the walk today? HB107: Probably the yachting pool actually, it's lovely … all the ducks were out and the geese. And everyone round there, everyone I met, like the blokes with their little boats, there were lots of joggers … it's just a nice place to be. It was really relaxing. (Interview with Participant HB107, 9 August 2016) HB107 was one of three participants (along with HB103 and HB108) who displayed a similar drop in EDA whilst walking around the lake in the Valley Parkway. Because we were not specifically researching the impact of green/blue space on stimulation levels, we do not have enough data to make a claim here to support the literature on the calming effects of green space (Kaplan & Kaplan, 1989). Nonetheless, an environment-led examination of the biosensing data raises some interesting questions about how the relationship between green space and arousal could be examined in future research.
Although the virtual city project does not have GPS data to draw on, one can use the video material to identify locations and environmental stimuli of interest to the researchers. Given that the researchers will likely have asked about these events during the playback-led interviews, the biosensing data theoretically gives an opportunity to see whether those events created a significant somatic response in the participants. This offers the potential to sidestep issues of power and participants attempting to please the researcher by giving the 'right' answer to a question about whether an event was significant to them. In common with McMahan et al. (2015), we often saw peaks in EDA response where death events happened within the virtual environment. Similarly, fights, carriage chases, and parkour activities sometimes (but not always) coincided with arousal. Again, however, given the significant variability of EDA response between individuals we would be cautious about claiming that this is a reliable means of being able to determine whether someone found one event more or less important than another. Conventional qualitative interviews, even with the intendant questions of power relations, thus remain crucial when attempting to determine the relative significance of events.

Thematic-led
This approach begins with a conventional process of coding qualitative interviews, looking for key themes that emerge when reading across the interview dataset. Having identified key themes, the video footage is re-examined to identify any environmental stimuli that prompted discussion on such topics. The timing on the  Was previously carefully looking around but then had to cross a road avoiding a horse and cart. 7:11 Getting stuck trying to walk past some pedestrians. 12:00 A series of events: bumping into someone, then carefully avoiding bumping into a child, running before slowing down to avoid a man and cross a road. 16:00 Crossing a busy road while avoiding several horses and carts. (Fig. 7) video footage can then be used to identify whether or not those stimuli are associated with an arousal response within the biosensing data and give an indication of relative importance to a participant.
In the virtual city project, one of the strongest themes that came out was participants layering their personal experience into their understanding of the videogame space. Two participants (V16 and V24) mentioned their fear of heights, talking about how the virtual experience was triggering their real world physical response when looking over the edge of high buildings: "I could feel that in my tummy. You know, that kind of duuuuuu." (Interview with Participant V24, 18 November 2016). In neither case, however, did the EDA data show any significant arousal response associated with the events being described. Another participant talked about relating a walk next to the Thames to a memory of an early morning walk around that part of London after attending a John Carpenter movie marathon. In the interview he expressed frustration about the virtual walk being interrupted after being attacked by some other characters: … actually, I was enjoying the whole walk thing and I don't want to get involved in a fight, so I started running away. But I kept getting shot. (Interview with Participant V5, 25 August 2016) This irritation was reflected in a sudden EDA peak. Given the rarity of these kinds of event spikes, as discussed above, it is quite unusual to find examples such as this one where themes that emerged strongly in the qualitative coding map neatly onto a response in the biosensing data.
Thematic analysis can, therefore, identify issues that participants identify as being of importance which may otherwise be missed when starting the analysis from the biosensing data. In the heritage project, a key theme that emerged was nostalgia. For example: HB105: Two weekends in June and July, one weekend is the Bournville festival, which is pretty much set in stone since 1955. And then there is CocoMad, which is the festival here, which changes every year with things that are new and different. So yeah, we don't want this place to be the same, we want it to change all the time. It's always doing new things.
Interviewer: Which one is better?
HB105: Oh we are way better! Their's is ridiculous with weird nostalgic stuff. (Interview with Participant HB105, 7 August 2016) These kinds of discussions framing the contested identity of the case study sites simply do not appear to map onto the biosensing data. The qualitative analysis indicates, however, that these issues are accompanied by a verbalized strong emotional response even if that is not visible in the participant's somatic response to landscape at a given moment.

Discussion
Mixed methods approaches have two distinct advantages. First, they give additional confidence where findings from one type of data can also be seen in another, particularly when cutting across quantitative and qualitative methods. Secondly, some techniques can capture things that others cannot thus giving a different perspective on the research topic (Creswell & Clark, 2007). Both advantages are evident here in the way that we deployed the biosensing data alongside environmental and narrative data.
Each of the three datasets raise questions, some of which can be answered by examining the others. A lab setting allows for multisensory factors and physical exertion to be isolated and controlled, which makes it easier to interpret biosensing data, both in terms of simple arousal and also inferred emotional valence. As we have demonstrated here by contrasting the two case studies, however, stepping outside the lab introduces a great deal of 'noise' into the biosensing recordings, through multisensory and physiological stimuli. This can make them challenging to interpret and brings a high failure rate.
Even in the lab, however, without an understanding of what prompted a particular somatic response, biosensing data taken in isolation tells us the what but not the why. An understanding of environmental stimuli (spatial location and activity) can often be used to infer the reason behind a somatic response. In turn, some somatic responses may be very specific to an individual's memories and experience, which needs to be captured through qualitative interviewing.
We suggest that the most productive approach is the one we have adopted here i.e. looking at the physiological (biosensing), environmental (video/GPS) and personal (interviewing) datasets in turn and examining how interesting elements identified in each can be contextualized by the other two. This reinforcing effect does not occur in every case, as things that seem interesting to the researchers in, say, the video footage, may have been of little significance to the participant, thus not appearing in the interview or biosensing data. Nonetheless, taking the different measures in combination gives the opportunity to cross-examine findings from individual datasets, both for contextualizing and deepening understanding.

Conclusion
In an ideal world, one could imagine biosensing technologies as an inexpensive and objective way of gathering people's somatic and (by inference) emotional responses to different environments. In practice, however, state of mind, physical fitness, the prevailing environmental conditions, and a host of other factors influence the generation of biosensing data. Although we have concentrated here on the Empatica E4, these problems are common to the range of biosensing devices and measures. Geographers will be interested in these techniques primarily for their potential to be deployed in the field. Even within the controlled environs of the lab, however, it is difficult to isolate and interpret the multiple factors that can shape an individual's somatic response; with the additional multisensory stimulation and physical exertion of a field setting it becomes exceedingly challenging to do so.
We have argued that, taken alone, biosensing data needs to be treated with caution. Indeed, there is always the danger of using an exciting new technique simply for the sake of novelty (Merriman, 2013). Nonetheless, biosensing offers a potentially powerful tool for understanding individuals' responses to environments, but only when used in combination with other datasets. This paper has demonstrated the potential for integrating biosensing into mixed methods approaches alongside environmental and narrative data. Combining biosensing with other techniques allows insights to be generated not only about the intensity of somatic response but also, crucially, the underlying causes of that response. Biosensing cannot be seen as giving unfettered access to an individual's unexpressed emotional responses. It does, however, serve as a useful quantified measure adding value to mixed methods projects interested in examining the relationship between bodies and environments.
Cultural geographers have been at the forefront of research into embodiment and the approach that we have outlined here offers considerable scope for enhancing this area of scholarship through a critical examination of somatic response to environments. It is important to highlight, however, that the circumplex model we use in this paper offers a somewhat different understanding of affect than that familiar to most cultural geographers (Thien, 2005;Thrift, 2008). These philosophically rather different approaches to embodiment will need to be reconciled in order for the potential of biosensing data within geography to be fully realized.