Cross‐cultural mobile game evaluation shows improvement in environmental learning, but not behavior

Games have a wide range of potential uses, from mediating conservation conflicts to changing behaviors. However, there have been few impact evaluations of serious games published in the environmental field. We conducted the first cross‐cultural evaluation of an environmental mobile game, Save The Purple Frog. In a randomized control trial with both UK and Indian participants, we found very strong evidence that Save The Purple Frog had a positive impact on learning (effect size 0.62, p = 0.0001), but no evidence that it affected behaviors or attitudes. Our study contributes to the evidence base for environmental serious games, demonstrating their potential as an engaging tool for environmental education. We suggest examining the impact of longer gameplay as a future research direction, as well as exploring different gameplay styles.


| INTRODUCTION
Serious games, "entertainment games with nonentertainment goals," have been increasingly used for learning and behavior change in many fields, including public health and education (Bellotti et al., 2013;Marsh, 2011;DeSmet et al., 2014). In the process of educating or promoting behavior change, these games are still designed to appeal to a broad audience like an entertainment game. Other goals for serious games have included mediating conflicts, crowdsourcing human computing power, and collecting data on decision-making (Duthie et al., 2021;Khatib et al., 2011;Redpath et al., 2018).
This interest in serious games extends to the environmental field, but as yet there have been few evaluations published. The evaluations that do exist tend to focus on enjoyment of the game, rather than any societal or environmental impacts (e.g., Alcid et al., 2017;Koutromanos et al., 2018;Zualkernan et al., 2009). For example, researchers created a location-based multiplayer mobile game to encourage recycling and tested its usability and gameplay with 52 teenagers (Centieiro et al., 2007). However, only 15 people were asked about their behavioral intentions to recycle. The limited evidence suggests that environmental games may struggle to change donation behaviors, but have positive impacts on a range of other variables. For example, Wildeverse is an augmented reality mobile game, which focused on great apes . When compared with a BBC documentary series, researchers found both interventions increased knowledge, attitudes and social norms without affecting monetary donations. Similarly, Kakapo Run is an infinite runner mobile game, which focused on Kakapo conservation and the impact of invasive species (Dunn & Veríssimo, 2021). Evaluation showed positive impact on knowledge, attitudes, and behavioral intentions to volunteer and actively manage pet cats, but no change in donation behavior.
Despite this research, there are still substantial gaps in our understanding of environmental serious games. For example, if we are to design more effective serious games in the future, it is important we understand the mechanisms by which a game impacts (or fails to impact) audiences. Research from the commercial game field suggests that flow may be a significant component in determining the impact of a game. Flow describes the state of being fully engaged in an activity, where the relationship between one's skills are optimally challenged by demands of the task (Hamari & Koivisto, 2014). In the gaming context, the concept of flow has been structured into eight elements, which form a general model of player enjoyment called GameFlow (Sweetser et al., 2017). These elements are: • Games should require concentration • Games should be sufficiently challenging • Games must support player skill development • Players should feel a sense of control over their actions in the game • Games should provide the player with clear goals at appropriate times • Players must receive appropriate feedback at appropriate times • Players should experience deep but effortless involvement in the game • Games should support and create opportunities for social interaction Further, there have been no cross-cultural evaluations of environmental serious games published. This is a recognized issue in behavioral science generally, where studies typically focus on a fairly narrow audience, usually Western, Educated, Industrialized, Rich, and Democratic (WEIRD) populations such as students in Europe and North America (Henrich et al., 2010). Research in behavioral science shows that there is substantial variability in experimental results across populations, and insights gleaned from WEIRD subjects are unlikely to generalize (Henrich et al., 2010). Expanding the evidence base to include audiences from the Global South is needed to enable us to more confidently predict the impact of future games in different cultures. Indeed, this is particularly important in the game field as 2020 alone saw a huge growth (92 million) in new players from emerging markets in the Asia-Pacific region and the Middle East and Africa (App Annie, 2020;Warman et al., 2020), but most evaluations of serious games are from Western countries (Boyle et al., 2016).
1.1 | Save the purple frog "Save the Purple Frog" is an example of a serious environmental game, designed by the charity On The Edge Conservation. It is a free-to-play mobile platform game, which aims to educate players about the threats faced by the tropical purple frog (Nasikabatrachus sahyadrensis). We evaluated the impact of playing Save The Purple Frog on the pro-conservation attitudes and behaviors of residents in the UK and India. We had the following hypotheses: Hypothesis H1. Playing "Save the Purple Frog" will improve knowledge of the threats faced by purple frogs compared to a nonenvironmental mobile game.
Hypothesis H2. Playing "Save the Purple Frog" will increase positive attitudes towards purple frogs compared to a nonenvironmental mobile game.
Hypothesis H3. Playing "Save the Purple Frog" will increase willingness to support actions that would help protect the purple frog compared to a nonenvironmental mobile game.

| METHODS
This was a preregistered study with a between-subjects design (link). Participants were randomly assigned to one of two treatment conditions: 1. Playing OTEC's Save the Purple Frog. 2. Playing a nonenvironmental mobile game, a generic version of the classic arcade game Frogger Retro Jumping Frog (game links in SI 1).
We had two study contexts, the UK and India, which differ in key ways, including levels of affluence and education. Most notably however, the central protagonist in the game, the purple frog, is endemic to Southern India, but is not found in the UK. We wanted to compare the effectiveness of the mobile game on participants from these two contexts. This research was approved by the University of Stirling's General University Ethics Panel (reference 2021 3683 3457).
To determine sample size, we used G*Power 3.1.9.7 for power calculations, estimating we would need a total of 187 participants per country to detect effect sizes of 0.1 in linear multiple regressions with alpha of .05 and beta of .95. Therefore, we aimed to recruit 400 participants aged 18-25 across the UK and India. To do this, we used market research companies in both countries and paid UK study participants 35 GBP and Indian participants 39 USD. The market research companies shared the study description among eligible individuals on their member lists, and recruited until we reached our goal sample size. Although there was no formal pilot, there was an opportunity for participants to feedback to the recruiters if they had any issues. In addition, they ensured Indian participants had sufficient English language skills to participate in the study through a verbal interview. After answering an initial questionnaire (SI 2), each participant was given 3 days to spend at least 1 h playing the assigned mobile game. At the end of this time period, they were given a follow-up survey. We verified gameplay through mobile battery screenshots from the participants detailing the amount of time they had spent on the game.
We structured our study according to the Theory-based Framework for Gamification Research (Treiblmaier et al., 2018; Figure 1). This is an association of constructs from models such as various behavioral (intention) theories, Flow Theory, and the Elaboration Likelihood Model, which are relevant to gamification research. From this framework, we identified changes in environmental motivations and attitudes as potential pathways to behavioral change, with flow as additional moderator. We also wanted to look at impacts on learning.
With the exception of knowledge, we used existing validated scales for the questionnaire variables. To measure knowledge, in both surveys participants were asked four factual multiple-choice questions about purple frogs, and we calculated the change in number of correct answers. To assess motivations and attitudes, we used two scales from Milfont's Environmental Attitudes Inventory; namely nonanthropogenic motivations and ecocentric concern (Milfont & Duckitt, 2010). Flow was based on seven elements of Sweetser's GameFlow model, although we omitted social interaction as there is none in these mobile games (Sweetser et al., 2017).
For behaviors, we needed to identify an area that was relevant to both the UK and India, and could conceivably be influenced by environmental content in a mobile game. We decided to use Alisat's Environmental Action Scale, which measures the level of engagement in collective civic actions (Alisat & Riemer, 2015). We did adapt the time period in the questions to coincide with the duration of our study. Instead of asking "In the last six months, how often, if at all, have you engaged in the following environmental activities and actions?," we split the scale into past behaviors and future behavioral intentions. These were measured as "In the last week, how many times have you engaged in the following environmental activities and actions?" and "In the next six months, how often, if at all, do you think you will engage in the following environmental activities and actions?," respectively. We checked the Cronbach's Alpha on these scales after splitting them and the internal consistency for past behavior and future behavior remained high (0.85 and 0.94 respectively). Finally, we also asked participants how much time they typically spent playing mobile games, and collected demographic details such as age and gender.

| Deviations from preregistration
There were three deviations from the preregistration. First, due to budgetary constraints we shortened the required gameplay time from three to 1 h. Second, for the Indian participants we collected data on one additional outcome measurement: the decision to donate to an environmental charity. This was coded as a binary outcome, and did not include UK participants who we could not request donations from. Finally, we also collected data on player experiences and issues with the games. Players were asked to rate the tutorial, mission, facts, music, graphics, and the game as a whole, as well as answer two open-ended questions about any problems they experienced and their overall impression. We used this feedback data to propose hypotheses that might explain the impact of Save The Purple Frog on players.

| Data analysis
We calculated scales by summing and using the mean of the numeric answers given to each question. We used F I G U R E 1 Theory-based framework for gamification research (adapted from Treiblmaier et al., 2018) general linear models to observe the relationship between game play and knowledge, ecocentric concern, conservation motivations, and behaviors (SI 3). We also controlled for age, gender, education, flow, country, rating, and time spent playing mobile games. To examine country differences in the player feedback, we used oneway ANOVAs. All data analyses were conducted in R version 4.0.0. Wherever we found a statisticallyinsignificant effect we used the TOSTER package in R (Lakens, 2017) to provide support for the absence of any meaningful effect, setting equivalence bounds at Cohen's d of 0.5 to demonstrate that the observed effect is statistically equivalent to zero.

| RESULTS
The experiment was conducted between November and December, 2021. In total we collected data from 422 participants, 206 from the UK and 216 from India. Demographic comparisons between the treatment and control groups can be found in SI 4. It is worth noting that F I G U R E 2 Changes in outcome measurements among study participants, attributable to playing Save The Purple Frog. (a) across both countries, and (b) specific to the UK and India. Effect estimates and SE come from GLMs despite randomization, a higher percentage of the treatment group were male (53%) than in the control group (46%). However, as gender was not significant in any of the analyses, this is likely not important in terms of the robustness of the results. Dependent variable histograms can also be seen in SI 5, and unadjusted comparisons of the treatment vs. control groups at baseline and after treatment in SI 6.
Overall, we found a significant improvement in learning about the purple frog from playing the treatment game ( Figure 2). This change was greater in UK participants compared to Indian participants. However, there was no meaningful effect on motivations, ecocentric concern, or behaviors. This included the decision to donate in India, and was not affected by interactions with time spent playing mobile games or flow.
Limiting the analysis to learning, we examined which other variables may be influential (Table 1). We found that those how gave the game a higher rating, had completed a university degree, and had spent more time playing mobile games learnt more. Those aged 18 and 19 learnt slightly more, while gender had no effect.

| Player feedback
Of the 210 participants who played Save the Purple Frog, only 30 reported experiencing problems with the game in response to the qualitative questions. Among the issues raised was the quality of the music, and the difficulty of the levels. Some participants found the time limit for each level too restrictive, while others found the game too repetitive. One major theme pertained to the touch controls for the frog navigation, which had a tendency to lag. There were some country differences in this feedback, with UK participants more likely to criticize the touch controls and difficulty, and India participants more likely to complain about the music and slowness. However, when answering the "overall impressions" question, many more players specifically mentioned enjoying the music and that they found the game entertaining with an appropriate difficulty level. Indeed, the quantitative data in Figure 3 shows that the average rating for all aspects measured was above the midway point. There were country differences in the ratings of the Tutorial (

| DISCUSSION
This is the first cross-cultural evaluation of an environmental serious game. Overall, we found very strong evidence that Save The Purple Frog had a positive impact on learning (effect size 0.62, p = .0001), but no evidence that it affected behaviors or attitudes. These findings were consistent between the two countries, and UK and Indian participants also gave similar ratings to the game as a whole.
The results of our study suggests that serious games are a promising avenue for environmental education, a common theme in the literature Dunn & Veríssimo, 2021). However, the gain in knowledge attributable to playing Save The Purple Frog was greater in UK participants compared to Indian participants, because Indian participants in the control condition also gained some knowledge about purple frogs. As the control condition consisted of playing a frog-themed mobile game and all participants were asked questions about the purple frog in the initial survey, it is possible this prompted participants to seek out information about the purple frog in their own time. This may have happened more among Indian participants because the purple frog is more salient to them, being endemic to India but not the UK. Another possibility is that cross-cultural demographic differences were influential (SI 7). Although the two study populations were comparable in terms of gender and age, UK participants were more highly educated and education level is often associated with environmental knowledge (Strieder Philippsen et al., 2017;Sun et al., 2020). UK participants also rated the quality of the facts within the game more highly.
There were no other significant differences in the impact of Save The Purple Frog on UK and Indian participants, and their overall ratings of the game were similar. The few cross-cultural evaluations of serious games in other domains vary in quality, but have compared the impact of serious games on outcomes such as learning about emotions in children with autism or decisionmaking in water management (e.g., Fridenson-Hayo et al., 2017;Li et al., 2018;Mcgrady & Trentacoste, 2014). Commonly, differences were found in some but not all measurements. For example, a maths game was equally effective in driving motivation to learn between schoolchildren in China and the Netherlands, but enjoyability differed. This underlines the importance of testing games with different audiences, rather than assuming any impacts are generalizable.
The lack of evidence to support a change in behavior attributable to playing Save The Purple Frog is not entirely unexpected, and echoes findings from previous environmental serious games Dunn & Veríssimo, 2021). There is also comparatively little evidence documenting behavior change arising from playing serious games in other fields, and behaviors may be more resistant to change compared to other variables such as knowledge (Bamberg & Möser, 2007;Boyle et al., 2016;Maibach, 1993). However, the lack of evidence to support a change in ecocentric concern or nonanthropocentric motivations is more surprising, and contradicts the previous research on environmental serious games. One possible reason for the failure of the game to affect these variables, and indeed behavior, is the limited duration ($1 h) of the exposure. Longer playing times may be more effective at creating change (Rahmani & Boren, 2012).
Future research directions for environmental serious games should include the impact of longer playing times, as well as different gaming styles. Notably, the survey we developed (excluding knowledge) can be used by researchers to measure the impact of future environmental games regardless of game play type. It is based on existing validated scales and a model specific to the games context which, compared to general models such as the Theory of Planned Behavior, may have greater predictive validity (SI 2; Ajzen, 1985;Treiblmaier et al., 2018). Our survey may also be useful for evaluating whether nonserious games, such as Pokémon Go or Animal Crossing, have unintended but potentially beneficial effects on environmental outcomes (Dorward et al., 2017;Fisher et al., 2021). However, it is worth noting that two of the scales used, Alisat's Environmental Action Scale and Sweetser's GameFlow model, have not yet been validated in diverse contexts, for example, using participants from the Global South.
We also recommend comparing different games to each other in terms of enjoyability, as a pleasurable gameplay experience may be associated with positive outcomes (Boot et al., 2008;Fjaellingsdal & Klöckner, 2017;Fu et al., 2009). Indeed, this may explain the failure of Save The Purple Frog to affect variables other than learning. Although the player feedback for all aspects was positive (above 50%), at no point does it exceed 80% (Figure 3). This was also the case for the flow measurement, which impacted learning outcomes despite only having an average rating of 3.4 out of 5 (Table 1). It suggests that elements such as immersive stimuli or smooth game mechanics could improve player engagement (Fjaellingsdal & Klöckner, 2017;Sweetser et al., 2017). Further, intrinsic motivation to keep playing a serious game is likely related to how enjoyable it is (Bisson & Luckner, 1996;Rangel et al., 2022). Future iterations of the game should try to improve player satisfaction, and test whether this improves environmental outcomes. Ideally, future evaluations would also include a local pilot with more qualitative data collection to explore reactions and perceptions in greater depth.

| CONCLUSION
Our study contributes to the evidence base for environmental serious games, demonstrating their potential as an engaging tool for environmental education. We recommend future researchers use models specific to the games context which may have greater predictive validity, rather than relying on general models such as the Theory of Planned Behavior. We also suggest future research direction, such as examining the impact of longer gameplay.

AUTHOR CONTRIBUTIONS
DV designed the experimental set-up. LTW developed the survey materials, analyzed the data, and led the writing. Both authors reviewed, edited, and accepted the final version of the manuscript.

ACKNOWLEDGMENTS
We thank On The Edge Conservation for developing the mobile game, Save The Purple Frog, and providing the funding for this study.

CONFLICT OF INTEREST
DV works for On The Edge Conservation.

SUPPORTING INFORMATION
Additional supporting information can be found online in the Supporting Information section at the end of this article.