The influence of mortality and socioeconomic status on risk and delayed rewards: a replication with British participants

Background

In recent years scientists in fields ranging from biomedicine to psychology have become increasingly concerned with the difficulties of replicating findings that are often assumed to be universal and reproducible (Cesario, 2014; Ferguson & Mann, 2014; Moonesinghe, Khoury & Janssens, 2007). In particular, experimental psychologists have been concerned about the fragility of priming effects, highlighting the need for replication of priming experiments (Cesario, 2014; Moonesinghe, Khoury & Janssens, 2007). In this paper, we report our attempts to replicate a finding of particular interest to us, which has been influential and highly cited.

The study we sought to replicate was that of Griskevicius et al. (2011b). Across three experiments, they found that, following exposure to a mortality-risk prime (a fake newspaper article about rising violent crime, designed to elicit the sense that the world is dangerous and unpredictable), participants who grew up wealthier took fewer risks in a lab-based risky choice task, while those who grew up poorer took more risks. The same mortality-risk prime led participants who grew up wealthier to prefer future rewards to immediate ones in a lab-based delay discounting task. Meanwhile, those who grew up in poorer environments preferred more immediate rewards after mortality priming. Thus, for both risk acceptance and delay discounting, there were interactions between priming condition and childhood socioeconomic status in predicting the outcome. Only the interactions were significant in the original study: there were no overall directional effects of either the prime, or childhood socioeconomic status. This finding of an interaction between acute mortality priming and childhood socioeconomic background has been widely cited. However, there have been no direct replications of the original experiments. The original study used student participants from a large university in the USA. Here, we report three replications using British samples, one sample of university students, and two online samples. As the three experiments reported in the original paper were extremely similar to one another, both in design and in outcome, we focussed on a single one (experiment 2), performing multiple replications to maximize the precision of our estimates of the effects. We chose to focus on replicating experiment 2 because the control condition did not use a prime. Using only one adapted prime, rather than two, meant that our replication was closer to the original study.

Methods

We carried out three experiments that replicated experiment 2 from Griskevicius et al. (2011b). These studies were granted ethical approval by the Newcastle University Faculty of Medical Sciences ethics committee (reference number 00554). The pre-registered protocols are available online at https://osf.io/6ucmq/ and https://osf.io/drq98/. All aspects of the replications including the informed consent and debrief screens, demographic questions, prime presentation, and outcome measures (see ‘Measures’), were presented in a web browser via Qualtrics (Qualtrics, Provo, UT, ©2016). Participants in the lab-based study (replication 1) received an additional verbal debrief.

As in Griskevicius et al. (2011b), participants in all three experiments were told that the newspaper article they read in the mortality priming condition (see ‘Priming material’) was part of a memory test. They were told that, after reading the article, they would complete questionnaires related to financial preferences to allow for memory decay. Participants were randomly allocated to either the mortality-priming condition or the control condition by Qualtrics. In the control condition, there were no priming materials. Control participants simply indicated their preferences for the risky and delayed rewards (see ‘Measures’). The order of presentation for the risky choice and delay discounting outcome measures was randomised in all conditions. Following this, participants in all conditions indicated their childhood socioeconomic status (SES) by answering the questions outlined below (see ‘Measures’).

Results

Individual replications

The results of the general linear models for our three replications are summarised in Table 3. The critical interaction was not significant in any of the experiments, either by two-tailed p-values or directed tests. The main effects were largely non-significant. Figures 1–3 reproduce for our three replications the plots used by Griskevicius et al. (2011b) to illustrate the interactions in their data.

Table 3:

Results from the main general linear models for the three replications. Df = 1, 68 (replication 1); 1,155 (replication 2); 1,158 replication 3. P-values are two-tailed; p_dir represents p-value from a directed test as described by Griskevicius et al. (2011b).

Replication	Predictor	F	p	B	SE(B)
Replication 1	Risk acceptance
	Condition	1.78	0.19	0.43	0.32
	Child SES	0.57	0.45	−0.20	0.26
	Condition * Child SES	0.77	0.38 (pdir = 0.24)	0.29	0.33
	Delay discounting
	Condition	0.03	0.87	0.09	0.50
	Child SES	0.24	0.62	0.20	0.40
	Condition * Child SES	0.10	0.76 (pdir = 0.78)	−0.16	0.52
Replication 2	Risk acceptance
	Condition	2.53	0.11	−0.40	0.25
	Child SES	3.40	0.07	0.30	0.16
	Condition * Child SES	0.03	0.87 (pdir = 0.70)	−0.04	0.26
	Delay discounting
	Condition	2.03	0.16	−0.54	0.38
	Child SES	0.81	0.37	−0.22	0.24
	Condition * Child SES	1.82	0.18 (pdir = 0.11)	0.52	0.38
Replication 3	Risk acceptance
	Condition	1.53	0.22	0.35	0.28
	Child SES	5.26	0.02	0.45	0.19
	Condition * Child SES	0.59	0.44 (pdir = 0.97)	−0.22	0.28
	Delay discounting
	Condition	0.10	0.75	−0.13	0.42
	Child SES	1.72	0.19	0.38	0.29
	Condition * Child SES	0.01	0.93 (pdir = 0.58)	0.04	0.43

DOI: 10.7717/peerj.3580/table-3

Meta-analysis

When we meta-analysed the findings of the three experiments, the 95% confidence intervals for the parameter estimates overlapped zero in most cases, for both main effects and interactions (Fig. 4). Thus, we did not detect interaction effects in the individual replications, or when all three replications were combined.

Discussion

We have reported three attempts to replicate findings by Griskevicius et al. (2011b), who found consistent interaction effects between childhood SES and mortality-priming condition for two outcomes: risk acceptance and delay discounting. We did not replicate this main finding in our three experiments which, unlike the original study, used British participants. In none of our individual replications was the predicted interaction significantly different from zero. We found no evidence of an interaction when we combined the results of the three replications in a meta-analysis. Our samples were diverse (one student, two online), and contained a reasonable degree of variation in childhood SES, as would have been necessary to reveal an interaction. We cannot directly compare the childhood SES variation in our sample with that in the sample of Griskevicius et al. (2011b), as they do not report descriptive statistics, but it seems unlikely our samples contained much less variation.

There are a number of potential reasons for our failure to replicate the original findings of Griskevicius et al. (2011b). Firstly, we used British undergraduate students (replication 1) and online participants from the more general British population (replications 2 & 3), whilst the original study used a North American undergraduate sample. Thus, demographic and cultural differences could explain differences in either our participants’ behavioural responses to the priming material, or their willingness to believe the information given in the prime. Indeed, we altered the original priming material in order to make it more convincing to a British audience (for example by replacing references to gun violence with references to knife attacks—see ‘Priming material’). Using the same piloting questions as Griskevicius et al., we found that our prime had similar effects on perceived danger (dangerous, unsafe: Table 2) in its respective audience to the original prime. However, our prime had significantly less effect on perceived uncertainty (uncertain, unpredictable: Table 2). In addition, we asked piloting participants how convincing they found the priming article. In both the initial pilot (Table 2), and in replication 3 (Table S13), participants were only moderately convinced by the priming information (4–5 points on a 7-point scale, with a score of 7 signifying that the prime was very convincing). Griskevicius et al. (2011a) did not ask their pilot participants to rate the prime for convincingness, and so we are unable to directly compare, but it is possible that our participants found our version of the prime less convincing than participants in the original experiments.

Our replications were also limited in other ways. Although our first replication matched the original study sample size (n₁ = 72, Table 1), it may well have been underpowered. Indeed, power analyses indicated that, assuming 80% power and a significance level of 0.05, replication 1 was powered to detect a minimum detectable main effect (MDE) of 0.11. This is a small-to-medium effect according to convention (Champely et al., 2017). Replications 2 and 3 had much greater power (n₂ = 159, MDE₂ = 0.05, n₃ = 162, MDE₃ = 0.05), but used an online crowdsourcing platform for participant recruitment. We applied rigorous quality controls (see ‘Participants and recruitment’) to ensure that the data came from participants living in the UK (the target audience for the modified prime), and that the participants didn’t take too long (an indicator that priming may have been interrupted) or too short (an indicator that the participants may have answered questions without reading them thoroughly) a time to complete the study. Nonetheless, we had no control over the environments in which the online participants experienced the experiment—something which may have affected the efficacy of priming. Finally, crowdsourcing platforms have limited systems in place to prevent their users “cheating” (e.g., by taking part in a study twice, Peer et al., 2017). We tried to reduce this problem by excluding repeat attempts from the same IP address (see ‘Participants and recruitment’). However, this may not be sufficient to prevent a few users repeatedly participating from different devices.

Across our three experiments, we saw some evidence that participants from higher-childhood SES backgrounds were more accepting of risk. This result was marginally non-significant overall. However, the associations were stronger in the two non-student samples, where the variation in childhood SES was larger. We also found, for delay discounting, an interaction between sex and priming that was significant across the three studies in meta-analysis: men tended to respond to the prime by becoming more patient, whereas women tended to become less patient (see Figs. S1 & S2). As this effect was not the subject of an a priori prediction, and was not reported by Griskevicius et al. (2011b), we interpret it with caution. We found no evidence of a three-way interaction between prime, sex, and childhood SES (see Section S3), and thus there is no reason to believe that differing sex balances in our samples and those of Griskevicius et al. (2011b) explain this difference in results.

Although it is possible that cross-cultural, demographic or methodological differences might account for our failure to replicate the original finding, it is also possible that the original findings were a false positive. Set against this possibility is that fact that Griskevicius et al. (2011b) reported three similar experiments using the same materials, and obtained the same result each time. We note that one attempt at conceptually replicating the effect in US undergraduate students also found no evidence of the interaction (Frederick, Khan & Ancona, 2016). However, this study by Frederick et al. used different priming material and, as the authors note, priming effects seem to be particularly sensitive to methodological differences (Cesario, 2014). Thus we suggest that further attempts at replication, both in the US population and in other populations internationally, are needed.

Supplemental Information