Testing the decoy effect to improve online survey participation: Evidence from a field experiment

As low participation and retention in online survey participation decrease confidence in its results, traditional research has focused on ways to optimize the survey structure and design. So far, no study has tested the decoy effect (i.e. offering an inferior decoy option to increase the attractiveness of the target option) to increase survey participation. In a field experiment with 203 students, we tested the effectiveness of adding a decoy questionnaire to the choice set to improve online survey participation. The decoy questionnaire featured open-ended instead of close-ended questions and delayed remuneration (1 week vs. 4 weeks). We found that the presence of the decoy questionnaire increased the probability of the target questionnaire being completed from 32.7 % to 55.9 %. Furthermore, while the decoy did not affect response behavior or cause non-response bias, a significant order effect was observed. When the target questionnaire was presented before the decoy, the participation was 82.7 % in comparison to 28.0 % when the decoy was presented first. This study is the first to test the decoy effect in the context of aiming to improve survey participation. These findings offer a ‘proof of principle ’ that decoys have the potential to increase participation without negatively influencing question response behavior.


Introduction
Quantitative social research relies heavily on survey methodology (Jansen, 2010).While in the last few decades, the majority of survey studies have already been moved online via emails or posted on websites, instead of postal questionnaires to improve data collection and research participation, the coronavirus pandemic has further increased online surveys, as data collection methods were restricted (Ilieva, Baron, & Healey, 2002Hlatshwako et al., 2021).Participation is measured as the proportion of the population providing consent and complete responses to items in a survey among those who were invited to participate (Schwarz, 2013).A high proportion of online survey respondents increases the generalizability of the outcomes as well as minimizes self-selection and desirability biases (Brtnikova et al., 2018).However, according to recent research, there is a reduction in participation in online surveys among target populations (Glass et al., 2015), which could decrease confidence in research findings.Therefore, increasing participation in online research is very critical.
Previous studies have identified that the response to online surveys is influenced by the length of questionnaires, survey structure, communication methods, interests of participants and identity of the sender, or remuneration (Edwards et al., 2009;Rolstad, Adler, & Rydén, 2011;Schwarz, 2013;Saleh & Bista, 2017).Recently, interventions based on behavioral sciences have shown that individual decision-making can be influenced or 'nudged' in a predicted direction through careful manipulation of the choice setting (Thaler & Sunstein, 2008).An example of such intervention is the decoy effect (or asymmetric decoy effect or attraction effect), which states that individuals' preferences can be influenced by the introduction of a less attractive alternative (i.e. a decoy) into a choice set increases the attractiveness and probability of the more attractive target or action being chosen (Huber, Payne, & Puto, 1982, 2014).The effectiveness of the decoy effect has been consistently demonstrated in various settings and meta-analyses (Heath & Chatterjee, 1995;Milberg, Silva, Celedon, & Sinn, 2014;Yang & Lynn, 2014).The fundamental idea underlying the decoy effect is that, individual preferences are context-dependent and that they can, therefore, be influenced (Vlaev, Chater, Stewart, & Brown, 2011).To be more specific, the decoy effect increases the attractiveness of the target and thus its likelihood to be chosen by offering a less attractive decoy alternative (Huber et al., 1982).
So far, no study has evaluated the effectiveness of the decoy effect to be used in survey participation as part of research invitations.It is hypothesized that among those who are likely to take part in a survey, the presence of a decoy version of the same survey on the invitation will increase the chances of completing the target version.In this study, we set out to test whether the decoy effect could be used in improving the participation in online studies.Specifically, we aimed to test two survey attributes based on previous research on preferences, namely; the question type (open vs close-ended questions) and the duration of the remuneration (immediate/shorter vs. delayed) (Reja, Manfreda, Hlebec, & Vehovar, 2003;Rubinstein, 2003).We manipulated both attributes to ensure the decoy survey is clearly inferior to the target survey (Stoffel, Yang, Vlaev, & von Wagner, 2019).
In a preliminary questionnaire, we verified whether potential study participants perceived open-ended questions and delayed remuneration as less attractive than close-ended questions and sooner remuneration.In the subsequent experiment, we investigated how the addition of the decoy questionnaire in the choice set affected the completion of the target questionnaire.Additionally, as the order of the presentation of the decoy and target influences individual perception of the choice set (Mantonakis, Rodero, Lesschaeve, & Hastie, 2009), we randomized the presentation order to test how it influences the decoy effect.

Recruitment
We carried out the study in August 2022 inviting students, registered at a UK-based university through its social media channels such as Facebook, WhatsApp and WeChat, for a survey on the fear of the coronavirus.Individuals were told in the invitation text that the study would consist of two individual surveys, (1) a short preliminary questionnaire to collect email addresses of individuals willing to participate in the main survey and to assess attitudes towards question type and remuneration and (2) the main survey on the fear of coronavirus.Moreover, the invitation text stated that participants would receive a remuneration upon completion of both the preliminary and main surveys.Fig. 1 shows the flow through the though preliminary questionnaire and experiment.

Preliminary questionnaire
Individuals were contacted for a survey on fear of coronavirus.They were asked to fill out a short initial questionnaire about their attitudes about the two questionnaire attributes question type (i.e.close-ended vs open-ended questions) and time of remuneration (i.e. one week vs four weeks after submitting the questionnaire) and their demographics.Individuals, who were interested in answering the main survey for an Amazon voucher of £2 could leave their email addresses at the end of the questionnaire.
The preliminary questionnaire served two purposes; (1) recruiting individuals for the experiment and (2) validating if planned manipulations based on the two survey attributes to be tested with the decoy conditions (question type and late remuneration) were perceived as worse.The rationale for proposing open-ended questions as a decoy was based on the literature showing that individuals prefer close-ended questions (Reja et al., 2003).Similarly, the late remuneration was informed by the literature on hyperbolic discounting, which states that individuals prefer rewards that arrive sooner than later (Rubinstein, Fig. 1.Flow through the study. S.T. Stoffel et al. 2003).

Experiment
For the experiment, one week after the preliminary questionnaire, we sent individuals, who had provided us with their email addresses, an email invite with the link to the main questionnaire.Individuals were randomized, by random function in Microsoft Excel, into two conditions; (1) the control condition and (2) the decoy condition.In the control condition, individuals received an invitation email to fill out the Fear of Coronavirus Questionnaire (FCQ, Mertens, Gerritsen, Duijndam, Salemink, & Engelhard, 2020), featuring eight close-ended questions and paying out the remuneration in one week.In the decoy condition, the decoy option was added to the invitation email and individuals could choose between filling out the standard FCQ (target) or an alternative questionnaire (decoy), featuring two open-ended questions and later remuneration in four weeks.Additionally, individuals were told that their answers to the open-ended questions would be needed to be coded before individuals can be paid, explaining so the late remuneration.The target and decoy questionnaires in the decoy condition featured debrief questions.These included questions about their reasons for their choice of the questionnaire and whether they felt stimulated by the decoy option.Table S1 in the supplementary file summarizes the key features of the different questionnaires.
The information about the two surveys was presented in table form so that individuals could easily compare them.Note that within the decoy condition, individuals were randomized, with equal probability, to one of two versions (target is shown first in the table vs. decoy is shown first, see Figs.S1-S3 in the supplementary file for the invitation emails).This was to control for the order effect, which refers to the tendency that individuals choose the questionnaire response options that are relatively closer to the beginning of the given list (Mackinnon & Wang, 2020).
No reminder emails were sent out to the individuals in the experiment.The primary outcome of the experiment was the completion of the target questionnaire.The secondary outcomes focused on the impact of the decoy on the FCQ and non-response bias.The answers to the FCQ were summed up to create a scale ranging from 8 to 40 [Mertens et al., 2020].

Ethics approval and data availability
The protocols for the study received ethics approval from the university's Humanities and Social Science Research Ethics Committee (HSSREC).All data files and materials are publicly available via the Open Science Framework and can be accessed at https://osf.io/tvd2y/.

Sample size calculation and data analysis
The sample size for the experiment was calculated before data collection based on estimates derived from the literature.It was assumed that around 30 % would complete the main survey in the control condition and around 45 % in the decoy condition.With at least 100 study participants per experimental condition, the experiment was sufficiently powered to detect such a difference, with a power of 80 % and an alpha value of 0.05 (Cohen, 1988).For the primary outcome, we used a Chi-square test of independence and multivariable logistic regression adjusting for sociodemographic variables to investigate the effect of the decoy on survey participation.While we only report odds ratios (ORs) for the experimental manipulation in the text, the full models showing all the covariates are displayed in Supplemental Tables S3 and S4.Please note that in line with prior literature, the 6 individuals who completed the decoy questionnaire were categorized as not having completed the target questionnaire (Stoffel et al., 2019;Stoffel, Kerrison, Vlaev, & Von Wagner, 2020).For the secondary outcomes, we used Chi-Square test of independency for sociodemographic variables and Wilcoxon-Mann-Whitney test for the FCQ.The statistical analysis was conducted with Stata/IC version 16.0 (StataCorp LP, College Station, TX).

Attitudes towards the manipulated survey attributes
Fig. 2 shows that in the preliminary survey 60.0 % stated that they prefer open-ended questions and 86.1 % stated that they would not respond to a survey if they would receive the remuneration in four weeks.

Study sample
Out of the 210 invitation emails sent out, 7 could not be delivered (4 from control condition and 3 from decoy condition 2), thus 203 individuals received the survey invitation for the experiment.Table 1 shows that the sociodemographic characteristics were similar to the preliminary questionnaire and no statistical differences were found in sociodemographic characteristics, except for ethnicity, between the respondents in control and decoy conditions.There were significantly more individuals with a White ethnic background in the decoy condition than in the control condition (67.7 % vs 47.5 %, Fisher's exact test, p = 0.006).

Effect on survey participation
Fig. 3 shows that the proportion of individuals who completed the target survey was significantly higher in the decoy condition than in the control condition (55.9 % vs. 32.7 %, χ2(1, N = 203) = 11.08,p<0.001).
The analysis of the debrief question in the target survey for the 57 individuals in the decoy condition reveals that most individuals (57.9 %, 33/57) at least somehow agreed that the decoy survey had influenced their decision to participate in the survey.

Effect on FCQ scale and non-response bias
Univariable analyses of the sociodemographic characteristics (see Table S5 in the Supplementary files) and the responses to the FCQ (see Fig. S4 in the Supplementary files) shows little evidence for nonresponse bias or adverse question response behavior.There was no difference in sociodemographic characteristics of individuals completing the target survey across the two experimental conditions, except for ethnicity.Similar to the analysis of the random allocation, we find that there were significantly more individuals with a White ethnic background completing the target survey in the decoy condition than in the control condition (70.2 % vs 36.4 %, Fisher's exact test, p = 0.006).The analysis of the FCQ scale reveals, that there was no statistically significant difference between the underlying distributions of the FCQ scores across the two experimental conditions (z = 0.488, p = 0.629).

General discussion
This is the first study that tested the decoy effect in the context of survey participation.In a field experiment, we showed that online survey participation can be increased by offering a decoy survey option to the choice set.Importantly, we found little evidence that the decoy caused adverse effects on question response behavior or non-response bias.Furthermore, we found a strong order effect within the decoy condition.Participation was similar to the control condition when the decoy option was shown first.This may be explained by the individuals' higher awareness of the decoy in this situation.
This study provides insights into using the decoy effect in increasing research participation.Previous studies on the decoy effect have mainly focused on purchase behavior (Heath & Chatterjee, 1995;Milberg et al., 2014;Yang & Lynn, 2014).Little is known about how the decoy effect influences informed decision-making (Stoffel et al., 2019;Stoffel et al., 2020).In this study, we investigated whether the decoy effect can facilitate participation among individuals who initially stated that they would fill out the questionnaire, by increasing the attractiveness of the motivation.
Our study has several limitations, which call for follow-up research.Firstly, the experiment featured a small analytical sample of university students, which limits the possibility to generalize the findings to the public.Secondly, our study involved deception, in that individuals were told that the study would be about fear of coronavirus, while it was about the decoy effect and the responses to the questionnaire were of little interest to this study.Future studies could test the decoy effect as part of a larger survey study for which there is genuine interest in the  .Furthermore, it's important to note that since individuals who completed the decoy survey also need to be remunerated, the decoy survey should offer valuable information for the research as well.Sixthly, we employed the same set of study participants for both the preliminary and main surveys.It's plausible that asking them about the two survey attributes in the preliminary survey could have potentially impacted their perception of the main survey and, consequently, the decoy effect.Finally, our study only tests one way of presenting the target and decoy in table form.Future research should look at alternative ways (i.e.top and bottom).

Conclusion
In this study, we sought to examine whether the decoy effect can be used to improve online survey participation.We recruited 210 individuals for a field experiment through a preliminary survey that also assessed attitudes towards the attributes of the survey.The results show  that the inclusion of a decoy survey option, which featured open-ended questions and later remuneration, increased survey participation without causing adverse question response behavior or non-response bias.The observed order effect calls for future research on the optimal way to present the decoy option.We believe that this study can be the first step in applying the decoy effect to solve the challenge of decreasing online survey participation.Future studies should be carried out off campus and test different ways of presenting target and decoy options.

Fig. 2 .
Fig. 2. Attitudes towards question type and time of remuneration.

Table 1
Description of the study sample for the experiment (N = 203).
questions, this approach might not be practically feasible to vary since some individuals expressed a preference for open-ended questions over closed-ended ones.Future research should therefore test a more practical decoy survey around a different attribute, such as different type of remuneration or survey length (e.g.short or long version of a validated questionnaire)

Table 2
Binary logistic regressions on completion of target survey.
Model 1: Effect of decoy on survey completionModel 2: Untangling the order within the decoy condition