Experimental methods: Extra-laboratory experiments-extending the reach of experimental economics

https://doi.org/10.1016/j.jebo.2013.04.002Get rights and content

Highlights

  • We propose a new classification scheme for lab and field experiments.

  • We describe examples of how different experiments fit into different categories.

  • We provide organizing principle for classifying research experiments.

Abstract

We propose a new organizing scheme for classifying types of experiments. In addition to the standard categories of laboratory and field experiments, we suggest a new category: “extra-laboratory experiments.” These are experiments that have the same spirit as laboratory experiments, but are conducted in a non-standard manner. We also suggest some organizing principles to distinguish amongst the three types of experiments.

Introduction

We propose that experiments that are conducted outside the typical on-campus lab and with participant knowledge should be considered extra-laboratory experiments. These are akin to classic lab experiments in every way except for the venue and the subject pool. Given the recent increase in popularity of online experimentation, targeting non-student populations and use of experiments in developing countries, it is important that these studies are anchored to their lab precursors as a means of comparison.

Extra-lab experiments differ from lab experiments along a number of dimensions. They typically entail a different subject pool and/or a different venue. They incorporate a wide variety of selection processes: mandatory participation in the classroom, voluntary participation from a select population online, targeted sampling in rural villages and many others. Extra-lab experiments have provided observations of behavior under a broad spectrum of stakes, from pennies online to months of income in developing countries. The administrators of the experiment are often very different: online interfaces remove all human interaction while local translators and community members have a closer connection to the subjects. Whether or not these differences are desirable depends primarily on the question being asked; often the direct effects of these differences are objects of economic interest.

Our paper relates to the taxonomy of Harrison and List (2004), which expanded the definition of field experiments to include experiments conducted outside of the lab and with non-student participant pools. In contrast with this taxonomy, we propose that extra-lab experiments are different than field experiments, and offer some clear organizing principles that define extra-laboratory experiments as separate from both laboratory experiments and field experiments.

In our classification, a sufficient, but not necessary, condition for an experiment to be classified as a field experiment is that the participants are not aware of the existence of an experiment. Participant awareness of being involved in an experiment does not, by itself, disqualify that experiment from being a field experiment, nor does its location: by our classification, a field experiment could be conducted in the laboratory. Gneezy et al. (2013) conduct an experiment in the laboratory that involves overpaying participants from a previous, unrelated experimental task and observing whether they report the overpayment. The overpayment is designed to look like a casual mistake: instead of ten $1 bills for the correct payment of $10, subjects receive nine $1 bills with a $5 bill mixed somewhere in the middle. By mimicking an experience that could happen anywhere, the experimenters bring the field to the lab.1 More generally, while the venue for an experiment is important, we do not feel that this factor is what distinguishes a field experiment from a lab experiment.2

Instead, one primary factor for determining whether an experiment is a field experiment is whether the activity of interest would have happened but for the actions of the experimenter. If this is the case, the experimenter is taking advantage of an existing natural environment. Examples include Lucking-Reiley (1999), an experiment that involves auctions of collectible cards on the Internet, List (2001), which experiments in an in-person sports-card market, and Garratt et al. (2012), which involves E-bay auctions for goods common to the site. However, on this basis, neither the public-goods experiments conducted in these venues nor the experiments in rural third-world settings with standard laboratory games (e.g., Gneezy et al., 2009, Henrich et al., 2001) would be classified as field experiments.

A similar approach to defining what constitutes a natural environment is to say that when the data is non-observational, the experiment is likely not a field experiment. In other words, when the subjects report their decisions or opinions directly to the experimenter, more likely than not, the natural environment has been compromised in some way.

Of course, this leaves some gray areas, and generates exceptions. Consider Gneezy et al. (2004), where the experimenters vary how a restaurant bill is split amongst the participants (and the experimenter). The restaurant is clearly a natural environment and choosing how to split the bill is a natural feature, but the diners are aware that the experimenter has altered the costs and benefits of splitting the check in certain ways. Even though all other aspects of the experiment meet the “but-for” definition provided earlier, the experimental manipulation of costs does not. Hence, we would classify it as an extra-lab experiment and not a field experiment. Another case in point is Charness and Gneezy (2009), in which participants are paid to go to the gym over time. While it is natural for students to go to the gym, paying them to do so is an unusual intervention.

Importantly, experiments in education or health interventions (e.g., Angrist and Lavy, 2009, Loewenstein et al., 2012) in which participants sign consent forms and are paid for performance improvements could fit into either category. As an analogy, consider an experiment that incentivizes students to get a high GPA. The students are engaged in an activity they would do regardless of the study, and the experimenter just adds incentives.

What distinguishes a standard laboratory experiment from an extra-lab experiment? Once again, the venue is not at the heart of the matter. Running standard lab experiments with special populations such as soccer players (Levitt et al., 2010), older and younger workers on location at French firms (Charness and Villeval, 2009) is, in our classification, an extra-lab experiment. Some primary factors in separating standard laboratory experiments from extra-lab experiments include having access to a broader range of age, culture, and experience than is otherwise available in student populations, having a population more attuned to the research question, and permitting activities to be carried out continuously over a longer period of time than is feasible in the laboratory.

Importantly, we do not argue for the any general superiority of any of these methodological approaches over any other. We simply feel that one should use the best tool available for the task at hand. In this respect, in many cases it is particularly useful to take the lab out to the field and conduct tests using standard stylized tasks, but with special subject pools and special locations.

The paper is organized as follows: Section 2 highlights motivations for taking experiments out of the lab, Section 3 discusses the methodology of doing so, Section 4 presents some of the differences in results between the two paradigms and we conclude in Section 5.

Section snippets

Changing the subject pool

Beyond looking for the differential importance of a particular theory, a primary motivation for extra-lab experiments is that the variation in individuals found on campus is simply insufficient to answer some questions. Socio-economic and demographic differences are likely to be important in determining the scope and magnitude of many phenomena that we commonly study in the lab. Consider research into whether or not gender differences in competitiveness are a result of nature or nurture. Gneezy

Methodology

Laboratory experiments set the gold standard for environmental control. Any uncontrolled complexities or confounds in the environment push researchers in the direction of the laboratory, since it is easier to exercise strong control over an experiment with (by design) fewer moving parts. Nevertheless, one might wish to perform an extra-lab experiment with a highly specialized group of experts at work for the purpose of implementing a very complicated protocol. Such versatility is one of the

Do laboratory and extra-laboratory results differ?

We consider a number of canonical games and tasks to compare across the lab and extra-lab paradigms.

Conclusion

We suggest a new classification of experiment methodologies that includes extra-laboratory experiments, which are generally somewhere between classic laboratory experiments and field experiments. We present organizing principles for this classification, highlighting a number of the distinct advantages (and thus motivations) for performing extra-lab experiments. These include changing the selection process, changing the subject pool, leveraging stakes, decreasing cost, or finding variation that

References (55)

  • S. Andersen et al.

    Stakes Matter in Ultimatum Games

    American Economic Review

    (2011)
  • S. Andersen et al.

    Eliciting risk and time preferences

    Econometrica

    (2008)
  • Andreoni, J., Kuhn, M.A., Sprenger, C., 2013. On measuring time preferences, UC San Diego. Working...
  • J. Angrist et al.

    The effects of high stakes high school achievement awards: evidence from a randomized trial

    American Economic Review

    (2009)
  • D. Ariely et al.

    Large stakes and big mistakes

    The Review of Economic Studies

    (2009)
  • Babcock, P., Bedard, K., Charness, G., Hartman, J., Royer, H., 2011. Letting down the team? Evidence of social effects...
  • Babcock, P., Hartman, J., 2010. Exercising in herds: treatment size and status specific peer effects in a randomized...
  • J.A. Bargh et al.

    Automaticity of social behavior: direct effects of trait construct and stereotype activation on action

    Journal of Personality and Social Psychology

    (1996)
  • A. Bosch-Domènech et al.

    One, two, (three), infinity,…: newspaper and lab beauty-contest experiments

    American Economic Review

    (2002)
  • Bosch-Domènech, A., Nagel, R., 1997. Guess the number: comparing the financial times and expansion's results. Financial...
  • L.A. Cameron

    Raising the stakes in the ultimatum game: experimental evidence from Indonesia

    Economic Inquiry

    (1999)
  • G. Charness et al.

    Incentives to exercise

    Econometrica

    (2009)
  • G. Charness et al.

    Cooperation and competition in intergenerational experiments in the field and the laboratory

    The American Economic Review

    (2009)
  • Cleave, B.L., Nikiforakis, N., Slonim, R., 2011. Is there selection bias in laboratory experiments? The case of social...
  • C.C. Eckel et al.

    Volunteers and pseudo-volunteers: the effect of recruitment method on subjects’ behavior in experiments

    Experimental Economics

    (2000)
  • E. Fehr et al.

    Cooperation and punishment in public goods experiments

    The American Economic Review

    (2000)
  • R.J. Garratt et al.

    Behavior in second-price auctsions by highly experienced eBay buyers and sellers

    Experimental Economics

    (2012)
  • Cited by (0)

    View full text