A practical guide to controlled experiments of software engineering tools with human participants

Abstract

Empirical studies, often in the form of controlled experiments, have been widely adopted in software engineering research as a way to evaluate the merits of new software engineering tools. However, controlled experiments involving human participants actually using new tools are still rare, and when they are conducted, some have serious validity concerns. Recent research has also shown that many software engineering researchers view this form of tool evaluation as too risky and too difficult to conduct, as they might ultimately lead to inconclusive or negative results. In this paper, we aim both to help researchers minimize the risks of this form of tool evaluation, and to increase their quality, by offering practical methodological guidance on designing and running controlled experiments with developers. Our guidance fills gaps in the empirical literature by explaining, from a practical perspective, options in the recruitment and selection of human participants, informed consent, experimental procedures, demographic measurements, group assignment, training, the selecting and design of tasks, the measurement of common outcome variables such as success and time on task, and study debriefing. Throughout, we situate this guidance in the results of a new systematic review of the tool evaluations that were published in over 1,700 software engineering papers published from 2001 to 2011.

Appendix

Past literature has considered the role of human participants in software engineering research before 2002 (Sjøberg et al. 2005). We still know little, however, about the trends in empirical evaluations with human participants in software engineering from the past decade. To address this gap, and to support the methodological discussion in this paper, we conducted a systematic literature review (Kitchenham et al. 2010) of research from the past decade.

We focused on publications from four of the top software engineering research venues: the International Conference on Software Engineering (ICSE), the ACM SIGSOFT Symposium on Foundations of Software Engineering (and associated European Software Engineering Conference in alternating years), ACM Transactions on Software Engineering and Methodology, and IEEE Transactions on Software Engineering. We chose these primarily because of their status as flagship software engineering journals and conferences, but also because of their focus on software engineering tools (as opposed to studies focused on understanding software engineering practice). Our sample included all of the research publications in these four venues that were published from 2001 to 2011. For conferences, we included only the technical proceedings, excluding experience reports and software engineering education research papers. The resulting set included 1,701 publications.

To begin, we first sought to identify the subset of these papers that reported on new technologies, methods, formalisms, design patterns, metrics, or other techniques intended to be used by a software professional of some kind. The first two authors redundantly classified a random sample of 99 of the 1,701 papers (the Cohen’s Kappa of this classification was 0.8, considered near perfect agreement). We then split the remaining papers and classified them independently in random order, identifying 1,392 papers (81 % of all publications) describing some form of software engineering “tool.” As seen in Fig. 3, the proportion of papers contributing tools has not changed significantly in the past decade (χ ²(10,N = 1,701) = 11.1, p = .35).

Fig. 3

Proportion of papers contributing software engineering tools

To narrow the sample of “tool papers” to “tool papers with evaluations,” we checked each paper for an evaluation of the tool (including evaluations of both internal properties such as performance or accuracy and external properties of how the tool is used by people). We counted an evaluation as “empirical” if it included any quantitative or qualitative data from observation of the tool’s behavior or use. Therefore, if the paper only described a task that could be performed with a tool, but was not actually performed, the paper was not included. Achieving high reliability when redundantly classifying the same 99 papers (Kappa = 0.83), we split and classified the remaining papers, identifying 1,065 reporting on both a tool and empirical evaluation of it; 77 % of tool papers included an empirical evaluation. As seen in Fig. 4, the proportion of papers contributing tools and reporting empirical evaluations of them has increased significantly in the past decade from about 50 % to over 90 % (χ ²(10,N = 1,392) = 117.9, p < .001). This shows that empirical evaluations are now widely adopted and perhaps even expected for publication of novel tool contributions.

Fig. 4

Proportion of papers contributing tools that also reported an empirical evaluation of the tool

To further narrow our sample to tool papers that evaluated some aspect of the use of a tool, we identified the individual studies reported in each paper. We counted any study with a unique method within a paper as a separate study (lab studies that were run multiple times or case studies run on multiple programs were counted as a single study). After achieving high reliability on this definition (Kappa = 0.62), we found 1,141 studies across 1,065 papers.

With the studies in each paper identified, we then classified each of these studies as either involving the human use of the tool or not. Any person using the tool for some task, including the paper authors themselves if they described their own use of the tool, was included. After achieving high reliability on this classification (Kappa = .70), we classified the papers, finding 345 studies across 289 papers that involved human use of a tool. As shown in Fig. 5, the proportion of studies that involve developers or other software professionals using a tool is actually on a slow decline in our sample, from a peak in 2002 of 44 % to only 26 % of papers in 2011 (χ ²(10,N = 1141) = 21.0, p < .05). This means that although more evaluations are being done, a smaller proportion of them are evaluating a tool’s use by humans (instead evaluating its internal characteristics such as speed, accuracy, or correctness).

Fig. 5

Proportion of studies contributing evaluations examining how the authors or recruited participants used the tool. Contrary to the graph in Fig. 4, this graph shows a significant decline, suggesting that papers are increasingly evaluating the non-human use of tools

The subset of tool evaluations of human use that recruited human participants and did not use authors is shown in Fig. 6. This plot shows that although studies evaluating tool use are less common (Fig. 5), an increasing proportion of these studies involve human participants. Therefore, when software engineering researchers are studying how a tool is used, they are increasingly recruiting participants rather than using themselves.

Fig. 6

Proportion of studies evaluating tool use that involved non-author human participants. Although studies of use are less common (Fig. 5), they are increasingly involving human participants

Most of the studies instead used lab studies (conducting an study in a controlled setting with participants), interviews (demonstrating the tool and asking participants to respond to spoken questions about it), surveys (demonstrating the tool and asking participants to respond to written questions about it), field deployments (observations of the use of the tool in real settings on real tasks), and a method we will call tool use experience reports ² (the use of the tool with a specific program or data set, either by the authors or some other person). As shown in Fig. 7, the most common method by far was the tool use experience report, which was the method of 67 % of the 345 studies evaluating human use of a tool. As seen in Fig. 7, the other four methods were much less common. The relative proportion of tool application studies compared to other types has not changed significantly in the past decade (χ ²(10,N = 345) = 9.6, p = .473). The frequency of experiments per year has also not changed significantly over the past decade (χ ²(10,N = 345) = 8.2, p = .611). Figure 8 shows that only a small subset of these studies are controlled experiments. In fact, the number of experiments evaluating tool use has ranged from 2 to 9 studies per year in these four venues, for a total of only 44 controlled experiments with human participants over 10 years.

Fig. 7

Proportion of methods used in studies evaluating human use of a tool

Fig. 8

Proportion of papers evaluating use that were experiments. The number of experiments ranged from 2 to 9 per year

These results indicate several major trends in the methods that software engineering researchers use to evaluate tools: (1) empirical evaluations are now found in nearly every paper contributing a tool, (2) the proportion of these evaluations that evaluate the human use of the tool is on the decline, (3) an increasing proportion of human use evaluations involve non-author participants, but (4) experiments evaluating human use are still quite rare.

These findings are subject to several threats to validity. We considered only 4 journals and conference proceedings in our review, focusing on those with a strong reputation for contributing new tools. There are many other software engineering publication venues where such work appears. It is possible that the trends we observed are particular to the venues we chose; for example, Buse et al. (2011) found that while ICSE and FSE showed no signs of increases in user evaluations, OOSPLA, ISSTA, and ASE did. There may also be evaluations with human participants that were never published or that were published in other venues after being rejected by the venues that we did analyze.