Advanced Training in Web-Based Psychology Research
Trends and Future Directions
Abstract
Abstract. This article describes Advanced Training Institutes (ATIs) and workshops on psychological experimentation conducted via the Internet. These programs, conducted since 2002, presented instruction that evolved over the decades to reflect changes in Web-based methods and techniques. The need for instruction in the methods and methodology of Web-based research has increased over the years, as more researchers adopted these methods without necessarily learning the methodological lessons developed from theory and experience. Links to instructional materials created for the workshops are provided. From events and trends that played out over time, including the story of methods that were once state-of-the-art and are no longer functional today, lessons hard-won from the past can be used to anticipate and plan future directions in Web-based experimentation.
Web-based research often provides advantages over lab-based methods. Because of lower costs, ease of recruiting large and diverse samples, ability to reach participants with rare characteristics, and other advantages (such as the ability to precisely communicate, replicate, and build upon exact experimental methods), the number of researchers employing these methods has grown exponentially since such studies became possible in the mid-1990s (Birnbaum, 2004a; Gosling & Mason, 2015; Wolfe, 2017).
Despite the increasing acceptance of Web-based methods, there has been and remains a need for proper education in computer technology and Internet methodology appropriate for this type of research (Birnbaum, 2010; Krantz & Reips, 2017; Wolfe, 2017). As noted by Krantz and Reips (2017), most current textbooks on experimental methods and methodology for psychological research do not address the technical, methodological, and ethical issues crucial to this type of research. In the age of COVID-19, laboratory experiments were proscribed in most universities because of the risks of COVID, which caused many researchers to adopt Web methods without any formal training. The present article describes a series of Advanced Training Institutes (ATIs) and workshops in Web-based psychology experiments that sought to fill this need. Trends over two decades help us understand how the field evolved and provide perspective to foresee future directions.
The Initial ATI Sessions: 2002–2005
With funding provided by US National Science Foundation (BCS-0129453), a series of ATI sessions specializing in Social Psychology experiments were held at California State University, Fullerton, starting in 2002 (Birnbaum, 2002). Six instructors (Michael Birnbaum, John Krantz, Gary McClelland, Ken McGraw, Ulf-Dietrich Reips, and William Schmidt) designed and presented the curriculum, a description of which can be found in Williams (2007). Instructional materials, links, and exercises (updated for a 2019 workshop) can be found in the following URL: http://ati-birnbaum.netfirms.com/
Michael Birnbaum presented a basic introduction to HTML, including the methods for collecting data provided by the FORMS components of HTML. Birnbaum provided “bare bones” examples that illustrated techniques in the simplest way possible, devoid of frills, decorations, or additions that might distract a participant, clutter a page, or slow downloading pages without adding functionality. An introduction to JavaScript (https://www.w3schools.com/js/), a scripting language that is widely used to add functionality to Web pages, was taught using material from Birnbaum (2000a, 2000b, 2001a) and Birnbaum and Wakcher (2002).
Ulf-Dietrich Reips presented his approach to Web research (Reips, 2000), including WEXTOR (https://wextor.eu), a program for constructing sets of Web pages that can implement a variety of experimental designs (Reips & Neuhaus, 2002). By using the “GET” method in HTML forms, data from a form are sent to the server’s logfile from which they can be extracted by a program like Scientific LogAnalyzer (http://sclog.eu), along with much other useful information about a participant’s interactions with the experiments (Reips & Stieger, 2004).
Such information can be used to examine exactly where a participant might decide to drop out of a study, for example. Because it is easy for a participant to drop out of a Web study (no social pressure to stay), and because drop-outs can harm the interpretation of between-subjects studies [even with equal rates of drop-out in the treatment and control groups, drop-outs can make a harmful treatment look beneficial and vice versa (Birnbaum & Mellers, 1989)], Reips (2000) proposed methods intended to reduce drop-outs or to cause people to drop-out before they are randomly assigned to conditions and analyzed drop-outs. “High hurdle” and “warm-up” are techniques intended to encourage early drop-out before the real experiment begins.
Asking people to supply personal information at the beginning of a study (rather than at the end) may help keep participants committed to a study (Frick et al., 2001). One “high-hurdle” technique proposed to induce people to drop out was to employ a page at the beginning of a study that loaded very slowly on the hope that those who would eventually depart would do so early, but Göritz and Stieger (2008) were unable to show that this technique improved data quality. Although incentives can induce people to join and remain in a study (Göritz, 2006), there is concern that those who stay only for cash incentives might respond randomly (Chandler & Paolacci, 2017). Howell (2021b) reviewed factors thought to affect drop-outs in Web studies.
John Krantz taught methods for transmitting media via the Web (Krantz, 2001). Images could be presented in sequence to create simple animations, for example, to implement a study of eyewitness testimony. When few had digital cameras, and no one yet had a smartphone, each participant received an inexpensive digital camera to capture images used in the exercises. Krantz taught the use of ImageJ, free public domain software for image analysis and processing, available from https://imagej.nih.gov/ij/.
William C. Schmidt presented lessons on the server-side of psychology experiments (Schmidt, 2000). He taught how the language PERL could be used as a CGI language to check data, capture the participant’s IP address (useful for detecting multiple submissions, among other purposes), save the data (along with time and date, referring document and other information), and then to redirect the participant to another Website following their submission (e.g., to a thank you and debriefing page, see also Fraley, 2004). A useful PERL script, generic.pl, written by Schmidt, that can be used to collect, organize, and save data is linked at http://ati-birnbaum.netfirms.com/Install_Perl_script.htm.
Reips and Birnbaum (2011) describe how to run your server (Apache server: https://httpd.apache.org/) and install this script. Göritz and Birnbaum (2005) presented an alternative, generic PHP script that can also be used with forms created by Birnbaum’s (2000b) SurveyWiz and FactorWiz programs that create Web forms.
Ken McGraw introduced the Authorware program and the extremely well-constructed PsychExps site at Ole Miss, created by McGraw et al. (2000). The Authorware program allowed one to create complex learning modules (and experiments) by moving icons representing the sequence of operations to be performed by the computer. These experiments could then be delivered via the Web and run on the client’s computer, using a special plug-in that the user had to download. McGraw and his group organized several workshops teaching these methods, and a substantial community of teachers and researchers developed around the Ole Miss site.
Gary McClelland taught an introduction to Java, an extremely powerful programming language that could be used to create dynamic graphics such as those used with McClelland’s book, Seeing Statistics (McClelland, 1999), or in the Cognitive Psychology Online Laboratory, described in Francis et al. (2000). Java could be used to create and animate visual images and to control and measure timing. Java apps were precompiled and sent as byte codes, and ran on the participant’s machine, if Java was properly installed.
All of the instructors contributed to group discussions of methodological issues pertinent to Web studies, and they addressed frequently asked questions such as the following:
What Techniques Can Be Used To Recruit Participants? How Do We Find People With Rare Characteristics?
One can recruit participants by traditional methods (those used to recruit participants for lab research at your institution) and test them via computers in the lab, using the Internet as a network. For more diverse samples (with respect to age, education, etc.), one can advertise via Websites posting lists of active online studies, such as the one maintained by Krantz (1996) or the Web Experiment List at Konstanz (https://wexlist.uni-konstanz.de/), or one might try social media sites such as Reddit’s/r/SampleSize (https://www.reddit.com/r/SampleSize/). One can recruit and maintain an online panel, that is, a list of people willing to participate in studies. Alternatively, use a panel maintained by a research organization or commercial enterprises, such as Amazon’s Mechanical Turk, or those listed in Qualtrics (https://www.qualtrics.com/support/survey-platform/sp-administration/brand-customization-services/purchase-respondents/). Several articles have discussed recruiting methods (Birnbaum, 1999, 2001a, 2004a; Buhrmester, 2018; Gosling & Mason, 2015; Göritz, 2009; Howell, 2021a). Specialized or rare cases can be recruited from mailings to organizations devoted to a rare characteristic or from emails sent to a list, preferably by a trusted member or officer of the organization who can vouch for the study and its relevance to the goals of the members.
How Does One Recruit and Maintain One’s Own Panel of Survey Participants That Can Be Tested Over a Long Period of Time?
Tips for online panels are given in Göritz (2009) and Conner (2012).
How Do We Know Who the Participants Are, or if They Are Truthful in Their Self-Descriptions? What Methods Can Be Used to Control or to Assess the Conditions Present When a Person Is Performing a Task Remotely?
The first method is to request honesty ask that participants follow instructions, remove incentives that encourage dishonesty or recklessness, and introduce incentives to encourage honesty. A second method is to ask people if they followed the instructions or ask the same questions in different ways or at different points in a questionnaire to check for reliability. In some cases, one might request uploading of images of a passport or require other identifying or confirming information such as a sign-in code (Reips, 2000; Howell, 2021a). The same issues can arise in lab-based studies, and observation of participants in the lab can inform what might be anticipated from Web participants. For example, I wanted to test a random response technique to inquire about the use of illegal drugs without requiring anyone to reveal his or her own usage: participants were instructed to toss two coins and to respond “yes” to a sensitive question if both coins were heads, to respond “no” if both were tails, and to tell the truth if they were mixed. While watching participants complete the questionnaire, my assistant noticed that only one person out of 44 tested in the lab took out any coins and actually followed the instructions, and even that person asked first, “It says here we are to take out two coins. Does that mean we should actually do that?” When asked to explain later why they did not follow the procedure, many “reasons” were given. Ultimately, one must concede that there is less control in Web studies compared to the laboratory.
What Types of Studies Are Not Really Appropriate for Delivery via the Web?
It is impossible to test certain populations (e.g., babies who do not yet use a computer) or deliver certain stimuli via the Web such as lifted weights, olfactory stimuli, or foods. In some such cases, however, the Internet can be used as a network by an experimenter who tests such cases on-site or by a participant who can follow instructions with a test kit delivered in the mail.
How Do Data Collected in the Lab and via the Web Compare?
Early studies of this type were reviewed by Krantz and Dalal (2000). Studies that used comparable populations (e.g., college students) tested in a lab or via the Web (e.g., Birnbaum, 2001b) find comparable results with either procedure. When different participants are recruited via the Internet or tested in the lab (e.g., PhDs vs. college students), systematic differences correlated with demographics such as age, gender, and education may be observed (Birnbaum, 1999). The issues of data quality and multiple submissions have been addressed in several papers, including Clifford and Jerit (2014), Howell (2021a), Bowen et al. (2008) and Teitcher et al. (2015), among others.
There were also group discussions of ethical and scientific standards of Web and lab studies. In Web studies, there is no social pressure to stay and complete the task, so a person can easily withdraw at any time. Psychology studies in the lab have not been likely to produce injury or death, and Web studies remove even the greatest dangers of lab research, which are the risks of driving or riding to the lab and the risk of acquiring disease from others (Birnbaum, 2004b). During the COVID-19 pandemic, the relative safety of Web studies was suddenly appreciated as never before.
Online methods for informed consent and debriefing were described. In addition, the importance of checking one’s work to avoid wasting people’s time with poorly constructed studies was emphasized, as well as the concept of saving one’s materials and data in an online archive for completed studies (Birnbaum, 2001a, Chapter 21). Web-based research makes it easy to save completed experiments on the Web, so that other researchers can see clearly how an experiment was implemented; Web publication of experiments makes scientific communication more transparent than possible with brief descriptions in a Method section, which facilitates replication of studies.
ATI Sessions: 2005–2014
In 2005, Birnbaum organized a five-day ATI for the American Psychological Association (APA), described by Adelson (2005). At this session, John Williams replaced McGraw for Authorware, and Anja Göritz joined the team to teach PHP, MySQL, and methods for managing online panels (Göritz & Birnbaum, 2005; Göritz, 2009). Williams (2007) later organized another ATI for APA.
Funded by the Decision, Risk and Management Science division of (US) NSF (DRMS-0721126), Birnbaum organized another series of workshops from 2007–2013. In addition to the longer ATI sessions, these included several smaller, 1-day workshops, held just before or after the Edwards Bayesian Research Conference, with the last one in 2014.
In 2015, 2017, and 2019 (with one planned for 2021), Reips organized week-long Summer Schools in Konstanz, in which Reips and Birnbaum taught Web methods and methodology similar to their presentations at the earlier ATIs, and where McClelland and Charles M. Judd taught statistical analysis, focused on topics of mediation and moderation, with emphasis on the use of R and data visualization.
The Konstanz workshops included not only fundamental methodological issues that persist as issues in the field (Krantz & Reips, 2017; Wolfe, 2017) but also discussion of evolving directions in Web-based research, including cautionary remarks regarding certain practices that have been adopted widely, including Mechanical Turk (Buhrmester, 2018; Buhrmester et al., 2018) using “professional” participants.
In addition to the Konstanz workshops, formal instruction on Web-based research has developed around the approach of de Leeuw (2015). Josh de Leeuw developed jsPsych, a library of additions to JavaScript that allows a researcher with a rudimentary understanding of HTML, CSS, and JavaScript to build new projects from pre-programmed components that can be assembled to build psychology experiments (de Leeuw, 2015). Tutorials from 2020 based on this promising approach (https://www.jspsych.org/) are available via YouTube at the following URL: https://www.youtube.com/watch?v=BuhfsIFRFe8
However, the limited opportunities for instruction in Web-research have not been able to reach many people who have adopted these methods. Experienced Web-researchers continue to find studies online with common mistakes; for example, programming an input device with preselected responses, such that the data saved do not distinguish whether a participant endorsed the preselected answer or may have simply skipped an item. There remains a strong need for programmatic instruction in Web-based research.
Past Trends and Future Directions
Over the last two decades of Web-based research, computer screens grew larger and larger – then smaller, as tablets and smartphones became ubiquitous. Certain techniques became problematic as Web Browser standards evolved and some methods were pushed aside. Among the methods that have fallen out of use for Web studies are Java, Authorware, and Flash. The story of the demise of these methods provides a lesson from the recent past with a message for the future.
The Java language, once a method for programming powerful Web applets with graphics and control of timing, ideal for cognitive psychology studies created headaches for programmers as the language evolved (Francis et al., 2000). Revisions to the language did not solve security concerns, and users turned off Java in their browsers, and it eventually lost out as a viable method for Web-based research. At this writing, studies created in Java are no longer functioning on the Web, but Java is still taught as a programming language.
McClelland presented Canvas to program dynamic graphics via HTML5 and JavaScript to replace what was previously done via Java applets, which were losing out as a viable method for Web presentation. McClelland (personal communication, February 17, 2021) has subsequently transitioned to a JavaScript approach known as D3.js (Data-Driven Documents), examples of which can be found at the following URL: https://observablehq.com/@d3/gallery
Some of McClelland’s examples originally programmed in Java are now available in the public domain as JavaScript at the following URL: https://bolderstats.com/AB/. Because D3 uses scalable graphics, the pages adjust reasonably well to fit on smartphones and tablets as well as on computer screens.
The Cognitive Psychology Online Lab was originally written in Java (Francis et al., 2000), so it could not survive as Java lost out as a viable Web language. Fortunately, HTML and JavaScript were expanded, allowing the authors to recreate their experiments in JavaScript. With more than 50 cognitive psychology demonstrations and experiments, this very successful site has been used by thousands of students (Gregory Francis, personal communication, May 14, 2021). It can be found at the following site: https://coglab.cengage.com
Like Java, Authorware and Flash were once used to construct perceptual and cognitive psychology demonstrations and experiments. All three were proprietary commercial products and were bought out by corporations that eventually stopped supporting them. Those researchers who had invested much effort in creating active laboratories saw their work become obsolete. If one conducted a study using Authorware or Flash, others could no longer view how the studies functioned when still in action, so they cannot replicate those studies exactly. So an important component of scientific communication and preservation became lost when commercial concerns canceled the work of scientists.
The Ole Miss site had been transferred to the American Psychological Association (2021), where it functioned for a time as a community resource that could be used to collect data for research or teaching. But this site was rebuilt without input from its creators when Adobe dropped support of Authorware. The APA hired programmers to re-create the experiments in Flash, which was later also depreciated by Adobe, which delivered a second blow to the site. It is hoped that the American Psychological Association will work with active researchers to rebuild a functioning and comprehensive site again.
In contrast with content created by Java, Authorware, or Flash, Websites built using HTML and JavaScript 25 years ago continue to function. Similarly, Research Websites that used the Apache server and server-side programming (e.g., using Perl) continue to function and have become so ubiquitous on the Web that they seem likely to remain stable for many years to come. It appears that current researchers would be advised to avoid using commercial products, especially if they depend on nonstandard configurations or special installations, as these risk being canceled by corporate decision or dying out, and to work instead with scientifically evaluated, non-commercial, open-source software that is supported by large and active communities.
Thanks are due to Gregory Francis, Kenneth McGraw, Gary McClelland, and Ulf-Dietrich Reips for discussions of this material and Bonny Quan for suggestions on the manuscript.
References
(2005). Web experimentation: A new APA Science Directorate institute taught psychologists and students to conduct online research. APA Monitor, 36(9), 38. https://www.apa.org/monitor/oct05/web
. (2021). APA online psychology laboratory. https://opl.apa.org/
(1999). Testing critical properties of decision making on the Internet. Psychological Science, 10(5), 399–407. https://doi.org/10.1111/1467-9280.00176
Birnbaum, M. H. (Ed.). (2000a). Psychological experiments on the Internet. Academic Press. https://doi.org/10.1016/b978-012099980-4/50001-0
(2000b). SurveyWiz and factorWiz: Javascript Web pages that make HTML forms for research on the Internet. Behavior Research Methods, Instruments, & Computers, 32(2), 339–346. https://doi.org/10.3758/BF03207804
(2001a). Introduction to behavioral research on the Internet. Prentice Hall.
(2001b).
A Web-based program of research on decision making . In U.-D. ReipsM. Bosnjak (Eds.), Dimensions of Internet science (pp. 23–55). Pabst Science.(2002). Advanced training institute in social psychology experiments via the WWW. http://psych.fullerton.edu/mbirnbaum/www/
(2004a). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832. https://doi.org/10.1146/annurev.psych.55.090902.141601
(2004b).
Methodological and ethical issues in conducting social psychology research via the Internet . In C. SansoneC. C. MorfA. T. Panter (Eds.). The Sage handbook of methods in social psychology (pp. 359–382.) Sage Publications. https://doi.org/10.4135/9781412976190.n16(2010).
An overview of major techniques of Web-based research . In S. D. GoslingJ. A. Johnson (Eds.). Advanced methods for conducting online behavioral research (pp. 9–25). American Psychological Association. https://doi.org/10.1037/12076-002(1989). Mediated models for the analysis of confounded variables and self-selected samples. Journal of Educational Statistics, 14(2), 146–158. https://doi.org/10.2307/1164604
(2002). Web-based experiments controlled by JavaScript: An example from probability learning. Behavior Research Methods, Instruments & Computers, 34(2), 189–199. https://doi.org/10.3758/BF03195442
(2008). Identifying multiple submissions in Internet research: Preserving data integrity. AIDS and Behavior, 12(6), 964–973. https://doi.org/10.1007/s10461-007-9352-2
(2018). M-Turk guide. https://michaelbuhrmester.com/mechanical-turk-guide/
(2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13(2), 149–154. https://doi.org/10.1177/1745691617706516
(2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8(5), 500–508. https://doi.org/10.1177/1948550617698203
(2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1(2), 120–131. https://doi.org/10.1017/xps.2014.5
(2012). 15 tips for running an online panel. https://www.researchworld.com/15-tips-for-running-an-online-panel/
(2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47(1), 1–12. https://doi.org/10.3758/s13428-014-0458-y
(2004). How to conduct behavioral research over the Internet: A beginner’s guide to HTML and CGI/Perl. Guilford Press.
(2000).
The cognitive psychology online laboratory . In M. H. BirnbaumEd., Psychological experiments on the Internet (pp. 267–283). Academic Press. https://doi.org/10.1016/b978-012099980-4/50012-5(2001).
Financial incentives, personal information and drop-out in online studies . In U.-D. ReipsM. BosnjakEds., Dimensions of internet science (pp. 209–219). Pabst.(2006). Incentives in Web studies: Methodological issues and a review. International Journal of Internet Science, 1, 58–70. https://www.ijis.net/ijis1_1/ijis1_1_goeritz.pdf
(2009). Building and managing an online panel with phpPanelAdmin. Behavior Research Methods, Instruments & Computers, 41(4), 1177–1182. https://doi.org/10.3758/BRM.41.4.1177
(2005). Generic HTML form processor: A versatile PHP script to save Web-collected data into a MySQL database. Behavior Research Methods, 37(4), 703–710. https://doi.org/10.3758/BF03192743
(2008). The high-hurdle technique put to the test: Failure to find evidence that increasing loading times enhances data quality in Web-based studies. Behavior Research Methods, 40(1), 322–327. https://doi.org/10.3758/BRM.40.1.322
(2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902. https://doi.org/10.1146/annurev-psych-010814-015321
(2021a). Online psychology experiments: Everything you need to know. https://www.psychstudio.com/articles/online-experiments/
(2021b). Why do participants drop out of online surveys and experiments. https://www.psychstudio.com/articles/dropout/
(1996). Psychological research on the net. http://psych.hanover.edu/research/exponnet.html
(2001).
Stimulus delivery on the Web: What can be presented when calibration isn’t possible . In U.-D. ReipsM. BosnjakEds., Dimensions of Internet science (pp. 113–130). Pabst Science.(2000).
Validity of Web-based psychological research . In M. H. BirnbaumEd., Psychological experiments on the Internet (pp. 35–60). Academic Press. https://doi.org/10.1016/B978-012099980-4/50003-4(2017). The state of Web-based research: A survey and call for inclusion in curricula. Behavior Research Methods, 49(5), 1621–1629. https://doi.org/10.3758/s13428-017-0882-x
(1999). Seeing statistics. Cengage. http://www.seeingstatistics.com/
(2000).
PsychExps: An online psychology laboratory . In M. H. BirnbaumEd., Psychological experiments on the Internet (pp. 219–233). Academic Press. https://doi.org/10.1016/b978-012099980-4/50010-1(2000).
The Web experiment method: advantages, disadvantages, and solutions . In M. H. BirnbaumEd., Psychological experiments on the Internet (pp. 89–117). Academic Press. https://doi.org/10.1016/b978-012099980-4/50005-8(2011).
Behavioral research and data collection via the Internet . In R. W. ProctorK.-P. L. VuEds., Handbook of human factors in Web design (2nd ed., pp. 563–585). Erlbaum. https://doi.org/10.1201/b10855-32(2002). WEXTOR: A Web-based tool for generating and visualizing experimental designs and procedures. Behavior Research Methods, Instruments, & Computers, 34(2), 234–240. https://doi.org/10.3758/BF03195449
(2004). Scientific LogAnalyzer: A Web-based tool for analyses of server log files in psychological research. Behavior Research Methods, Instruments & Computers, 36(2), 304–311. https://doi.org/10.3758/BF03195576
(2000).
The server side of psychology Web experiments . In M. H. BirnbaumEd., Psychological experiments on the Internet (pp. 285–310). Academic Press. https://doi.org/10.1016/b978-012099980-4/50013-7(2015). Detecting, preventing, and responding to “fraudsters” in Internet research: Ethics and tradeoffs. The Journal of Law, Medicine & Ethics, 43(1), 116–133. https://doi.org/10.1111/jlme.12200
(2007). Internet based research. American Psychological Association. https://www.apa.org/science/about/psa/2007/05/williams
(2017). Twenty years of Internet-based research at SCiP: A discussion of surviving concepts and new methodologies. Behavior Research Methods, 49(5), 1615–1620. https://doi.org/10.3758/s13428-017-0858-x
