Using Mechanical Turk to Assess the Effects of Age and Spatial Proximity on Inattentional Blindness

Few studies have used online data collection to study cognitive aging. We used a large ( N = 515) online sample to replicate the findings that inattentional blindness increases with age and with the distance of the unexpected object from the focus of attention. Critically, we assessed whether distance moderates the relationship between age and noticing. We replicated both age and distance effects, but found no age by distance interaction. These findings disconfirm a plausible explanation for age differences in noticing (restricted field of view), while for the first time highlighting the advantages and disadvantages of using Mechanical Turk for the study of cognitive aging.


Introduction
More than half (59%) of individuals 65 years of age and older in the United States report using the Internet [1], meaning that the Internet provides a new way to conduct studies of cognitive aging. Conducting aging research online would allow for larger and more heterogeneous samples, faster data collection, and reduced costs [2]. Extrapolating from a recent sample of Amazon Mechanical Turk (MTurk) workers in the United States [3], we estimate that 32,025 of the estimated 381,250 American workers on MTurk are aged 55 and above, and more than 5,337 are aged 65 and above. With larger samples, age could be treated as a continuous rather than categorical variable, and age effects could be fitted to more complex functions. MTurk has already been validated as a means to collect behavioral [4] and clinical data [5] but no study has assessed the usefulness of MTurk as a way to conduct aging research.
We used MTurk to replicate and extend a previously reported age difference in inattentional blindness. Inattentional blindness refers to the failure to notice an unexpected object when attention is directed elsewhere [6]. Although a number of studies have explored whether stable individual differences predict inattentional blindness susceptibility (e.g., [7]), only one published study examined age-related differences in noticing [8]. Consistent with the original study [9], 40% of younger adults failed to notice a person in a gorilla suit walking through a ball game. Yet, close to 90% of older adults failed to notice the gorilla [8].
In addition, our study replicated the effect of distance from fixation on inattentional blindness: inattentional blindness is higher for unexpected objects appearing further from the focus of attention [6,10,11]. For instance, when counting the number of times one set of objects crossed a line while ignoring other objects, participants were less likely to notice unexpected objects that appeared further from the line [10].
Given that older adults tend to have a more restricted breadth of attention than younger adults [12], and younger adults are typically less likely to notice unexpected objects that appear further from the focus of attention [10], we predicted that the effect of distance of the unexpected object from the focus of attention would be even greater for older adults. If age moderates the effect of distance on inattentional blindness, then age differences in inattentional blindness might result at least in part from reduced attention breadth in older participants. Alternatively, if the relationship between distance and inattentional blindness is unaffected by age, then other mechanisms likely are responsible for the increased inattentional blindness rates for older participants.

Materials and Methods
This study was pre-registered at the Open Science Framework (https://osf.io/6hkx8/).

Participants
Turk participants were recruited using the following text: "Participants are needed for a study on visual attention. In the study, participants will be asked to complete

ORIGINAL RESEARCH REPORT
Using Mechanical Turk to Assess the Effects of Age and Spatial Proximity on Inattentional Blindness a short visual attention task and answer a number of questions. This study will take up to 10 minutes to complete and participants will receive $0.25 for completing it. At the end of the study, participants will receive a completion code to paste into the box below to receive payment for participating. You may only complete this study once." In order to calculate more precise estimates of the noticing rates as a function of age and distance, we aimed for a sample size of 500. Therefore, we continuously collected data until we reached a minimum of 500 participants, excluding those who did not meet the following inclusion criteria: (a) from the United States (screened before data collection), (b) reported having normal or corrected-to-normal vision (n = 77 excluded), (c) participated only one time (determined by matching MTurk IDs between observations; we included data from the first time a person participated, and excluded data from 55 respondents who had previously participated), (d) reported no problems with the appearance or presentation of the task (participants were asked if the task appeared to function properly and two coders, who were blind to the condition assignment or other responses of the participants, independently judged whether or not they should be excluded based on their response; n = 33), (e) correctly answered an attention check item (remember the middle of 5 integers and report it correctly on the next screen; n = 60), (f) showed no inconsistencies in self-reported age across questions (i.e., their self-reported age in years was within one year of what we calculated based on their subsequently reported birth year; n = 20), or (g) at least one letter crossed the line on the critical trial (n = 2). In total, 650 participants were tested and data from 135 were excluded for one or more of these reasons (see Table 1 for demographics on both excluded and included participants). Importantly, those who were included (M = 36.24, SD = 12.37, Mdn = 34.00, IQR = 18.00) and those who were not (M = 35.83, SD = 13.46, Mdn = 32.00, IQR = 21.00) did not significantly differ in age, t(647) = 0.378, M difference = 0.013, 95% CI [-0.052, 0.77] (log age was used to correct for non-normality). Of the remaining 515 participants, 311 were female (Mean Age = 36.88, SD = 13.18) and 204 were male (Mean Age = 35.25, SD = 10.97). The mean age of the sample was 36.24 years (SD = 12.37, Median = 34, IQR = 18, Min = 18, Max = 75). All participants were paid $0.25 for completing the study. This study was approved by the Institutional Review Board at Florida State University, with a waiver of the requirement for signed consent due to the anonymous nature of the study (participants read a consent screen before participating).

Materials and Procedure
In the sustained inattentional blindness task (based on [10]; see Figure 1), participants were presented with four L block letters (2 white, 2 black) and four T block letters (2 white, 2 black), with each letter having a width and height of 43px and a thickness of 11px. During a single trial, these letters moved independently about a 666px by 546px gray (#777777) window at speeds ranging from approximately 90px to 180px per second, bouncing-off of the window edges whenever they came in contact. Both the speed and direction of each letter changed randomly every 1 to 4 seconds, making their trajectories difficult to predict. Each object was assigned a unique, fixed change interval so all letters changed speed and direction at different times throughout each trial. A blue (#0000FF) 2pxthick line spanned the window horizontally at its vertical midpoint, and an 11px by 11px blue fixation square was positioned at its midpoint.
During each of the six 15s trials, participants were asked to maintain fixation on the blue square and to count the number of times the white-colored letters completely crossed the blue line. After each trial, they entered their total count. Five seconds into the final trial, a light gray (#B2B2B2) cross unexpectedly entered the window on the right, traveled horizontally across the display window at a rate of 90px/second, and exited on the left. The cross  also had a height and width of 43px and a line thickness of 11px. It traveled on 1 of 7 possible horizontal paths: on the blue line, 103px above or below the blue line, 206px above or below the blue line, or 254px above or below the blue line. The vertical position of the cross was chosen randomly for each participant. After completing the critical trial, participants were asked, "on that last trial of the task, did you notice anything that was not there on previous trials?" Regardless of their answer, they then were asked forced-response questions about the color, shape, and motion of the unexpected object. Participants were classified as having noticed the unexpected object if they answered that they had noticed something new on the critical trial and were able to correctly answer one of the forced-choice questions about the cross's appearance. Finally, participants answered a set of demographic questions and reported whether they noticed any obvious problems with the quality or appearance of the displays.

Pre-Registered Data Analyses
Using a logistic regression, we predicted noticing from age, distance, and the interaction between age and distance. The model also included as predictors the critical trial error rate to control for performance of the primary task, the mean frame rate to control for the speed of object movement, and the available vertical monitor resolution to control for differences in the amount of available monitor space the task could use. We controlled for error rates on the primary task because differences in the ability to perform the primary task accurately might be confounded with age. In addition, we wanted to assess age-related differences in susceptibility to inattentional blindness that are not explained by differences in primary task performance; after accounting for primary task performance, do inattentional blindness rates differ? The error rate was calculated by taking the absolute difference between the participant's line-crossing count and the actual number of line-crosses made on the critical trial and dividing that difference by the actual number of line-crosses made on the critical trial.

Logistic Regression
The results from the logistic regression analysis can be found in Table 2 and Figure 2. Overall, 49% [95% confidence interval: 44%, 53%] of participants reported noticing the unexpected object. Replicating earlier results from a laboratory study [10], the further away from the attended line the unexpected object traveled, the less   Table 2: Logistic regression predicting inattentional blindness from participant age and the distance of the unexpected object from the attended line. Note. Hosmer-Lemeshow R 2 = .09, Cox-Snell R 2 = .12, Nagelkerke R 2 = . 16. Model X 2 (1) = 64.27, OR = odds ratio. The odds ratio for age is so small because it represents every one year increase in age. So, although every one year increase in age is associated with only a 1 + 1 − exp(−0.0357) = 1.035 fold increase in the probability of displaying inattentional blindness, every 10 years increase in age is associated with a larger 1 + 1 − exp (−0.0357 * 10) = 1.30 fold increase. This is the same for the distance odds ratio, which represents every one pixel increase in distance from the line.

Figure 2:
Probability of Noticing the Unexpected Object. The probability of noticing the unexpected object as a function of participant age and distance from the fixation while holding mean frame rate, monitor y resolution, and the error rate on the critical trial at their means. The points at the top of the plot represent participants who noticed the object while the points at the bottom represent participants who did not notice the object. Note that these points were jittered on the y-axis in order to make their distribution across age more clear. Also consistent with previously reported age differences in inattentional blindness [8], the probability of noticing the unexpected object decreased with increasing age. Holding other variables at their means, an increase of 10 years of age was associated with a 1.30-fold [1.18, 1.40] increase in the probability of inattentional blindness.
In addition to replicating these effects of distance and age, our study tested the novel prediction that the effect of distance would increase with age; if differences in attention breadth are predictive of inattentional blindness, then we should observe an interaction between age and distance. Contrary to our prediction, we found no evidence for such an interaction-the odds ratio was 1.0, indicating no age differences in the pattern of noticing across different distances 1 . Given our large sample and the lack of evidence for an interaction, increases in inattentional blindness rates with age do not appear to result from differences in the effect of distance on noticing. This pattern suggests that age-related differences in inattentional blindness likely do not result from decreases in attention breadth with age.

Supplemental Data Collection and Analysis
Our sample contained only 22 participants over 60, and this restricted age range might have hindered the ability to detect any differential effects of age on the spatial proximity effect occurring among the elderly. To address this concern, we recruited additional participants between the ages of 18 and 35 or aged 61 or above. We did this by screening potential participants by first asking for their age and then allowing participation only by those in our desired age ranges. Our original plan was to assess the interaction between age group and distance on noticing in these new samples (see our pre-registration at https://osf.io/v4i3c/), but we were unable to recruit enough older participants for an independent analysis (total new samples: younger n = 198, older n = 25). Rather than conducting a separate analysis of these small samples, as an exploratory analysis, we added these new data to our original sample and re-ran the same analyses. Even with the additional data, we still found little evidence for an interaction between age and distance on noticing, B = 0.0001, SE = 0.0001, OR = 1.0001, 95% CI [0.9999, 1.0002].
We also conducted an additional control analysis with this larger sample to address the possibility that monitor size and resolution might vary systematically with participant age, masking an age by distance interaction. In two separate linear regressions, neither self-reported monitor size (B = 0.075, 95% CI [−0.119, 0.269], t(742) = 0.762) nor diagonal resolution (B = −0.002, 95% CI [−0.005, 0.004], t(743) = −1.093) predicted age. Consequently, differences in the types of displays used by older and younger participants are unlikely to have masked an interaction between age and distance in noticing.

Discussion
Our findings directly replicated the effect of unexpected object distance on inattentional blindness [10] and conceptually replicated the effect of age on inattentional blindness [8] using an online sample from Amazon Mechanical Turk. Inattentional blindness increased with increasing distance of the unexpected object from the focus of attention, and greater age was associated with less noticing of unexpected objects. Contrary to our predictions, increasing age was not associated with a greater effect of distance on noticing. Our finding, tested with a relatively large sample size, suggests that age-related differences in attention breadth do not play a large role in the relationship between age and inattentional blindness. Furthermore, the constant effect of distance on inattentional blindness across the age range, despite overall differences in noticing rates, means that the effect of distance on inattentional blindness is robust.
Although it could be argued that the small number of subjects aged 60+ in our sample limited our ability to detect any slope differences beyond 60 years of age, there is no reason to suspect that an interaction between distance and age would appear only beyond that age. Additionally, even when we more than doubled the number of participants aged 60+, we still found no evidence for an interaction.
Although this result increases our confidence that there is no interaction between age and distance in the population, it suggests some limitations of MTurk when it comes to collecting data from participants over the age of 60. Does this mean that MTurk is limited when it comes to conducting aging research? Not necessarily. First, aging is a continuous process and many of the cognitive decrements associated with it onset well before 60 [14]. Second, Pew Research Center's Internet and American Life Project has tracked a rapid increase in the number of older adults online, from 14% in the year 2000 to 59% in 2013, suggesting the sample of older adults using Mechanical Turk may expand as well [1].
Other limitations of using MTurk to conduct aging studies should be considered as well. An older adult MTurk sample is undoubtedly self-selected. If our older participants happened to have perceptual and cognitive abilities comparable to those of younger adults, that might explain the lack of an interaction between age and distance. However, our successful replication of an overall effect of age on inattentional blindness suggests that our older participants were not just functioning like younger participants. Moreover, the concerns about self-selection should be viewed in the broader context of aging research: older adults who participate in lab-based studies may also be self-selected and unrepresentative of the broader aging population.
Our results also extend earlier findings in a number of ways. First, our MTurk sample was more heterogeneous: Although we sampled exclusively from the United States, our participants varied in their education, race, and income. Second, our sample size was substantially larger than that used in earlier studies, meaning we could calculate more precise estimates of the noticing rates. Third, we sampled across a large age range rather than recruiting only old or young participants, allowing us to use age as a continuous rather than a categorical predictor. Finally, our results provide a better sense of the size of the effect compared to studies using extreme group designs because such designs tend to overestimate effect sizes [15,16]. Despite these implementation differences, we replicated the results of earlier studies that had used smaller and