The role of object history in establishing object correspondence

Abstract

Our visual system establishes correspondence between objects and thus enables us to perceive an object, like a car on the road, as moving continuously. A central question regarding correspondence is whether our visual system uses relatively unprocessed image-based information or further processed object-based information to establish correspondence. While it has been shown that some object-based attributes, such as perceived lightness, can influence correspondence, manipulating object-based information typically involves at least minimal changes of image-based information as well, making it difficult to clearly distinguish between the two levels. To avoid this confound, we manipulated object-based information prior to the task in which we measured correspondence. We used 3-element Ternus displays to assess correspondence. These are ambiguous apparent-motion displays that, depending on how correspondence is solved, are perceived as either one element jumping across the others or as all three elements moving together as a group. We manipulated object-based information by presenting one of two object histories prior to the Ternus display. In one, they moved or changed luminance independently, and thus appeared independent from each other. In the other, the elements moved or changed their luminance all together and thus appeared grouped with each other. We found that the object history did influence how the Ternus displays were perceived, thereby confirming that object-based information alone can be used as a basis for establishing correspondence in line with object-based theories of correspondence.

Introduction

We perceive a moving object, like a car, as a distinct entity with a continuous history based on visual information that may itself be discontinuous. To do this our visual system must establish correspondence between objects across time, i.e., represent whether an object at a current time is the same as one seen previously. If correspondence is established, then a single object is perceived. In contrast, if correspondence is not established, then separate individual objects are perceived (e.g., Dawson, 1991; Wertheimer, 1912). How the visual system establishes object correspondence, therefore, has an important impact on how the world is perceived. It could even determine whether one or two objects are represented. Beyond the direct function of determining the number and continuity of objects perceived, object correspondence is thought to support other cognitive functions such as change detection (Flombaum & Scholl, 2006), perceptual stability across eye movements (Tas, Moore, & Hollingworth, 2012), and object recognition across eye movements (Poth, Herwig, & Schneider, 2015; Poth & Schneider, 2016). Object correspondence is especially important when the scene is complex. For example, when multiple moving objects are present, like many cars at a crossroad, they may temporarily occlude each other. In such situations, which objects belong together is ambiguous because any current object could, in principle, correspond to any object from a moment ago (e.g., ambiguous apparent motion; Pikler, 1917; Ternus, 1926). This kind of ambiguity is known as the correspondence problem (Dawson, 1991; Ullmann, 1979).

A central question regarding the correspondence problem concerns at which level of processing correspondence is established. One possibility is that correspondence is image based, which means that correspondence is established on the basis of relatively unprocessed visual information that makes up the retinal image, e.g., luminance, spectral content, textures (Adelson & Bergen, 1985; Breitmeyer & Ritter, 1986a, 1986b; van Santen & Sperling, 1985; Werkhoven, Sperling, & Chubb, 1993). A set of features in one part of the retina at one time is perceived as corresponding to a similar set of features at another part of the retina at a later time, and is thereby perceived as a single object such as a car. Under this view, information about the car as an object (e.g., that it was recently occluded by another car) cannot be used for establishing correspondence because it is unavailable at an image level of processing. An alternative possibility is that correspondence is object based (Hein & Cavanagh, 2012; Hein & Moore, 2014), which means that correspondence is established on the basis of associating episodic representations of objects. Such object representations include information about the history of the object, including for example changes in appearance or position, from the time the object representation was initially established. For the moving car example this would mean that where it was previously, and how it interacted with other perceived objects in the scene, would be used in the correspondence process. If the car becomes occluded, for example, the occluded information would be maintained as part of the representation and could influence later correspondence processes even though it is no longer present in the image. The focus of this study is to differentiate between these two levels of visual processing.

There are proponents of both image-based and object-based theories of object correspondence. Motion energy models are examples of image-based theories (Adelson & Bergen, 1985; van Santen & Sperling, 1985; Werkhoven et al., 1993). In these models, simple low-level motion detectors (i.e., Reichardt detectors; Reichardt, 1961) compute the direction in which the most motion energy occurs, i.e., the greatest change of features at subsequent locations in the image is detected. Based on this outcome, motion and direction are perceived, and correspondence is determined. Evidence of such image-based mechanisms comes in part from studies using apparent motion displays (Wertheimer, 1912). In these displays, movement can be perceived between two stimuli that are presented successively and at two different positions. The perception of motion in these displays is referred to as apparent motion because the display consists of two stationary stimuli, but through correspondence processes they are together perceived as a single object moving from one location to another. According to image-based theories, the perception of apparent motion emerges from the establishment of correspondence between contrast energy at one location and time with contrast energy at another location and time. Studies that show that apparent motion between two elements is dependent on the spatiotemporal gap, i.e., the time and distance between them, are in line with these theories (e.g., Kolers, 1972; Korte, 1915).

In contrast to image-based theories of correspondence, object-based theories assert that correspondence can be established between higher-order representations. An example is the use of attentional pointers to associate image elements across time that are identified as corresponding to a single object within the scene (e.g., Hein & Cavanagh, 2012; Hein & Moore, 2014). The object representation is then updated accordingly (e.g., via an object-mediated updating process; Enns et al., 2010; Lleras & Moore, 2003; Moore & Lleras, 2005; Moore, Mordkoff, & Enns, 2007). Evidence of object-based correspondence comes in part from ambiguous apparent-motion displays such as the Ternus display (Pikler, 1917; Ternus, 1926). In the classic Ternus display (see Fig. 1), three identical stimuli (e.g., black discs) are presented next to each other with uniform separation. From the first frame to the second, the three stimuli are presented in locations that are shifted by one element position. Depending on how correspondence is solved between these two stimulus frames, motion in the Ternus display is perceived either as one element jumping across the other two elements (element motion) or as all elements moving together (group motion). These different percepts reflect different, mutually exclusive, object correspondences. And consistent with object-based theories of correspondence, higher-order information – as for example the motion context or perceived lightness, but also semantic and lexical information – can influence which type of motion, element or group, is perceived (Aydın, Herzog, & Öğmen, 2011; Chen & Zhou, 2011; He & Nakayama, 1994; He & Ooi, 1999; Hein & Moore, 2014; Hsu, Taylor, & Pratt, 2015; Ramachandran & Anstis, 1983; Yu, 2000). Hein and Moore (2014), for example, showed that the match between the perceived lightness of the stimuli in the Ternus display, rather than their physical luminance, determined whether element or group motion was perceived, indicating that correspondence was based on the higher-order attribute of lightness not the image-level attribute of luminance.

Fig. 1

Ternus display and the two alternative motion percepts. (a) Ternus display. The two successively presented Ternus frames are separated by a variable interstimulus interval (ISI). (b) Group motion. Correspondence between elements compatible with the group motion percept. (c) Element motion. Correspondence solution compatible with the element motion percept

While the studies cited above are consistent with an object-level correspondence process, a challenge to testing between object-based and image-based theories of correspondence is that object representations are initially defined on the basis of information in the image. Therefore, manipulating information at the object level usually involves image-level changes to displays. While conditions can be established such that local parts of the image are identical while object-level aspects differ, these strategies require different image-level information as context, such as different sets of transparent filters (Hein & Moore, 2014) or occluding surfaces (Hein & Moore, 2014; Moore et al., 2007). An alternative strategy that we take in the current study is to define object structure on the basis of spatiotemporal history, and measure how correspondence is established when displays are physically identical but have different object histories. Using this strategy, the possibility that any differences that are observed were driven by differences in image-level context can be ruled out because there will be no differences in image-level context at the time that correspondence is measured. To the best of our knowledge this is the first time this approach is used to investigate the influence of object-based information on object correspondence.

The aim of this study was to distinguish between theories that maintain that the correspondence process is entirely image based and theories that maintain that it can be object based. We measured perceived Ternus motion (i.e., element vs. group motion) as a measure of correspondence, and manipulated the object history of the stimuli used in the Ternus displays. This provided a measure of correspondence under different object conditions, but identical image conditions. Following previous studies (Moore & Lleras, 2005; Mordkoff & Danek, 2011), we manipulated object history by presenting a short movie prior to the Ternus display that showed the Ternus stimuli as either spatiotemporally grouped together (see Fig. 2 common history, e.g., movie 1), or spatiotemporally independent from each other (see Fig. 2 separate history, e.g., movie 2). Any difference in how the Ternus displays are perceived following these two different types of history would indicate that object-based information played a role in correspondence, consistent with object-based theories and inconsistent with pure image-based theories. If, however, perceived Ternus motion is the same across the two different object-history conditions, then it would indicate that object history was insufficient to drive correspondence, consistent with image-based theories.

Fig. 2

Overview experiments. Each trial consisted of two phases. In phase 1, the object history was manipulated in order to change the representations of the Ternus elements as being spatiotemporally grouped together (common history) or independent from each other (separate history). Examples of the type of movies used in this phase can be found online (movies ESM 1-6). In phase 2, after the object history, the Ternus display was presented and participants were asked to indicate whether they perceived element or group motion

Experiment 1: Continuous motion object history

Each trial in Experiment 1 began with a short movie in which the three stimuli from a Ternus display were shown in smooth random motion. In the common history condition, the elements moved together along a single random trajectory, whereas in the separate history condition, the three elements moved independently along different random motion trajectories (see Fig. 2 Experiment 1 and movies 1 and 2). The movie was then followed by a standard Ternus display, which was identical across the two history conditions. On the basis of the principle of grouping-by-common-fate (Wagemans et al., 2012; Wertheimer, 1923), the stimuli in the common history condition should be represented as a single group, whereas in the separate history condition, they should be represented as separate entities. If more group motion is perceived following common history movies than separate history movies, it will indicate that object-level information influenced correspondence.

Method

Participants

Thirty-nine individuals from the University of Tübingen and the surrounding community participated in the experiment, and received 8 €/h or course credit in compensation for their time. One participant was excluded because he/she had participated in a similar experiment and was therefore not naïve as to the purpose of this experiment. Two others were excluded because of technical errors during data collection. A total of 36 participants (27 female, nine male; mean age = 25.4 years, range: 19–56), therefore, contributed to the reported data. All were naïve as to the purpose of the experiment and reported normal or corrected-to-normal vision.

Apparatus

The experiment was controlled by a PC running Windows XP, driving a 17-in. color cathode ray tube monitor (1,024 x 768 pixel) with a refresh rate of 100 Hz. MATLAB software (Version R2012a, 7.14, Mathworks Inc., MA, USA) with Psychtoolbox 3 extensions (Brainard, 1997; Kleiner, Brainard, & Pelli, 2007; Pelli, 1997) was used to run the experiment. The experiment was conducted in a dimly lit individual testing room with a fixed viewing distance of 65 cm to the monitor.

Stimuli

A fixation cross was presented at the center of the screen, which had a line width of 0.08° and measured 0.2° x 0.2°. The Ternus display (Pikler, 1917; Ternus, 1926) consisted of two frames with three elements aligned next to each other, each with a diameter of 1.6°. The center-to-center distance between the elements was 2°. The elements were presented 2° above the fixation cross, the middle of the center elements being vertically aligned with it. The color of the Ternus elements and the fixation cross was black (RGB: 0, 0, 0; 0 cd/m²). They were presented on a gray background (RGB: 127.5, 127.5, 127.5; 24 cd/m²). For the motion history identical elements were used to those in the Ternus display.

Continuous motion object history

To create the motion history of the elements, a sequence of five element positions was chosen. The starting position of the elements in the motion history was always the same as for the first Ternus frame. From there the elements moved away for five positions and then moved back taking the same positions in reversed order until the motion history ended again on the start position of the Ternus display. For each trial a new random motion sequence was calculated using a new set of five positions. Each element position of the motion sequence was calculated as follows: the x-value of an element position was calculated by multiplying the sine (in degrees) of a randomly chosen degree value (range: 0–359°) by a randomly chosen radius value (range: 30–60 pixels; element diameter: 58 pixels). For the y-value of this element position the cosine of the same degree was multiplied with the same radius used for the x-value. In the common history condition the same motion sequence was used for all three elements, which started and finished at the same time. For the separate history condition separate motion sequences were calculated for each element. The elements started their movement one after another in random order and finished in reversed order. To realize the sequential start of the elements in the separate history condition, while at the same time having the same movie duration in both conditions, two element presentations without a position change were added to the common history condition. The entire object history sequence lasted for 2,400 ms.

Task

During the presentation of the object history movie, participants only had to watch the history. For the following Ternus display participants had to report whether they perceived all elements in the Ternus display as moving together (group motion) or as one element jumping across the others (element motion). They gave their response by pressing the “j” or “f” key, respectively.

Procedure

The experiment lasted about 60 min and began with informing participants about the experimental procedure. After they gave informed consent according to the ethical principles of the World Medical Association (2013; Declaration of Helsinki), the experiment started with written instructions on the screen followed by clear demonstrations of group and element motion (using the most extreme interstimulus intervals (ISIs) tested of 0 and 160 ms) and a practice trial block of 12 trials. After asking questions if something about their task remained unclear participants completed 12 experimental blocks of 24 trials each. Each trial began with the presentation of the first frame of the three Ternus elements and the fixation cross for 200 ms. The fixation cross remained on the screen during the entire trial and participants were asked to fixate it. Then the motion history movie was presented (2,400 ms). After the motion history the fixation cross was presented alone for another 500 ms (short-pause condition) or 1,200 ms (long-pause condition) to test whether the time between the movie and the Ternus display influences the processing of the history. The pause was followed by the first Ternus frame, which was presented for 200 ms. After a variable ISI (0, 10, 20, 40, 80, or 160 ms), during which again only the fixation cross was presented, the second Ternus frame was shown for another 200 ms. The fixation cross stayed on the screen until the participants responded. The next trial started after 800 ms.

Design

A 6 (ISI: 0, 10, 20, 40, 80, 160 ms) x 2 (history: common, separate) x 2 (pause: short, long) within-subject design was used in which all factors were counterbalanced and randomly mixed within all trials. Each participant completed 288 trials, resulting in 12 observations per condition.

Results and discussion

Figure 3a shows mean group motion responses as a function of object history and ISI for Experiment 1. Trials with responses other than the two response keys were eliminated (0.85%). In addition, all trials with response times (RTs) longer than 8,000 ms were excluded (0.06% ; mean RT: 1,046 ms). We performed a 6 x 2 x 2 repeated-measures ANOVA on the subject means with the factors ISI, object history and pause on the mean group motion responses. Effect sizes are reported throughout in terms of adjusted partial eta-squared ($adj\ {\hat{\eta}}_p^2$), which is an estimate of partial eta-squared that adjusts for known bias (Mordkoff, 2019). As expected, mean group motion responses increased with increasing ISI, F(5,175) = 115.57, p < .001, $adj\ {\hat{\eta}}_p^2$ = .76, replicating the typical ISI effect found for the Ternus display in many studies (e.g., Breitmeyer & Ritter, 1986a; Petersik & Pantle, 1979). The pause showed no significant effect, F(1,35) = 0.86, p = .361, $adj\ {\hat{\eta}}_p^2$ < .01, but there was a significant interaction between pause and ISI, F(5,175) =3.86, p = .002, $adj\ {\hat{\eta}}_p^2$= .07. To examine this interaction further, we conducted Holm’s -corrected paired t-tests for the pause condition for each ISI condition separately. They showed that at an ISI of 20 ms more group motion was perceived in the long- than in the short-pause condition (85% vs. 80% , t(35) = 2.98, p_holm = .031, $adj\ {\hat{\eta}}_p^2$=.18). All other comparisons were not significant, 1.93 <= t <= 2.38, p_holm >= .116, $adj\ {\hat{\eta}}_p^2$<= -.11. Most importantly, however, we found that object history had no significant effect on the motion percept, F(1,35) = 0.70, p = .409, $adj\ {\hat{\eta}}_p^2$= -.01, and there was also no significant interaction of object history with any of the other factors, F_s <= 0.97, p_s >= .436, $adj\ {\hat{\eta}}_p^2$<= -.01.

Fig. 3

Mean group motion responses as a function of interstimulus interval (ISI) and object history for all three experiments (N = 36 in each). (a) Experiment 1: Continuous motion object history. (b) Experiment 2: Apparent motion object history. (c) Experiment 3: Luminance flicker object history. Standard errors represent within-subject SEs after Cousineau-Morey (Cousineau, 2005; Morey, 2008)

The main result was that there was no influence of object history on perceived Ternus motion, which is consistent with pure image-based theories of correspondence. We manipulated object history in Experiment 1 using smooth continuous motion to establish (un)grouping by (un)common fate. Previous experiments that manipulated object history, however, used apparent motion (Moore & Lleras, 2005; Mordkoff & Danek, 2011). Because Ternus motion (group or element) is itself apparent motion, it is possible that the Ternus display was not perceived as related to the object-history part of the display, and information from it was therefore not used for resolving correspondence in the Ternus display. This possibility was examined in Experiment 2.

Experiment 2: Apparent motion object history

Experiment 2 was analogous to Experiment 1, except that object history was manipulated using apparent motion instead of smooth motion by including a blank screen between the stimulus frames (see Fig. 2 Experiment 2 and movies 3 and 4). This change made the quality of motion in the object-history movies more similar to the quality of motion in the Ternus displays. If more group motion is perceived following common history movies than following separate history movies in Experiment 2, then it would suggest that the mismatch of motion quality across motion history and Ternus displays in Experiment 1 lead to object information not carrying over to the Ternus displays for use in the correspondence process. If, however, there is no difference in perceived group versus element motion as a function of object history, it would reinforce the possibility that correspondence is a purely image-based process.

Following the main experiment, we asked participants three short questions in order to assess the extent of the participants’ understanding of the objective of the experiment. We used the questionnaire, because of the concern that participants might strategically base their responses “group” versus “element” on the history part of the display, rather than only on the Ternus display as instructed. This possibility would seem to be more likely for participants who were aware of the objective than those who were not. We therefore sought to identify those subjects in order to be able to have an independent measure of this potential influence.

Method

Participants

Thirty-seven individuals participated in Experiment 2, and received 8 €/h or course credit in compensation for their time. One participant was excluded because he/she had participated in a similar experiment and was therefore not naïve as to the purpose of this one. A total of 36 participants (26 female, ten male; mean age = 24.5 years, range: 18–47), therefore, contributed to the reported data. All were naïve as to the purpose of the experiment and reported normal or corrected-to-normal vision.

Apparatus

The same apparatus was used as in Experiment 1.

Stimuli

With the following exceptions the stimuli used in this experiment were the same as in Experiment 1: The fixation cross had a line width of 0.05° and was 0.15° x 0.15° in size. The black Ternus elements were surrounded by a small white (RGB: 255, 255, 255; 150 cd/m²) outline (linewidth: 1 pixel), in order to make sure that the individual elements were easily perceived even when they overlapped with each other. The fixation cross was first presented in white during the object history and then changed its color to green (RGB: 100, 255, 60; 103 cd/m²) with the start of the Ternus display. The object history elements looked the same as the Ternus elements.

Apparent motion object history

The element positions for the motion sequence of the object-history elements were calculated in the same way as for Experiment 1, but the radius was increased compared to Experiment 1 and randomly chosen between 60 and 100 pixels. Additionally, a blank screen divided each presentation of the object history elements, which was presented for the same duration as the ISI used for the Ternus display in each trial. To keep the duration of the motion history similar to Experiment 1, a motion sequence of only six element positions was used (total object history time: 1,400–2,520 ms).

Task

The task was the same as in Experiment 1.

Questionnaire

The following three questions were used to get a more and more precise idea of what the participants understood of the purpose of the experiment: Question 1: “What do you think this experiment was about?”; Question 2: “Do you have any idea why you have seen different movie sequences before the actual task? If yes, what is your idea?”; Question 3: “In your opinion, what was the author’s hypothesis?”. To analyze the questionnaire, one of the authors and a undergraduate research assistant rated the answers independently in the following way: for each answer to one of the three questions the raters gave a score of either 0 (no idea), 1 (a vague idea, but incorrect), or 2 (correct). As a measure for the agreement between both raters we used Cohen's (1968) weighted kappa (weight: squared). We used the average across both raters of the sum of the scores (ranging between 0 and 6) for each question as the final experimental understanding score of the participant.

Procedure

The procedure was the same as for Experiment 1 apart from the following exceptions: The practice trial block consisted of 24 trials and participants completed ten experimental blocks of 24 trials. The trial sequence in Experiment 2 was similar to Experiment 1, but between the object history and the start of the Ternus display no different pause conditions were used, the time interval being the same as the ISI. To make the start of the Ternus display sequence easily recognizable, the color of the fixation cross changed from white to green simultaneously with the start of the Ternus display. In addition, we introduced two different motion directions of the Ternus display to make the Ternus direction unpredictable. The second frame of the Ternus display could therefore be presented to the right or the left of the first frame. At the end of the main experiment the questions of the questionnaire were presented on the screen. Participants responded to each question one after the other by typing their answer on the keyboard.

Design

A 6 (ISI: 0, 10, 20, 40, 80, 160 ms) x 2 (history: common, separate) x 2 (Ternus direction: left, right) within-subject design was used in which all factors were counterbalanced and randomly mixed within all trials. Each participant completed 240 trials, resulting in ten observations per condition.

Results and discussion

Ternus percept. Figure 3b shows mean group motion responses as a function of object history and ISI for Experiment 2. As in Experiment 1, trials with responses other than the two response keys were eliminated (0.75%). In addition, all trials with RTs larger than 8,000 ms were also excluded (0.02% , mean RT 1,134 ms). We performed a 6 x 2 x 2 repeated-measures ANOVA on the subject means with the factors ISI, object history and Ternus direction on the mean group motion responses. We found no main effect of Ternus direction, F(1,35) = 0.06, p = .816, $adj\ {\hat{\eta}}_p^2$= -.03. Ternus direction also did not interact with any of the other factors and the three-way interaction was also not significant, F_s <= 0.77, p_s >= .413, $adj\ {\hat{\eta}}_p^2$<= -.01. As expected, there was a significant effect of ISI, F(5,175) = 47.94, p < .001, $adj\ {\hat{\eta}}_p^2$=.57, as mean group motion responses increased with increasing ISI. Most importantly and in contrast to Experiment 1, we found a significant effect of object history, F(1,35) = 12.77, p = .001, $adj\ {\hat{\eta}}_p^2$=.25, with more group motion responses in the common object history condition compared to the separate object history condition. We also found an interaction of object history and ISI, F(5,175) = 3.55, p = .013, $adj\ {\hat{\eta}}_p^2$=.07. To examine this interaction further, we conducted Holm’s-corrected t-tests for object history and each ISI condition separately. These post hoc analyses revealed that the object history effect was reliable for all ISI conditions, including the 0 ms condition, t(35) >= 2.33, p_holm <= .026, $adj\ {\hat{\eta}}_p^2$<=.12.

A subset of eight participants showed no effect of the ISI on reports of Ternus motion (collapsed across both object history conditions). These participants may have used a strategy in which they based their responses on the history part of the display, which is independent of ISI, rather than on the Ternus display. Such a strategy would be especially problematic as it concerns our question because it would yield data consistent with the object history influencing the perception of Ternus motion, when in fact the responses are unrelated to the perception of Ternus motion. We therefore repeated our analyses excluding this subset of participants, and found a very similar pattern to the original analyses. Figure 4b shows the mean data for Experiment 2, excluding the eight subjects who showed no effect of ISI. Allthough the difference between the two object conditions was reduced, there was still a main effect of object history, F(1,27) = 6.06, p = .021, $adj\ {\hat{\eta}}_p^2$=.15. The main effect of ISI was significant, not surprisingly given that inclusion in this analysis was contingent on this, F(5,135) = 78.02, p < .001, $adj\ {\hat{\eta}}_p^2$=.73. In addition, the interaction between object history and ISI was significant, F(5,135) = 5.26, p = .001, $adj\ {\hat{\eta}}_p^2$=.13. No other reliable effects were observed, F_s <= 1.10, p_s >= .358, $adj\ {\hat{\eta}}_p^2$<=.004. Separate Holm’s-corrected t-tests for each ISI condition showed that the object history effect was only significant for the 10-ms condition, t(27) = 3.67, p_holm = .006, $adj\ {\hat{\eta}}_p^2$=.31, the condition, in which the Ternus motion is most ambiguous and therefore presumably most susceptible to be influenced from other sources of information such as object history instead of the ISI. No other comparisons were significant, t_s(27) <= 2.38, p_s;holm >= .124, $adj\ {\hat{\eta}}_p^2$<=.14. For comparison, Fig. 4a shows the analagous data for Experiment 1, with three participants who showed no effect of ISI excluded.

Fig. 4

Mean group motion responses as a function of ISI and object history for all three experiments for a subset of participants. Participants that showed no effect of ISI across both object history conditions were excluded from the data set for these graphs. (a) Experiment 1: Continuous motion object history (three participants excluded). (b) Experiment 2: Apparent motion object history (eight participants excluded). (c) Experiment 3: Luminance flicker object history (eight participants excluded). Standard errors represent within-subject SEs after Cousineau-Morey (Cousineau, 2005; Morey, 2008)

Questionnaire

We calculated the difference in mean group motion responses between the common and the separate history condition for each participant and compared this value with the experimental understanding score from the questionnaire. As can be seen in Fig. 5a, participants’ understanding of the experiment varied strongly, ranging from 0 to 6 with a mean of 3.17 and a standard deviation of 2.17. The agreement between raters across all questions was very good, K = 0.81, and much greater than would be expected by chance, Z = 5.38, p < .001 (Question 1: K = 0.84, Z = 5.18, p < .001; Question 2: K = 0.73, Z = 4.99, p < .001; Question 3: K = 0.73, Z = 5.01, p < .001). If all participants are included the size of the history effect is negatively correlated with the experimental understanding (gray regression line), as participants who had a better understanding of the experiment showed smaller history effects, Kendall’s tau = -0.27, z = -2.20, p = .028. This significant correlation, however, depends on the inclusion of the eight participants who showed no ISI effect (green circles) than the rest of the participants. If only the participants that showed an ISI effect were included (purple circles), we did not find evidence that the history effect was related to the experimental understanding of the participants (purple regression line), as the same history effects were found for all levels of experimental understanding, Kendall’s tau = -0.04, z = 0.30, p = .762.

Fig. 5

Comparison of the size of the object history effect (difference of group motion responses in the common versus separate history condition; higher values indicate larger object history effect) with the experimental understanding score of the participant (higher values indicate more understanding). Purple (darker) circles represent the participants that showed ISI effects and green (lighter) circles represent participants that showed no ISI effect. The gray (light) regression line is based on all participants (purple and green circles together), the green (lightest) regression line is based only on the subset of participants that showed no ISI effect, and the purple (dark) regression line represents the subset of participants that showed an ISI effect and that were included in the second analysis. Regression lines for illustrative purposes only. (a) Experiment 2: Apparent motion object history. (b) Experiment 3: Luminance flicker object history

Discussion

In contrast to Experiment 1, participants perceived more group motion in the common compared to the separate history condition. The analysis of the questionnaire revealed that with the subjects who showed no ISI effect included, the size of the history effect was related to the understanding of the experiment, albeit in the opposite direction of what we expected, as participants with larger knowledge of the experiment showed smaller history effects. Nevertheless, this finding is consistent with the concern that participants’ understanding might affect the size of the history effect, therefore highlighting the importance of the separate analyses we conducted. For the subset of participants who showed the usual ISI effect, there was no evidence that the understanding of the experiment had an effect on the size of the history effect, suggesting that for this subset of participants at least the history effect is not likely due to strategic responses biased by the object-history movie. This suggests that the object history can affect how correspondence is solved, if the appropriate, i.e., similar enough, motion information is given during the history.

Experiment 3: Luminance flicker object history

In Experiment 2 we used apparent motion to manipulate object history and found an influence of it on perceived motion in the Ternus display. Both the dependent and the independent variables therefore involved motion. Given this, a possible concern is that participants may have strategically based their response to the Ternus display on the type of motion that they perceived in the history displays, especially when they were unsure about their percept in the Ternus display. Consistent with this idea, the questionnaire based on all participants, including the eight participants without any ISI effect, revealed a correlation between the experimental understanding and the size of the history effect (even though the correlation disappeared for the subset of participants with ISI effect alone). Thus, the concern remains that the results of Experiment 2 might have at least in part been affected by strategic responses due to the similarity between the motion during the object-history movie sequence and the subsequent Ternus motion. We therefore conducted a third experiment in which we replaced the object-motion history with an object-luminance history (grouping by common luminance changes; Sekuler & Bennett, 2001). During the history part of the trial, stimuli were stationary but changed in luminance (see Fig. 2 Experiment 3 and movies 5 and 6), either all together (common history) or independently (separate history). Otherwise, the logic was identical to that of Experiment 2. If the results of Experiment 2 were driven by the matched motion-type decision strategy described above, then perceived Ternus motion should not depend on the object history in Experiment 3. If, however, they were driven by differences in the perceived object organization that was established by object history, then we should observe the same pattern of results in Experiment 3 as we did in Experiment 2.

Method

Participants

Thirty-seven individuals participated in Experiment 3, and received 8 €/h or course credit in compensation for their time. One participant was excluded because he/she had accidentally participated in a similar experiment and was therefore not naïve as to the purpose of this one. A total of 36 participants (24 female, 12 male; mean age = 23.4 years, range: 18–47), therefore, contributed to the reported data. All were naïve as to the purpose of the experiment and reported normal or corrected-to-normal vision.

Apparatus

The same apparatus was used as in the previous two experiments.

Stimuli

The same stimuli were used as in Experiment 2 except that the Ternus elements had no white contours.

Luminance flicker object history

For the luminance object history used in this experiment the elements were presented stationary at the later start position of the first Ternus frame. To create the luminance flicker the color of the object history elements changed between black (RGB: 0, 0, 0; 0 cd/m²), light gray (RGB: 200, 200, 200; 82 cd/m²), and dark gray (RGB: 80, 80, 80; 6 cd/m²). For the common history condition all elements had the same luminance change at the same time in the following sequence: black – light gray – dark gray – black – light gray – dark gray – black. For the separate history the elements started their luminance change one after the other in a random order. Each element passed the following sequence: light gray – dark gray – black – light gray – dark gray. In order to maintain the similarity in appearance across the history and Ternus parts we created a flicker in the luminance history by presenting a blanc frame between each element frame with the same duration as the ISI in the Ternus display in each trial. In total the luminance history lasted about 1,400-2,520 ms and included seven presentations of luminance changes and ISIs.

Task

Participants had the same task as in the other two experiments. At the end of the experiment they were asked to answer the same questionnaire as in Experiment 2.

Procedure

The procedure was the same as for Experiment 2. Apart from the type of motion history the trial sequence was the same as in Experiment 2.

Design

As in Experiment 2, a 6 (ISI: 0, 10, 20, 40, 80, 160 ms) x 2 (history: common, separate) x 2 (motion direction: left, right) within-subject design was used, in which all factors were counterbalanced and randomly mixed within all trials. Again, each participant completed 240 trials, resulting in ten observations per condition.

Results and discussion

Ternus percept

Figure 3c shows mean group motion responses as a function of object history and ISI for Experiment 3. As before, trials with responses other than the two response keys were eliminated (1.94%), as well as all trials with RTs larger than 8,000 ms (0.02% , mean RT 1,193 ms). On this data set we performed a 6 x 2 x 2 repeated-measures ANOVA with the factors ISI, object history and Ternus direction on the mean group motion responses. There was no significant effect of the Ternus direction, F(1,35) = 1.29, p = .264, $adj\ {\hat{\eta}}_p^2$=.01, but a significant interaction with ISI, F(5,175) = 2.51, p = .032, $adj\ {\hat{\eta}}_p^2$=.04. This interaction was due to the ISI of 10 ms, in which the largest difference between the Ternus direction from right to left compared to the other direction was found (6.45%), but this difference did not reach significance in a Holm’s-corrected post hoc comparison, t(35) = 2.15, p_holm = .231 (p_uncorrected = .038), $adj\ {\hat{\eta}}_p^2$=.09. All other comparisons were not significant, -1.67 >= t_s <= 1.32, p_s;holm >= .517, $adj\ {\hat{\eta}}_p^2$<=.05 (direction differences ranging from -2.5% for the ISI of 160 ms to 5.5% for the ISI of 20 ms). No other factor interacted significantly with the Ternus direction, F_s <= 0.89, p_s >= .472, $adj\ {\hat{\eta}}_p^2$<=-.01. As in the previous experiments we found a significant effect of ISI, F(5,175) = 48.72, p < .001, $adj\ {\hat{\eta}}_p^2$=.57. In addition, and most importantly, we found again a significant effect of object history, F(1,35) = 22.96, p < .001, $adj\ {\hat{\eta}}_p^2$=.38, as well as a significant interaction of both factors, F(5,175) = 3.62, p = .008, $adj\ {\hat{\eta}}_p^2$=.07. As in the previous experiment we examined this interaction further using Holm’s-corrected t-tests for each ISI condition separately. These analysis showed that the comparisons were significant for all ISIs, t_s >= 3.40, p_s;holm <= .003, $adj\ {\hat{\eta}}_p^2$>=.23.

As in Experiment 2, we repeated the analysis excluding the subset of participants who showed no effect of ISI (N=8; Fig. 4c). The general pattern of results was the same as before, with the exception that the interaction between the ISI and direction was not reliable F(5,135) = 2.12, p = .067, $adj\ {\hat{\eta}}_p^2$=.04 (largest direction difference of 6.7% was again found for the ISI of 10 ms, t(27) = 1.77, p_holm=.528, (p_uncorrected=.088), $adj\ {\hat{\eta}}_p^2$=.07. Most relevant to our question, the main effect of object history was still significant, F(1,27) = 26.24, p < .001, $adj\ {\hat{\eta}}_p^2$=.47. Of course, given that the analysis was contingent on it, the main effect of ISI was also significant, F(5,135) = 78.78, p < .001, $adj\ {\hat{\eta}}_p^2$=.74. And the interaction between these two factors was significant, F(5,135) = 4.09, p = .002, $adj\ {\hat{\eta}}_p^2$=.10. Holm’s-corrected post hoc comparisons for the history effect at each ISI level showed that with the exception of the 80-ms ISI all comparisons were significant, t_s >=2.59, p_s;holm <= .030, $adj\ {\hat{\eta}}_p^2$>=.14. The object history effect at the ISI of 80 ms showed a trend, t(27) = 1.90, p_holm = .069,$adj\ {\hat{\eta}}_p^2$=.07. No other significant effects were found, F_s <= 1.13, p_s >= .333, $adj\ {\hat{\eta}}_p^2$<=.01.

Questionnaire

As in the previous experiment we calculated the difference in mean group motion responses between the common and the separate history condition for each participant and compared this value with the experimental understanding score. As can be seen in Fig. 5b, participants’ understanding of the experiment varied again strongly, ranging from 0 to 6 with a mean of 3.03 and a standard deviation of 2.00. The agreement between raters across all questions was substantial, K = 0.61, and greater than would be expected by chance, Z = 4.71, p < .001 (Question 1: K = 0.52, Z = 3.95, p < .001; Question 2: K = 0.61, Z = 4.24, p < .001; Question 3: K = 0.58, Z = 4.32, p < .001). Furthermore, small and large history effects were found for each level of experimental understanding and there was no evidence for a correlation between both factors. This is true for all participants, including those eight participants that showed no ISI effect (green circles), Kendall’s tau = - 0.06, z = - 0.48, p = .631, as well as the subset of participants who showed an ISI effect (purple circles), Kendall’s tau = 0.14, z = 1.04, p = .300.

Discussion

As in Experiment 2, we found an effect of object history with more group motion reported in the common compared to the separate object history condition. In contrast to Experiment 2, even with all participants included, there was no evidence that the size of the object-history effect was related to the understanding of the experiment, suggesting that it is unlikely that participants adopted a strategy of basing their Ternus responses on the appearance of the object-history displays. Indeed, the effect of the luminance object history was even stronger than the effect of the motion object history. One could speculate that this difference comes from the fact that, in contrast to the motion history, in the luminance flicker history the elements were always presented at the position of the first Ternus frame. This could have strengthend the spatial connection between the elements in the history and the Ternus display and thereby the role of object-based information, i.e., the common or separate luminance change, was more pronounced. Experiment 6 of He and Ooi (1999) is in line with this idea. They presented the two inner Ternus elements flickering together twice prior to the Ternus display. Their results showed more element motion compared to a Ternus display without the flicker history. He and Ooi (1999) suggested that this could be due to a stronger spatial grouping between the flicker elements and the inner Ternus elements leading to a ‘no motion’ trace for the inner Ternus elements. The results of Experiment 3, therefore, reinforce our interpretation of those from Experiment 2, and together suggest that object history can affect perceived Ternus motion.

General discussion

This study investigated whether correspondence is resolved based entirely on image-based information (Adelson & Bergen, 1985; van Santen & Sperling, 1985; Werkhoven et al., 1993) or whether higher-order object-based information is used as well (e.g., Enns et al., 2010; Hein & Cavanagh, 2012; Hein & Moore, 2012, 2014). We used Ternus motion (group versus element) as a measure of the correspondence process and manipulated the object history of the elements used in the Ternus display to bias a perception of them as spatiotemporally grouped togehter (common object history) or spatiotemporally independent from each other (separate object history). This allowed us to compare correspondence across displays that were identical in terms of image-level information, but differed in terms of object-level information. While the smooth-motion history (Experiment 1) did not influence subsequent Ternus motion, the apparent-motion history (Experiment 2) did. Moreover, the luminance flicker history similarly influenced Ternus motion. Together the results indicate that object-level information can play a role in resolving correspondence.

Why did smooth-motion history fail to influence correspondence (Experiment 1)? We suspect it is because the quality of smooth motion is so different from the quality of apparent motion that the two parts of the trial (object history and Ternus display) were perceived as distinct and therefore did not interact. The fact that object history defined by apparent motion, which is the same type of motion as the Ternus motion, did influence correspondence in the Ternus display (Experiment 2) supports this interpretation. Unfortunately, the other way of matching motion type – smooth motion for both object history and measuring correspondence – cannot be tested with our design because our measure of motion is Ternus motion, a kind of apparent motion. Furthermore, the influence of the luminance flicker history (Experiment 3) also supports this interpretation that there must be a certain element similiarity between the history and the Ternus part. In this case both parts could be linked by the similarity of the presentation frequency of the flicker in the history part and the ISI in the Ternus part. Future work that uses a different measure of correspondence or manipulates the similarity between the object history and Ternus display could provide converging evidence and give insight how similar the history part and the Ternus part have to be for the representations to be intergrated. In the meanwhile, it is important to note that it is the fact that any version of object history influenced correspondence that indicates that the correspondence process can incorporate object-level information.

A general concern regarding the approach used in this study is that differences in reports of element versus group motion could reflect strategical responses based on the object history part of the trial, rather than differences in object correspondence and the perception of the displays as our logic assumes. Specifically, it is possible that when participants did not have a clear perception of the Ternus motion type, they responded based on the nature of the object history displays, which were clearly separate or common history displays. Because separate and common are conceptually similar to element and group, it may be natural to associate the two types of history display with the two alternative responses to the Ternus display. While this is a concern, there are several aspects of our results that reduce it. First, in Experiments 2 and 3, we administered a questionnaire at the end of the experiment, that asked participants about their understanding of the experiment. There was no correlation between the size of the object history effect and the level of understanding of the relationship between the history conditions and Ternus responses for the subset of participants that showed an ISI effect in Experiment 2, and no correlation at all for Experiment 3. This indicates that at least no conscious strategy of basing responses on object history drove the pattern of responses. Second, the strategy was available in all three experiments, yet object history influenced reports of element versus group motion in Experiments 2 and 3, but not in Experiment 1. Finally, the conceptual similarity between the separate versus common history conditions and the element versus group response choices seems less strong in Experiment 3 (luminance change) than 2 (apparent motion), and yet the effect of object history was largest in Experiment 3. In addition to concious response biases, the object history may have induced implicit, i.e., unconscious, response biases which are difficult to access. The typical ISI effect that we found in all experiments reduces this concern by showing that participants must have based there response at least to a certain extent on how they perceived the motion in the Ternus display. Future work that identifies alternative measures of correspondence from which converging evidence can be sought, would be most helpful in assuaging remaining concerns. In the meanwhile, however, the results from these three experiments taken as a whole, suggest that object history can influence the correspondence process separate from response strategies.

The current findings extend our understanding of the correspondence process. First, it is important to note that the conclusion that object-level information influences correspondence does not imply that image-level information does not. Numerous studies have demonstrated the use of image-level information in correspondence, as unprocessed retinal information e.g., luminance, spectral content, textures, that affect motion energy and persistence (e.g., Adelson & Bergen, 1985; Breitmeyer & Ritter, 1986b, 1986a; Petersik & Pantle, 1979; van Santen & Sperling, 1985; Werkhoven et al., 1993). In addition the effect of ISI in the current (and previous) Ternus studies (Breitmeyer & Ritter, 1986a; Petersik & Pantle, 1979) is a demonstration of image-level information influencing correspondence. Second, while previous studies have shown that higher-order information can influence correspondence (Chen & Zhou, 2011; He & Nakayama, 1994; He & Ooi, 1999; Hein & Moore, 2014; Hsu et al., 2015; Ramachandran & Anstis, 1983; Yu, 2000), none before this one has done so without manipulating the level of potentially influencing information by changing the image-level context in which it was embedded. The current findings demonstrate that identical displays can be resolved differently in terms of object correspondence depending on higher-order information that is no longer present.

In summary, we found strong evidence that our visual system uses object-based information as a basis for solving correspondence, further strengthening object-based theories of correspondence (Hein & Cavanagh, 2012; Hein & Moore, 2014). Identical displays were resolved differently in terms of object correspondence depending on their history. Pure image-based theories such as motion-energy models (Adelson & Bergen, 1985; van Santen & Sperling, 1985; Werkhoven et al., 1993) cannot account for these findings. Overall, it looks like the correspondence process happens at different levels of visual processing, including low but also higher, object-based processing levels. Further studies need to examine more closely the circumstances under which different correspondence processes are engaged.