1.1 Time-resolved dimension reduction of VEP data

The EEG data that were used to construct visual evoked potentials (VEPs) were recorded from human participants (n = 24) while they viewed 80 scene images sampled from a variety of environments (see the Materials & Methods section for further detail). We focused our analyses on the posterior scalp electrodes (54 in total; see S1A Fig) because VEPs recorded at those sites carry retinotopically selective spatial frequency (SF) information [36]. To reduce the high-dimensionality of each participant’s VEP dataset (54 electrodes X 500 time points), we applied time-resolved PCA at each electrode across all scene evoked VEPs (variables) and time points (observations) in steps of 5 ms (for computational efficiency), centered within a 41 ms temporal window (±20 ms from a given time point), with each time point in the temporal window serving as feature. We chose a 41 ms temporal window as it is sufficiently broad to capture the full width at half magnitude of most VEP deflections. We tested narrower (21 ms) and broader (61 ms) window sizes and found that they only had modest effects on the mapping results. Across all participants and time points, the first two PCs were found to explain 93.2% (median) of the VEP variance, with PC 1 accounting for 76.8% of the variance. The first PC’s eigenvector for each electrode and window-centered time point was therefore used to define a uni-dimensional ‘space’ that could be mapped to each scene location within a space defined by our log-Gabor encoder model. That is, the goal of the DETI mapping procedure is to find a representational space that maps neural responses to images. By reducing the dimensionality of the VEP responses using PCA, we can represent each scene as a data point in a neural response space. From there, we can assess the extent to which that space can be explained with an image representation, in this case, a log-Gabor filter energy encoder.

1.2 Log-Gabor encoder model

We used log-Gabor filter response power to model the response variability of differently tuned neurons at each location in our stimuli [37–38] (see the Materials & Methods section for more detail). Briefly, the model consisted of seven filters, each tuned to a different peak SF (0.25, 0.50, 0.75, 1, 2, 4, or 8 cycles per degree; cpd) and all orientations (i.e., a log ‘doughnut’ filter in the Fourier domain)–refer to Fig 3 for examples. The code to build the encoders and generate the encoder space can be downloaded here https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/EncoderModel.zip. While the focus of this study was on SF, we nevertheless built a set of filter encoders that spanned all SFs centered on eight different orientations (0–157.5° in steps of 22.5°) and report some of those mapping results as Supplementary Material. We emphasize that this procedure can generalize to any parameterized encoding model.

1.3 Linking VEP Data to encoder space and DETI mapping procedures

The linking operation serves to map each encoder’s peak SF to each pixel coordinate across a set of scenes to each encoder’s peak SF. First, each filter’s response at a given pixel coordinate across all images is first assembled into an 80 x 1 array (one filter response for each image as illustrated in Fig 3). Next, the natural log of each filter’s response array is regressed against the first PC at each 5 ms time step and electrode. This process is repeated on a pixel-by-pixel basis. The result of that procedure is a 2D encoder map for each of the seven SFs (for each electrode and time step), with each cell in a given map containing an R² value (see Fig 4A for example encoder R² maps). That value therefore provides an account of how encoder response variability across images (at any given pixel location) correlates with the variability of the first PC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4.

A) Example R² maps from two different electrodes and time points. The DETI maps for each example are shown in the upper left of each set of R² maps. Each R² map shows significant R²s at each pixel location in image space. The color bar for each map shows R². B) Example encoder R² tuning functions for the two DETI maps shown in (A), averaged over all instances of each encoder’s tag in the DETI maps (y-axis is averaged R², x-axis is encoder peak SF). The shaded region of each trace shows the 95% confidence interval over all instances of pixels for each encoder.

https://doi.org/10.1371/journal.pcbi.1009456.g004

From here, two different mapping procedures can take place: 1) an image-general view of SF coding over the entire stimulus set, and 2) an image-specific view of the SF code for each scene in the stimulus set. Specifically, the first approach builds maps based on local (i.e., pixel-level) encoder variability across all stimuli at each time point (Fig 3 lower right). Each pixel is ‘tagged’ with the encoder’s peak SF that best correlates with the first PC’s eigenvector. The second approach finds the encoder that best correlates with the first PC at each pixel coordinate within each stimulus, thereby providing scene-specific maps over time. Specifically, each pixel coordinate for a given scene is tagged with the encoder’s peak SF that had the shortest distance (smallest residual) between that scene and any one of the encoder regression lines across all images (Fig 3 lower left). The code for the pipeline operations and both mapping procedures can be downloaded here https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/DETI_Code.zip.

2.0 Winner-take-all mapping validation

One of the more powerful features of the DETI mapping procedure is that it allows one to characterize the local transformational states of scenes in a space defined by populations of neural responses. By using a log-Gabor filter power encoder space, our primary goal was to assess which SF encoder best captured the local visual code at every pixel location across and within a large set of images and provide an account of how that local code changes over time (see Section 1.3). To achieve that goal, we reduced the encoder dimensionality at each pixel to that which accounted for the most variance in the VEPs (i.e., the peak SF of the encoder with the highest R²) and constructed the DETI maps based on that value (i.e., tagged each pixel with the encoder SF with the highest R²). While this approach may amplify the magnitude of the differential mapping, the goal of the DETI maps is to bring out those differences over time and electrodes. However, for that type of winner-take-all dimension reduction to be meaningful, there should be an underlying R² tuning function in the encoder array at each pixel so that the highest R² will capture the peak of the underlying R² tuning function. Specifically, each pixel coordinate in a DETI map has only a single array of seven R²s associated with it. If there is no consistent underlying tuning structure (e.g., a distribution drawn from a random permutation of the images) across the seven R²s at all pixel coordinates, then a winner-take-all average of those arrays would be largely flat, with a single elevated point at the tagged SF, as opposed to a relatively gradual monotonic reduction in R² magnitude with increased distance from the highest R². We tested whether or not that was the case by averaging the R² arrays across all pixels associated with each tagged SF for each DETI map separately at each electrode and time step. The result of that analysis showed clear evidence of R² tuning (i.e., a gradual monotonic reduction in R² magnitude with increased distance from the highest R²) for pixels tagged with any given encoder peak SF along with narrow confidence intervals. Example tuning functions are shown in Fig 4B, with the results from the comprehensive analysis shown in S1B Fig. Finally, to provide further support for this approach, we conducted the same tuning curve assessment analysis reported in Fig 4 on another VEP dataset that was collected for another study that used different images than those used in the current study (see Materials & Methods) [39]. The results of that analysis (reported in S2A Fig) show tuning curve structure for each tagged set of pixels and are thus consistent with the results of the analyses reported in Fig 4.

3.1 Image-general DETI mapping results & mapping analysis

Prior to building the image-general DETI maps, all encoder R² maps were corrected for multiple comparisons using the Benjamini-Hochberg procedure with a false discovery rate of 5% [40] across all pixels for each encoder separately at each time point and each electrode. We opted for that particular correction as it does a good job controlling for false positives while also being computationally efficient (computing DETI maps is very time intensive). As an added precaution, we ran permutation tests on the DETI mapping procedure. Specifically, the trial-averaged VEP data for each image was randomly shuffled for each participant (i.e., images were shuffled), and then averaged across participants. We then ran the DETI mapping procedure on that dataset and repeated that process 100 times (i.e., 100 simulated experiments). The resulting Benjamini-Hochberg corrected shuffled data maps over all electrodes at each time point were either completely empty (~94.5% over all electrodes and time points) or contained ~1.41% (median) pixels with encoder tags, demonstrating that this mapping procedure is largely resistant to noise. Using the same permutation tests, we found that the Benjamini-Hochberg correction remained robust to false positives out to a false discovery rate of 10% (~3.37% pixels with encoder tags).

Example DETI maps from participant-averaged data are shown in Fig 1. Please view the accompanying movie for the complete depiction of how different DETI maps evolve over time https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/HansenGreeneField_Figure1_Movie.mp4. Fig 4A shows example DETI maps for the image-general analysis. Here, the same participant-averaged data for two electrodes at two different time points are shown alongside the encoder R² maps that were used to tag the encoder’s peak SF with the largest significant R². Fig 4B shows the averaged R² tuning curves as described in Section 2.0. The image-general DETI maps provide an opportunity to analyze and explore the general coding principles of scenes over time on an electrode-by-electrode basis. In the following two subsections, we first sought to further validate the mapping procedure by examining the extent to which the image-general DETI maps replicate the results of existing studies that used traditional VEP analyses to understand the neural dynamics related to real-world scenes (Section 3.1.1). In Section 3.1.2, we assess some of the general coding principles of scenes over time afforded by the image-general DETI maps.

3.1.1 Relationship between the image-general DETI maps and traditional VEP analyses.

The time-resolved maps shown in Fig 1 (and the corresponding movie) reveal differences with respect to scalp topography and the relative contribution of SF encoders to each DETI map. Importantly, many of those differences are consistent with the results of traditional VEP analyses of real-world scene processing–namely those related to SF scalp topography and coarse-to-fine processing. To quantify those observations, we reduced the dimensionality of the DETI maps in several ways. To reduce the dimensionality from 206,643-pixel maps to a vector of seven SFs, we calculated the probability of observing pixels tagged with any given encoder’s peak SF by summing the number of pixels tagged by each encoder and divided each sum by the total number of visible pixels in the stimuli. This process was carried out for each electrode at each time step on a participant-by-participant basis. The resulting encoder probability-by-time calculations for each electrode and time step were then averaged across participants and shown in Fig 5 (see S3 Fig for the results from the orientation analysis). Only probabilities where the lower bound of their 95% confidence interval (empirical confidence intervals computed over participants) did not cross zero are marked in that figure. The results plotted in Fig 5 show that the posterior electrodes vary with respect to scalp location of the dominant encoder SF, likely reflecting their underlying calcarine sources in accordance with the cruciform model [41]. Specifically, the higher SFs (HSFs; 2–8 cpd) tend to dominate the DETI maps along the ventral-posterior electrodes, with the lower SFs (LSFs; 0.25–0.75) dominating the DETI maps along the dorsal-posterior electrodes–both observations are consistent with previous literature [42]. We verified that observation by averaging the probabilities for the ventral electrodes and dorsal electrodes (separately) for each participant and conducted a time step by time step paired samples t-test. The results of that analysis showed statistically significant differences between ~48 ms and 149 ms (all P’s < 0.05, corrected for multiple comparisons by setting the p value to the proportion of t values produced by 1000 permutations that were above the observed t values). The extent of SF lateralization shown in Fig 5 is less clear, but the lowest SF does tend to be somewhat right lateralized between 52 ms and 108 ms (paired samples t-test, all P’s < 0.05, corrected for multiple comparisons as stated above), which is consistent with the fMRI literature [43], though caution must be used in further consideration of that result due to the poor spatial resolution of EEG.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Encoder probability over time, averaged across participants.

The y-axes show participant-averaged encoder probability (note that the axes are different across SF peak), with time (ms) on the x-axes. Each trace is from a specific electrode. Probabilities that were above the lower bound of 95% confidence intervals across participants are indicated with a marker point. The electrode traces are color-coded topographically in two ways (illustrated at the top of each set of plots). The left-hand plots are coded from ventral-posterior to dorsal-posterior portions of the scalp, with the right-hand side coded from left to right across the scalp.

https://doi.org/10.1371/journal.pcbi.1009456.g005

The time-resolved maps shown in Fig 1 (and the corresponding movie) also show evidence for coarse-to-fine processing [44–45] whereby the LSFs tend to tag DETI maps across electrodes more frequently before the HSFs (e.g., compare the posterior electrode montage at 50 ms and 160ms in Fig 1). To test that observation with the data plotted in Fig 5, we averaged those data across electrodes for the LSFs and HSFs (separately) for each participant and conducted paired samples t-tests time point by time point. The results revealed that the LSFs dominate between 40 ms and 55 ms (all P’s < 0.05). Another way to visualize the SF probability differences over time is by electrode density associated with each encoder’s SF. To do that, we tagged each electrode with the encoder SF that was most prevalent in its map and summed the number of electrodes dominated by each encoder at each time point (Fig 6). The results show that LSFs dominate early (~50 ms), followed by the HSFs, with the 2 cpd and 4 cpd encoders preceding the 8 cpd encoder. Interestingly, every time point yields electrodes that are dominated by different encoder SFs, revealing a multiscale representation of scenes over time. In other words, every SF that we explored is explaining VEP variance simultaneously across the posterior scalp, though the HSF neural sources tend to dominate the scalp.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Participant averaged electrode counts based on encoder dominated DETI maps.

The left-hand plot shows the number of electrodes that were dominated (largest sum) by each of the seven encoders. The right-hand side is a summary of that plot and shows the electrode counts summed over LSFs (0.25–0.75 cpd) and HSFs (2.0–8.0 cpd). The shaded area shows the 95% confidence intervals over participants.

https://doi.org/10.1371/journal.pcbi.1009456.g006

Finally, the results from the orientation-based encoders do not show a clear topographic organization, but instead show that orientations at and near horizontal have the lowest probability in the maps over time (S3 Fig). Such a result is consistent with normalization operations in visual cortex and scene selective cortices that operate to reduce the magnitude of horizontal information [46–47].

The consistency between the image-general DETI mapping results and existing results from traditional VEP analyses provides an important validation for the DETI mapping approach and shows that this procedure can successfully map the SF scalp topography without resorting to multiple experiments that present stimuli at different locations in the visual field to avoid dipole cancellation. Further, this approach provides a much more detailed account of the code that underlies the topography with full images that span a sizable portion of the visual field.

3.1.2 The general coding principles of scenes over image space across time.

Having validated DETI mapping through the replication of existing results of scene processing obtained with traditional VEP analyses, we turn our attention to what this approach can tell us about how general scene regions are coded over time. Towards that end, we analyzed DETI map variation over all electrodes at each time point. Because cortical folding varies from person-to-person (which influences how VEP signals register on the scalp), we could not simply collapse across participants at corresponding electrodes. Instead, we split each electrode’s fully tagged DETI map into a set of seven binary maps, one for each encoder, and then summed those maps (pixel-by-pixel) across all electrodes at each time point. That process results in 2D histograms for each encoder, which were then summed across participants. To provide a comprehensive view of the spatial biases in encoder maps over time, we again grouped the lower SF encoders (0.25–0.75 cpd) and higher SF encoders (2.0–8.0 cpd) and conducted an upper-to-lower and left-to-right marginal analysis on the LSF and HSF maps at each time point (Fig 7A). To assess participant agreement between the different marginal analyses, we regressed the upper/lower and left/right biases at each time point across participants (Fig 7B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7.

A) Marginal analyses of the 2D DETI map histograms for the lower (LSF) and higher (HSF) encoder peak SF. The row-wise marginal analysis consists of an average across the histogram maps from left to right for each time point. The columns of that plot were first normalized over time and then normalized again within each column to emphasize encoder fit magnitude over time and space. The column-wise marginal analysis was carried out the same way, but from top to bottom of the R² maps (the normalization therefore took place row-wise). For a dynamic view of each marginal analysis, follow this link https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/HansenGreeneField_Figure4_Movie.mp4. B) Averaged cross-participant histogram maps for each column in the row-wise marginal analysis (left) and each row of the column-wise marginal analysis (right). The shaded region shows the 95% confidence interval across participants.

https://doi.org/10.1371/journal.pcbi.1009456.g007

The results shown in Fig 7A reveal that the upper and lower portions of images are coded by LSFs, with the HSFs coding the central portion of images. These biases were largely consistent across participants, especially during the earlier time window (e.g., 50–180 ms) (Fig 7B). Interestingly, those spatial biases are not static, but vary in a nonuniform manner over time. Specifically, in the row-wise marginal analysis, the lower portion of images is dominantly coded early (~50 ms) by LSFs, with the upper portion of image space dominating later (~250 ms), again with LSFs (which is also apparent in the movie associated with Fig 1). The central portion of image space shows two waves of HSFs, with the first wave more centralized than the second. When compared to Fig 6, the first wave corresponds to 2–4 cpd, with the spatially broader wave corresponding to 8 cpd. The column-wise marginal analysis revealed additional asymmetries that were observed laterally across images with LSFs showing an early (~70 ms) central-to-left hand bias, with HSFs showing a slightly later (~80 ms) bias toward the right and far left portions of images. The left-right asymmetries are consistent with the fMRI literature [43] that reports a right-lateralized occipitotemporal LSF bias (which would show up here as a bias on the left), and a left-lateralized temporal HSF bias (which would show up here as a bias on the right). Lastly, we conducted a similar analysis using the raw R² maps from each participant. The results of that analysis are largely consistent with the spatiotemporal patterns revealed by the marginal analyses conducted on the 2D histograms (see S4 Fig), as well as the encoder probability over time analysis (see S5 Fig) and are largely replicated using the additional dataset mentioned in Section 2.0 and reported in S2B Fig).

Crucially, the nonuniformities over image region and time would not be expected from a simple linear model based on retinotopic mapping of SF preference and suggest that the neural code for different image regions changes SF preferences differently over time, thereby providing insight into a possible prioritization of different image regions as time advances. For instance, an early prioritization of the ground plane may support rapid judgements regarding scene navigation [48–49], with a later upper image region analyses focused on landmark organization [50].

3.2 Image-specific DETI mapping results & mapping analysis

In addition to providing useful insights into how image regions are coded across a set of images, the DETI mapping procedure offers the ability to examine the local code for individual images over time. All image-specific mapping reported here was based on the regression fitting procedure that was carried out for the image-general analysis, but here was focused on mapping based on minimal residual error from each encoder’s regression line (refer to Fig 3). The random permutation analysis reported in Section 3.1 suggests that the Benjamini-Hochberg correction procedure tended to be overly conservative, so we adjusted the false discovery rate to 10% for the analyses reported here (the results from analyses with a 5% false discovery rate were largely consistent with those reported here and are shown in S6 Fig).

Example image-specific DETI maps from participant averaged data are shown in Fig 2. Please view the accompanying movie for the complete depiction of how different DETI maps evolve over time https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/HansenGreeneField_Figure2_Movie.mp4. Additional image-specific map examples are shown in Fig 8. Together, Figs 2 and 5 (and the corresponding movie) illustrate some of the diversity of image transformations over time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Additional stimulus examples and their DETI maps at different time points from electrode 90.

https://doi.org/10.1371/journal.pcbi.1009456.g008

While the image-general maps revealed the general coding principles across images for our set of stimuli, here we analyzed the image-specific maps to provide an account of the various encoder-based transformations that each scene undergoes over time. To do that, we first constructed a summary statistic (illustrated in Fig 9 and described later in this section) that would characterize the transformational state of each image at each time step. Doing so would enable us to track (over time) where any given image is in the neural response space. To that end, we first built a low-dimensional DETI map space to plot each image and then tested how well our summary statistic predicted the location of each image in that space. The results of that analysis allowed us to proceed with that statistic in subsequent analyses of the transformational states of images over time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Illustration of the procedure for projecting all encoder-tagged images into a state-space representation at each time point for each participant using all electrodes.

This procedure was either carried out for entire images (top left) or within patches located at nine different regions of the images (bottom left). Encoder probabilities were calculated as described in the text for whole images or image patches. At each time point and across all electrodes, each encoder’s probability was stored in a matrix as an array on an image-by-image basis (e.g., a 378 x 80 matrix for each time point). Each matrix was then submitted to PCA, with the first two PCs defining the encoded image’s state-space. Each image was then characterized by taking the log₁₀ HSF-to-LSF probability ratio (see text for further detail).

https://doi.org/10.1371/journal.pcbi.1009456.g009

To build a low dimensional image-specific DETI map space, we chose to reduce the dimensionality of those maps by calculating the probability of observing pixels tagged with any given encoder’s peak SF, thereby reducing any given image-level map to seven probability values (similar to the image-general analysis). That process was carried out for each electrode and time step on a participant-by-participant basis. Example probabilities (averaged across participants) for each encoder over time and electrode for two example images are shown in Fig 10. Consistent with the image-general mapping, individual images exhibit a tendency for HSFs to dominate the electrode maps along the ventral-posterior electrodes, with LSFs dominating the electrode maps along the dorsal-posterior electrodes. Importantly, each image showed a relatively unique probability-by-time ‘signature’ between and across electrodes. For example, the image on the left-hand side of Fig 10 shows an early dominant multimodal distribution for lower SF which gives way to a weaker bimodal HSF distribution. The image on the right-hand side shows a much weaker multimodal distribution at the lowest SFs which changes to a dominant bimodal distribution for SFs > = 0.75 cpd.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Encoder probability over time for two stimulus examples (upper left of each set of plots), averaged across participants.

The y-axes show participant-averaged encoder probability, with time (ms) on the x-axes. Each trace is from a specific electrode and color-coded from ventral-posterior to dorsal posterior electrode (illustrated at the upper right of each set of plots).

https://doi.org/10.1371/journal.pcbi.1009456.g010

With the probability-by-time data in hand, we next projected all image-specific DETI maps into a state-space representation based on all electrodes at each time point for each participant. Briefly, for any given time point, each encoder’s probability for each electrode was stored in an array on a map-by-map basis, resulting in a 378 x 80 matrix for each time point (i.e., 54 electrodes * 7 encoders X 80 scenes) (see Fig 9). Thus, each image’s map is represented as patterns of SF probability across encoders and electrodes. Including all 54 electrodes allowed representations based on visual signals from different regions of the of the calcarine sulcus [41–42]. The resulting matrix was then submitted to time-resolved PCA (with electrodes as observations and scenes as variables). The first two PCs accounted for 87.4% (median) of the variance across time, images, and participants, with the first PC accounting for 80.7% (median). All images could then be represented in a 2D encoder probability state-space defined by the first two PCs at each time point and participant. Examples of that space for a given participant at three different time points are shown in Fig 11A and show how images are organized in the low-dimensional DETI map space.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11.

A) Principal component (PC) defined VEP-mapped image state-space from an example participant at three different time points (refer to text and Fig 9 for further details). Data points are images, and the color bar corresponds to the log₁₀ HSF-to-LSF probability ratio for each image in that space. The x- and y-axes are PC 1 and 2 respectively. B) Characterization metric comparison for explaining the relative location of each image in PC-defined state-space over time. The y-axis shows the average adjusted R² across participants. The shaded region shows the 95% confidence interval of the fits across participants.

https://doi.org/10.1371/journal.pcbi.1009456.g011

Next, we sought a summary statistic that could characterize the relative positioning of the images in the DETI map space according to encoder probability, thereby enabling us to ‘track’ the different transformational states of any given image. Specifically, we characterized each image by the log₁₀ ratio of the summed HSF probabilities (2.0–8.0 cpd) to the summed LSF (0.25–0.75 cpd) probabilities (an SF probability ratio) at each time point as depicted in Fig 9. Characterizing images according that summary statistic defines any given transformational state as the relative amount of the image coded by HSFs vs LSFs. To assess the ability of that characterization to explain the first two PCs, we submitted the SF ratio for each image (and at each time point) to multiple regression and show the adjusted R² over time in Fig 11B. The result of that analysis shows that SF probability ratio provides an excellent account of the relative positioning of images in the DETI map state-space for all participants. We also compared the SF probability ratio to that which could be explained by simple Fourier image statistics based on: 1) the amplitude spectrum slope of each image, and 2) the ratio of local filter response slopes (shallow slopes to steep slopes) at each pixel (also shown in Fig 11B). While the Fourier image summary statistics were successful at explaining image position in that space, the SF probability ratio is far superior, meaning that the different transformational states that the images undergo are only minimally explained by simple Fourier image statistics.

The success of our summary statistic enabled us to visualize each image’s transformation over time. Fig 12A plots the participant-averaged SF probability ratio at each time step for each of the 80 images and illustrates how the relative HSF to LSF coding of images changes over time. The most prominent variation in Fig 12A is an interesting, possibly two-stage [51–53] pattern of transformations in that the images first show an initial LSF based code (~50 ms), followed by a relative HSF code (~70 ms to ~140 ms) where the encoder SF ratio variance is ~3 times larger than any other time window. Around 150 ms, the image transformations undergo what appears to be intermittent LSF transformations at ~180 ms and ~260 ms, possibly indicative of recurrent processes [52–53]. The variation of the SF probability ratios over time was verified with a one-way ANOVA including each image’s ratio and factor of time, F_(55,1288) = 2.7, P < .001. Because the SF probability ratio characterization provides a strong account of where each image is located in the DETI map state-space, we next sought to assess the stability of the relative positioning of the images in that space over time. That is, we aimed to assess the relative similarity of the transformational states of images over time thereby testing the relative persistence of those states over time. To do that, we ran a time-time regression analysis on each participant’s SF probability ratios. Specifically, we regressed the ratios of all images at any given time point against the ratios at every other time point, and then averaged the resulting R²s across participants (Fig 12B). The results of that analysis show an early period (~70 ms to ~160 ms) of relative stability, meaning that the transformational states of the images during that window of time covary with one another. However, after that window of time, there is an increase in the temporal variability (which can be interpreted as a reduction in temporal similarity) between transformational states, suggesting that each transformation becomes more unique as time advances past ~160 ms.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12.

A) Participant averaged log₁₀ HSF-to-LSF probability ratio for each encoder tagged image over time across the entire stimulus space (i.e., all pixels). B) Participant averaged time-time R² matrix assembled by regressing the log₁₀ HSF-to-LSF probability ratios across images for each time point against every other time point (color bar is R²). The contour line encapsulates the top 12% of the R²s.

https://doi.org/10.1371/journal.pcbi.1009456.g012

While the results reported in Fig 12 provide insight into the different transformational states (as defined by SF probability ratios) that scenes undergo, they do not provide an account of how local image regions shape those states. Understanding local variation is important because as receptive field size increases along the visual processing stream, image coded by smaller receptive fields likely contribute differentially to larger receptive fields that shape high-level representations. Understanding the local variation therefore offers some insight into how different image regions contribute to the higher representations. We therefore focused on an assessment of the transformations within localized image windows. Out of practicality, we ran this analysis within nine different spatial windows using a polar coordinate system to define the position of each spatial window (refer to Fig 9 for an example). Each spatial window had a diameter of 128 pixels (4.7° of visual angle). The center-to-center distance between the center window and the outer windows was 190 pixels (7.1° of visual angle), with the center-to-center distance between each outer window being 160 pixels (6° of visual angle). The analysis described above to produce Fig 12A was carried out on each window location across all images on a participant-by-participant basis and then averaged across participants (Fig 13).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Participant averaged log₁₀ HSF-to-LSF probability ratio for each encoder tagged image over time, measured at nine different locations within stimulus space (see text for more details).

The format of the plots is the same as in Fig 12A.

https://doi.org/10.1371/journal.pcbi.1009456.g013

Fig 13 reveals that different image locations undergo a relatively unique transformational process over time, with the largest difference being between the upper and lower portions of the image, with the central portion being somewhat intermediate. Specifically, the upper and central portions of different images show transformational states over time that are similar to that found with the whole image analysis (Fig 12A), with the lower image regions showing transformational states that follow a monotonic rise from LSF-based representations to HSF-based representations. To assess the relative similarity of the transformational states of the nine image regions over time, we ran the same time-time regression analysis mentioned before on the SF probability ratios for each region (Fig 14). The results of that analysis were somewhat similar to the whole image analysis (Fig 12B), with the upper and central portions of image space showing the gradual reduction of similarity between transformational states over time. On the other hand, the lower portion shows overall less similarity between transformation states. That plot illustrates the tendency of the lower image windows to result in more temporal variation, thereby suggesting more unique transformational states on a time point by time point basis. That observation was first verified with a one-way ANOVA including each image’s ratio and factor of time, F_(8,207) = 2.9, P = .005, with post-hoc t-tests showing significant differences between all lower windows and the upper middle, upper right, and central right windows (P’s < .02). This suggests more dynamic representations in the lower portion of image space compared to the upper portion and may reflect a shift from a higher-dimensional representations to lower-dimensional representations across the image that may be relevant for perceptual decision making [54]. Further, such a differential shift in dimensionality across the image may suggest a need to maintain different degrees of spatial dimensionality to allow for flexibility in perceptual decision making (though such an account is in need of further research). Nevertheless, what is striking about this local analysis is that the representations contained within each region (upper, central, lower) is more similar than the representations between each region. This further supports the finding that there are different neural coding operations at different image locations over time. Importantly, the differential transformational states within different image regions suggest that the temporal coding of visual information is far more complex than a simple coarse-to-fine analysis and subsequent mapping to higher cortical representations [44–45].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14.

Top: Results from the time-time regression analysis on the participant averaged SF probability ratios within each of the nine local image windows. Bottom: The percentage of time-time R²s that fell within the top 12% contour area was calculated for each participant, and then averaged across participants (gray area shows the 95% confidence interval).

https://doi.org/10.1371/journal.pcbi.1009456.g014

Finally, to provide an approximate visualization of how the images may appear in different transformational states, we ‘reconstructed’ example images using an approach that is similar to the SF bubbles technique in the spatial domain [55] (see Materials & Methods for details). The examples shown in Fig 15 show two time points, with an earlier time point (140 ms) showing the relatively constant SF mapping across image region, with a later (250 ms) inconsistent representation across image region where the upper portion of the images are in an LSF transformational state and lower portion of the images are in an HSF transformational state.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Example stimuli along with their SF filter reconstructions based on the 2D encoder histograms at two time points (see Materials & Methods for further details).

https://doi.org/10.1371/journal.pcbi.1009456.g015

Ethics statement

This study was reviewed and approved by Colgate University’s Institutional Review Board, and all participants gave written informed consent before participating.

Apparatus

All stimuli were presented on a 23.6” VIEWPixx/EEG scanning LED-backlight LCD monitor with one ms black-to-white pixel response time. Maximum luminance output of the display was 100 cd/m², with a frame rate of 120 Hz and resolution of 1920 x 1080 pixels. Single pixels subtended .0382° of visual angle as viewed from 35 cm. Head position was maintained with an Applied Science Laboratories (ASL) chin rest.

Participants

A total of 35 participants were recruited for this experiment. Of those, 8 failed to complete both recording sessions and 3 were excluded for having fewer than 50% valid trials following artifact rejection. The age of the remaining 24 participants (13 female, 22 right-handed) ranged from 18–21 (median age = 18). All participants had normal (or corrected to normal) vision as determined by standard ETDRS acuity charts and were compensated for their time.

Stimuli

Stimuli were selected from a large database of real-world scenes consisting of 2500 photographs that varied in content from purely natural to purely carpentered (both indoor and outdoor), with various mixtures of natural/carpentered environments in between [38]. All images were 512 x 512 pixels and converted to grayscale using the standard weighted sum conversion in MatLab.

For the purposes of stimulus presentation and analysis, all images were calibrated according to the following procedures. First, each image was fit with a hard-edge circular window (with a diameter of 512 pixels) whereby all pixels that fell outside of the circular window were set to zero (i.e., we’re only interested in the 206,643 pixels that were presented to the participants as described later in this section). Next, each image was converted to an array, I(y), that included only the pixels that fell within the circular window and were made to possess the same root mean square (RMS) contrast and mean pixel luminance.

Root mean square contrast is defined as the standard deviation of all pixel luminance values divided by the mean of all pixel luminance values. Image arrays were set to have the same RMS contrast and zero mean using the following operations. with I_zm defined as:

The pixels values of each image array are first normalized to fall between [-.5 .5] with zero mean as follows,

We then calculated an RMS scaling factor, S_rms = (2*RMS_t)/RMS_sc, with RMS_t set to a reasonable target RMS value. By reasonable, we mean a value that did not result in significant (> 5%) clipping of the resulting pixel values. That value was 0.20 for the images used in the current study. Finally, each image array was scaled to have an RMS equal to RMS_t and reassign to I(y) as follows: I(y) = 127*(I_sc*S_rms). Note that scaling by 127 puts the scaled pixel values of I(y) back in the original range of I_zm.

Stimulus images were selected according to an image state-space sampling procedure as follows. All images in the database were left in vector form after RMS normalization (described above). When in array form, each cell constitutes a coordinate in a high-dimensional image state-space where each coordinate takes on a pixel luminance value ranging from [–127, 127]. An angular distance matrix was then constructed by calculating the angular distance (in degrees) between each image array and every other image array as follows:

From there, the angular distance matrix was projected into a lower dimensional space (3D) via t-distributed stochastic neighbor embedding (t-SNE) [62]. Projecting the images into this space enables a general lower dimensional organization of images based on their structural attributes defined by pixel luminance.

Stimuli were selected by uniformly sampling 80 images from the lower dimensional t-SNE space in order to increase the probability of the different regions in our image state-space being represented in the stimulus set. We chose to use 80 images to have as many images as possible while keeping the overall length of the recording sessions to a practical limit. All selected images maintained their RMS of .20 (defined above) but had their mean pixel luminance set to 127 and then fit with a circular linear edge-ramped window (512-pixel diameter, ramped to the mean pixel luminance) to obscure the square frame of the images. That step ensures that the contrast changes at the boundaries of the image were not biased to any particular orientation [63–64].

Experimental procedure

The experiment consisted of two recording sessions, each ranging between 50–55 min. Within each session, all 80 stimuli were presented 15 times, resulting in a total of 30 presentations per image over both recording sessions (stimulus presentation order was randomized). Each trial began with a 500 ms fixation followed by a variable duration (500–750 ms) blank mean luminance screen to allow any fixation-driven activity to dissipate. The blank screen was immediately followed by the stimulus interval (500 ms) that was then followed by a variable 100–250 ms blank mean luminance screen, followed by a response screen. The response screen prompted the participant to rate the visual clutter of the scene stimuli on a scale of 1–4 (1 being not cluttered and 4 being very cluttered) using a button box (response time was unlimited).

EEG recording and processing

All Continuous EEGs were recorded in a Faraday chamber using Electrical Geodesics Incorporated’s (MagStim EGI) Geodesic EEG acquisition system (GES 400). All EEGs were obtained by means of Geodesic Hydrocel sensor nets consisting of a dense array of 128 channels (electrolytic sponges). The on-line reference was at the vertex (Cz), and the impedances were maintained below 50 kΩ (EGI amplifiers are high-impedance). All EEG signals were amplified and sampled at 1000 Hz. The digitized EEG waveforms were first highpass filtered at a 0.1 Hz cut-off frequency to remove the DC offset, and then lowpass filtered at a 45 Hz cutoff frequency to eliminate 60 Hz line noise.

Continuous EEGs were divided into 600 ms epochs (99 ms before stimulus onset and 500 ms of stimulus-driven response). Trials that contained eye movements or eye blinks during data epochs were excluded from analysis via magnitude thresholding followed by visual inspection. Additionally, all epochs were subjected to algorithmic artifact rejection whereby voltages exceeding +/- 100 μV or transients greater than +/- 100 μV were omitted from further analysis. These trial rejection routines resulted in a median of 9% (range 3% - 29%) of trials being rejected across participants. Each epoch was then re-referenced offline to the net average, and baseline-corrected to the last 99 ms of the blank interval that preceded the image interval. Finally, VEPs were constructed for each participant by averaging the processed epochs across trials for each image at each electrode, resulting in a 128 x 600 x 80 VEP data matrix for each participant. The full dataset (34Gb) can be downloaded here https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/HansenGreeneField_Data.zip.

Encoder model details

Visual evoked potentials constitute the sum of neural activity (post-synaptic potentials) at the circuit level. Given the retinotopic mapping of the visual cortices [34], the VEPs measured on the scalp likely stem from a summation of the underlying responses tuned to different image attributes at different image locations. If the majority of the summation arises from early visual cortical processes [1], then we can expect a good portion of the sum to be explained by contrast in different bands of spatial frequency and orientation. As a first approximation to model the relative response of differently tuned neurons at each location in our stimuli, we used a filter-power encoding model based on log-Gabor filters [38]. Specifically, the model consists of 7 filters, each tuned to a different peak spatial frequency (0.25, 0.50, 0.75, 1, 2, 4, 8 cpd) and all orientations (i.e., a log ‘doughnut’ filter in the Fourier domain). The spatial frequency bandwidths (full width at half height) of the filters scaled with peak spatial frequency such that they were broader at lower spatial frequencies and narrower at higher spatial frequencies: 2.3, 2.3, 2.0, 2.0, 1.75, 1.5, and 1.0 octaves respectively [2–3]. We chose 7 peak encoder frequencies in order to tile as much of the frequency domain as possible within minimal overlap while minimizing filter clipping at the highest frequencies in the Fourier domain.

Stimulus representation in encoder space

All image filtering was conducted in the Fourier domain using the images in matrix form. To minimize edge effects in the Fourier domain due to the non-periodic nature of scene images, the images were symmetrized prior to taking the Fourier transform. Each symmetrized image was submitted to the 2D discrete fast Fourier transform to obtain H(u,v) as follows: where I(x,y) represents a given image, with X and Y representing the dimensions of the symmetrized image. Next, the amplitude spectrum was calculated according to: with H_R(u,v) and H_I(u,v) representing the real and imaginary parts of H(u,v), respectively. For filtering convenience, the amplitude spectrum, A(u,v) was shifted to polar coordinates and in this form will be denoted as A(f,θ), with f serving as the index along the radial (i.e., spatial frequency) dimension, and θ as the index along the theta (i.e., orientation) dimension.

Each image’s amplitude spectrum was then multiplied by a 2D log-Gabor filter. Log-Gabor filters in the Fourier domain consist of a log-Gaussian function along the f axis and a Gaussian function along the θ axis, which are then combined by multiplying a 2D log-Gaussian filter (i.e., a log ‘doughnut’ filter) with a 2D Gaussian ‘wedge’ filter. The construction of the 2D log-Gaussian filter, L_gaus(f, θ), took place in the same polar coordinate frame as A(f,θ). Thus, for each θ axis, L_gaus(f) was modulated as follows.

Where f increases with spatial frequency (radial distance), f_peak represents the peak of the function, and f_σ represents the SF bandwidth of the filter. Next, a 2D Gaussian function (modulated across θ in radians) about a central orientation was generated as follows.

The log-Gabor filter, LG(f, θ), was then constructed by multiplying G_θ(f, θ) by L_gaus(f, θ).

The filtered amplitude spectra and corresponding phase spectra were then inverse Fourier transformed back into the spatial domain with the image in its original orientation cropped from the symmetrized version. We then took the natural log of the squared filter responses (i.e., each pixel location across all filters and images was expressed as log power). Representing the filtered images in the spatial domain allowed us access to the encoder responses at each pixel coordinate across all images. The code used to create the encoders and generate the encoder space can be downloaded here https://pbsc.colgate.edu/~bchansen/HansenGreeneField2021/EncoderModel.zip.

Visualizing the transformational states of scenes over time

To provide a rough visualization of how the images may appear in different transformational states, we ‘reconstructed’ example images using an approach that is similar to the SF bubbles technique in the spatial domain [55] using the participant-averaged VEP data. That process began by first choosing an image and filtering it with the same set of log-Gabors that were used in the DETI mapping procedure. Those filter responses were later mapped to different locations in the reconstruction. Next, we converted each electrode’s fully-tagged image-specific DETI map for a given image for a given time point to a set of seven binary maps, one for each encoder, and then summed those maps (pixel-by-pixel) across all electrodes. That process resulted in 2D histograms for each encoder (similar to what was done for the marginal analysis carried out in the image-general DETI map analysis). Next, for any given pixel coordinate, we selected the encoder that had the highest electrode sum. We then sampled a window centered on the corresponding pixel of the image that had been filtered with the selected encoder. The diameter of that window scaled with encoder peak SF such that lower SFs had larger windows. Specifically, window diameter allowed for 1.5 periods of a given encoder’s SF. That sample was then weighted with a normalized Gaussian (normalized by area) and then summed with the corresponding pixel location in the reconstructed image template. This process was repeated for all pixels.

Replication data set

EEG data were collected to construct VEPs were recorded from human participants (n = 23) while they viewed 150 scene images sampled from a variety of environments (using the same image state-space sampling procedure described above) as a part of another study [39]. The experiment consisted of one recording session (50–55 min in length). Each of the 150 stimuli were presented 6 times. The presentation sequence was identical to that used in the current study. However, the task required participants to categorize (via mouse click) each image as indoor, urban, or natural. EEG recording and processing were identical to that used in the current study. The final data set was constructed by averaging across participants.