1.1 Modeling neural activity via feature-disentangled generative latents

Although neural representations are constructed from experience, an infinite amount of visual phenomena can be represented by the brain to successfully interact with the environment. That is, novel yet plausible situations that respect the regularities of the natural environment can also be mentally simulated or imagined [38]. From a machine learning perspective, generative models achieve the same objective by capturing the probability density underlying a huge set of observations. We can sample from this modeled distribution and synthesize new instances that appear as if they belong to the real data distribution yet are suitably different from the observed instances thereof. Particularly, generative adversarial networks (GANs) [39] are among the most impressive generative models to date which can synthesize novel yet realistic-looking images (e.g., images of human faces, bedrooms, cars and cats [40–43] from latent vectors. In the context of generative models, like GANs, a latent space refers to a lower-dimensional data distribution (e.g., a standard Gaussian distribution) in which a more complex data distribution (e.g., face- or natural images) is encoded; it is a compressed and abstract space that captures the most essential features of the more complex data. A GAN consists of two neural networks: a generator network that synthesizes images from randomly-sampled latent vectors and a discriminator network that distinguishes synthesized from real images. During training, these networks are pitted against each other until the generated data are indistinguishable from the real data. The one-to-one (bijective) mapping from latents to images by the generator effectively models the ‘synthesis’ operation (as specified in Fig 2) which can be exploited in neural coding to disambiguate the images from brain activity via their latents, since the visual content is deterministically specified by their underlying latents (such an approach was earlier suggested by [44]), and perform analysis by synthesis [45]. Note that, while the generator’s latent-to-image transformation performs the reconstruction of the perceived stimuli, it is the feature-response correspondence that enables the interpretation of neural activity as variations in the latent features.

Traditional GANs are known to suffer from feature entanglement where the generator has learned to fuse multiple features into a single latent dimension (i.e., a hyperplane in the multidimensional latent space) [46]. As a consequence of this fusion, the latent space contains biases inherited from the training dataset. To illustrate this, consider an example of generating images of human faces. A conventional GAN may entangle features like “gender” and “hair length” when predominantly exposed to feminine-looking faces with long hair and masculine-looking faces with short hair. Entanglement of these two features would result in biased outputs, hindering the generator’s ability to synthesize a masculine face with long hair, even if such combinations exist in reality. The concept of feature disentanglement, on the other hand, refers to the independence of different visual features, allowing variations in one feature to be untangled from others [47]. In a feature-disentangled GAN, the generator has learned to encode each facial feature independently. For example, changing the latent dimension corresponding to “hair length” would only modify the hair region of the generated face while keeping other features invariant. Here, we posit that feature-disentangled GAN latents exhibit a stronger alignment with neural representations in the ventral visual stream.

One member of the family of feature-disentangled GANs is StyleGAN [42] (Fig 3)—which maps the conventional z-latent via a multilayer perceptron (MLP) to an intermediate and less entangled w-latent space. Feature disentanglement is an emergent property that arose as the MLP learned to control diverse aspects of the image synthesis process within the training framework of StyleGAN. That is, the interplay between the generator’s evolving architecture, the injection of w-latents at different levels, and the network’s optimization for image generation contributes to the disentanglement of features in the w-latent space. Here, we propose feature-disentangled w-latents as a promising feature candidate to explain neural responses during visual perception. In brief, visual stimuli were synthesized by a feature-disentangled GAN and presented during a passive fixation task to a macaque with cortical implants in visual areas V1, V4 and IT (Fig 4). In contrast to many previous studies that relied on noninvasive fMRI signals with limited temporal resolution and low signal-to-noise ratio, the current use of multi-unit activity (MUA) [48] via 15 chronically implanted multielectrode arrays (each with 64 channels) provided opportunities for spatiotemporal analysis of brain activity in unprecedented detail. The electrode placings across these three visual areas are visualized in Fig 2. For neural encoding, we predicted brain activity from StyleGAN’s z- and w-latent representations, as well as Contrastive Language-Image Pre-training (CLIP; ViT-L/14@336px) latents which represent images and text in a shared representational space that captures their semantic relationships [49]. CLIP-latents are not just abstract representations of visual content, but they are also pivotal in the generative processes of contemporary latent diffusion models like Stable Diffusion [50]. Their key strength for our purposes lies in their ability to capture the essence of images in a way that reflects how the visual system of the brain processes visual inputs into semantic representations [51, 52].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. StyleGAN3 generator architecture.

The generator takes a 512-dim. z-latent (entangled or correlated dimensions) as input and maps this to its 512-dim. w-latent (disentangled or decorrelated dimensions) via the MLP, f(), for feature disentanglement. Then, the w-latent is transformed into a 1024 × 1024 px RGB image. Face images in this figure are replaced for copyright reasons. The original version of the figure can be accessed here.

https://doi.org/10.1371/journal.pcbi.1012058.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Passive fixation task.

The monkey was fixating a red dot with gray background for 300 ms followed by a fast sequence of four face images (500² pixels): 200 ms stimulus presentation and 200 ms inter-trial interval. The stimuli were slightly shifted to the lower right such that the fovea corresponded with pixel (150, 150). The monkey was rewarded with juice if fixation was kept for the whole sequence.

https://doi.org/10.1371/journal.pcbi.1012058.g004

The contributions of this work are as follows: first, our encoding analysis revealed that w-latents, compared to the z- and CLIP-latents, were the most successful at predicting high-level brain activity in the inferior temporal (IT) cortex, which is located at the end of the visual ventral pathway. Second, neural decoding using w-latents resulted in highly accurate reconstructions that matched the stimuli in their specific visual characteristics. This was done by fitting a decoder to the recorded brain responses and the ground-truth w-latents of the training stimuli. We then used this decoder to predict the w-latents from responses of the held-out test set and fed these to the generator of the GAN for reconstruction [36]. Our findings indicate that the brain’s representation of visual information, in the context we studied, exhibits a degree of structured organization that aligns with our model, offering a new way forward for the previously limited yet biologically more plausible unsupervised models of brain function. Third, time-based neural decoding showed how the brain captured meaningful information about the stimulus in time. Finally, the interpretation of neural activity via the established response-latent relationship was explored by the application of linear operations to control specific visual features in the images. Taken together, the high quality of the neural recordings and feature representations resulted in novel experimental findings that not only demonstrate how advances in machine learning extend to neuroscience but also serve as an important benchmark for future research.

1.2 Earlier work

Visual experience is partially determined by the selective responses of neuronal populations along the visual ventral “what” pathway [53] where the receptive fields of neurons in early cortical regions are selective for simple features (e.g., local edge orientations [54]) and those in more downstream regions respond to more complex patterns of combined features [55, 56]. At first, neural coding studies primarily relied on retinotopy to infer visual content since the spatial organization of images is reflected in the stimulus-evoked responses in the primary visual cortex (V1) [57]. As such, visual content was mainly inferred from neural responses in early cortical areas, and the stimuli often consisted of low-resolution contrast patterns or digits [24, 25, 27, 29, 32]. Attempts to reconstruct more complex naturalistic images from activations in early regions were taken [28] but still fell short of capturing the full complexity of high-level neural activity required for reconstructing more intricate visual content. To successfully decode more high-level information from anterior regions, suitable feature representations were required that captured similar information about the stimulus as these responses, as attempted with more high-level hand-engineered features by [26] and [58] to reconstruct naturalistic images and scene backgrounds, respectively.

Next, the complexity gradient in visual processing, where increasingly complex features are represented across the ventral stream, was also identified in deep neural networks (DNNs): the alignment of DNN layers with neural activations revealed that early layers were mainly predictive of responses in upstream visual areas whereas deeper layers were more predictive of more downstream visual areas in humans [3–8, 11] as well as in primates [1, 12]. At present, DNNs are commonly used to decode more high-level neural activity during visual perception, imagery and dreaming [19, 33–36, 59–62]. For reconstruction, the decoded feature representations of discriminative DNNs were used, for instance, by providing them directly as input to a decoder DNN (feature-to-image) [33] or by using a feature loss to iteratively optimize the pixel values in an input image [34] or decoder weights [63] so that the reconstruction features matched those of the stimulus. At this time, unsupervised learning paradigms, although more biologically plausible, seemed to appear less successful in modeling neural representations in the primate brain than their supervised counterparts [5].

Recent advancements have shifted attention towards the potential of unsupervised generative (rather than discriminative) models and their latent spaces, such as Variational Autoencoders (VAEs) [35, 64] and GANs [36, 60–62, 65]. In contrast to discriminative features, generative latents offer a distinctive advantage by aligning neural representations with generative processes the brain might perform during various cognitive functions (e.g., anticipation and mental imagery). Also, it is not possible to directly model the synthesis operation from discriminative features because they are primarily optimized to differentiate between classes rather than generate new visual content. However, the challenge posed by the scarcity of neural data, along with the substantial data requirements for properly training data-hungry DNNs with numerous parameters, has hindered effective GAN training from scratch (see [61], for an attempt using 6000 training examples). To address this issue, [60] trained an encoder model to generate synthetic neural activity to a much broader set of images which was then utilized to train a GAN. However, biases and inaccuracies in the synthetic neural data fail to capture the intricate details of authentic neural responses, leading to discrepancies between the reconstructions and stimuli. Rather than training our own models from limited data, we can also leverage pretrained GANs and their latent spaces as a proxy for brain activity. For this, access to the latents of the visual stimuli is required so that a linear model can be fit on these latents and the neural data, after which predicted latents from held-out brain activity can be fed to the GAN for image reconstruction. Yet, the inherent nonlinearities in the transformation from latent space to image space by the generator render it inherently unidirectional. Post-hoc approximate inversion has shown to work to some extent but entails information loss [62, 65] (note that VAEs do approximate inference by design). Instead, to have direct access to the ground-truth latents, [36] used synthesized stimuli by a pretrained progressively grown GAN, which was the state-of-the-art generative model for generating high-quality and high-resolution images at the time. The current work adopted and improved this experimental paradigm to study neural representations in the ventral visual stream during visual perception.

Finally, an earlier study already showed that disentangled latent units learned by a β-VAE better explained the coding of single neurons in the primate inferior temporal (IT) cortex at the end of the ventral stream during face perception [66]. This further underscores the potential of such generative models to unravel intricate neural representations and their interactions with complex visual stimuli.

2.1 Neural encoding

We studied how well neural responses were predicted from latents of recent generative models. Specifically, we focused on three types of latents: z-latents of StyleGAN3/StyleGAN-XL (512-/128-dim.), feature-disentangled w-latents of StyleGAN3/StyleGAN-XL (512-/512-dim.) and language-regularized CLIP-latents (768-dim.). In the case of natural images, we used the embedding that integrated both z-latent and class information, which serves as the input for the first layer of the mapping MLP. For each individual unit within a multi-unit microelectrode (960 individual units in total), we fit three distinct kernel ridge regression models on the aforementioned z-, w- and CLIP-latents, of which the optimal regularization parameter λ was determined per visual area using 5-fold cross-validation.

For reference, we also fit three distinct encoding models on feature representations extracted from the discriminative VGG16 network, which was pretrained for either face or object recognition. Concretely, we used early (1; layer 2/16, after max pooling), middle (2; layer 7/16, after max pooling) and deep (5; layer 13/16, after max pooling) activations for this purpose. Note that the numbering system ‘1, 3, 5’ refers to the max pooling operations in VGG16, which has a total of five max pooling layers. This numbering is used for the remainder of this manuscript. The encoding performance was quantified by Pearson product-moment correlation coefficients. Notably, among the generative-based encoders, the w-latent-based encoder statistically outperformed those of z- and CLIP-latents in predicting the neural activity (Figs 5 and 6). For face images, the w-latent-based encoder demonstrated significant superiority over the z-based encoder (2-Sample T-Test; t(1918) = −13.8067, p = 2.07e–41) as well as the CLIP-based encoder (2-Sample T-Test; t(1918) = 16.0527, p = 1.65e–54). Additionally, the CLIP-based encoding also outperformed the z-based encoding (2-Sample T-Test; t(1918) = 2.1068, p = 0.0353) although the difference was not as pronounced. Similarly, for natural images, the w-latent-based encoder significantly outperformed the z-based encoder (2-Sample T-Test; t(1918) = −44.4495, p = 3.13e–297) and the CLIP-based encoder (2-Sample T-Test; t(1918) = 6.2957, p = 3.78e–10). And CLIP-based encoding also outperformed z-based encoding (2-Sample T-Test; t(1918) = −35.3777, p = 1.79e–211). Fig 7 directly compares the raw w-based encoding performance across visual areas and shows that the w-latents of natural images mainly captured visual information relevant to high-level neural activity, as indicated by the increasing variance explained from V1 to IT. This pattern was however not observed for face images.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Encoding performance.

The effectiveness of each encoding model is assessed using the Pearson correlation coefficients between predicted and recorded neural responses. For each dataset, the first and second graphs denote discriminative and generative representations, respectively. The correlation distribution across each encoding model shows a robust level of accuracy.

https://doi.org/10.1371/journal.pcbi.1012058.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Generative-based encoding performance.

For each individual microelectrode unit, we fit three encoding models based on three distinct feature representations: z-, w− and CLIP-latent representations. As such, we fit 3× 960 independent encoders, resulting in 3× 960 predicted neural responses because there were seven, four and four microelectrode arrays (64 units each) for V1, V4 and IT, respectively (i.e., 7 × 64 = 448 in V1, 4 × 64 = 256 in V4 and 4 × 64 = 256 in IT). The scatterplots display the prediction-target correlation (r) of one encoding model on the X-axis and another encoding model on the Y-axis to investigate the relationship between the two. Each dot represents the performance of one modeled microelectrode unit in terms of both encoding models (960 dots per plot, in total). Negative correlation values were set to zero. The diagonal represents equal performance between both models. The critical r-value at Bonferonni-corrected α = 5.21e–5 is at r = 0.3895 and r = 0.2807 for faces (df = 100) and natural images (df = 200), respectively, and is denoted by the shaded area. It is clear that w-latents outperform both z- and CLIP-latents because most dots lie in the direction of the w-axis (above the diagonal). The stars indicate the mean correlation coefficient per region of interest based on the data points outside the shaded area. Face images in this figure are replaced for copyright reasons. The original version of the figure can be accessed here.

https://doi.org/10.1371/journal.pcbi.1012058.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. w-based encoding performance across visual areas.

The left panel presents the distribution of correlation coefficients for face images using a swarm plot, with mean values indicated for V1 (0.53), V4 (0.52) and IT (0.53). The right panel displays the distribution for natural images, with mean values for V1 (0.40), V4 (0.47) and IT (0.56).

https://doi.org/10.1371/journal.pcbi.1012058.g007

As discussed in ‘earlier work’, the complexity gradient observed across the ventral stream in the brain is reflected in the multi-layered architecture of discriminative DNNs [1, 3–8, 11, 12]. As such, the representations extracted from early layers are more predictive of responses in early visual areas, while the deeper representations are more predictive of responses in more downstream areas. This refers to the progression from simpler, lower-level visual processing in early visual areas, like V1, to more complex, higher-level processing in areas like IT. We reproduced this complexity gradient by assigning the discriminative representation with the highest encoding performance to each microelectrode unit on the brain (Fig 8, bar graphs in the first column). Subsequently, we explored the positioning of generative w-latents along this complexity gradient. To this end, we replaced each level of discriminative feature representation with the w-latent representation to see where along this gradient the w-latents have the most predictive power (Fig 8). The results of this comparative analysis revealed that w-latents of both image types were predominantly assigned to the higher end of the complexity spectrum. This indicates that w-latents captured visual features particularly relevant to high-level neural activity. This positioning should not be interpreted as a competition between discriminative and generative latents; rather, it highlights their complementary nature as high-level representations in the overall hierarchy of neural encoding. It is possible that while w-latents explain more variance in higher visual areas for both face and natural images, the increase in variance explained from V1 to IT (the gradient) was more pronounced for natural images than for face images (as suggested by Fig 7) and therefore not as apparent when looking at w-latents in isolation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. w-latents explain high-level brain activity.

Three encoding models were fit on early (1; layer 2/16), middle (3; layer 7/16) and deep (5; layer 13/16) feature representations of VGG16 pretrained for face/object recognition. The representation that led to the highest encoding performance was assigned to each microelectrode unit, resulting in the complexity gradient where more low-level and high-level representations are assigned to earlier and more downstream brain areas, respectively (see most-left graph for reference). In each of the three plots, one VGG16 representation was replaced by the w-latent representation to see where it falls on the complexity gradient. The results illustrate that w-latents predominantly accounted for neural responses in downstream IT. Face images in this figure are replaced for copyright reasons. The original version of the figure can be accessed here.

https://doi.org/10.1371/journal.pcbi.1012058.g008

It is worth noting that encoders based on discriminative models appear to generally outperform those based on generative models. The performance proximity of w-latents to some discriminative-based predictions suggests that feature disentanglement in generative models may enhance their predictive capabilities. However, generative models inherently differ from discriminative models in their primary function and approach to data. Nevertheless, given that the w-based and discriminative-based encoders are compared, we statistically analyzed these comparisons for a more informed understanding of the relative strengths of each encoding approach. For faces, the comparison between w-based and VGGFace-1-based encoders shows no significant difference (2-Sample T-Test; t(1918) = −0.2859, p = 0.78) but we found highly significant differences when comparing w-based encoders with VGGFace-3 (2-Sample T-Test; t(1918) = 16.5817, p = 8.21e − 58) and with VGGFace-5 (2-Sample T-Test; t(1918) = 15.0820, p = 1.17e − 48). For natural images, the disparity between w- and VGGFace-1-based encoders also shows no significance (2-Sample T-Test; t(1918) = 0.7771, p = 0.4372). In contrast, we observed a highly significant difference between w-based and VGGFace-3-based encoders (2-Sample T-Test; t(1918) = 12.7855, p = 5.56e–36) and a significant difference between w-based and VGGFace-5-based encoders (2-Sample T-Test; t(1918) = 3.7425, p = 0.0002). Despite their statistical similarity to the encoders based on early activations of VGG16–1, w-based encoders were mainly predictive of high-level brain activity in IT.

2.2 Neural decoding

The ‘analysis’ component of neural decoding was modeled by multiple linear regression from neural responses to the feature-disentangled w-latents, which were subsequently fed to the generator for ‘synthesis‘. This resulted in remarkably accurate reconstructions that closely resembled the stimuli in their specific characteristics; Figs 9 and 10. Perceptually, we can notice a high similarity between stimuli and their reconstructions in terms of their specific attributes (e.g., gender, age, pose, haircut, lighting, hair color, skin tone, smile and eyeglasses for faces; shapes, colors, textures, object locations, (in-)animacy for natural images). We repeated the experiment with another macaque that had silicon-based electrodes in V1, V2, V3 and V4 (S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9.

Qualitative reconstruction results: The 100 test set stimuli (top row) and their reconstructions from brain activity in V1, V4 and IT (bottom row) via w-latents. Face images in this figure are replaced for copyright reasons. The original version of the figure can be accessed here.

https://doi.org/10.1371/journal.pcbi.1012058.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10.

Qualitative reconstruction results: The 200 test set stimuli (top row) and their reconstructions from brain activity in V1, V4 and IT. (bottom row) via w-latents.

https://doi.org/10.1371/journal.pcbi.1012058.g010

The supplementary materials contain decoding results from z-latents (S2 Appendix) and another reconstruction approach based on [28] (S3 Appendix). The former not only demonstrates superior performance using w-latents over z-latents in conditional image generation but also that this disentanglement enables unconditional image generation using GANs. Furthermore, a leave-one-class-out analysis confirmed that our approach extends beyond mere classification (S5 Appendix).

The quantitative metrics in Table 1 show the similarity between stimuli and their reconstructions from brain activity in terms of six metrics that evaluated reconstruction quality at different levels of abstraction (see S7 Appendix for the visual guide). Specifically, a stimulus and its reconstruction were both fed to VGG16 (pretrained on face- and object recognition for faces and natural images, respectively) and we extracted five intermediate activations (the five MaxPool layers) thereto. The early layers capture more low-level features (e.g., edges and orientations) whereas deeper layers capture increasingly higher-level features (e.g., textures to object parts to entire objects). We then compared the cosine similarity between these extracted representations of stimulus and reconstruction. Next, to study the decoder that resulted in these accurate reconstructions, the contribution of each visual area was determined by the occlusion of the microelectrode recordings in the other two brain areas (rather than fitting three independent decoders on subsets of brain activity). It is reasonable to say that, of the three cortical areas, the area that resulted in the highest similarity contains the most information about that representation. For faces, decoding performance was for the largest part determined by responses from IT—which is the most downstream site we recorded from. For natural images, we found that the lower-level representations (VGG16 layers 1–2) were most similar when decoded from V1 and the higher-level representations (VGG16 layers 3–5) and latent space were most similar when decoded from area IT. We validated our quantitative results with a permutation test as follows: per iteration, we sampled a hundred/two-hundred random latents from the same distribution as our original test set and generated their corresponding images. We assessed whether these random latents and images were closer to the ground-truth latent and images than our predictions from brain activity, and found that our predictions from brain activity were always closer to the original stimuli than the random samples for all metrics, yielding statistical significance (p < 0.001) (in S6 Appendix, the results of random permutation analyses can be found).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Quantitative results.

The upper and lower block display model performance (mean ± std.error) when reconstructing face images and natural images, respectively, in terms of six metrics of perceptual cosine similarity using the five MaxPool layer outputs of VGG16 for face recognition (face images) / object recognition (natural images) and latent cosine similarity between w-latents of stimuli and their reconstructions. The rows display decoding performance when using the recordings from all recording sites (i.e., V1, V4 and IT together) or the recordings within a specific brain area.

https://doi.org/10.1371/journal.pcbi.1012058.t001

2.2.1 Time-based neural decoding.

Time-based neural decoding showed the gradual extraction of stimulus-related information over the trial of 300 ms, with stimulus presentation occurring at 100 ms, by sliding a 100 ms time window across the entire time course using a stride of 25 ms, resulting in nine averaged points of neural activity across time (Fig 11A). We fit separate decoders for individual time points but decoding via the original decoder, which was fit on brain activity within the predefined time windows, yielded similar results. Initially, the reconstructions exhibited an average appearance, but then gradually acquired their distinct visual features upon stimulus onset (Fig 11B and 11D). Noteworthy, the reconstructions prior to stimulus onset exhibit an average-looking appearance because we averaged multiple repetitions in the test set, where each repetition was preceded by a different stimulus due to the randomized order of stimulus presentation. Although canceled out following our approach, it remains highly plausible that the information about the preceding stimulus is still preserved in the brain. Moreover, the area-based reconstructions and performance graphs revealed that V1 generally displayed stimulus-like visual features earlier in time whereas IT consistently outperformed the other two in the final reconstruction of stimulus information (Fig 11C and 11E). Albeit trivial, the finding that reconstruction from all rather than isolated areas yields the highest performance confirms that visual perception involves a distributed process across multiple areas that each hold distinct information about the stimulus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Time-based decoding.

A For each trial, responses were recorded for 300 ms with stimulus onset at 100 ms. Rather than taking the average response within the original time windows (see the three color-coded windows for V1, V4 and IT), we slid a 100 ms window with a stride of 25 ms over the entire time course, resulting in nine average responses across time. B, D Two stimulus-reconstruction examples evolve over time for faces and natural images, respectively. C, E Decoding performance over time for faces and natural images, respectively. The error bars denote the standard error of the cosine similarities between features of stimuli and reconstructions. It can be noted how V1 performance climbs up slightly earlier in time than the other two visual areas. For faces, IT outperforms V1 and V4 in most instances. For natural images, V1 outperforms V4 and IT for low-level feature similarity, after which V4 and IT climb up together and outperform V1 for the more high-level feature similarity metrics. Face images in this figure are replaced for copyright reasons. The original version of the figure can be accessed here.

https://doi.org/10.1371/journal.pcbi.1012058.g011

2.2.2 Linear operations.

The application of linear operations to GAN-latents directly translates to meaningful perceptual changes in the generated images because visual data that look perceptually similar in terms of certain features are also closely positioned in latent space. As such, pathways through the well-structured latent landscape can be explored by interpolating two distinct latents, resulting in an ordered set of images whose semantics vary smoothly with latent codes [67] (Fig 12A1 and 12B) and simple arithmetic operations [68] (Fig 12A2). Since such operations can be performed to traverse the latent space (Fig 12, row 1), without needing to understand the intricate details of the underlying generator network, the latent-response correspondence also opens the door to interpreting neural activity in terms of such operations within the latent space (Fig 12, row 2). To illustrate, consider having a neural response to a neutral face and another neural response to a smiling face, interpolating their respective decoded latents yields a sequence of latents, and consequently, a series of images transitioning from neutral to smiling expressions. Note that both the operations applied to neural activity as well as the decoder are linear, resulting in “linearity stacking”. This means that we can also apply the linear operations directly to the neural responses themselves, decode them into latents, and feed them to the GAN for reconstruction. This would yield the same images as those in row 2 of Fig 12.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Linear operations to latent codes.

(row 1) shows linear operations to two ground-truth w-latents and (row 2) to two predicted w-latents from brain activity. The linearly-manipulated latents were then fed to the generator for image generation. (A1, A2) face images, also contains vector arithmetic. (B) As for (A1, A2) but for natural images. Face images in this figure are replaced for copyright reasons. The original version of the figure can be accessed here.

https://doi.org/10.1371/journal.pcbi.1012058.g012

The linear relationship between neural activity patterns and latent codes, coupled with the feature-disentangled nature of the GAN’s latent space, enables synthesis (and analysis) of specific aspects of the visual experience captured by the neural responses—which is the key idea of our decoding approach.

3.1 Uncovering principles of neural coding

The primary goal of our study was to uncover principles that govern neural coding of the visual world and gain a more interpretable understanding of high-level neural representations underlying visual perception using deep generative modeling. As such, similarities between w-latents and the brain could provide further insights into what drives the organization of visual processing in the brain. First, GANs are trained in an unsupervised setting; they learn directly from raw visual data without explicit labels or annotations. Not only does this make GANs more biologically plausible than their supervised counterparts since it resembles more closely how the brain learns from its environment but they may also lead to more flexible and generalizable representations that are better able to capture the underlying structure and patterns in the observed data. Note that our finding that discriminative-based encoders (supervised) outperform w-based encoders (unsupervised) in neural encoding does not directly challenge this notion since these models were optimized for different objectives (i.e., image recognition and image generation, respectively). Second, StyleGAN was designed to disentangle different visual semantics into separate w-latent features. The superior performance of w-latents relative to other generative latents highlights the role of feature disentanglement in explaining the high-level neural representations and the ability to disentangle the object manifold [69]. Keep in mind that StyleGAN itself has never been optimized on neural data which implies a general principle of shared encoding of real-world phenomena. Finally, there is a conceptual analogy between the adversarial training of GANs and the predictive coding theory of perception where the brain uses top-down predictions, based on prior knowledge and experience, to guide bottom-up sensory processing and adjusts its internal models based on the mismatch between expectations and actual observations. In GANs, the discriminator and generator engage in a similar process with the discriminator evaluating the “real” sensory input and the “predicted/imagined” instances by the generator. Based on the mismatch, as determined by the discriminator, the internal model of the generator is refined such that its outputs match the real-world data closer; the generator harnesses the knowledge of the discriminator to learn how to represent the world in its latent space. And like generative models interpolate between latent vectors to create intermediate outputs, the brain might engage in similar processes that interpolate between neural representations to accommodate variations in mental simulation. So, while the exact mechanisms used by the brain and GANs differ significantly, their conceptual similarities could provide insights into the nature of perception and the potential of machine learning to capture some of the same principles underlying this ability.

3.2 Limitations and future directions

It is essential to clarify that the brain’s overall functioning is far more complex than a linear system; our approach merely exploits the linearity within a particular representation space. Further investigation of the correspondence between latents and responses is needed by, for instance, obtaining neural responses to more diverse stimuli and stimulus manipulations to identify which visual properties could be effectively translated to the latent space and where this approach falls short. We did observe that, within the limitations of StyleGAN-XL’s design tied to the natural image distribution, the generator exhibits the capability to synthesize abstract stimuli (see S8 Appendix), which offers a promising perspective for future investigations in this direction. Further, this study solely used synthesized stimuli with known latent representations generated by StyleGAN. While this allowed for a controlled and systematic examination of neural representations of visual information, future studies should also include real photographs to see how this method generalizes. This requires accurate inversion methods of the generator’s ‘synthesis’ operation, yet this endeavor is intricate due to the inherent information loss associated with post-hoc inference. That said, the current study still performed valid neural encoding and reconstruction from brain activity despite the nature of the presented images themselves. Another limitation is the small sample size of one subject (note that we did include face reconstructions from a second subject with different cortical implants in S1 Appendix). Although small sample sizes are common in studies using invasive recordings—larger sample sizes are needed to further confirm the robustness of our findings. Finally, it is worth noting that the use of deep neural networks to model brain activity is still a developing field and the models used in this study are not flawless representations of the underlying neural processes.

4.1 Ethics statement

All procedures complied with the NIH Guide for Care and Use of Laboratory Animals and were approved by the local institutional animal care and use committee of the Royal Netherlands Academy of Arts and Sciences.

In conjunction with the evolving field of neural decoding grows the concern regarding mental privacy [70]—a concept that safeguards the sanctity of individual cognitive experiences. Importantly, our methodology included extensive datasets for which constant and complete subject cooperation was required throughout the process to decode very specific information from the brain. Together with the invasive nature of our approach, which entails surgical interventions, this presents substantial barriers to any unsolicited invasion of mental privacy. Furthermore, it is important to at all times strictly follow ethical rules and regulations that govern data extraction, storage, and protection. Finally, this work solely concentrated on reconstructing visual perception; it has not extended into the domains of imagery or dreams which are more closely aligned with private cognitive experiences.

4.2 Stimuli

StyleGAN [42, 71] was developed to optimize control over the semantics in the synthesized images in single-category datasets (e.g., only-faces, -bedrooms, -cars or -cats) [43]. This generative model maps z-latents via an MLP to an intermediate w-latent space in favor of feature disentanglement. That is, the original z-latent space is restricted to follow the data distribution that it is trained on (e.g., old-looking faces wear eyeglasses more often than young-looking faces) and such biases are entangled in the z-latents. The less entangled w-latent space overcomes this such that unfamiliar latent elements can be mapped to their respective visual features.

4.3 Features

As the in-between feature candidates, we used the (generative) z-latents of StyleGAN3/StyleGAN-XL (512-/ 128-dim.), w-latents of StyleGAN3/StyleGAN-XL (512-/512-dim.) and CLIP-latents (768-dim.). In the case of natural images, we used the embedding that integrated both z-latent and class information, which serves as the input for the first layer of the mapping MLP. We also used the five (discriminative) layer activations of VGG16 for face recognition [76] and object recognition [77]. Specifically, we utilized the outputs from layers 2/16, 4/16, 7/16, 10/16, 13/16, referred to as layers 1–5, following max pooling. Because the features from layer 1 and 2 were very large (∼ 10⁶), we performed downsampling, as done in [11]. That is, for each channel in the activation, the feature map was spatially smoothed with a Gaussian filter and subsampled with a factor 2. The kernel size was set to be equal to the downsampling factor.

4.4 Responses

We recorded multi-unit activity (MUA) [48] with 15 chronically implanted electrode arrays (64 channels each) in one macaque (male, 7 years old) upon presentation with images (resized to 500 × 500 px) in a passive fixation experiment (Fig 4). During the experiment, 4000 training images were each presented once, which ensured that these training set responses covered a diverse set of stimulus variations (note that repetitions would limit the total number of distinct images presented). In contrast, 100/200 test set images were each presented twenty times to increase the signal-to-noise ratio, which facilitated more reliable assessment and interpretation. The images were presented in a randomized order. Next, neural responses were recorded in V1 (7 arrays), V4 (4 arrays) and IT (4 arrays) leading to a total of 960 channels (see electrode placings in Fig 2). For each trial, we averaged the early response of each channel using the following time windows: 25–125 ms for V1, 50–150 ms for V4 and 75–175 ms for IT. To capture feedforward processing in each region, the time windows were centered on the response peaks and averaged across trials and channels, as determined on an independent dataset of responses to 22k natural images. The 100 ms window length accounted for the variability of response latency across channels and stimuli. Normalization was carried out as in [78], such that for each channel, the mean response was subtracted from all the values which were then divided by the standard deviation.

To determine the contribution of the activity in each brain region to the overall model performance, we evaluated the decoder using partially occluded test set data. Concretely, we used our main decoder which was trained on neural data from all three brain areas and evaluated it using test set recordings from one brain area. To do this, the responses from the other two areas were occluded by the average response of all but the corresponding response. Alternatively, one could also evaluate the contribution per region by training three independent decoders on subsets of neural data (V1-only, V4-only and IT-only) which would allow for evaluation of the contribution of each brain area independently of one another. But in our case, we used the occlusion approach to investigate the area-specific contribution to the same decoder’s performance by keeping the contributions from the other two areas constant.

4.5 Models

We used linear mapping to evaluate our claim that the feature- and neural representation effectively encode the same stimulus properties, as is standard in neural coding [6, 79]. A more complex nonlinear transformation would not be valid to support this claim since nonlinearities will fundamentally change the underlying representations.

4.6 Evaluation

Decoding performance was evaluated by six metrics that compared the stimuli from the held-out test set with their reconstructions from brain activity: perceptual cosine similarity using the five MaxPool layer outputs of VGG16 and latent cosine similarity. For perceptual cosine similarity, we computed the cosine similarity between layer activations (rather than pixel space which is the model input) extracted by VGG16 pretrained for object recognition. This metric reflects human perception of similarity better because it takes more high-level visual cues into account (e.g., color, texture and spatial information) and human perception is often not directly related to the pixel values themselves. Specifically, we fed the stimuli and their reconstructions to the DNN and then considered the cosine similarity per activation unit: where x and are the visual stimuli and their reconstructions, respectively, n the number of activation elements and f(.) the image-activation transformation. For latent similarity, we considered the cosine similarity per latent dimension between predicted and ground-truth latent vectors: where and w are the 512-dimensional predicted and ground-truth feature-disentangled latent vectors, respectively.

4.7 Implementation details

All analyses were carried out in Python 3.8 on a cloud-based virtual machine with Intel(R) Xeon(R) CPU @ 2.20GHz and NVIDIA Tesla T4 GPU (Driver Version: 510.47.03, CUDA Version: 11.6) on a Linux-based operating system. We used the original PyTorch implementations of StyleGAN3 and StyleGAN-XL to generate the faces and natural images in this manuscript. We used VGG16 for face recognition and object recognition for analysis of the faces and natural images. The scripts to generate the visual datasets as well as our implementations of neural encoding and -decoding can be found on our GitHub repository.