The Unreliable Influence of Noise Normalization on the Reliability of Neural Dissimilarity

Representational similarity analysis (RSA) is increasingly part of the standard analytic toolkit in neuroimaging. Core to RSA is the measuring of neural dissimilarity between the response patterns for different conditions to construct neural representational dissimilarity matrices (RDMs). It has been proposed that noise normalizing these patterns, and using crossvalidated distances as a dissimilarity measure, is superior for characterizing the structure of neural RDMs. This assessment has been motivated by improvement in within-subject neural dissimilarity after noise normalization. However, between-subject reliability is more directly related to determining the amount of explainable variance, and the evaluation of observed effect sizes when they are correlated with behavioral or model RDMs. Across three datasets we did not find that noise normalization consistently boosts within-subject reliability, between-subject reliability or correlations with behavioral or model RDMs. Overall, our results provide equivocal support for the utility of noise normalization to RSA.


Introduction
Representational similarity analysis (RSA) is increasingly part of the standard analytic toolkit utilized in neuroimaging. Central to RSA is the measuring of neural dissimilarity between the neural response patterns for different conditions to construct neural representational dissimilarity matrices (RDMs). Previously, it has been proposed that noise normalizing these patterns, and using cross-validated distances as a dissimilarity measure, is superior for characterizing the structure of neural RDMs (Walther et al., 2016). This assessment has been motivated by resulting suggesting improvement in within-subject neural dissimilarity after noise normalization.
However, when it comes to carrying out RSA, between-subject reliability is also related to determining the amount of explainable variance, and the evaluation of observed effect sizes when they are correlated with behavioral or model RDMs. Therefore, we sought to further investigate this issue by revisiting the impact of noise normalization on the reliability of measures of neural dissimilarity. Across three datasets, we did not find that noise normalization consistently boosted within-subject reliability, between-subject reliability or correlations with behavioral or model RDMs. Overall, our results provide equivocal support for the utility of noise normalization to RSA.

Datasets
Datasets from three previously published studies were reanalyzed. Preprocessing (e.g. motion correction) and first-level analysis of the data followed standard pipelines (e.g. SPM). Further methodological details can be found in those works. Notably, unlike in the original studies, in the present reanalysis data was not smoothed.

501
This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0

Dissimilarity Measures
Four measures were used to calculate pairwise dissimilarity (Walther et al., 2016). First, the Correlation Distance (COR) dissimilarity was estimated based on 1 -r, where r is the pairwise Pearson's correlation between response patterns averaged across runs. Second, the Classification Accuracy (CLA) Dissimilarity was estimated based on the pairwise accuracy of a linear discriminant analysis (LDA) classifier using leave-one-run-out crossvalidation. Third, the Euclidean Distance (EUC) Dissimilarity was estimated based on the (leave-onerun-out) cross-validated Euclidean distance: the pairwise distance between responses in the training data was multiplied by the transposed distances from the test data. Finally, the Malahanobis Distance (MAL) Dissimilarity was estimated based on the (leave-onerun-out) cross-validated Malahanobis distance: pairwise distances between training data responses were multiplied by the training data covariance, and transposed distances of the test data.

Noise Normalization
Following Walther et al., the data of individual subjects was normalized using the run-wise covariance between voxels as estimated by the GLM residuals from the first level analysis. To render the covariance matrices invertible, they were regularized using a shrinkage function.

Estimating Reliability and Effect Sizes
To estimate the within-subject reliability, for each subject data was split into odd and even runs, single RDMs (for each dissimilarity measure) were constructed for each split and correlated together (Pearson's r). For between-subject reliability, for each subject a single RDM (for each dissimilarity measure) was constructed, the RDM of one subject was correlated with the average of the remaining subjects, and then values were averaged across all leave-onesubject-out folds.
Neural RDMs of individual subjects were correlated with one of three behavioral or model RDMs: for Dataset 1, pairwise similarity judgments based on stimulus orientation and spatial frequency; For Dataset 2, multiple-arrangement similarity based on object shape similarity; and For Dataset 3, a binary model matrix differentiating social vs. non-social videos.
In order to assess the impact of noise normalization on the within-and between-subject reliability of RDMs, and correlation effects, a linear mixed effects model was fit to the data with normalization and measure as fixed effects, and subject as a random grouping effect. Normalization had a significant effect on the reliability for D1 (F(1,72) = 19.36, p = 3.67e-5). There was also a significant effect of measure (F(3,72) = 11.63, p = 2.66e-6), and a significant interaction between the two (F(3,72) = 10.72, p = 6.56e-6). When visualized ( Figure  1), one can see that across measures noise normalization improve within-subject reliability, but the size of this improvement varies with measure, which also vary in baseline reliability.

Discussion
Noise normalization has been billed as a useful method for improving the reliability of neural RDMs (Walther et al., 2016). We revisited this issues to assess the extent to which noise normalization provides a net benefit to within-subject reliability of neural RDMs, while also assessing its impact on between-subject reliability, and correlations with behavioral or model RDMs.
Our findings were mixed. For only 1/3 datasets did we see a clear positive impact of noise normalization on within-subject reliability, while for 1/3 datasets reliability decreased after normalization. Across all three datasets, there was either no effect, or no positive effect, of noise normalization on between-subject reliability, which is typically used as the estimate of the noise ceiling for RSA. Finally, for 2/3 datasets noise normalization did seem to somewhat improve the correlations of neural RDMs with behavioral ones, while again for one dataset it made the correlations worse.
Taken as a whole, our findings provide equivocal support for making noise normalization core to the RSA pipeline. It is notable that for D3, noise normalization consistently made both within-subject reliability and model RDM correlations worse. While these findings could be related to the fact that there were far fewer runs than for D1 and D2 (6 compared to 12 and 16), they may also be related to the fact that the ROI was TPJ, which lacks the sort of topographic organization known to exist in the ROIs for the other datasets (object-LOTC, and especially V1). Therefore, one possibility is that methods like noise normalization may only provide a benefit when regions are known to have some clear topographic organization.