Music can be reconstructed from human auditory cortex activity using nonlinear decoding models
Fig 1
Protocol, data preparation, and encoding model fitting.
(A) Top: Waveform of the entire song stimulus. Participants listened to a 190.72-second rock song (Another Brick in the Wall, Part 1, by Pink Floyd) using headphones. Bottom: Auditory spectrogram of the song. Orange bars on top represent parts of the song with vocals. (B) X-ray showing electrode coverage of 1 representative patient. Each dot is an electrode, and the signal from the 4 highlighted electrodes is shown in (C). (C) HFA elicited by the song stimulus in 4 representative electrodes. (D) Zoom-in on 10 seconds (black bars in A and C) of the auditory spectrogram and the elicited neural activity in a representative electrode. Each time point of the HFA (yi, red dot) is paired with a preceding 750-ms window of the song spectrogram (Xi, black rectangle) ending at this time point (right edge of the rectangle, in red). The set of all pairs (Xi, yi), with i ranging from .75 to 190.72 seconds constitute the examples (or observations) used to train and evaluate the linear encoding models. Linear encoding models used here consist in predicting the neural activity (y) from the auditory spectrogram (X), by finding the optimal intercept (a) and coefficients (w). (E) STRF for the electrode shown in red in (B), (C), and (D). STRF coefficients are z-valued and are represented as w in the previous equation. Note that 0 ms (timing of the observed HFA) is at the right end of the x-axis, as we predict HFA from the preceding auditory stimulus. The data underlying this figure can be obtained at https://doi.org/10.5281/zenodo.7876019. HFA, high-frequency activity; STRF, spectrotemporal receptive field.