Temporal Dynamics of Meaning

Language is a hallmark of human cognition. It allows for efficient communication between humans and is a building block for high-level cognition. The question of how lingual semantic meaning is represented in and produced by the brain is one of the great mysteries of cognitive neuroscience. In this work we compare the processes of semantic comprehension and production, focusing on neural processes that take place in the speaker’s and listener’s brains before the word was articulated. To that end, we are building a big dataset of high-quality brain data as patients speak and comprehend language in real-life conversations using intracranial EEG recordings. Using linear and deep models, we were able to track the triggered temporal dynamic for semantic comprehension and semantic production before and after word onset. In particular, we were able to read semantic information from the speaker’s brain and utilize it to identify the spoken words, hundreds of milliseconds before the moment of articulation, with some hints of pre-verbalization predictive coding in the listener’s brain much closer to word onset. These results demonstrate the difference between semantic comprehension and production, and shed light on the underlying processes allowing for the verbalization of inner neural states.


Introduction
The attempt to understand the neural representations of semantic concepts is fundamental to our quest to understand how language is embedded in the human brain. This line of research has produced some groundbreaking research on the representation of words and semantic concepts in the human brain (Akbari, 2019;Huth, et al., 2016;Pereira, 2018). However, the nature of the temporal dynamic of semantic meaning formation is still a mystery. Recent advances in deep machine learning methods opened new ways for modeling complex real-life data. However, such techniques rely on the availability of big data. Big data is lacking in most cognitive neuroscience research, which relies on the collection of minutes long data sets in highly controlled experimental conditions. In this study, we devised a novel pipeline for collecting large, high quality intracranial neural recordings of brain activity during the production and comprehension of natural speech in real-life conversations.

Method
Speech and intracranial neural recordings were recorded continuously (24/7) during free conversations at the epileptic unit at NYU's medical center from two epileptic patients. For each patient, the brain signals are recorded from ~100-200 implemented electrodes at a sampling rate of 512Hz during the entire stay of the patient at the epileptic unit (about ~3-6 days for each patient). This yielded about 42 hours of speech (comprehension ~120k words, production ~120k words) across the two patients. The speech was transcribed and aligned with the brain signal at milliseconds precision.

Results
Each word was represented by a 50-dimensional GLoVe (Pennington, Socher & Manning, 2014) semantic vector embedding. We started by using a linear model to predict the brain signal from the word embedding for different temporal lags relative to the onset of the words. For each lag we compute the Pearson correlation ( r ) between the predicted signal and the actual signal. In figure 1 we present the lag which denoted the maximal correlation between the 192 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0 predicted signal and actual signal. We present only electrode which correlation turned out significant.
Generally, we can see activation that precedes articulation during production in the motor, premotor and language areas, indicating processing involved with "translating" the brain state into semantic verbal units (words). During comprehension, on the other hand, maximal correlation is achieved post word onset (figure 1).
In motor related electrodes along the precentral gyrus (see one example in figure 2, electrode 1), we managed to encode information during production (blue) but not comprehension (red). Interestingly, we were able to encode semantic information for spoken words up to a few seconds before the moment of articulation (figure 3).
In high order language areas, such as the inferior frontal gyrus (where Broca's area is located) and inferior parietal cortex (where Wernicke's area is located) we managed to encode both production related and comprehension related information, at different moments in time (figure 2, electrode 2). In particular, it seems that high order language areas are involved with the processing of words before they are articulated, and the processing of articulated words during comprehension (figure 4).  Interestingly, some language related electrodes in the inferior frontal gyrus (figure 2, electrode 3) seem to be able to predict information about the upcoming word during the comprehension of incoming speech (red) even before it was articulated (figure 5).
In on-going work in the lab, we are applying deep models to predict the upcoming words.

Conclusion
Examining the brain dynamic during free conversations provides a window to numerous neurocognitive processes related to the production and comprehension of speech in real-life contexts. We witness the temporal dynamic of deciphering the semantic meaning for spoken words. The interpretation of significant semantic related activity that precedes word articulation (before lag 0) is different for production and comprehension. For production, this activity indicates preparatory processes. In comprehension, it indicates cognitive prediction of the following words. Our results start to shed light on the mysterious process by which pre-articulation neural patterns lead to the embedding of a thought in a spoken sentence.