PhD Project 15

Data driven investigation of intrinsic dynamic brain states underlying language processing

 

PhD-candidate: Julia Berezutskaya
PIs: Nick Ramsey (WP6) and Peter Desain (WP7)
Start date: 01 March 2015

(last update 2019-06-27)

Research Content

State of the art in sensory, and recently associative, cortex research reveals a detailed topographical representation, with indications that functions are represented in groups of neuronal ensembles.
The goal of this project is to extract structures in language processing from brain activity dynamics, informed by symbolic and high-dimensional representations of recurring and dominant brain events. Extensive sets of unlabelled ECoG data from epilepsy patients will be used in combination with machine learning to identify canonical brain responses occurring during conversations and listening to television. These structures may map upon existing conceptual language frameworks or generate novel avenues for future research.

Highlight

Data driven investigation of intrinsic dynamic brain states underlying language processing
Team members: Freudenburg (UMCU), Ramsey, Van Gerven, Güçlü (DCC), and Ambrogioni

The goal of this project is to extract structures in language processing from brain activity dynamics, informed by symbolic and high-dimensional representations of recurring and dominant brain events. Extensive sets of unlabelled ECoG data from epilepsy patients and machine learning are used to identify canonical brain responses occurring during conversations and listening to television. These structures may map upon existing conceptual language frameworks or generate novel avenues for future research.

Affinity propagation clustering of regression coefficients in ECoG electrodes tuned to speech. For each of the six clusters, a number of properties are shown: the number of electrodes belonging to the cluster (N), the normalized electrode locations on the MNI brain, the distribution of electrodes over patients (half pie charts with number of patients out of 15), and the feature tuning profiles over lags, temporal modulations (TMs), and spectral modulations (SMs). Right hemisphere electrodes were projected on the left hemisphere and marked with star symbols; circles show left hemisphere electrodes. The size of electrodes on the MNI brain reflects the similarity of the electrode's feature tuning profile to the exemplar electrode of the cluster (framed in pink). The sectors of the half pie charts show percentage of each patient's electrodes relative to the overall number of electrodes in the cluster (N). The lags were used in the range of 0 to −500 ms with respect to audio onset; TMs were in the range of 0.25–64 Hz; SMs were in the range of 0.03–8 cyc/oct. Per cluster, mean (colored bold curve) and SD (colored shading) over regression coefficients are plotted for each dimension. The mean (gray bold curve) and SD (gray shading) of the null distribution based on permutations of cluster assignments are shown. The black thick line at the bottom of several plots represents significant segments of the tuning profile compared with the null distribution (p < 0.01, Bonferroni corrected).

One of the findings is that using a completely data-driven approach information can be extracted about which acoustic properties of perceived speech are encoded in the neural responses, at which latency and with which specific spatial distribution of the neural sources.
Specifically, it is found that various temporal modulations of speech are encoded in a form of a gradient along the perisylvian cortex with posterior superior temporal gyrus encoding fast modulations (>15 Hz) and anterior regions reaching inferior frontal gyrus encoding slow modulations (< 3 Hz).
In the context of a naturalistic audio-visual perception (as in watching a movie), a flow of information between the posterior superior temporal gyrus, inferior frontal gyrus and the medial temporal region within the first 400 ms of the speech onset was also observed.

A tentative timeline of neural processing of audiovisual speech in -50 to 300 ms around the sound onset. The plots contain cortical weight maps for selected key BO-NN features (|β| > 1.3). Six selected key features are shown together with the associated cortical distributions (cortical weight maps per feature). Larger weights indicate a stronger positive relationship between the time course of the key feature and the corresponding cortical location (ECoG electrode), negative weights denote a negative relationship. The ECoG electrode locations were projected from individual subjects (n=29) to a standard MNI brain. Below each cortical weight map is displayed a STMF-correlation profile for two dimensions: temporal modulations (TMs) and frequency (Spearman correlation, p < 0.001, Bonferroni corrected). This shows the relationship between the features extracted in a data-driven way (BO-NN) with a theory-driven approach based on spectrotemporal modulation filters (STMFs). Below the STMF-correlation profiles are example time courses of each key feature (black trace) during a speech fragment. Moments of speech are highlighted in red. The bold black dashed line at the bottom denotes step in the timeline of -50 to 300 ms around the speech onset.

Different parts of the project combine recent advances in machine learning, neuroscience, natural language processing, and computer vision. One of the innovative aspects is a data-driven approach to extracting stimulus features that driven the brain responses. Focus is also on building up large multi-purpose dataset of neural recordings to share with the neuroscience community.

During the project multiple collaborations have been forged with RU, the University of Freiburg and colleagues in the USA. Collaboration allows for joining and sharing of expertise across different labs and using non-standard approaches to problem solving.

Progress 2018

Extension of the last project on extraction of automatic data-driven audio features lead to optimal predictions of the neural responses to natural sound. Currently this work is being finalized and the manuscript is being prepared.
Two more projects are in their final stages as well: (1) a project on high-density ECoG motor responses to naturalistic speech perception, and (2) a project on decoding of visual semantics from whole-brain ECoG responses to a short film. Work is also done towards compiling and publishing large ECoG datasets to promote data sharing and open science in the neuroscience community.
Additionally, in collaboration with Ambrogioni (postdoc, see pages 17 and 18), a probabilistic programming toolbox Brancher (Python) is currently under construction as a basis for a larger project on neuroprobabilistic programming aimed at construction of biologically plausible models of whole-brain neural computations integrated with behavioural and cognitive evidence in the framework of Bayesian deep learning.

Groundbreaking characteristics

This project bridges clinical neuroscience with artificial intelligence, translating mathematical principles to the clinical setting, with potential direct applications in the clinic for language brain mapping. It contributes joint development of data mining techniques and new research strategies to obtain greater insight into cortical organization of language.