Big Question 1

New Advances in Neural Networks for Encoding and Decoding of Neural Representations

Supervisors: Marcel van Gerven, Stefan Frank, and Roel Willems
Postdoctoral Research Associate: Umut Güçlü (Nov 2017 - Oct 2018) and Luca Ambrogioni (Oct 2018 onward)
Starting date: November 01, 2017

(last update 2019-07-03)

Research content

The goal of this postdoc project is to advance the state of the art in both encoding and decoding of linguistic and semantic representations using advanced neural network architectures. To this end, it will build on and significantly expand on research in this field conducted in the Artificial Cognitive Systems lab by developing a general computational framework for encoding and decoding of neural representations.
The framework will make extensive use of state-of-the-art neural networks and other machine learning techniques, which will be developed throughout this project. The models to be developed will be based on novel and efficient, deep, generative and/or recurrent neural network architectures that aim to capture biologically plausible human cognitive processing at an unprecedented level of detail. Specifically, artificial neural networks (with different architectures and tasks) and neural coding (in different modalities) will be used to answer the following questions:
- How are representations of language encoded in the human brain?
- How are short-term memory and working memory of language encoded in the human brain?
- How is inner speech encoded in the human brain?


Cortical information flow for system identification in neuroscience
Team members: Ambrogioni and Van Gerven

Cortical information flow (CIF) is a new framework for system identification in neuroscience. CIF models represent neural systems as coupled brain regions that each embody neural computations. These brain regions are coupled to observed data specific to that region. Neural computations are estimated via stochastic gradient descent. Using a large-scale fMRI dataset it was shown that, in this manner, models can be estimated that learn meaningful neural computations. The framework is general in the sense that it can be used in conjunction with any (combination of) neural recording techniques. It is also scalable, providing neuroscientists with a principled approach to make sense of the high-dimensional neural datasets.

Using fMRI data collected during prolonged naturalistic stimulation it was shown that BOLD responses across different brain regions could successfully be predicted. Furthermore, meaningful receptive fields emerged after model estimation. Importantly, the learnt receptive fields are specific to each brain region but collectively explain all of the observed measurements.  These results demonstrate for the first time that biologically meaningful neural information processing systems can be estimated directly from neural data.  CIF allows neuroscientists to specify hypotheses about neuronal interactions and test these by quantifying how well the resulting models explain observed measurements.

Features of V1 extracted using the Cortical Information Flow model.

The cortical information flow model has proven to be a powerful encoding model that can be used to jointly reproduce brain and behavioural data. The approach is very flexible as it learns all its parameters from the data. Consequently, this approach can be used to analyse neural data acquired using heterogeneous measurement devices and experimental settings in a common computational framework.

The project is the fruit of a strict collaboration between people with complementary fields of expertise. This is required as the development of the Cortical Flow model requires both sophisticated mathematical expertise and deep knowledge of the computational principles behind the human cortex.

Progress 2018

Work has been done on developing temporal and hierarchical language models for investigating temporal and hierarchical language representations in the brain. Several fMRI datasets are analysed to test how well the memories of the temporal language models and the layers of the hierarchical language models can explain brain activity in different regions.

At the same time, toolboxes (Python packages) are being developed to make it easy for others (both in and outside of the consortium) to use and/or understand these sophisticated methods, fostering collaboration and further facilitating progress toward the overarching quest of LiI. Brancher, a probabilistic programming toolbox that will be the core of future computational frameworks, is in the final stage of development. Brancher automatizes Bayesian inference problems using automatic differentiation and stochastic optimization. The toolbox is designed to be easy to use and to have an intuitive semantic interface.

The first paper on the Cortical Flow project is being finalized. The aim of this project is to model the brain’s information flow by training deep convolutional architectures directly on brain data. Work is also done on a CorticalFlow toolbox for designing brain models.