PhD Project 12

Neural processing of action, gesture and language in healthy and autistic individuals

PhD-candidate: James Trujillo
PIs: Asli Özyürek (WP4) and Harold Bekkering (WP4)
Start date: 01 February 2015

(last update 2019-06-27)

Research Content

It is often suggested that language and action on the one hand, and language and gestures on the other hand, share similar neurocognitive processes. It is investigated in more detail if this is true. fMRI repetition suppression paradigm will be used to identify the shared neuronal substrate for processing communicative and semantic information across these modalities. The features of the cross-modal processing in autism will be further explored. Finally, interpersonal differences in the processing of the communicative component in normal population will be assessed. The project will inform on the neural mechanism and relation between the different communicative modalities.


Recognizing communicative intent on the basis of kinematic cues
Team members: Trujillo, Simanova, Özyürek, and Bekkering

fMRI and motion tracking data was used to investigate the brain activation and connectivity patterns underlying recognition of communicative intent using only subtle kinematic cues.
It was found that the mentalizing and mirroring systems act together in response to the communicativeness of gesture kinematics. The brain activation and connectivity patterns are directly in line with what is seen in intention recognition in the action literature, as well as brain responses to unexpected or unusual stimuli.  This effect was specific to when participants paid attention to the communicative/social aspect of kinematics.

The findings show that intentions can be seen in movement kinematics, providing evidence for a domain-general mechanism of intention recognition. This has implications for human and non-human communication in general, evolution of communicative ability, and human-machine interaction.

Overview of brain activation correlating with communicatively modulated kinematics in a social and a non-social (handedness) perception task. Red areas indicate significant (p < 0.001) correlation between BOLD response and video communicativeness. The red colour bars show the corresponding T values. Panels A and B provide a slice by slice overview of the two tasks, while panels C and D provide a 3D rendering of the same data, with significant areas of interest highlighted (mPFC = medial prefrontal cortex; IFG = inferior frontal gyrus; TPJ = temporoparietal junction; MFG = middle frontal gyrus)

Progress 2018

Collaborative project with Drijvers (PhD, see page 75) led to large data collection and public outreach at Lowlands music festival. Methodological paper published in Behavior Research Methods. Manuscript on the ‘Semantic comprehension of gestures’ was completed and submitted. fMRI paper on the recognition of communicative intent was completed and submitted.
Ethics proposal was submitted in June of 2018 (accepted as of January 2019) for a new project in which gesture kinematics and semantic comprehension in high functioning autistic adults and neuro-typical adults is assessed.

Groundbreaking characteristics

The work was made possible by bringing together diverse expertise in naturalistic gesture recordings, fMRI and connectivity analyses, and the theoretical framework that had been built up primarily in the action domain.