Work Package 4

Language in action

The objective of this WP is to understand language processing and its variability during social interactions in multimodal settings. Language users operate in a constantly variable and dynamic context provided by multimodal sources of information. Although at a behavioural level there are detailed descriptions of multimodal language use in social interaction, the neural and cognitive infrastructure of this unique human ability is poorly understood.

Although language can be distinguished from other cognitive functions, such as perception, action and cognitive control, it is intertwined with these functions in many ways. Regarding the link between action and language, for instance, the actor needs to be able to predict the physical consequences of her own verbal and non-verbal behaviour and that of others. Moreover, in order to establish the actor’s goals, cognitive control is required in the service of selecting appropriate actions and monitoring the consequences of one’s own and others’ behaviour. Variability in these other (i.e. non-language) domains might, at least partly, explain individual differences in language skills. For instance, language production and comprehension performance tends to be better for language users with good versus poor cognitive control ability. Differences in cognitive control might also influence multimodal integration, assessing other’s goals, turn taking, and using contextual cues for understanding language. One might find that social situations which require a lot of task shifting (an example is early bilingualism) result in the development of a strong cognitive control system, which could be an important prerequisite for developing certain language skills in some individuals, such as the capacity for simultaneous translation.

Thus, three key questions in this WP concern the role of multimodal integration, the actions and goals understanding and cognitive control in linguistic communication. The WP is composed of renowned scientists who are specialists in the areas of perception, action, prediction, multisensory integration and cognitive control systems outside the domain of language (Bekkering, Cools, Medendorp), and those who are experts at investigating relations between action simulations, meaningful communicative actions (i.e., signs and gestures) and cognitive control in the context of language (Bekkering, Özyürek, Roelofs, Zwaan). Together, we aim to achieve a better understanding of the cognitive and neural infrastructures involved in language use in social context.

People involved

WP-leaders

Harold Bekkering (RU)
Asli Ozyurek (RU)

Principal Investigators

Roshan Cools (RU)
Pieter Medendorp (RU)
Ardi Roelofs (RU)
Rolf Zwaan (EUR)

Post Docs

Irina Simanova (RU)