Big Question 3

(last update 2019-06-18)

Creating a shared cognitive space: How is language grounded in and shaped by communicative settings of interacting people?

Language is a key socio-cognitive human function predominantly used in interaction. Yet, linguistics and cognitive neuroscience have largely focused on individuals’ coding-decoding signals according to their structural dependencies. Understanding the communicative use of language requires shifting the focus of investigation to the mechanisms used by interlocutors to share a conceptual space.

This big question considers the influence of two dimensions over multiple communicative resources (speech, gestures, gaze) and linguistic structures (from phonology to pragmatics), namely the temporal structure of communicative interactions and the functional dynamics of real-life communicative interactions.

There is deep collaboration between all BQ3 subprojects. The qualitative results that follow from the simulation studies will be related to the empirical findings from the other subprojects and vice versa, the empirical observations from the other subprojects will inspire the qualitative hypotheses to be tested. The cognitive agent-based simulation studies go beyond the empirical paradigm in the BQ3 project, because they allow us to test for qualitative differences in interactive behaviour by manipulating the cognitive capacities of the agents—something that is difficult to do with human test subjects—while simultaneously leading to explicit theories of computational mechanisms.

Cognitive agent-based modeling of communication

In this project, the aim is to explain how the different communicative components—language, cognition and embodiment—can lead to two interlocutors coming to mutually understand each other. Our position is that mutual understanding is an emergent property of the interaction dynamic between interlocutors. To study this, cognitive agent-based models are developed. These models allow us to explore the necessary and sufficient conditions for successful communication beyond the conditions available in laboratory settings.

The simulation framework developed builds on the rational speech act model (Goodman and Frank, 2012). A simulation study is showcased that investigates under what conditions two agents that perform a repeated 1-shot director-matcher task can understand each other. Each trial, the interlocutors switch between communicator and listener. These agents communicated under various conditions of ambiguity and asymmetry. Furthermore, the agents’ order of pragmatic inference (zero, first or second order) was manipulated. It was found that higher-order asymmetrical agents can understand each other better than zero-order agents with exactly matched vocabularies, but only when vocabularies are ambiguous. This counterintuitive role of ambiguity under asymmetry shows the importance of exploring the boundary conditions under which communication can be successful.

Left: Illustration of simulation setup, each trial one agent communicates an intention the other infers. Agents switch roles each trial. Right: Results of the simulation experiment, where ambiguity and asymmetry of agent lexicons is manipulated. Here, we observe that pragmatic communicators with asymmetry can even outperform zero-order communicators with no asymmetry, but only when ambiguity is present. This suggests that under realistic conditions (some asymmetry and some ambiguity) ambiguity seems to help (and not hinder) pragmatic communicators overcome asymmetry.

This framework guides the integration of intuitive theories from the subprojects in BQ3 in a unified, formal theoretical framework, which is instrumental to BQ3’s interdisciplinary goal. Moreover, the project is innovative on multiple fronts: novel simulation methodology based on interacting agents, accessibility and open-science.

The project and its team members have proven to be highly successful in translating difficult computational notions to non-expert collaborators. Through focus sessions, it has been the foundation of BQ3 internal collaboration, giving the team members a common language to speak.

Empirical paradigm

The empirical part of BQ3 is an ongoing study on 80 pairs of participants engaged in face-to-face communicative interactions. Each pair needs to find their way of uniquely identifying novel objects (“Fribbles”, see Barry et al., 2014). Through repeated interactions about each Fribble, pair-specific labels emerge. We consider several elements of the face-to-face interactions in each pair, e.g. speech is transcribed, co-speech gestures and pragmatic devices are identified. These multimodal measurements are meant to identify regularities in how pairs achieve mutual understanding. We focus on linguistic alignment (e.g., syntactic, phonological/phonetic, lexical, and semantic alignment); gestural alignment; and pragmatic devices (e.g., backchannels and repair). Before and after each pair engages in the face-to-face interactions, we measure the participants’ individual representations of the Fribbles, using both fMRI and behavioural metrics. We hypothesise that the individual representations of the Fribbles will change as a function of the level of alignment achieved during the face-to-face interaction. We aim to uncover which variables measured during the interaction are the major contributors to the emergence of conceptual alignment during communication.

Creating shared (neural) representations

The aim of this project is to investigate how the conceptual representations of two communicators change and become more similar as a result of communication. To investigate this, we measure participants’ individual representations of the Fribbles (novel objects) both before and after a series of communicative interactions about these objects (see overview of the empirical paradigm above), using both neural and behavioural metrics. The communicative interactions are structured in a ‘director-matcher’ task (e.g., Clark & Wilkes-Gibbs, 1986) in which each member of a pair takes turns describing the Fribbles to the other.

In the naming task, participants name each Fribble, using one to three words. We compare the names given by the members of a pair to the same Fribble, before and after their communicative interactions, using similarity scores between the vector-based models of those words (Mandera, Keuleers, & Brysbaert, 2017). The similarity score reflects the co-occurrences of those words in large text corpora. A preliminary analysis of 51 pairs collected so far shows increased similarity scores after those pairs engaged in face-to-face communicative interactions over those Fribbles (left panel). The increased similarity score is driven by the communicative interactions of each pair: there is no change in the similarity scores of random pairs, i.e. pairs of participants that performed the same tasks and experienced the same interaction, but not with each other (right panel).

 

Similar observations emerge from an independent behavioural measure of participants’ mental representations of the Fribbles. In the features task, participants rate each Fribble across a number of features based on more visual (e.g., “How rounded is this Fribble?”), and more abstract (e.g., “How human is this Fribble”) properties of the objects (based on Binder et al., 2016). For each Fribble, we correlate the scores obtained across 29 features between two members of a pair (real pairs, right panel) as well as between members of random pairs (left panel). Preliminary results suggest that there is indeed an increase in feature-based similarity for real pairs but not random pairs after the interaction.

This project aims to characterize the neural mechanisms that lead to the increased similarity in communicators’ mental representations of the Fribbles. We do that by using fMRI to measure neurovascular responses to visual presentations of the Fribbles, one by one, before and after each participant is engaged in the communicative interactions. By using Representational Similarity Analysis (RSA, Kriegeskorte, Mur, & Bandettini, 2008) – a type of analysis which uses correlations of fMRI activity patterns as a proxy for similarity  of neural responses to different Fribbles – we can quantify how the relations between the Fribbles change within as well as between members of real pairs and random pairs. More precisely, we first measure the activation pattern elicited by each Fribble in a particular brain region of each participant. Second, we correlate the activation patterns for different Fribbles in the same participant, leading to an RSA matrix of correlations that is specific to that participant (see the figure below). These matrices will then be correlated between participants to see how similar their Fribbles’ representations are. We hypothesise that Fribbles’ representations become more similar following an interaction in real pairs, but not in random pairs, and we hypothesise that this effect would vary as a function of the level of alignment achieved during the face-to-face interaction. This project also aims to understand how these changes in neural representations across different brain regions are influenced by particular metrics of the communicative interaction.

References

  • Frank, M. C., & Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science 336, 25.
  • Barry, J., Griffith, J.W., De Rossi, S., & Hermans, D. (2014). Meet the Fribbles: novel stimuli for use within behavioural research. Front Psychol. 5, 103.
  • Binder, J. R., Conant, L. L., Humphries, C. J., Fernandino, L., Simons, S. B., Aguilar, M., & Desai, R. H. (2016). Toward a brain-based componential semantic representation. Cognitive Neuropsychology, 33(3–4), 130–174.
  • Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22(1), 1–39.
  • Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2.
  • Mandera, P., Keuleers, E., & Brysbaert, M. (2017). Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation. Journal of Memory and Language, 92, 57–78.

Synergy with other Big Questions

Synergies between BQ3 and BQ5 are anticipated, given a shared interest in understanding how agents navigate and organize conceptual spaces. Methodological tools are sharpened by interacting with the LiI Toolkit work package for developing automatic analysis of the co-speech gestures acquired during communicative interactions in BQ3.

Team members

Coordinator

Ivan Toni