Big Question 3

Computational characterization of mutual understanding during interaction

 

Supervisors: Ivan Toni (DCCN) and Iris van Rooij (DCC)
Postdoctoral Research Associate: Mark Blokpoel

Starting date: November 01, 2017

(last update 2019-07-03)

Research content

Imagine you want to tell your friend an anecdote about a class mate. Neither of you know the person’s name. How can you convey who you are referring to? You could refer to features of the person (e.g., ‘the one with red hair’ or ‘the tall one’) or use more metaphorical terms (i.e., ‘the teacher’s favourite’ or ‘the gamer’). Of all the things you could say, in principle, how do you decide what to say? This communication problem cuts at the core of some of the key challenges in explaining the computational cognitive infrastructure of human communication.

In this project, the aim is to explain how the different communicative components—language, cognition and embodiment—can lead to two interlocutors coming to mutually understand each other.  Our position is that mutual understanding is an emergent property of the interaction dynamic between interlocutors. To study this, cognitive agent-based models are developed. These models are the basis of computer simulations, designed to investigate communication dynamics during interaction by manipulating the cognitive and linguistic abilities of these agents. At the basis of the models is the Rational Speech Act (RSA) model. This model characterizes pragmatic inference using recursive Bayesian reasoning. In this project, RSA is extended to computationally characterize the interaction between two interlocutors, multimodal communication and interactional repair strategies.

Subproject A.1

Cognitive agent-based modelling of communication
Collaboration with: Van Rooij, Toni, and Dingemanse (MPI)

The project will produce a novel cognitive agent-based simulation platform for modelling language in interaction. The project involves expertise from different fields: Artificial Intelligence, Computer Science, Cognitive (Neuro)science, and Psycholinguistics from the DI, MPI, and CLS.

Progress in 2018
Development of the first generation of agent-based model simulation software was finished. Using cognitive agent-based simulations it was investigated which cognitive capacities are necessary and sufficient to achieve mutual understanding. Two key properties were manipulated, namely signal ambiguity and knowledge asymmetry, as well as pragmatic inference to study the effects on success in a communication game.

It was shown that agents can communicate successfully under a range of conditions. In the limiting case where both agents share the exact same 1-to-1 signal-referent mappings (no ambiguity and no asymmetry), communication is obviously successful. However, this success is highly vulnerable to increased ambiguity or asymmetry. Some measure of success may be recovered by endowing agents with pragmatic inference abilities, but with diminishing returns for higher orders (cf. Frank and Goodman, 2012; Frank et al., 2017). Crucially, it was found that higher-order asymmetrical agents can understand each other better than zero-order agents with exactly matched vocabularies, but only when vocabularies are ambiguous. This counterintuitive role of ambiguity under asymmetry shows the importance of exploring the boundary conditions under which referential communication between two interacting agents can be successful.

Subproject A.2

Multimodality and repair
Postdoctoral Research Associate: Mark Blokpoel
Collaborators: Van Rooij, Dingemanse (MPI), Rasenberg, and Ozyurek

A successful focus session led to insights from computational modelling and identification of what theoretical challenges lie within the field of multi-modal communication and cognitive linguistics. Two main challenges were identified: Explaining the computational mechanisms that underlie the integration of gestural and verbal communicative signals; and explaining the computational mechanisms that underlie interactional repair. These initial explorations have so far yielded relevant empirical grounding for the computational modelling project; they have led to computationally grounded insights; and they have led to a collaborative grant application. A computational mechanism for lexical updating has been identified, which is a necessary cognitive component for interactional repair. This mechanism will be explored using the agent-based simulation platform developed in Subproject A.1.

Highlight

Cognitive agent-based modelling of communication
Team members: Blokpoel, Dingemanse (MPI), Kachergis (SU), Toni, and Van Rooij

In this project, the aim is to explain how the different communicative components—language, cognition and embodiment—can lead to two interlocutors coming to mutually understand each other.  Our position is that mutual understanding is an emergent property of the interaction dynamic between interlocutors. To study this, cognitive agent-based models are developed. These models allow us to explore the necessary and sufficient conditions for successful communication beyond the conditions available in laboratory settings.

The simulation framework developed builds on the rational speech act model (Goodman and Frank, 2012; Frank et al., 2017). A simulation study is showcased that investigates under what conditions two agents that perform a repeated 1-shot director-matcher task can understand each other. Each trial, the interlocutors switch between communicator and listener. These agents communicated under various conditions of ambiguity and asymmetry. Furthermore, the agents’ order of pragmatic inference (zero, first or second order) was manipulated. It was found that higher-order asymmetrical agents can understand each other better than zero-order agents with exactly matched vocabularies, but only when vocabularies are ambiguous. This counterintuitive role of ambiguity under asymmetry shows the importance of exploring the boundary conditions under which communication can be successful.

Left: Illustration of simulation setup, each trial one agent communicates an intention the other infers. Agents switch roles each trial. Right: Results of the simulation experiment, where ambiguity and asymmetry of agent lexicons is manipulated. Here, we observe that pragmatic communicators with asymmetry can even outperform zero-order communicators with no asymmetry, but only when ambiguity is present. This suggests that under realistic conditions (some asymmetry and some ambiguity) ambiguity seems to help (and not hinder) pragmatic communicators overcome asymmetry.

This framework guides the integration of intuitive theories from the subprojects in BQ3 in a unified, formal theoretical framework, which is instrumental to BQ3’s interdisciplinary goal. Moreover, the project is innovative on multiple fronts: novel simulation methodology based on interacting agents, accessibility and open-science.

The project and its team members have proven to be highly successful in translating difficult computational notions to non-expert collaborators. Through focus sessions, it has been the foundation of BQ3 internal collaboration, giving the team members a common language to speak.