Big Question 3

(last update 2020-05-12)

Creating a shared cognitive space: How is language grounded in and shaped by communicative settings of interacting people?

Language is a key socio-cognitive human function predominantly used in interaction. Yet, linguistics and cognitive neuroscience have largely focused on individuals’ coding-decoding signals according to their structural dependencies. Understanding the communicative use of language requires shifting the focus of investigation to the mechanisms used by interlocutors to share a conceptual space.

This big question considers the influence of two dimensions over multiple communicative resources (speech, gestures, gaze) and linguistic structures (from phonology to pragmatics), namely the temporal structure of communicative interactions and the functional dynamics of real-life communicative interactions.

There is deep collaboration between all BQ3 subprojects. The qualitative results that follow from the simulation studies will be related to the empirical findings from the other subprojects and vice versa, the empirical observations from the other subprojects will inspire the qualitative hypotheses to be tested. The cognitive agent-based simulation studies go beyond the empirical paradigm in the BQ3 project, because they allow us to test for qualitative differences in interactive behaviour by manipulating the cognitive capacities of the agents—something that is difficult to do with human test subjects—while simultaneously leading to explicit theories of computational mechanisms.

Highlights

Highlight 1: Cognitive agent-based modelling of communication

Team members: Blokpoel, Dingemanse, Woensdregt, Kachergis (Stanford University), Bögels, Toni, and van Rooij

How can we understand each other in the face of word ambiguity and knowledge asymmetry? Using cognitive agent-based models of probabilistic pragmatic inference, we systematically vary ambiguity and asymmetry in the lexicons of interacting agents. We show that pragmatic communicators successfully deal with ambiguity by means of recursive pragmatic reasoning, and, remarkably, exploit the ambiguity of their lexicons to overcome knowledge asymmetries. Our formalization offers a principled explanation for the success of pragmatic inference in adverse conditions, even before recourse to computationally costlier contextual scaffolding and interactive feedback.

We investigated the effect of ambiguity , asymmetry and order of pragmatic inference on the agents' ability to communicate successfully. Agents produce and interpret signals based on Rational Speech Act (Frank & Goodman, 2012). Simulations show that pragmatic communicators can overcome asymmetry by exploiting the ambiguity in their lexicons. Figure 1 shows the results, where there is a clear increased asymmetry tolerance for pragmatic agents with moderately ambiguous lexicons. This proves that there exist computationally leaner mechanisms that allow interlocutors to counter the detrimental effects of asymmetry prior to resorting to more costly forms of pragmatic inference and interactive repair.

Figure 1.

Figure 1. Main simulation results. A: Mean communicative success for interacting agents with moderately ambiguous lexicons , where  indicates peak performance of literal (zero order) agents,  the first point where performance of pragmatic (first order) agents drops below that, and the dotted line between them is the additional amount of asymmetry pragmatic agents can tolerate by exploiting ambiguity. B: Increased asymmetry toleration for each possible combination of lexicon ambiguity, showing that for pragmatic agents there is a 'sweet spot' where moderate ambiguity on both sides helps rather than hinders communication. Surrounding smaller panels illustrate communicative success for different combinations of lexicon ambiguity.

This framework guides the integration of intuitive theories from the subprojects in BQ3 in a unified, formal theoretical framework, which is instrumental to BQ3’s interdisciplinary goal. Moreover, the project is innovative on multiple fronts: novel simulation methodology based on interacting agents, accessibility and open-science (see tutorials here).

The project and its team members have proven to be highly successful in translating difficult computational notions to non-expert collaborators. Through focus sessions, it has been the foundation of BQ3 internal collaboration, giving the team members a common language to speak. The full paper is available online as a preprint.

Highlight 2: Multimodal and pragmatic alignment in dialogue

Team members: Rasenberg, Özyürek, and Dingemanse

We aim to understand how people reach mutual understanding, for example when talking about novel objects or abstract ideas. Previous work has shown that people often repeat each other’s words and gestures, something referred to as “alignment”. However, little is known about when and why people exactly align to each other. Furthermore, we do not yet have a complete understanding of the relationship between different types of alignment (e.g., alignment of words vs. alignment of gestures). Our research sheds new light on these questions and enables us to unify different theoretical perspectives.

An initial step has been to review the diverging theoretical interpretations and empirical operationalizations of lexical and gestural alignment. To capture the multidimensional nature of the phenomenon of ‘alignment’, we identified five key dimensions to formalize the relationship between any pair of behaviors: sequence, time, semantics, form and modality.

The integrative framework proposed here (in Figure 2) draws upon work from a range of disciplines, and introduces a novel perspective on alignment by considering it from a multidimensional as well as a multimodal perspective. It also functions as a stepping stone for future empirical and theoretical work within BQ3, and contributes to the overall goal of Language in Interaction to understand the dynamics of language in social interaction.

Figure 2. Conceptual framework for understanding and investigating alignment.

The review is a direct result of BQ3’s endeavor to investigate various kinds of alignment (both in terms of behaviour as well as conceptual representations), which highlighted the need for an integrative framework. The combination of expertise within BQ3 – from psycholinguistics to gesture studies and from joint action to computational cognitive science – consequently shaped the development of a framework that is both applicable and relevant across a wide range of disciplines.

Highlight 3: Alignment of pitch and articulation rate

Team members: Eijk, Ernestus, and Schriefers

Previous studies have shown that speakers align their speech with each other at multiple linguistic levels. Conflicting results were reported about whether speakers adapt their speech to the directly preceding stretch of speech or to the general speech characteristics of their interlocutor. This study investigates whether alignment is mostly the result of priming from the immediately preceding speech materials, focussing on pitch and articulation rate (AR).

Native Dutch speakers completed sentences, first by themselves (pre-test), then in alternation with Confederate 1 (Round 1), with Confederate 2 (Round 2), with Confederate 1 again (Round 3), and lastly by themselves again (post-test). Results indicate that participants aligned to the confederates and that this alignment lasted during the post-test, as visible in Figures 3A and 3B.

Furthermore, the confederates’ directly preceding sentences were not good predictors for the participants’ pitch and AR. These results contribute to the main question of Big Question 3 by showing that alignment seems to be a global effect on the phonetic level, more than being a local priming effect.

Figure 3A. Participants’ median F0 over pre-test, Rounds 1, 2 and 3 and post-test; lines were fitted using lm. Points represent Confederates’ means.

Figure 3B. Participants’ AR over pre-test, Rounds 1, 2 and 3 and post-test; lines were fitted using lm. Points represent Confederates’ means.

Synergy with other Big Questions

Synergies between BQ3 and BQ5 are anticipated, given a shared interest in understanding how agents navigate and organize conceptual spaces. Methodological tools are sharpened by interacting with the LiI Toolkit work package for developing automatic analysis of the co-speech gestures acquired during communicative interactions in BQ3.

Collaboration between BQ3-SP4 (Blokpoel and van Rooij) and BQ5-SP3 (Martin) has been started. The goal for BQ5-SP3 is to develop a formal theory and implement it in a computational model with the purpose of understanding the cognitive neural transformations that underlie structure generation in language and in action planning. This is a natural extension of the computational-level, cognitive and interactional focus of the theory development in BQ3-SP4, and vice versa. Both projects are expected to be mutually informing and constraining, further refining their respective theories; and taken together the theories aim to explain communication spanning the neural/linguistic and cognitive/interactional levels.

Team members

Coordinator

Ivan Toni