PhD Project 31
Multimodal and Pragmatic Alignment in Dialogue
(last update 2019-07-01)
In social interaction, mutual understanding is achieved interactively and incrementally by means of sequences of communicative turns. That is, every turn at talk provides people with opportunities to review, revise and recalibrate mental representations. While turns have long been treated as speech-only constructs, in their most common realization (face-to-face social interaction) they are multimodal, combining both visuospatial and verbal resources. In this project it will be investigated how interlocutors use these communicative resources (speech and gestures) and interactional mechanisms (repair, backchannels) to create and negotiate shared conceptual representations. Both in-depth qualitative, as well as quantitative approaches will be adopted to answer various questions. The (current) focus is on: a) the interrelation between lexical and gestural alignment, b) the role of gesture in repair sequences, and c) how various interactional mechanisms and resources relate to measurable outcomes of mutual understanding and shared mental representations.
Together with the BQ3 team, two experimental studies have been set up. A lot of progress has been made for these studies in terms of working out the research design, as well as various practical and technical affairs, such as installing a new technical set-up in one of the DCCN behavioural labs, finalizing scripts and testing protocols, creating information brochures and consent forms for participants and setting up a data management/transfer infrastructure.
For the first behavioural study, data collection has been completed; for the more elaborate (behavioural + neuroimaging) study, the tasks and protocol have been piloted. A large amount of the data from the first study has been pre-processed and annotated. Pre-processing pipelines (for audio, video, and Kinect) have been established, together with templates and protocols to annotate the data. For a substantial amount of the data the speech has been transcribed and a subset of gestures coded.
The data from the first study has so far been used to support theory development and preliminary, qualitative interpretations of the data have been presented in an invited talk.
A key goal of the larger project is to integrate distinct levels of analysis, which calls for a high degree of collaboration with the closely related subprojects. Moreover, there is an important synergy with the outcomes of empirical work that can be related to the computational agent-based models, which will be mutually informative for all involved subprojects.