PhD Project 1

Feedback loops in learning to perceive and produce non-native speech contrasts

 

PhD-candidate: Jana Thorin
PIs: James McQueen (WP1) and Peter Desain (WP7)
Start date: 16 June 2014

(last update 2019-06-27)

Research Content

How do perception and production interact during L2 speech category learning? Emerging categories will be tracked with EEG (MMN, ERP, ERN) measures, using on-line single-trial multivariate pattern-classification techniques. The effects of different feedback loops on learning will be compared: 1) behavioural feedback on perceptual decisions and/or produced speech; 2) self-monitoring in production and imitation tasks; 3) neurofeedback on perceived and produced speech. These comparisons will advance understanding of the nature of phonological category representations in perception and production, and their inter-dependence. Findings should also guide construction of Brain-Computer Interface systems for L2 learning that outperform behavior-based training methods.

Highlight

Mismatch negativity as measure of subtle perceptual learning not (yet) present in conventional behavioural measures
Team members: McQueen, Sadakata (DCC) and Desain

It was investigated how the perception of a challenging non-native vowel contrast was affected by a 4-day perceptual training which was either complimented by also producing words containing target sounds or by producing unrelated material. Interestingly, despite no evidence of behavioural differences between the two types of training, neither in the perception nor the production modality, findings from electrophysiological measurements after the training do indicate enhanced perceptual ability following combined perception-production training on the relevant contrast. Evidence comes in the form of a mismatch negativity response present in the related production group but absent in the unrelated production and untrained control groups (see figure).
This innovative finding shows for the first time positive effects of a combined production-perception training on the perception of a challenging non-native contrast and thereby contributes to the discussion of how the perception and production modalities interact in the course of second language sound learning which can be directly related to the overarching goal of the consortium to study language at different levels and between domains. The results also underline the power of electrophysiological measures as sensitive tool to identify fine-grained differences in perceptual ability.

Post-test word oddball. [Left] Difference curves between grand average ERP responses to standard and deviant responses time locked to word onset of three stimulus sets: English /pæn/-/pɛn/ (top), Dutch /pɔt/-/pʏt/ (middle) and Dutch /pɑn/- /pɛn/ (bottom). The three training groups are distinguished by colour. Responses are averaged across a fronto-central cluster of electrodes and shaded areas indicate standard errors. The typical time window for the MMN response, which was also used for the cluster-based permutation test, is highlighted by grey frame. [Right] Corresponding topographic maps averaged across the MMN time window.

Progress 2018

Behavioural results were published in JASA. Results contribute to the discussion on how perception and production modalities interact during second language speech learning. Complimentary findings based on EEG-measurements were submitted to “Bilingualism: Language and Cognition” and further contribute to the scientific understanding of bilingualism. Data were collected for a production training study. The data is currently analysed and will be written up to a scientific paper. Lastly, behavioural and EEG data from an experiment investigating error-monitoring in non-native speech production was analysed to large extents and will also be written up to a scientific paper.

Groundbreaking characteristics

This project is an interdisciplinary collaboration between the artificial intelligence and the language department of the Donders Institute. The project combines psycholinguistics and brain-computer interfacing translating findings of linguistic research into applications in the AI domain. It is at the cutting edge of single-trial EEG classification and seeks to develop novel language-learning technology using neurofeedback.