PhD Project 1

Feedback loops in learning to perceive and produce non-native speech contrasts


PhD-candidate: Jana Krutwig
PIs: James McQueen (WP1) and Peter Desain (WP7)
Start date: 16 June 2014

Research Content

How do perception and production interact during L2 speech category learning? Emerging categories will be tracked with EEG (MMN, ERP, ERN) measures, using on-line single-trial multivariate pattern-classification techniques. We will compare the effects of different feedback loops on learning: 1) behavioural feedback on perceptual decisions and/or produced speech; 2) self-monitoring in production and imitation tasks; 3) neurofeedback on perceived and produced speech. These comparisons will advance understanding of the nature of phonological category representations in perception and production, and their inter-dependence. Findings should also guide construction of Brain-Computer Interface systems for L2 learning that outperform behavior-based training methods.

Groundbreaking characteristics

This project is an interdisciplinary collaboration between the artificial intelligence and the language department of the DI. The project combines psycholinguistics and brain-computer interfacing translating findings of linguistic research into applications in the AI domain. It is at the cutting edge of single-trial EEG classification and seeks to develop novel language-learning technology using neurofeedback.

Progress 2016

The data collection for the first EEG experiment has been completed. The purpose is to investigate whether Dutch learners of English can establish a native-like vowel category (i.e. a-e), which role actively producing instead of only listening to good examples of the targeted vowels has and whether their EEG signal can be classified on a single trial level (offline analysis as basis for the coming experiments). Behavioural results show that participants were able to significantly improve in both the perceptual as well as the production domain. Whether participants had to actively produce the trained vowels additional to the perceptual task was not determining behavioural improvement. Preliminary results of post-training EEG measurements, however, show differences in the neural domain. This indicates that (additional) production training can enhance the neural perceptual response to L2 sounds. EEG analyses of the during-training data are still ongoing and are hoped to shed more light on the time-course of those differences in neural signature.

Next to the main study, two other projects have been pursued. Firstly, all participants of the above mentioned study as well as an equally sized control group, were also tested in a phoneme substitution task (in collaboration with Dr. Eliana Garcia-Cossio). The focus of this study is error-monitoring during and after non-native speech production and its potential changes due to perceptual and production training.  Both EEG and behavioural analyses are ongoing. Secondly, in preparation for a next experiment, production-feedback software has been developed and is currently improved. It is specifically evaluating the English vowels, Dutch participants were trained on during the main study mentioned above and will eventually serve as the central tool for a multiple-day training study focusing on neural and behavioural changes occurring in response to external feedback on speech production.