Combining speech with sight and touch : investigating the benefits of audio-visual and audio-tactile speech on speech intelligibility and cortical speech-envelope tracking accuracy

O'Hanlon, Brandon and Nuttall, Helen and Plack, Christopher (2025) Combining speech with sight and touch : investigating the benefits of audio-visual and audio-tactile speech on speech intelligibility and cortical speech-envelope tracking accuracy. PhD thesis, Lancaster University.

Text (2025ohanlonphd)
2025ohanlonphd.pdf - Published Version
Available under License Creative Commons Attribution-NonCommercial-NoDerivs.
Download (10MB)

Abstract

Speech plays a critical role in communication in our everyday lives. Understanding how speech is processed in the brain, from mechanoelectrical transduction in the ear to processing in the subcortical and cortical regions of the auditory pathway, provides insight into how we may improve speech perception during difficult listening conditions which may hinder us. As discussed in Chapter 1, multisensory integration is one such method in which we may restore intelligibility to speech-in-noise. This thesis aims to contribute to the current multisensory speech integration literature through new investigations into audio-visual and audio-tactile speech and their benefits to cortical speech-envelope tracking in the auditory cortex and speech intelligibility. Chapter 2 reassessed the audio-visual benefits to speech perception in-noise when phoneme stimuli are selected from different visually distinct viseme categories. It found that, when visemes are considered, the benefits of audio-visual integration were reduced compared to previous literature, with some phonemes providing more benefit to intelligibility than others. Chapters 3 and 4 explored potential benefits of both bottom-up and top-down short-term audio-tactile training on cortical speech-envelope tracking accuracy and speech intelligibility. Within these chapters, findings indicated an initial enhancement on neural tracking accuracy but with no associated benefit to speech intelligibility. This unexpected outcome was further investigated with Chapter 5, which returned to audio-visual speech to examine the potential role of neural tracking in the prediction of oncoming speech. In all, this thesis provided evidence of multisensory benefits to speech perception, especially in difficult listening conditions. The thesis also contributes to the growing literature around neural tracking in the brain, with further evidence to suggest that tracking accuracy is not intrinsically linked to intelligibility. Chapter 6 discusses these findings, alongside future directions, to finalise this thesis.