A deep neural network model of audiovisual speech recognition reports the McGurk effect.

ABSTRACT: In the McGurk effect, perception of an auditory syllable changes dramatically when it is paired with an incongruent visual syllable, countering our intuition that speech perception is solely an auditory process. The dominant modeling framework for the study of audiovisual speech perception is that of Bayesian causal inference, but current Bayesian models are unable to predict the wide range of percepts evoked by McGurk syllables. We explored whether a deep neural network (DNN) known as AVHuBERT could provide an alternative modeling framework. AVHuBERT model variants were presented with McGurk syllables consisting of auditory "ba" paired with visual "ga" recorded from eight different talkers. AVHuBERT identified McGurk syllables as something other than "ba" at a rate of 59%, demonstrating a

SUBMITTER: Ma H 

PROVIDER: S-EPMC12917083 | biostudies-literature | 2026 Feb

REPOSITORIES: biostudies-literature