Speech recognition using an english multimodal corpus with integrated image and depth information.

ABSTRACT: Traditional English corpora mainly collect information from a single modality, but lack information from multimodal information, resulting in low quality of corpus information and certain problems with recognition accuracy. To solve the above problems, this paper proposes to introduce depth information into multimodal corpora, and studies the construction method of English multimodal corpora that integrates electronic images and depth information, as well as the speech recognition method of the corpus. The multimodal fusion strategy adopted integrates speech signals and image information, including key visual information such as the speaker's lip movements and facial expressions, and uses deep learning technology to mine acoustic and visual features. The acoustic model in the Kaldi toolkit

SUBMITTER: Wang B 

PROVIDER: S-EPMC11542033 | biostudies-literature | 2024 Nov

REPOSITORIES: biostudies-literature