A dual-channel language decoding from brain activity with progressive transfer training.

ABSTRACT: When we view a scene, the visual cortex extracts and processes visual information in the scene through various kinds of neural activities. Previous studies have decoded the neural activity into single/multiple semantic category tags which can caption the scene to some extent. However, these tags are isolated words with no grammatical structure, insufficiently conveying what the scene contains. It is well-known that textual language (sentences/phrases) is superior to single word in disclosing the meaning of images as well as reflecting people's real understanding of the images. Here, based on artificial intelligence technologies, we attempted to build a dual-channel language decoding model (DC-LDM) to decode the neural activities evoked by images into language (phrases or short sentences).

SUBMITTER: Huang W 

PROVIDER: S-EPMC8449118 | biostudies-literature | 2021 Oct

REPOSITORIES: biostudies-literature