Shared functional specialization in transformer-based language models and the human brain.

ABSTRACT: When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transform

SUBMITTER: Kumar S 

PROVIDER: S-EPMC11217339 | biostudies-literature | 2024 Jun

REPOSITORIES: biostudies-literature