Analog in-memory computing attention mechanism for fast and energy-efficient large language models.

ABSTRACT: Transformer networks, driven by self-attention, are central to large language models. In generative transformers, self-attention uses cache memory to store token projections, avoiding recomputation at each time step. However, graphics processing unit (GPU)-stored projections must be loaded into static random-access memory for each new generation step, causing latency and energy bottlenecks. Here we present a custom self-attention in-memory computing architecture based on emerging charge-based memories called gain cells, which can be efficiently written to store new tokens during sequence generation and enable parallel analog dot-product computation required for self-attention. However, the analog gain-cell circuits introduce non-idealities and constraints preventing the direct mapping of p

SUBMITTER: Leroux N 

PROVIDER: S-EPMC12457188 | biostudies-literature | 2025 Sep

REPOSITORIES: biostudies-literature