Adaptive noise-augmented attention for enhancing Transformer fine-tuning on longitudinal medical data.

ABSTRACT: Transformer models pre-trained on self-supervised tasks and fine-tuned on downstream objectives have achieved remarkable results across a variety of domains. However, fine-tuning these models for clinical predictions from longitudinal medical data, such as electronic health records (EHR), remains challenging due to limited labeled data and the complex, event-driven nature of medical sequences. While self-attention mechanisms are powerful for capturing relationships within sequences, they may underperform when modeling subtle dependencies between sparse clinical events under limited supervision. We introduce a simple yet effective fine-tuning technique, Adaptive Noise-Augmented Attention (ANAA), which injects adaptive noise directly into the self-attention weights and applies a 2D Gaussian

SUBMITTER: Amirahmadi A 

PROVIDER: S-EPMC12484044 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature