Distinguishing word identity and sequence context in DNA language models.

ABSTRACT: Transformer-based large language models (LLMs) are very suited for biological sequence data, because of analogies to natural language. Complex relationships can be learned, because a concept of "words" can be generated through tokenization. Training the models with masked token prediction, they learn both token sequence identity and larger sequence context. We developed methodology to interrogate model learning, which is both relevant for the interpretability of the model and to evaluate its potential for specific tasks. We used DNABERT, a DNA language model trained on the human genome with overlapping k-mers as tokens. To gain insight into the model's learning, we interrogated how the model performs predictions, extracted token embeddings, and defined a fine-tuning benchmarking task to pr

SUBMITTER: Sanabria M 

PROVIDER: S-EPMC11395559 | biostudies-literature | 2024 Sep

REPOSITORIES: biostudies-literature