BiRNA-BERT allows efficient RNA language modeling with adaptive tokenization.

ABSTRACT: Transformer-based models have achieved remarkable success in biological sequence modeling, yet their application to RNA remains constrained by sequence length limitations. Existing RNA language models often truncate inputs, discarding distal nucleotide context crucial for full-length tasks. Additionally, advanced NLP tokenization methods do not directly apply to biological sequences, where nucleotide-level resolution is essential for tasks like secondary structure prediction. To address these challenges, we introduce BiRNA-BERT, a 117M-parameter Transformer encoder trained on 36 million non-coding RNA sequences. At its core is an adaptive dual-tokenization framework that combines nucleotide-level (NUC) encoding for fine-grained structural tasks with byte-pair encoding (BPE) for efficient l

SUBMITTER: Tahmid MT 

PROVIDER: S-EPMC12635123 | biostudies-literature | 2025 Nov

REPOSITORIES: biostudies-literature