Integrating convolution and self-attention improves language model of human genome for interpreting non-coding regions at base-resolution.

ABSTRACT: Interpretation of non-coding genome remains an unsolved challenge in human genetics due to impracticality of exhaustively annotating biochemically active elements in all conditions. Deep learning based computational approaches emerge recently to help interpret non-coding regions. Here, we present LOGO (Language of Genome), a self-attention based contextualized pre-trained language model containing only two self-attention layers with 1 million parameters as a substantially light architecture that applies self-supervision techniques to learn bidirectional representations of the unlabelled human reference genome. LOGO is then fine-tuned for sequence labelling task, and further extended to variant prioritization task via a special input encoding scheme of alternative alleles followed by adding

SUBMITTER: Yang M 

PROVIDER: S-EPMC9371931 | biostudies-literature | 2022 Aug

REPOSITORIES: biostudies-literature