Proformer: a hybrid macaron transformer model predicts expression values from promoter sequences.

ABSTRACT: The breakthrough high-throughput measurement of the cis-regulatory activity of millions of randomly generated promoters provides an unprecedented opportunity to systematically decode the cis-regulatory logic that determines the expression values. We developed an end-to-end transformer encoder architecture named Proformer to predict the expression values from DNA sequences. Proformer used a Macaron-like Transformer encoder architecture, where two half-step feed forward (FFN) layers were placed at the beginning and the end of each encoder block, and a separable 1D convolution layer was inserted after the first FFN layer and in front of the multi-head attention layer. The sliding k-mers from one-hot encoded sequences were mapped onto a continuous embedding, combined with the learned positiona

SUBMITTER: Kwak IY 

PROVIDER: S-EPMC10877777 | biostudies-literature | 2024 Feb

REPOSITORIES: biostudies-literature