Understanding Language Model Scaling on Protein Fitness Prediction.

ABSTRACT: Protein language models, and models that incorporate structure or homologous sequences, estimate sequence likelihoods p(sequence) that reflect the protein fitness landscape and are commonly used in mutation effect prediction and protein design. It is widely believed in deep learning field that larger model performs better across tasks. However, for fitness prediction, language model performance declines beyond a certain size, raising concerns about their scalability. Here, we showed that model size, training dataset, and stochastic elements can bias the predicted p(sequence) away from real fitness. Model performance on fitness prediction depends on how well p(sequence) matches evolutionary patterns in homologs, which is best achieved at a moderate p(sequence) le

SUBMITTER: Hou C 

PROVIDER: S-EPMC12330683 | biostudies-literature | 2025 Aug

REPOSITORIES: biostudies-literature