Scaling down for efficiency: Medium-sized protein language models perform well at transfer learning on realistic datasets.

ABSTRACT: Protein language models (pLMs) can offer deep insights into evolutionary and structural properties of proteins. While larger models, such as the 15 billion parameter model ESM-2, promise to capture more complex patterns in sequence space, they also present practical challenges due to their high dimensionality and high computational cost. We systematically evaluated the performance of various pLMs across multiple biological datasets to assess the impact of model size on transfer learning. Surprisingly, we found that larger models not necessarily outperform smaller ones, in particular when data is limited. Medium-sized models, such as ESM-2 650M and ESM C 600M, demonstrated consistently good performance, falling only slightly behind their larger counterparts-ESM-2 15B and ESM C 6B-despite be

SUBMITTER: Vieira LC 

PROVIDER: S-EPMC11601519 | biostudies-literature | 2025 Jan

REPOSITORIES: biostudies-literature