biostudies-literatureAn JInnovative Research Team of Northeast Agricultural University467https://www.ebi.ac.uk/biostudies/studies/S-EPMC9641823biostudies-literatureUnknown23(1)Pre-trained natural language processing models on a large natural language corpus can naturally transfer learned knowledge to protein domains by fine-tuning specific in-domain tasks. However, few studies focused on enriching such protein language models by jointly learning protein properties from strongly-correlated protein tasks. Here we elaborately designed a multi-task learning (MTL) architecture, aiming to decipher implicit structural and evolutionary information from three sequence-level classification tasks for protein family, superfamily and fold. Considering the co-existing contextual relevance between human words and protein language, we employed BERT, pre-trained on a large natural language corpus, as our backbone to handle protein sequences. More importantly, the encoded knowledBMC bioinformaticsCollectively encoding protein properties enriches protein language models.PMC9641823No. 54941012Weng XAn JfalseCollectively encoding protein properties enriches protein language models.Pre-trained natural language processing models on a large natural language corpus can naturally transfer learned knowledge to protein domains by fine-tuning specific in-domain tasks. However, few studies focused on enriching such protein language models by jointly learning protein properties from strongly-correlated protein tasks. Here we elaborately designed a multi-task learning (MTL) architecture, aiming to decipher implicit structural and evolutionary information from three sequence-level classification tasks for protein family, superfamily and fold. Considering the co-existing contextual relevance between human words and protein language, we employed BERT, pre-trained on a large natural language corpus, as our backbone to handle protein sequences. More importantly, the encoded knowled2022-01-01T00:00:00Z2022 Nov2026-04-08T11:37:05.731Z2025-04-19T22:46:22.369ZS-EPMC96418233634828110.1186/s12859-022-05031-z