biostudies-literatureUnknown65(5)Huang DPretrained language models have demonstrated strong capability and versatility in natural language processing (NLP) tasks, and they have important applications in optoelectronics research, such as data mining and topic modeling. Many language models have also been developed for other scientific domains, among which Bidirectional Encoder Representations from Transformers (BERT) is one of the most widely used architectures. We present three "optoelectronics-aware" BERT models, OE-BERT, OE-ALBERT, and OE-RoBERTa, that outperform both their counterpart general English models and larger models in a variety of NLP tasks about optoelectronics. Our work also demonstrates the efficacy of a cost-effective domain-adaptive pretraining (DAPT) method with RoBERTa, which significantly reduces computationJournal of chemical information and modeling2476-2486https://www.ebi.ac.uk/biostudies/studies/S-EPMC11898057biostudies-literatureCost-Efficient Domain-Adaptive Pretraining of Language Models for Optoelectronics Applications.PMC11898057Huang DCole JMfalseCost-Efficient Domain-Adaptive Pretraining of Language Models for Optoelectronics Applications.Pretrained language models have demonstrated strong capability and versatility in natural language processing (NLP) tasks, and they have important applications in optoelectronics research, such as data mining and topic modeling. Many language models have also been developed for other scientific domains, among which Bidirectional Encoder Representations from Transformers (BERT) is one of the most widely used architectures. We present three "optoelectronics-aware" BERT models, OE-BERT, OE-ALBERT, and OE-RoBERTa, that outperform both their counterpart general English models and larger models in a variety of NLP tasks about optoelectronics. Our work also demonstrates the efficacy of a cost-effective domain-adaptive pretraining (DAPT) method with RoBERTa, which significantly reduces computation2025-01-01T00:00:00Z2025 Mar2025-04-03T23:57:17.276Z2025-04-03T23:57:17.276ZS-EPMC118980573993307410.1021/acs.jcim.4c02029