Cost-Efficient Domain-Adaptive Pretraining of Language Models for Optoelectronics Applications.

ABSTRACT: Pretrained language models have demonstrated strong capability and versatility in natural language processing (NLP) tasks, and they have important applications in optoelectronics research, such as data mining and topic modeling. Many language models have also been developed for other scientific domains, among which Bidirectional Encoder Representations from Transformers (BERT) is one of the most widely used architectures. We present three "optoelectronics-aware" BERT models, OE-BERT, OE-ALBERT, and OE-RoBERTa, that outperform both their counterpart general English models and larger models in a variety of NLP tasks about optoelectronics. Our work also demonstrates the efficacy of a cost-effective domain-adaptive pretraining (DAPT) method with RoBERTa, which significantly reduces computation

SUBMITTER: Huang D 

PROVIDER: S-EPMC11898057 | biostudies-literature | 2025 Mar

REPOSITORIES: biostudies-literature