Multimodal masked siamese network improves chest X-ray representation learning.

ABSTRACT: Self-supervised learning methods for medical images primarily rely on the imaging modality during pretraining. Although such approaches deliver promising results, they do not take advantage of the associated patient or scan information collected within Electronic Health Records (EHR). This study aims to develop a multimodal pretraining approach for chest radiographs that considers EHR data incorporation as an additional modality that during training. We propose to incorporate EHR data during self-supervised pretraining with a Masked Siamese Network (MSN) to enhance the quality of chest radiograph representations. We investigate three types of EHR data, including demographic, scan metadata, and inpatient stay information. We evaluate the multimodal MSN on three publicly available chest X-ra

SUBMITTER: Shurrab S 

PROVIDER: S-EPMC11439023 | biostudies-literature | 2024 Sep

REPOSITORIES: biostudies-literature