Multimodal contrastive learning for enhanced explainability in pediatric brain tumor molecular diagnosis.

ABSTRACT: Despite the promising performance of convolutional neural networks (CNNs) in brain tumor diagnosis from magnetic resonance imaging (MRI), their integration into the clinical workflow has been limited. That is mainly due to the fact that the features contributing to a model's prediction are unclear to radiologists and hence, clinically irrelevant, i.e., lack of explainability. As the invaluable sources of radiologists' knowledge and expertise, radiology reports can be integrated with MRI in a contrastive learning (CL) framework, enabling learning from image-report associations, to improve CNN explainability. In this work, we train a multimodal CL architecture on 3D brain MRI scans and radiology reports to learn informative MRI representations. Furthermore, we integrate tumor location, salie

SUBMITTER: Ketabi S 

PROVIDER: S-EPMC11955525 | biostudies-literature | 2025 Mar

REPOSITORIES: biostudies-literature