Visual-textual integration in LLMs for medical diagnosis: A preliminary quantitative analysis.

ABSTRACT:

Background and aim

Visual data from images is essential for many medical diagnoses. This study evaluates the performance of multimodal Large Language Models (LLMs) in integrating textual and visual information for diagnostic purposes.

Methods

We tested GPT-4o and Claude Sonnet 3.5 on 120 clinical vignettes with and without accompanying images. Each vignette included patient demographics, a chief concern, and relevant medical history. Vignettes were paired with either clinical or radiological images from two sources: 100 images from the OPENi database and 20 images from recent NEJM challenges, ensuring they were not in the LLMs' training sets. Three primary care physicians served as a human benchmark. We analyzed diagnostic accuracy and the models' explanations for a subset o

SUBMITTER: Agbareia R 

PROVIDER: S-EPMC11754970 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature