Performance evaluation of large language models on Korean medical licensing examination: a three-year comparative analysis.

ABSTRACT: Performance evaluation of large language models (LLMs) in non-English medical contexts remains limited, particularly for medical licensing examinations including both text- and image-based questions. Therefore, the performance and reliability of three LLMs-GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro was evaluated using the Korean Medical Licensing Examination (KMLE) questions from 2022 to 2024. We analyzed 942 KMLE questions encompassing text-only and image-based formats across various medical specialties. Reproducibility was evaluated through repeated testing, and inter-model agreement was analyzed using pairwise comparisons. GPT-4o achieved the highest accuracy (83.2%), followed by Claude 3.5 Sonnet (79.5%) and Gemini 1.5 Pro (76.6%). While GPT-4o and Claude 3.5 Sonnet outperformed tex

SUBMITTER: Kim HJ 

PROVIDER: S-EPMC12528361 | biostudies-literature | 2025 Oct

REPOSITORIES: biostudies-literature