Five advanced chatbots solving European Diploma in Radiology (EDiR) text-based questions: differences in performance and consistency.

ABSTRACT:

Background

We compared the performance, confidence, and response consistency of five chatbots powered by large language models in solving European Diploma in Radiology (EDiR) text-based multiple-response questions.

Methods

ChatGPT-4o, ChatGPT-4o-mini, Copilot, Gemini, and Claude 3.5 Sonnet were tested using 52 text-based multiple-response questions from two previous EDiR sessions in two iterations. Chatbots were prompted to evaluate each answer as correct or incorrect and grade its confidence level on a scale of 0 (not confident at all) to 10 (most confident). Scores per question were calculated using a weighted formula that accounted for correct and incorrect answers (range 0.0-1.0).

Results

Claude 3.5 Sonnet achieved the highest score per question (0.84 ± 0.26, mean

SUBMITTER: Pristoupil J 

PROVIDER: S-EPMC12364795 | biostudies-literature | 2025 Aug

REPOSITORIES: biostudies-literature