Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study.

ABSTRACT:

Background

Large language models (LLMs) show increasing potential for their use in healthcare for administrative support and clinical decision making. However, reports on their performance in critical care medicine is lacking.

Methods

This study evaluated five LLMs (GPT-4o, GPT-4o-mini, GPT-3.5-turbo, Mistral Large 2407 and Llama 3.1 70B) on 1181 multiple choice questions (MCQs) from the gotheextramile.com database, a comprehensive database of critical care questions at European Diploma in Intensive Care examination level. Their performance was compared to random guessing and 350 human physicians on a 77-MCQ practice test. Metrics included accuracy, consistency, and domain-specific performance. Costs, as a proxy for energy consumption, were also analyzed.

Results

GPT-

SUBMITTER: Workum JD 

PROVIDER: S-EPMC11809097 | biostudies-literature | 2025 Feb

REPOSITORIES: biostudies-literature