Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge Assessments.

ABSTRACT:

Background

Artificial intelligence (AI) has the potential to dramatically alter healthcare by enhancing how we diagnosis and treat disease. One promising AI model is ChatGPT, a large general-purpose language model trained by OpenAI. The chat interface has shown robust, human-level performance on several professional and academic benchmarks. We sought to probe its performance and stability over time on surgical case questions.

Methods

We evaluated the performance of ChatGPT-4 on two surgical knowledge assessments: the Surgical Council on Resident Education (SCORE) and a second commonly used knowledge assessment, referred to as Data-B. Questions were entered in two formats: open-ended and multiple choice. ChatGPT output were assessed for accuracy and insights by surgeon evalua

SUBMITTER: Beaulieu-Jones BR 

PROVIDER: S-EPMC10371188 | biostudies-literature | 2023 Jul

REPOSITORIES: biostudies-literature