Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework.

ABSTRACT:

Objective

Large language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT's performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare.

Design

We evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question d

SUBMITTER: Maitland A 

PROVIDER: S-EPMC10946340 | biostudies-literature | 2024 Mar

REPOSITORIES: biostudies-literature