Unknown

Dataset Information

0

Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments.

ABSTRACT: The United States Medical Licensing Examination (USMLE) has been a subject of performance study for artificial intelligence (AI) models. However, their performance on questions involving USMLE soft skills remains unexplored. This study aimed to evaluate ChatGPT and GPT-4 on USMLE questions involving communication skills, ethics, empathy, and professionalism. We used 80 USMLE-style questions involving soft skills, taken from the USMLE website and the AMBOSS question bank. A follow-up query was used to assess the models' consistency. The performance of the AI models was compared to that of previous AMBOSS users. GPT-4 outperformed ChatGPT, correctly answering 90% compared to ChatGPT's 62.5%. GPT-4 showed more confidence, not revising any responses, while ChatGPT modified its original answers 82.5% of the time. The performance of GPT-4 was higher than that of AMBOSS's past users. Both AI models, notably GPT-4, showed capacity for empathy, indicating AI's potential to meet the complex interpersonal, ethical, and professional demands intrinsic to the practice of medicine.

SUBMITTER: Brin D 

PROVIDER: S-EPMC10543445 | biostudies-literature | 2023 Oct

REPOSITORIES: biostudies-literature

Json Xml
altmetric image

Publications

Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments.

Brin Dana D   Sorin Vera V   Vaid Akhil A   Soroush Ali A   Glicksberg Benjamin S BS   Charney Alexander W AW   Nadkarni Girish G   Klang Eyal E  

Scientific reports 20231001 1

The United States Medical Licensing Examination (USMLE) has been a subject of performance study for artificial intelligence (AI) models. However, their performance on questions involving USMLE soft skills remains unexplored. This study aimed to evaluate ChatGPT and GPT-4 on USMLE questions involving communication skills, ethics, empathy, and professionalism. We used 80 USMLE-style questions involving soft skills, taken from the USMLE website and the AMBOSS question bank. A follow-up query was us  ...[more]