Leveraging Large Language Models to Evaluate the Quality of Narrative Feedback for Surgery Residents in Competency-Based Medical Education.

ABSTRACT:

Objective

This study aimed to investigate large language model (LLM) performance in evaluating narrative feedback quality in the entrustable professional activities (EPAs) assessments within a Surgical Foundations program.

Background

Transitioning to competency-based medical education (CBME) has increased the volume of narrative feedback for surgery residents. However, evaluating narrative feedback quality is time-consuming, requiring manual review by humans. LLMs show potential for automating this process.

Methods

An existing dataset of 2229 deidentified comments from EPA assessments for surgery residents in an academic program (2017-2022) was analyzed using generative pre-trained transformer (GPT)-3.5-turbo-1106 and GPT-4-1106-preview. LLM-generated scores were comp

SUBMITTER: Kwan BYM 

PROVIDER: S-EPMC12727304 | biostudies-literature | 2025 Dec

REPOSITORIES: biostudies-literature