AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI-Simulated Patient System: Cross-Sectional Validation Study.

ABSTRACT:

Background

Generative artificial intelligence (AI) is increasingly used in medical education, including AI-based virtual patients to improve interview skills. However, how much AI-based assessment (ABA) differs from human-based assessment (HBA) remains unclear.

Objective

This study aimed to compare the quality of clinical interview assessments generated via an ABA (GPT-o1 Pro [ABA-o1] and GPT-5 Pro [ABA-5]) with those generated via an HBA conducted by clinical instructors in an AI-based virtual patient setting. We also examined whether AI reduced evaluation time and assessed agreement across participants with different levels of clinical experience.

Methods

A standardized case of leg weakness was implemented in an AI-based virtual patient. Seven participants (2 medica

SUBMITTER: Takahashi H 

PROVIDER: S-EPMC12912650 | biostudies-literature | 2026 Feb

REPOSITORIES: biostudies-literature