Performance of o1 pro and GPT-4 in Self-Assessment Questions for Nephrology Board Renewal.

ABSTRACT:

Background

Large language models (LLMs) are increasingly evaluated in medical education and clinical decision support, but their performance in highly specialized fields, such as nephrology, is not well established. We compared two advanced LLMs, GPT-4 and the newly released o1 pro, on comprehensive nephrology board renewal examinations.

Methods

We administered 209 Japanese Self-Assessment Questions for Nephrology Board Renewal from 2014 to 2023 to o1 pro and GPT-4 using ChatGPT pro. Each question, including images, was presented in separate chat sessions to prevent contextual carryover. Questions were classified by taxonomy (recall/interpretation/problem-solving), question type (general/clinical), image inclusion, and nephrology subspecialty. We calculated the proportion of

SUBMITTER: Noda R 

PROVIDER: S-EPMC12685630 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature