Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study.

ABSTRACT:

Background

The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied.

Objective

This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages.

Methods

This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions.

Results

The results indicated that GPT-4 outperformed GPT-3.5 in terms of

SUBMITTER: Takagi S 

PROVIDER: S-EPMC10365615 | biostudies-literature | 2023 Jun

REPOSITORIES: biostudies-literature