Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan.

ABSTRACT:

Purpose

This study aims to investigate the effects of language selection and translation quality on Generative Pre-trained Transformer-4 (GPT-4)'s response accuracy to expert-level diagnostic radiology questions.

Materials and methods

We analyzed 146 diagnostic radiology questions from the Japan Radiology Board Examination (2020-2022), with consensus answers provided by two board-certified radiologists. The questions, originally in Japanese, were translated into English by GPT-4 and DeepL and into German and Chinese by GPT-4. Responses were generated by GPT-4 five times per question set per language. Response accuracy was compared between languages using one-way ANOVA with Bonferroni correction or the Mann-Whitney U test. Scores on selected English questions translated by a

SUBMITTER: Harigai A 

PROVIDER: S-EPMC11790683 | biostudies-literature | 2025 Feb

REPOSITORIES: biostudies-literature