Assessing the performance of large language models (LLMs) in answering medical questions regarding breast cancer in the Chinese context.

ABSTRACT:

Purpose

Large language models (LLMs) are deep learning models designed to comprehend and generate meaningful responses, which have gained public attention in recent years. The purpose of this study is to evaluate and compare the performance of LLMs in answering questions regarding breast cancer in the Chinese context.

Material and methods

ChatGPT, ERNIE Bot, and ChatGLM were chosen to answer 60 questions related to breast cancer posed by two oncologists. Responses were scored as comprehensive, correct but inadequate, mixed with correct and incorrect data, completely incorrect, or unanswered. The accuracy, length, and readability among answers from different models were evaluated using statistical software.

Results

ChatGPT answered 60 questions, with 40 (66.7%) compreh

SUBMITTER: Piao Y 

PROVIDER: S-EPMC11462564 | biostudies-literature | 2024 Jan-Dec

REPOSITORIES: biostudies-literature