The performance of large language model-powered chatbots compared to oncology physicians on colorectal cancer queries.

ABSTRACT:

Background

Large language model (LLM)-powered chatbots have become increasingly prevalent in healthcare, while their capacity in oncology remains largely unknown. To evaluate the performance of LLM-powered chatbots compared to oncology physicians in addressing colorectal cancer queries.

Methods

This study was conducted between August 13, 2023, and January 5, 2024. A total of 150 questions were designed, and each question was submitted three times to eight chatbots: ChatGPT-3.5, ChatGPT-4, ChatGPT-4 Turbo, Doctor GPT, Llama-2-70B, Mixtral-8x7B, Bard, and Claude 2.1. No feedback was provided to these chatbots. The questions were also answered by nine oncology physicians, including three residents, three fellows, and three attendings. Each answer was scored based on its consist

SUBMITTER: Zhou S 

PROVIDER: S-EPMC11487020 | biostudies-literature | 2024 Oct

REPOSITORIES: biostudies-literature