Unknown

Dataset Information

Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study.


ABSTRACT:

Background

Artificial intelligence (AI) chatbots have recently gained use in medical practice by health care practitioners. Interestingly, the output of these AI chatbots was found to have varying degrees of hallucination in content and references. Such hallucinations generate doubts about their output and their implementation.

Objective

The aim of our study was to propose a reference hallucination score (RHS) to evaluate the authenticity of AI chatbots' citations.

Methods

Six AI chatbots were challenged with the same 10 medical prompts, requesting 10 references per prompt. The RHS is composed of 6 bibliographic items and the reference's relevance to prompts' keywords. RHS was calculated for each reference, prompt, and type of prompt (basic vs complex). The average RH

SUBMITTER: Aljamaan F 

PROVIDER: S-EPMC11325115 | biostudies-literature | 2024 Jul

REPOSITORIES: biostudies-literature