Mixed methods assessment of the influence of demographics on medical advice of ChatGPT.

ABSTRACT:

Objectives

To evaluate demographic biases in diagnostic accuracy and health advice between generative artificial intelligence (AI) (ChatGPT GPT-4) and traditional symptom checkers like WebMD.

Materials and methods

Combination symptom and demographic vignettes were developed for 27 most common symptom complaints. Standardized prompts, written from a patient perspective, with varying demographic permutations of age, sex, and race/ethnicity were entered into ChatGPT (GPT-4) between July and August 2023. In total, 3 runs of 540 ChatGPT prompts were compared to the corresponding WebMD Symptom Checker output using a mixed-methods approach. In addition to diagnostic correctness, the associated text generated by ChatGPT was analyzed for readability (using Flesch-Kincaid Grade Level)

SUBMITTER: Andreadis K 

PROVIDER: S-EPMC11339520 | biostudies-literature | 2024 Sep

REPOSITORIES: biostudies-literature