Dataset Information

ChatGPT Influence on Medical Decision-Making, Bias, and Equity: A Randomized Study of Clinicians Evaluating Clinical Vignettes.

ABSTRACT: In a randomized, pre-post intervention study, we evaluated the influence of a large language model (LLM) generative AI system on accuracy of physician decision-making and bias in healthcare. 50 US-licensed physicians reviewed a video clinical vignette, featuring actors representing different demographics (a White male or a Black female) with chest pain. Participants were asked to answer clinical questions around triage, risk, and treatment based on these vignettes, then asked to reconsider after receiving advice generated by ChatGPT+ (GPT4). The primary outcome was the accuracy of clinical decisions based on pre-established evidence-based guidelines. Results showed that physicians are willing to change their initial clinical impressions given AI assistance, and that this led to a significant improvement in clinical decision-making accuracy in a chest pain evaluation scenario without introducing or exacerbating existing race or gender biases. A survey of physician participants indicates that the majority expect LLM tools to play a significant role in clinical decision making.

SUBMITTER: Goh E

PROVIDER: S-EPMC10705632 | biostudies-literature | 2023 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

ChatGPT Influence on Medical Decision-Making, Bias, and Equity: A Randomized Study of Clinicians Evaluating Clinical Vignettes.

Goh Ethan E Bunning Bryan B Khoong Elaine E Gallo Robert R Milstein Arnold A Centola Damon D Chen Jonathan H JH

medRxiv : the preprint server for health sciences 20231127

In a randomized, pre-post intervention study, we evaluated the influence of a large language model (LLM) generative AI system on accuracy of physician decision-making and bias in healthcare. 50 US-licensed physicians reviewed a video clinical vignette, featuring actors representing different demographics (a White male or a Black female) with chest pain. Participants were asked to answer clinical questions around triage, risk, and treatment based on these vignettes, then asked to reconsider after ...[more]

PMID: 38076944

Similar Datasets

Project description:BackgroundChatGPT has gained global attention recently owing to its high performance in generating a wide range of information and retrieving any kind of data instantaneously. ChatGPT has also been tested for the United States Medical Licensing Examination (USMLE) and has successfully cleared it. Thus, its usability in medical education is now one of the key discussions worldwide.ObjectiveThe objective of this study is to evaluate the performance of ChatGPT in medical biochemistry using clinical case vignettes.MethodsThe performance of ChatGPT was evaluated in medical biochemistry using 10 clinical case vignettes. Clinical case vignettes were randomly selected and inputted in ChatGPT along with the response options. We tested the responses for each clinical case twice. The answers generated by ChatGPT were saved and checked using our reference material.ResultsChatGPT generated correct answers for 4 questions on the first attempt. For the other cases, there were differences in responses generated by ChatGPT in the first and second attempts. In the second attempt, ChatGPT provided correct answers for 6 questions and incorrect answers for 4 questions out of the 10 cases that were used. But, to our surprise, for case 3, different answers were obtained with multiple attempts. We believe this to have happened owing to the complexity of the case, which involved addressing various critical medical aspects related to amino acid metabolism in a balanced approach.ConclusionsAccording to the findings of our study, ChatGPT may not be considered an accurate information provider for application in medical education to improve learning and assessment. However, our study was limited by a small sample size (10 clinical case vignettes) and the use of the publicly available version of ChatGPT (version 3.5). Although artificial intelligence (AI) has the capability to transform medical education, we emphasize the validation of such data produced by such AI systems for correctness and dependability before it could be implemented in practice.

Project description:Background and aimSelf-management support is an integral part of current chronic care guidelines. The success of self-management interventions varies between individual patients, suggesting a need for tailored self-management support. Understanding the role of patient factors in the current decision making of health professionals can support future tailoring of self-management interventions. The aim of this study is to identify the relative importance of patient factors in health professionals' decision making regarding self-management support.MethodA factorial survey was presented to primary care physicians and nurses. The survey consisted of clinical vignettes (case descriptions), in which 11 patient factors were systematically varied. Each care provider received a set of 12 vignettes. For each vignette, they decided whether they would give this patient self-management support and whether they expected this support to be successful. The associations between respondent decisions and patient factors were explored using ordered logit regression.ResultsThe survey was completed by 60 general practitioners and 80 nurses. Self-management support was unlikely to be provided in a third of the vignettes. The most important patient factor in the decision to provide self-management support as well as in the expectation that self-management support would be successful was motivation, followed by patient-provider relationship and illness perception. Other factors, such as depression or anxiety, education level, self-efficacy and social support, had a small impact on decisions. Disease, disease severity, knowledge of disease, and age were relatively unimportant factors.ConclusionThis is the first study to explore the relative importance of patient factors in decision making and the expectations regarding the provision of self-management support to chronic disease patients. By far, the most important factor considered was patient's motivation; unmotivated patients were less likely to receive self-management support. Future tailored interventions should incorporate strategies to enhance motivation in unmotivated patients. Furthermore, care providers should be better equipped to promote motivational change in their patients.

Project description:ObjectiveTo conceptualise the cognitive processes of early expert decision-making in urgent care.BackgroundExpert clinicians in the UK frequently determine suitable urgent care patient pathways via telephone triage. This strategy is promoted by policymakers but how it is performed, and its effectiveness has not been evaluated. Evaluation of early senior decision-making requires knowledge of decision-processes, influences, and goals. Previous research has focused on diagnostic decision-making and rarely studied clinicians in the field.MethodWe triangulated analytic autoethnography of early expert decision-making with focused ethnography of experts and trainee doctors performing the task. The study took place in a medium-sized Acute Medical Unit which provided internal medical emergency care for a mixed urban and rural population in the UK. A grounded theoretical model of expert decision-making was created via Gioia Methodology. Decision types were categorised to identify differences in solutions as well as decision processes.ResultsThe hallmarks of intuitive decision-making were found in most expert decisions. Experts made intuitive use of pattern-matching to extract key data from large volumes of information which triggered the spontaneous manifestation of solutions. Solutions were holistic and usually solitary. Upon manifestation solutions were consciously tested for viability with emotional affect playing a key role. Expert solutions were previously applied ones but were frequently entirely novel. Novel solution generation was not a feature of trainee decisions but moments of intuition were. Expert goals varied between optimal care for individual patients, system-wide efficiency, and equity of care. The decision environment had a large influence upon experts.ConclusionExpert clinicians employ intuitive decision-making supported by rational analysis in early urgent care decision-making. Expert solutions generated in this manner are pragmatic rather than optimal, context dependent, and seek to achieve goals which vary from moment-moment. Findings are crucial to inform research evaluating the effectiveness of early expert decision-making in urgent care as it is a high cost strategy. They also have implications for methodological approaches in future studies of expert clinical decision-making, developing artificial expert systems, and clinician training.

Dataset Information

ChatGPT Influence on Medical Decision-Making, Bias, and Equity: A Randomized Study of Clinicians Evaluating Clinical Vignettes.

Publications

ChatGPT Influence on Medical Decision-Making, Bias, and Equity: A Randomized Study of Clinicians Evaluating Clinical Vignettes.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets