Dataset Information

Completeness and readability of GPT-4-generated multilingual discharge instructions in the pediatric emergency department.

ABSTRACT:

Objectives

The aim of this study was to assess the completeness and readability of generative pre-trained transformer-4 (GPT-4)-generated discharge instructions at prespecified reading levels for common pediatric emergency room complaints.

Materials and methods

The outputs for 6 discharge scenarios stratified by reading level (fifth or eighth grade) and language (English, Spanish) were generated fivefold using GPT-4. Specifically, 120 discharge instructions were produced and analyzed (6 scenarios: 60 in English, 60 in Spanish; 60 at a fifth-grade reading level, 60 at an eighth-grade reading level) and compared for completeness and readability (between language, between reading level, and stratified by group and reading level). Completeness was defined as the proportion of literature-derived key points included in discharge instructions. Readability was quantified using Flesch-Kincaid (English) and Fernandez-Huerta (Spanish) readability scores.

Results

English-language GPT-generated discharge instructions contained a significantly higher proportion of must-include discharge instructions than those in Spanish (English: mean (standard error of the mean) = 62% (3%), Spanish: 53% (3%), P = .02). In the fifth-grade and eighth-grade level conditions, there was no significant difference between English and Spanish outputs in completeness. Readability did not differ across languages.

Discussion

GPT-4 produced readable discharge instructions in English and Spanish while modulating document reading level. Discharge instructions in English tended to have higher completeness than those in Spanish.

Conclusion

Future research in prompt engineering and GPT-4 performance, both generally and in multiple languages, is needed to reduce potential for health disparities by language and reading level.

SUBMITTER: Gimeno A

PROVIDER: S-EPMC11216721 | biostudies-literature | 2024 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Completeness and readability of GPT-4-generated multilingual discharge instructions in the pediatric emergency department.

Gimeno Alex A Krause Kevin K D'Souza Starina S Walsh Colin G CG

JAMIA open 20240701 3

<h4>Objectives</h4>The aim of this study was to assess the completeness and readability of generative pre-trained transformer-4 (GPT-4)-generated discharge instructions at prespecified reading levels for common pediatric emergency room complaints.<h4>Materials and methods</h4>The outputs for 6 discharge scenarios stratified by reading level (fifth or eighth grade) and language (English, Spanish) were generated fivefold using GPT-4. Specifically, 120 discharge instructions were produced and analy ...[more]

PMID: 38957592

Similar Datasets

Project description:BackgroundDischarge instructions are a key form of documentation and patient communication in the time of transition from the emergency department (ED) to home. Discharge instructions are time-consuming and often underprioritized, especially in the ED, leading to discharge delays and possibly impersonal patient instructions. Generative artificial intelligence and large language models (LLMs) offer promising methods of creating high-quality and personalized discharge instructions; however, there exists a gap in understanding patient perspectives of LLM-generated discharge instructions.ObjectiveWe aimed to assess the use of LLMs such as ChatGPT in synthesizing accurate and patient-accessible discharge instructions in the ED.MethodsWe synthesized 5 unique, fictional ED encounters to emulate real ED encounters that included a diverse set of clinician history, physical notes, and nursing notes. These were passed to GPT-4 in Azure OpenAI Service (Microsoft) to generate LLM-generated discharge instructions. Standard discharge instructions were also generated for each of the 5 unique ED encounters. All GPT-generated and standard discharge instructions were then formatted into standardized after-visit summary documents. These after-visit summaries containing either GPT-generated or standard discharge instructions were randomly and blindly administered to Amazon MTurk respondents representing patient populations through Amazon MTurk Survey Distribution. Discharge instructions were assessed based on metrics of interpretability of significance, understandability, and satisfaction.ResultsOur findings revealed that survey respondents' perspectives regarding GPT-generated and standard discharge instructions were significantly (P=.01) more favorable toward GPT-generated return precautions, and all other sections were considered noninferior to standard discharge instructions. Of the 156 survey respondents, GPT-generated discharge instructions were assigned favorable ratings, "agree" and "strongly agree," more frequently along the metric of interpretability of significance in discharge instruction subsections regarding diagnosis, procedures, treatment, post-ED medications or any changes to medications, and return precautions. Survey respondents found GPT-generated instructions to be more understandable when rating procedures, treatment, post-ED medications or medication changes, post-ED follow-up, and return precautions. Satisfaction with GPT-generated discharge instruction subsections was the most favorable in procedures, treatment, post-ED medications or medication changes, and return precautions. Wilcoxon rank-sum test of Likert responses revealed significant differences (P=.01) in the interpretability of significant return precautions in GPT-generated discharge instructions compared to standard discharge instructions but not for other evaluation metrics and discharge instruction subsections.ConclusionsThis study demonstrates the potential for LLMs such as ChatGPT to act as a method of augmenting current documentation workflows in the ED to reduce the documentation burden of physicians. The ability of LLMs to provide tailored instructions for patients by improving readability and making instructions more applicable to patients could improve upon the methods of communication that currently exist.

Project description:IntroductionPrevious studies have shown that patients in the emergency department (ED) are frequently given incomplete discharge instructions that are written at least four grade levels above the recommended sixth-grade reading level, leading to poor understanding. Our aims in this study were to implement standardized discharge instructions containing six key components written at a more appropriate reading level for common emergency department (ED) diagnoses to improve patient understanding.MethodsWe conducted this study in a 20-bed ED at an urban Veteran's Administration hospital. Data was collected via in-person patient and clinician interviews. Patient interviews were conducted after patients received their discharge instructions. We compared patient responses to clinician responses and marked them as incorrect, partially correct, or correct with a score of 0, 0.5, or 1, respectively. The maximum possible score for each interview was six. Six key components of discharge instructions were asked about: diagnosis; new medications; at-home care; duration of illness; reasons to return; and follow-up. There were 25 patients in the pre-intervention group and 20 in the intervention group with the standardized set of instructions. We performed a Mann-Whitney U test on the total interview scores in the control and intervention groups and conducted a sub-analysis on the individual scores for each of the six key components.ResultsThe patients in the intervention demonstrated a statistically significant increase in patient-clinician correlation when compared to the patients in the pre-intervention group overall (P < 0.05), and two of the six key components of the discharge instructions individually showed statistically significant increase in patient-clinician correlation when standardized discharge instructions were used.ConclusionPatients who received the standardized discharge instructions had improved understanding of their discharge instructions. Future opportunities extending off this pilot study include expanding the number of diagnoses for which standardized instructions are used and investigating patient-centered outcomes related to these instructions.

Project description:The aim was to evaluate if the addition of video discharge instructions (VDIs) to usual verbal information improved the comprehension of information provided to caregivers of patients who consult for acute gastroenteritis (AGE). We conducted an open-label, parallel, randomized trial, enrolling patients who consulted for AGE at a tertiary hospital. First, caregivers answered a written test concerning AGE characteristics and management. They were randomly allocated to a control group, which received the usual verbal instructions, or to an intervention group, which additionally received VDI. After discharge, caregivers were contacted by telephone and answered the same test, satisfaction questions, and follow-up information. From September 2019 to March 2020, 139 patients were randomized, 118 completed follow-up. The mean score was 3.13 (SD 1.07) over 5 points in the initial test and 3.96 (SD 0.96) in the follow-up test. Patients in the intervention group had a greater improvement (1.17 points, SD 1.11) than those in the control group (0.47 points, SD 0.94, p < 0.001). In the follow-up test, 49.1% in the intervention group and 18.6% in the control group answered all questions correctly (p < 0.001). There were no significant differences in return visits. Caregivers gave high satisfaction scores regardless of the allocation group.Conclusion: Video instructions improve caregivers' understanding of discharge information.Trial registration: [NCT04463355, retrospectively registered July 9, 2020]. What is Known: • Poor comprehension of discharge instructions leads to incorrect treatment after discharge, increased readmissions and a reduction of caregivers' satisfaction. • Video discharge instructions are useful providing concise information independently of the patients' health literacy level or communication skills of the health care provider What is New: • The addition of video discharge instructions to verbal instructions improves caregivers' knowledge about AGE improved with respect to those who only receive verbal instructions • Video instructions do not add extra time to the emergency department visits.

Project description:ObjectiveTo compare the completeness of Emergency Department (ED) discharge instructions before and after introduction of an electronic discharge instructions module by scoring compliance with the Centers for Medicare and Medicaid Services (CMS) Outpatient Measure 19 (OP-19).MethodsWe performed a quasi-experimental study examining the impact of an electronic discharge instructions module in an academic ED. Three hundred patients discharged home from the ED were randomly selected from two time intervals: 150 patients three months before and 150 patients three to five months after implementation of the new electronic module. The discharge instructions for each patient were reviewed, and compliance for each individual OP-19 element as well as overall OP-19 compliance was scored per CMS specifications. Compliance rates as well as risk ratios (RR) and risk differences (RD) with 95% confidence intervals (CI) comparing the overall OP-19 scores and individual OP-19 element scores of the electronic and paper-based discharge instructions were calculated.ResultsThe electronic discharge instructions had 97.3% (146/150) overall OP-19 compliance, while the paper-based discharge instructions had overall compliance of 46.7% (70/150). Electronic discharge instructions were twice as likely to achieve overall OP-19 compliance compared to the paper-based format (RR: 2.09, 95% CI: 1.75 - 2.48). The largest improvement was in documentation of major procedures and tests performed: only 60% of the paper-based discharge instructions satisfied this criterion, compared to 100% of the electronic discharge instructions (RD: 40.0%, 95% CI: 32.2% - 47.8%). There was a modest difference in medication documentation with 92.7% for paper-based and 100% for electronic formats (RD: 7.3%, 95% CI: 3.2% - 11.5%). There were no statistically significant differences in documentation of patient care instructions and diagnosis between paper-based and electronic formats.ConclusionWith careful design, information technology can improve the completeness of ED patient discharge instructions and performance on the OP-19 quality measure.

Project description:BACKGROUND:Previous studies have demonstrated that patients have poor understanding of the discharge instructions provided from the emergency department (ED). The aims of this study are to determine if patient factors, such as income and level of education, correlate with patient understanding of discharge instructions and to explore if different patient populations prefer different resources for receiving discharge instructions. METHODS:We conducted live observations of physicians providing discharge instructions in the ED to 100 patients followed by a patient survey to determine their understanding in four domains (diagnosis, treatment plan, follow-up instructions, and return to ED (RTED) instructions) and collect patient demographics. We enrolled patients over the age of 18 being discharged home. We excluded non-English- or French-speaking patients and those with significant psychiatric history or cognitive impairment. We performed a two-way ANOVA analysis of patient factors and patient understanding. RESULTS:We found that patients had poor understanding of discharge instructions, ranging from 24.0% having poor understanding of their follow-up plan to 64.0% for RTED instructions. Almost half (42%) of patients did not receive complete discharge instructions. Lower income was correlated with a significant decrease in patient understanding of discharge diagnosis (p = 0.01) and RTED instructions (p = 0.04). Patients who did not complete high school trended towards lower levels of understanding of their diagnosis and treatment plan (p = 0.06). Lower income patients had a preference for receiving a follow-up phone call by a nurse, while higher income patients preferred online resources. CONCLUSIONS:Lower income patients and those who have not completed high school are at a higher risk of poor understanding discharge instructions. As new technological solutions emerge to aid patient understanding of discharge instructions, our study suggests they may not aid those who are at the highest risk of failing to understand their instructions.

Project description:IntroductionThere are more than 80,000 emergency department (ED) visits for non-fatal bullet-related injuries (BRI) per year in the United States. Approximately half of these patients are discharged home from the ED. Our objective in this study was to characterize the discharge instructions, prescriptions, and follow-up plans provided to patients discharged from the ED after BRI.MethodsThis was a single-center, cross-sectional study of the first 100 consecutive patients who presented to an urban, academic, Level I trauma center ED with an acute BRI beginning on January 1, 2020. We queried the electronic health record for patient demographics, insurance status, cause of injury, hospital arrival and discharge timestamps, discharge prescriptions, and documented instructions regarding wound care, pain management, and follow-up plans. We analyzed data using descriptive statistics and chi-square tests.ResultsDuring the study period, 100 patients presented to the ED with an acute firearm injury. Patients were predominantly young (median age 29, interquartile range 23-38 years), male (86%), Black (85%), non-Hispanic (98%), and uninsured (70%). We found that 12% of patients did not receive any type of written wound care instruction, while 37% received discharge paperwork that included instructions to take both an NSAID and acetaminophen. Fifty-one percent of patients received an opioid prescription, with a range from 3-42 tablets (median 10 tablets). The proportion of patients receiving an opioid prescription was significantly higher among White patients (77%) than among Black patients (47%).ConclusionThere is variability in prescriptions and instructions provided to survivors of bullet injuries upon ED discharge at our institution. Our data indicates that standardized discharge protocols could improve quality of care and equity in the treatment of patients who have survived a BRI. Current variable quality in discharge planning is an entry point for structural racism and disparity.

Dataset Information

Completeness and readability of GPT-4-generated multilingual discharge instructions in the pediatric emergency department.

Objectives

Materials and methods

Results

Discussion

Conclusion

Publications

Completeness and readability of GPT-4-generated multilingual discharge instructions in the pediatric emergency department.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets