Exploring the capabilities and challenges of ChatGPT in disseminating urologic information.
Ontology highlight
ABSTRACT:
SUBMITTER: McMahon A
PROVIDER: S-EPMC10999024 | biostudies-literature | 2024 Mar
REPOSITORIES: biostudies-literature
Publications
Exploring the capabilities and challenges of ChatGPT in disseminating urologic information.
McMahon Amber A Jacob Cian L CL Bird Vincent G VG Whiles Bristol B BB
Translational andrology and urology 20240311 3
Similar Datasets
Project description:PurposeTechnological progress of machine learning and natural language processing has led to the development of large language models (LLMs), capable of producing well-formed text responses and providing natural language access to knowledge. Modern conversational LLMs such as ChatGPT have shown remarkable capabilities across a variety of fields, including medicine. These models may assess even highly specialized medical knowledge within specific disciplines, such as radiation therapy. We conducted an exploratory study to examine the capabilities of ChatGPT to answer questions in radiation therapy.Methods and materialsA set of multiple-choice questions about clinical, physics, and biology general knowledge in radiation oncology as well as a set of open-ended questions were created. These were given as prompts to the LLM ChatGPT, and the answers were collected and analyzed. For the multiple-choice questions, it was checked how many of the answers of the model could be clearly assigned to one of the allowed multiple-choice-answers, and the proportion of correct answers was determined. For the open-ended questions, independent blinded radiation oncologists evaluated the quality of the answers regarding correctness and usefulness on a 5-point Likert scale. Furthermore, the evaluators were asked to provide suggestions for improving the quality of the answers.ResultsFor 70 multiple-choice questions, ChatGPT gave valid answers in 66 cases (94.3%). In 60.61% of the valid answers, the selected answer was correct (50.0% of clinical questions, 78.6% of physics questions, and 58.3% of biology questions). For 25 open-ended questions, 12 answers of ChatGPT were considered as "acceptable," "good," or "very good" regarding both correctness and helpfulness by all 6 participating radiation oncologists. Overall, the answers were considered "very good" in 29.3% and 28%, "good" in 28% and 29.3%, "acceptable" in 19.3% and 19.3%, "bad" in 9.3% and 9.3%, and "very bad" in 14% and 14% regarding correctness/helpfulness.ConclusionsModern conversational LLMs such as ChatGPT can provide satisfying answers to many relevant questions in radiation therapy. As they still fall short of consistently providing correct information, it is problematic to use them for obtaining medical information. As LLMs will further improve in the future, they are expected to have an increasing impact not only on general society, but also on clinical practice, including radiation oncology.
| S-EPMC10831180 | biostudies-literature
Project description:PurposeTo investigate the potential use of large language models (LLMs) in orthopaedics by presenting queries pertinent to anterior cruciate ligament (ACL) surgery to generative pre-trained transformer (ChatGPT, specifically using its GPT-4 model of March 14th 2023). Additionally, this study aimed to evaluate the depth of the LLM's knowledge and investigate its adaptability to different user groups. It was hypothesized that the ChatGPT would be able to adapt to different target groups due to its strong language understanding and processing capabilities.MethodsChatGPT was presented with 20 questions and response was requested for two distinct target audiences: patients and non-orthopaedic medical doctors. Two board-certified orthopaedic sports medicine surgeons and two expert orthopaedic sports medicine surgeons independently evaluated the responses generated by ChatGPT. Mean correctness, completeness, and adaptability to the target audiences (patients and non-orthopaedic medical doctors) were determined. A three-point response scale facilitated nuanced assessment.ResultsChatGPT exhibited fair accuracy, with average correctness scores of 1.69 and 1.66 (on a scale from 0, incorrect, 1, partially correct, to 2, correct) for patients and medical doctors, respectively. Three of the 20 questions (15.0%) were deemed incorrect by any of the four orthopaedic sports medicine surgeon assessors. Moreover, overall completeness was calculated to be 1.51 and 1.64 for patients and medical doctors, respectively, while overall adaptiveness was determined to be 1.75 and 1.73 for patients and doctors, respectively.ConclusionOverall, ChatGPT was successful in generating correct responses in approximately 65% of the cases related to ACL surgery. The findings of this study imply that LLMs offer potential as a supplementary tool for acquiring orthopaedic knowledge. However, although ChatGPT can provide guidance and effectively adapt to diverse target audiences, it cannot supplant the expertise of orthopaedic sports medicine surgeons in diagnostic and treatment planning endeavours due to its limited understanding of orthopaedic domains and its potential for erroneous responses.Level of evidenceV.
| S-EPMC10598178 | biostudies-literature
Project description:The impact of microbial communities, also known as the microbiome, on human health and the environment is receiving increased attention. Studying translated gene products (proteins) and comparing metaproteomic profiles may elucidate how microbiomes respond to specific environmental stimuli, and interact with host organisms. Characterizing proteins expressed by a complex microbiome and interpreting their functional signature requires sophisticated informatics tools and workflows tailored to metaproteomics. Additionally, there is a need to disseminate these informatics resources to researchers undertaking metaproteomic studies, who could use them to make new and important discoveries in microbiome research. The Galaxy for proteomics platform (Galaxy-P) offers an open source, web-based bioinformatics platform for disseminating metaproteomics software and workflows. Within this platform, we have developed easily-accessible and documented metaproteomic software tools and workflows aimed at training researchers in their operation and disseminating the tools for more widespread use. The modular workflows encompass the core requirements of metaproteomic informatics: (a) database generation; (b) peptide spectral matching; (c) taxonomic analysis and (d) functional analysis. Much of the software available via the Galaxy-P platform was selected, packaged and deployed through an online metaproteomics "Contribution Fest" undertaken by a unique consortium of expert software developers and users from the metaproteomics research community, who have co-authored this manuscript. These resources are documented on GitHub and freely available through the Galaxy Toolshed, as well as a publicly accessible metaproteomics gateway Galaxy instance. These documented workflows are well suited for the training of novice metaproteomics researchers, through online resources such as the Galaxy Training Network, as well as hands-on training workshops. Here, we describe the metaproteomics tools available within these Galaxy-based resources, as well as the process by which they were selected and implemented in our community-based work. We hope this description will increase access to and utilization of metaproteomics tools, as well as offer a framework for continued community-based development and dissemination of cutting edge metaproteomics software.
| S-EPMC5874766 | biostudies-literature
Project description:Background and objectivePatients struggle to classify symptoms, which hinders timely medical presentation. With 35-75% of patients seeking information online before consulting a health care professional, generative language-based artificial intelligence (AI), exemplified by ChatGPT-3.5 (GPT-3.5) from OpenAI, has emerged as an important source. The aim of our study was to evaluate the role of GPT-3.5 in triaging acute urological conditions to address a gap in current research.MethodsWe assessed GPT-3.5 performance in providing urological differential diagnoses (DD) and recommending a course of action (CoA). Six acute urological pathologies were identified for evaluation. Lay descriptions, sourced from patient forums, formed the basis for 472 queries that were independently entered by nine urologists. We evaluated the output in terms of compliance with the European Association of Urology (EAU) guidelines, the quality of the patient information using the validated DISCERN questionnaire, and a linguistic analysis.Key findings and limitationsThe median GPT-3.5 ratings were 4/5 for DD and CoA, and 3/5 for overall information quality. English outputs received higher median ratings than German outputs for DD (4.27 vs 3.95; p < 0.001) and CoA (4.25 vs 4.05; p < 0.005). There was no difference in performance between urgent and non-urgent cases. Analysis of the information quality revealed notable underperformance for source indication, risk assessment, and influence on quality of life.Conclusion and clinical implicationsOur results highlights the potential of GPT-3.5 as a triage system for offering individualized, empathetic advice mostly aligned with the EAU guidelines, outscoring other online information. Relevant shortcomings in terms of information quality, especially for risk assessment, need to be addressed to enhance the reliability. Broader transparency and quality improvements are needed before integration into, primarily English-speaking, patient care.Patient summaryWe looked at the performance of ChatGPT-3.5 for patients seeking urology advice. We entered more than 400 German and English inputs and assessed the possible diagnoses suggested by this artificial intelligence tool. ChatGPT-3.5 scored well in providing a complete list of possible diagnoses and recommending a course of action mostly in line with current guidelines. The quality of the information was good overall, but missing and unclear sources for the information can be a problem.
| S-EPMC11567918 | biostudies-literature
Project description:PurposeThis study evaluated and compared the clinical support capabilities of ChatGPT 4o and ChatGPT 4o mini in diagnosing and treating lumbar disc herniation (LDH) with radiculopathy.MethodsTwenty-one questions (across 5 categories) from NASS Clinical Guidelines were input into ChatGPT 4o and ChatGPT 4o mini. Five orthopedic surgeons assessed their responses using a 5-point Likert scale for accuracy and completeness, and a 7-point scale for reliability. Flesch Reading Ease scores were calculated to assess readability. Additionally, ChatGPT 4o analyzed lumbar images from 53 patients, comparing its recognizable agreement with orthopedic surgeons using Kappa values.ResultsBoth models demonstrated strong clinical support capabilities with no significant differences in accuracy or reliability. However, ChatGPT 4o provided more comprehensive and consistent responses. The Flesch Reading Ease scores for both models indicated that their generated content was "very difficult to read," potentially limiting patient accessibility. In evaluating lumbar disc herniation images, ChatGPT 4o achieved an overall accuracy of 0.81, with LDH recognition precision, recall, and F1 scores exceeding 0.80. The AUC was 0.80, and the Kappa value was 0.61, indicating moderate agreement between the model's predictions and actual diagnoses, though with room for improvement.ConclusionWhile both models are effective, ChatGPT 4o offers more comprehensive clinical responses, making it more suitable for high-integrity medical tasks. However, the difficulty in reading AI-generated content and occasional use of misleading terms, such as "tumor," indicate a need for further improvements to reduce patient anxiety.
| S-EPMC11753088 | biostudies-literature
Project description:This study investigates the potential of ChatGPT 4 in the assessment of personality traits based on written texts. Using two publicly available datasets containing both written texts and self-assessments of the authors' psychological traits based on the Big Five model, we aimed to evaluate the predictive performance of ChatGPT 4. For each sample text, we asked for numerical predictions on an eleven-point scale and compared them with the self-assessments. We also asked for ChatGPT 4 confidence scores on an eleven-point scale for each prediction. To keep the study within a manageable scope, a zero-prompt modality was chosen, although more sophisticated prompting strategies could potentially improve performance. The results show that ChatGPT 4 has moderate but significant abilities to automatically infer personality traits from written text. However, it also shows limitations in recognizing whether the input text is appropriate or representative enough to make accurate inferences, which could hinder practical applications. Furthermore, the results suggest that improved benchmarking methods could increase the efficiency and reliability of the evaluation process. These results pave the way for a more comprehensive evaluation of the capabilities of Large Language Models in assessing personality traits from written texts.
| S-EPMC11865037 | biostudies-literature
Project description:BackgroundSocial media are potentially powerful channels for communicating relevant health information in culturally sensitive and influential ways to key audiences. Moreover, these channels hold promise for promoting awareness and knowledge of health risks, prevention, and treatment by utilizing opinion leaders for message dissemination. Despite limited empirical evidence to-date, early promising results suggest that blogs are a form of social media that should be examined as worthy channels for health communication.ObjectivesThis formative study explored mommy bloggers' perceptions about sharing health-related information on their blogs with their readers. It also sought to analyze which topics would be of most interest to mommy bloggers, what motivates them to write about health issues, and how they perceive interest in these topics among their readers.MethodsThis study employed survey methodology, including the use of open-ended questions, the responses to which were coded for analysis. Specifically, a 14-item survey was fielded with mommy bloggers between October 1 and October 28, 2016. Bloggers were recruited through The Motherhood network. A total of 461 mommy bloggers responded to the survey; 163 were removed for low quality responses and incomplete data. As a result, 298 eligible participants completed the survey. For open-ended questions in the survey, a sample of responses were coded and analyzed.ResultsThe majority of the respondents (87.2%, 260/298) reported that they have written about health issues in the past; 97.3% (290/298) of the respondents reported that they would consider writing about health issues sometime in the future, and 96.3% (287/298) of the respondents reported that their readers like to read about health issues on their blogs. In terms of content priorities for this sample of bloggers, Nutrition and Physical Activity dominate the current conversation and similarly, Physical Activity and Nutrition remain top content priorities for these bloggers for the future. Moreover, 21.3% of the respondents reported that their readers would be interested in these topics. Finally, having a personal connection with a health issue was found to be positively associated with likeliness to write about health issues on their blog (P<.001).ConclusionsThis study illustrates that there are potentially rich opportunities for working with mommy bloggers to communicate with key health decision makers (moms) on important health issues. There is a great support among mommy bloggers for health information dissemination as well as interest for accessing relevant health information from their readers. This presents an opportunity for public health research and communication campaigns to more broadly promote their messages, thereby contributing to their behavior change objectives. Limitations included overrepresentation of white, higher-educated, and younger women. It suggests a need for more targeted engagement of a diverse sample for future work.
| S-EPMC5941100 | biostudies-literature
Project description:We present a draft Minimum Information About Geospatial Information System (MIAGIS) standard for facilitating public deposition of geospatial information system (GIS) datasets that follows the FAIR (Findable, Accessible, Interoperable and Reusable) principles. The draft MIAGIS standard includes a deposition directory structure and a minimum javascript object notation (JSON) metadata formatted file that is designed to capture critical metadata describing GIS layers and maps as well as their sources of data and methods of generation. The associated miagis Python package facilitates the creation of this MIAGIS metadata file and directly supports metadata extraction from both Esri JSON and GEOJSON GIS data formats plus options for extraction from user-specified JSON formats. We also demonstrate their use in crafting two example depositions of ArcGIS generated maps. We hope this draft MIAGIS standard along with the supporting miagis Python package will assist in establishing a GIS standards group that will develop the draft into a full standard for the wider GIS community as well as a future public repository for GIS datasets.
| S-EPMC10275873 | biostudies-literature
Project description: Not available
| S-EPMC11697136 | biostudies-literature
Project description:SummaryProtein structure prediction aims to infer a protein's three-dimensional (3D) structure from its amino acid sequence. Protein structure is pivotal for elucidating protein functions, interactions, and driving biotechnological innovation. The deep learning model AlphaFold2, has revolutionized this field by leveraging phylogenetic information from multiple sequence alignments (MSAs) to achieve remarkable accuracy in protein structure prediction. However, a key question remains: how well does AlphaFold2 understand protein structures? This study investigates AlphaFold2's capabilities when relying primarily on high-quality template structures, without the additional information provided by MSAs. By designing experiments that probe local and global structural understanding, we aimed to dissect its dependence on specific features and its ability to handle missing information. Our findings revealed AlphaFold2's reliance on sterically valid C β for correctly interpreting structural templates. Additionally, we observed its remarkable ability to recover 3D structures from certain perturbations and the negligible impact of the previous structure in recycling. Collectively, these results support the hypothesis that AlphaFold2 has learned an accurate biophysical energy function. However, this function seems most effective for local interactions. Our work advances understanding of how deep learning models predict protein structures and provides guidance for researchers aiming to overcome limitations in these models.Availability and implementationData and implementation are available at https://github.com/ibmm-unibe-ch/template-analysis.
| S-EPMC11751578 | biostudies-literature