On collaborative reinforcement learning to optimize the redistribution of critical medical supplies throughout the COVID-19 pandemic.
Ontology highlight
ABSTRACT: Objective
This work investigates how reinforcement learning and deep learning models can facilitate the near-optimal redistribution of medical equipment in order to bolster public health responses to future crises similar to the COVID-19 pandemic.Materials and methods
The system presented is simulated with disease impact statistics from the Institute of Health Metrics, Centers for Disease Control and Prevention, and Census Bureau. We present a robust pipeline for data preprocessing, future demand inference, and a redistribution algorithm that can be adopted across broad scales and applications.Results
The reinforcement learning redistribution algorithm demonstrates performance optimality ranging from 93% to 95%. Performance improves consistently with the number of random states participating in exchange, demonstrating average shortage reductions of 78.74 ± 30.8% in simulations with 5 states to 93.50 ± 0.003% with 50 states.Conclusions
These findings bolster confidence that reinforcement learning techniques can reliably guide resource allocation for future public health emergencies.
SUBMITTER: Bednarski BP
PROVIDER: S-EPMC7799039 | biostudies-literature | 2021 Mar
REPOSITORIES: biostudies-literature
Publications
On collaborative reinforcement learning to optimize the redistribution of critical medical supplies throughout the COVID-19 pandemic.
Bednarski Bryan P BP Singh Akash Deep AD Jones William M WM
Journal of the American Medical Informatics Association : JAMIA 20210301 4
<h4>Objective</h4>This work investigates how reinforcement learning and deep learning models can facilitate the near-optimal redistribution of medical equipment in order to bolster public health responses to future crises similar to the COVID-19 pandemic.<h4>Materials and methods</h4>The system presented is simulated with disease impact statistics from the Institute of Health Metrics, Centers for Disease Control and Prevention, and Census Bureau. We present a robust pipeline for data preprocessi ...[more]
Similar Datasets
Project description:The aim of this work was to develop and evaluate the reinforcement learning algorithm VentAI, which is able to suggest a dynamically optimized mechanical ventilation regime for critically-ill patients. We built, validated and tested its performance on 11,943 events of volume-controlled mechanical ventilation derived from 61,532 distinct ICU admissions and tested it on an independent, secondary dataset (200,859 ICU stays; 25,086 mechanical ventilation events). A patient "data fingerprint" of 44 features was extracted as multidimensional time series in 4-hour time steps. We used a Markov decision process, including a reward system and a Q-learning approach, to find the optimized settings for positive end-expiratory pressure (PEEP), fraction of inspired oxygen (FiO2) and ideal body weight-adjusted tidal volume (Vt). The observed outcome was in-hospital or 90-day mortality. VentAI reached a significantly increased estimated performance return of 83.3 (primary dataset) and 84.1 (secondary dataset) compared to physicians' standard clinical care (51.1). The number of recommended action changes per mechanically ventilated patient constantly exceeded those of the clinicians. VentAI chose 202.9% more frequently ventilation regimes with lower Vt (5-7.5 mL/kg), but 50.8% less for regimes with higher Vt (7.5-10 mL/kg). VentAI recommended 29.3% more frequently PEEP levels of 5-7 cm H2O and 53.6% more frequently PEEP levels of 7-9 cmH2O. VentAI avoided high (>55%) FiO2 values (59.8% decrease), while preferring the range of 50-55% (140.3% increase). In conclusion, VentAI provides reproducible high performance by dynamically choosing an optimized, individualized ventilation strategy and thus might be of benefit for critically ill patients.
| S-EPMC7895944 | biostudies-literature
Project description:BackgroundOne of the significant changes in intensive care medicine over the past 2 decades is the acknowledgment that improper mechanical ventilation settings substantially contribute to pulmonary injury in critically ill patients. Artificial intelligence (AI) solutions can optimize mechanical ventilation settings in intensive care units (ICUs) and improve patient outcomes. Specifically, machine learning algorithms can be trained on large datasets of patient information and mechanical ventilation settings. These algorithms can then predict patient responses to different ventilation strategies and suggest personalized ventilation settings for individual patients.ObjectiveIn this study, we aimed to design and evaluate an AI solution that could tailor an optimal ventilator strategy for each critically ill patient who requires mechanical ventilation.MethodsWe proposed a reinforcement learning-based AI solution using observational data from multiple ICUs in the United States. The primary outcome was hospital mortality. Secondary outcomes were the proportion of optimal oxygen saturation and the proportion of optimal mean arterial blood pressure. We trained our AI agent to recommend low, medium, and high levels of 3 ventilator settings-positive end-expiratory pressure, fraction of inspired oxygen, and ideal body weight-adjusted tidal volume-according to patients' health conditions. We defined a policy as rules guiding ventilator setting changes given specific clinical scenarios. Off-policy evaluation metrics were applied to evaluate the AI policy.ResultsWe studied 21,595 and 5105 patients' ICU stays from the e-Intensive Care Unit Collaborative Research (eICU) and Medical Information Mart for Intensive Care IV (MIMIC-IV) databases, respectively. Using the learned AI policy, we estimated the hospital mortality rate (eICU 12.1%, SD 3.1%; MIMIC-IV 29.1%, SD 0.9%), the proportion of optimal oxygen saturation (eICU 58.7%, SD 4.7%; MIMIC-IV 49%, SD 1%), and the proportion of optimal mean arterial blood pressure (eICU 31.1%, SD 4.5%; MIMIC-IV 41.2%, SD 1%). Based on multiple quantitative and qualitative evaluation metrics, our proposed AI solution outperformed observed clinical practice.ConclusionsOur study found that customizing ventilation settings for individual patients led to lower estimated hospital mortality rates compared to actual rates. This highlights the potential effectiveness of using reinforcement learning methodology to develop AI models that analyze complex clinical data for optimizing treatment parameters. Additionally, our findings suggest the integration of this model into a clinical decision support system for refining ventilation settings, supporting the need for prospective validation trials.
| S-EPMC11525081 | biostudies-literature
Project description:BackgroundIn most cases, the abstracts of articles in the medical domain are publicly available. Although these are accessible by everyone, they are hard to comprehend for a wider audience due to the complex medical vocabulary. Thus, simplifying these complex abstracts is essential to make medical research accessible to the general public.ObjectiveThis study aims to develop a deep learning-based text simplification (TS) approach that converts complex medical text into a simpler version while maintaining the quality of the generated text.MethodsA TS approach using reinforcement learning and transformer-based language models was developed. Relevance reward, Flesch-Kincaid reward, and lexical simplicity reward were optimized to help simplify jargon-dense complex medical paragraphs to their simpler versions while retaining the quality of the text. The model was trained using 3568 complex-simple medical paragraphs and evaluated on 480 paragraphs via the help of automated metrics and human annotation.ResultsThe proposed method outperformed previous baselines on Flesch-Kincaid scores (11.84) and achieved comparable performance with other baselines when measured using ROUGE-1 (0.39), ROUGE-2 (0.11), and SARI scores (0.40). Manual evaluation showed that percentage agreement between human annotators was more than 70% when factors such as fluency, coherence, and adequacy were considered.ConclusionsA unique medical TS approach is successfully developed that leverages reinforcement learning and accurately simplifies complex medical paragraphs, thereby increasing their readability. The proposed TS approach can be applied to automatically generate simplified text for complex medical text data, which would enhance the accessibility of biomedical research to a wider audience.
| S-EPMC9719064 | biostudies-literature
Project description:BackgroundDecision support systems based on reinforcement learning (RL) have been implemented to facilitate the delivery of personalized care. This paper aimed to provide a comprehensive review of RL applications in the critical care setting.ObjectiveThis review aimed to survey the literature on RL applications for clinical decision support in critical care and to provide insight into the challenges of applying various RL models.MethodsWe performed an extensive search of the following databases: PubMed, Google Scholar, Institute of Electrical and Electronics Engineers (IEEE), ScienceDirect, Web of Science, Medical Literature Analysis and Retrieval System Online (MEDLINE), and Excerpta Medica Database (EMBASE). Studies published over the past 10 years (2010-2019) that have applied RL for critical care were included.ResultsWe included 21 papers and found that RL has been used to optimize the choice of medications, drug dosing, and timing of interventions and to target personalized laboratory values. We further compared and contrasted the design of the RL models and the evaluation metrics for each application.ConclusionsRL has great potential for enhancing decision making in critical care. Challenges regarding RL system design, evaluation metrics, and model choice exist. More importantly, further work is required to validate RL in authentic clinical environments.
| S-EPMC7400046 | biostudies-literature
Project description:BackgroundThe optimal indication, dose, and timing of corticosteroids in sepsis is controversial. Here, we used reinforcement learning to derive the optimal steroid policy in septic patients based on data on 3051 ICU admissions from the AmsterdamUMCdb intensive care database.MethodsWe identified septic patients according to the 2016 consensus definition. An actor-critic RL algorithm using ICU mortality as a reward signal was developed to determine the optimal treatment policy from time-series data on 277 clinical parameters. We performed off-policy evaluation and testing in independent subsets to assess the algorithm's performance.ResultsAgreement between the RL agent's policy and the actual documented treatment reached 59%. Our RL agent's treatment policy was more restrictive compared to the actual clinician behavior: our algorithm suggested withholding corticosteroids in 62% of the patient states, versus 52% according to the physicians' policy. The 95% lower bound of the expected reward was higher for the RL agent than clinicians' historical decisions. ICU mortality after concordant action in the testing dataset was lower both when corticosteroids had been withheld and when corticosteroids had been prescribed by the virtual agent. The most relevant variables were vital parameters and laboratory values, such as blood pressure, heart rate, leucocyte count, and glycemia.ConclusionsIndividualized use of corticosteroids in sepsis may result in a mortality benefit, but optimal treatment policy may be more restrictive than the routine clinical practice. Whilst external validation is needed, our study motivates a 'precision-medicine' approach to future prospective controlled trials and practice.
| S-EPMC9961939 | biostudies-literature
Project description:BACKGROUND:Modern clinical practice increasingly relies on collaborative and team-based approaches to care. Regulatory bodies in medical education emphasise the need to develop collaboration and teamwork competencies and highlight the need to do so from an early stage of medical training. In undergraduate medical education, the focus is usually on collaborative learning, associated with feedback and reflection on this learning This article describes a novel educational instrument, the Collaborative Learning Development Exercise (CLeD-EX), which aims to foster the development of key collaborative learning competencies in medical students. In this article we report on the effectiveness, feasibility and educational impact of the CLeD-EX. METHODS:In this study, the "educational design research" framework was used to develop, implement and evaluate the CLeD-EX. This involved adopting a systematic approach towards designing a creative and innovative instrument which would help solve a real-world challenge in developing collaborative learning skills. The systematic approach involved a qualitative exploration of key collaborative learning behaviours which are influential in effective collaborative learning contexts. The identified competencies were employed in the design of the CLeD-EX. The design of the CLeD-EX included features to facilitate structured feedback by tutors to students, complemented by self-evaluation and reflection. The CLeD-EX was field-tested with volunteer junior medical students, using a controlled pre-test post-test design. Analysis of the completed CLeD-EX forms, self-perception surveys (i.e. pre-test and post-test surveys) and analyses of reflective reports were used to explore the educational impact of CLeD-EX, as well as its utility and practicality. RESULTS:After using the CLeD-EX, students showed a significant improvement in critical thinking and group process as measured by a previously validated instrument. Both students and tutors recognised CLeD-EX as an effective instrument, especially as a structured basis for giving and receiving feedback and for completing the feedback loop. CLeD-EX was also found to be feasible, practical and focused, while promoting learning and effective interactions in small group learning. CONCLUSION:The findings of this study support the introduction of an effective and feasible educational instrument such as the CLeD-EX, to facilitate the development of students' skills in collaborative learning.
| S-EPMC7052979 | biostudies-literature
Project description:Traditional search and rescue methods in wilderness areas can be time-consuming and have limited coverage. Drones offer a faster and more flexible solution, but optimizing their search paths is crucial for effective operations. This paper proposes a novel algorithm using deep reinforcement learning to create efficient search paths for drones in wilderness environments. Our approach leverages a priori data about the search area and the missing person in the form of a probability distribution map. This allows the policy to learn optimal flight paths that maximize the probability of finding the missing person quickly. Experimental results show that our method achieves a significant improvement in search times compared to traditional coverage planning and search planning algorithms by over 160% , a difference that can mean life or death in real-world search operations Additionally, unlike previous work, our approach incorporates a continuous action space enabled by cubature, allowing for more nuanced flight patterns.
| S-EPMC11831046 | biostudies-literature
Project description:Enhancing deep learning performance requires extensive datasets. Centralized training raises concerns about data ownership and security. Additionally, large models are often unsuitable for hospitals due to their limited resource capacities. Federated learning (FL) has been introduced to address these issues. However, FL faces challenges such as vulnerability to attacks, non-IID data, reliance on a central server, high communication overhead, and suboptimal model aggregation. Furthermore, FL is not optimized for realistic hospital database environments, where data are dynamically accumulated. To overcome these limitations, we propose federated influencer learning (FIL) as a secure and efficient collaborative learning paradigm. Unlike the server-client model of FL, FIL features an equal-status structure among participants, with an administrator overseeing the overall process. FIL comprises four stages: local training, qualification, screening, and influencing. Local training is similar to vanilla FL, except for the optional use of a shared dataset. In the qualification stage, participants are classified as influencers or followers. During the screening stage, the integrity of the logits from the influencer is examined. If the integrity is confirmed, the influencer shares their knowledge with the others. FIL is more secure than FL because it eliminates the need for model-parameter transactions, central servers, and generative models. Additionally, FIL supports model-agnostic training. These features make FIL particularly promising for fields such as healthcare, where maintaining confidentiality is crucial. Our experiments demonstrated the effectiveness of FIL, which outperformed several FL methods on large medical (X-ray, MRI, and PET) and natural (CIFAR-10) image dataset in a dynamically accumulating database environment, with consistently higher precision, recall, Dice score, and lower standard deviation between participants. In particular, in the PET dataset, FIL achieved about a 40% improvement in Dice score and recall.
| S-EPMC11442468 | biostudies-literature
Project description:PurposeOne of the aims of medical education is to generate lifelong learners, leading to the identification of self-directed learning (SDL) as an essential component of medical education. While SDL is focused on an individual learner, collaboration is critically important in medicine. We developed an online course using the collaborative SDL (CSDL) framework. A goal for the course was for students to gain a better appreciation for the SDL process by exploring the COVID-19 pandemic.MethodsWe utilized CSDL to implement a 2-week elective attended by fifteen M3 and M4 medical students. Students submitted short videos reflecting on their course experience and the relevance of the material to their future training. Qualitative analysis of reflections was conducted to determine the effectiveness of the CSDL framework, and an assessment of the course evaluations was performed to explore student perceptions of the course and its effectiveness at preparing them for practice. A survey regarding student perceptions of SDL was offered to M3 and M4 students in order to explore their experiences with SDL and perceptions of its importance in the context of the school curriculum.ResultsThe CSDL framework was effective in making students aware of the importance of SDL in medical practice. Students gained basic and clinical knowledge about the subject, experienced increased confidence, and appreciated collaborating with their peers. The survey offered to the general student body reflected that all students perceived that they employed SDL in their time as medical students. However, many students indicated that they had not utilized components of SDL beyond synthesizing and assessing their learning needs.ConclusionCSDL is an effective method for promoting self-directed learning. Undergraduate medical course constructs utilizing CSDL will benefit students as they continue their career development.Supplementary informationThe online version contains supplementary material available at 10.1007/s40670-021-01493-7.
| S-EPMC8725963 | biostudies-literature
Project description:ProblemThe COVID-19 pandemic has presented a unique set of challenges to medical education globally. Low- and middle-income countries (LMICs) have faced unique barriers in transitioning to virtual modalities, and many medical students in LMICs experienced dramatically reduced educational time. The authors created the Global Medical Education Collaborative (GMEC) to address this problem by providing free, online, case-based tutorials to medical students in LMICs during the pandemic.ApproachThe authors developed a needs assessment to gauge students' educational requirements, which informed GMEC's 2 primary goals: to provide free access to interactive online tutorials for students in LMICs and to bridge the physical distance between educators and learners via an online platform. A pilot program in Nigeria (April 26-May 26, 2020) helped inform the current strategy and logistics. Tutors and students were recruited via social media and medical education networks at the authors' home institutions.OutcomesWithin the first 2 months (April 26-June 26, 2020), 324 students representing 12 countries and 20+ medical schools joined GMEC. Additionally, 95 physicians and trainees joined as tutors and, collectively, delivered 52 tutorials. Students responded to a needs assessment querying confidence in various clinical domains, interest in covering clinical topics, barriers to virtual learning, and the effect of the pandemic on their education. Tutors held 1-hour, interactive tutorials over Zoom covering a variety of clinical topics. According to surveys, 91% of students (71 of 78) felt more confident in the material related to the tutorial's topic after participating.Next stepsGMEC will continue to engage students, tutors, and collaborators to facilitate the delivery of innovative, high-quality tutorials to students affected by COVID-19 in LMICs. To ensure that the platform is sustainable and aligned with GMEC's mission to promote equity in global medical education, the collaborative will need to be agile and responsive.
| S-EPMC8541891 | biostudies-literature