Dataset Information

Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool.

ABSTRACT:

Background

Overuse of blood cultures (BCs) in emergency departments (EDs) leads to low yields and high numbers of contaminated cultures, accompanied by increased diagnostics, antibiotic usage, prolonged hospitalization, and mortality. We aimed to simplify and validate a recently developed machine learning model to help safely withhold BC testing in low-risk patients.

Methods

We extracted data from the electronic health records (EHR) for 44.123 unique ED visits with BC sampling in the Amsterdam UMC (locations VUMC and AMC; the Netherlands), Zaans Medical Center (ZMC; the Netherlands), and Beth Israel Deaconess Medical Center (BIDMC; United States) in periods between 2011 and 2021. We trained a machine learning model on the VUMC data to predict blood culture outcomes and validated it in the AMC, ZMC, and BIDMC with subsequent real-time prospective evaluation in the VUMC.

Findings

The model had an Area Under the Receiver Operating Characteristics curve (AUROC) of 0.81 (95%-CI = 0.78-0.83) in the VUMC test set. The most important predictors were temperature, creatinine, and C-reactive protein. The AUROCs in the validation cohorts were 0.80 (AMC; 0.78-0.82), 0.76 (ZMC; 0.74-0.78), and 0.75 (BIDMC; 0.74-0.76). During real-time prospective evaluation in the EHR of the VUMC, it reached an AUROC of 0.76 (0.71-0.81) among 590 patients with BC draws in the ED. The prospective evaluation showed that the model can be used to safely withhold blood culture analyses in at least 30% of patients in the ED.

Interpretation

We developed a machine learning model to predict blood culture outcomes in the ED, which retained its performance during external validation and real-time prospective evaluation. Our model can identify patients at low risk of having a positive blood culture. Using the model in practice can significantly reduce the number of blood culture analyses and thus avoid the hidden costs of false-positive culture results.

Funding

This research project was funded by the Amsterdam Public Health - Quality of Care program and the Dutch "Doen of Laten" project (project number: 839205002).

SUBMITTER: Schinkel M

PROVIDER: S-EPMC9294655 | biostudies-literature | 2022 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool.

Schinkel Michiel M Boerman Anneroos W AW Bennis Frank C FC Minderhoud Tanca C TC Lie Mei M Peters-Sengers Hessel H Holleman Frits F Schade Rogier P RP de Jonge Robert R Wiersinga W Joost WJ Nanayakkara Prabath W B PWB

EBioMedicine 20220716

<h4>Background</h4>Overuse of blood cultures (BCs) in emergency departments (EDs) leads to low yields and high numbers of contaminated cultures, accompanied by increased diagnostics, antibiotic usage, prolonged hospitalization, and mortality. We aimed to simplify and validate a recently developed machine learning model to help safely withhold BC testing in low-risk patients.<h4>Methods</h4>We extracted data from the electronic health records (EHR) for 44.123 unique ED visits with BC sampling in ...[more]

PMID: 35853298

Similar Datasets

Project description:ImportanceThe management of patients with syncope in the emergency department (ED) is challenging because no robust risk tool available has been recommended for clinical use.ObjectiveTo validate the Canadian Syncope Risk Score (CSRS) in a new cohort of patients with syncope to determine its ability to predict 30-day serious outcomes not evident during index ED evaluation.Design, setting, and participantsThis prospective multicenter cohort study conducted at 9 EDs across Canada included patients 16 years and older who presented to EDs within 24 hours of syncope. Patients were enrolled from March 2014 to April 2018.Main outcomes and measuresBaseline characteristics, CSRS predictors, and 30-day adjudicated serious outcomes, including arrhythmic (arrhythmias, interventions for arrhythmia, or unknown cause of death) and nonarrhythmic (myocardial infarction, structural heart disease, pulmonary embolism, or hemorrhage) serious outcomes, were collected. Calibration and discrimination characteristics for CSRS validation were calculated.ResultsA total of 3819 patients were included (mean [SD] age 53.9 [22.8] years; 2088 [54.7%] female), of whom 139 (3.6%) experienced 30-day serious outcomes, including 13 patients (0.3%) who died. In the validation cohort, there were no differences between the predicted and observed risk, the calibration slope was 1.0, and the area under the receiver operating characteristic curve was 0.91 (95% CI, 0.88-0.93). The empirical probability of a 30-day serious outcome during validation was 3.64% (95% CI, 3.09%-4.28%) compared with the model-predicted probability of 3.17% (95% CI, 2.66%-3.77%; P = .26). The proportion of patients with 30-day serious outcomes increased from 3 of 1631 (0.3%) in the very-low-risk group to 40 of 78 (51.3%) in the very-high-risk group (Cochran-Armitage trend test P < .001). There was a similar significant increase in the serious outcome subtypes with increasing CSRS risk category. None of the very-low-risk and low-risk patients died or experienced ventricular arrhythmia. At a threshold score of -1 (2145 of 3819 patients), the CSRS sensitivity and specificity were 97.8% (95% CI, 93.8%-99.6%) and 44.3% (95% CI, 42.7%-45.9%), respectively.Conclusions and relevanceThe CSRS was successfully validated and its use is recommended to guide ED management of patients when serious causes are not identified during index ED evaluation. Very-low-risk and low-risk patients can generally be discharged, while brief hospitalization can be considered for high-risk patients. We believe CSRS implementation has the potential to improve patient safety and health care efficiency.

Project description:ObjectivesTraditional approaches to safety and quality screening in the emergency department (ED) are porous and low yield for identifying adverse events (AEs). A better approach may be in the use of trigger tool methodology. We recently developed a novel ED trigger tool using a multidisciplinary, multicenter approach. We conducted a multicenter test of this tool and assess its performance.MethodsIn design and participants, we studied the ED trigger tool for a 13-month period at four EDs. All patients 18 years and older with Emergency Severity Index acuity levels of 1 to 3 seen by a provider were eligible. Reviewers completed standardized training modules. Each site reviewed 50 randomly selected visits per month. A first-level reviewer screened for presence of predefined triggers (findings that increase the probability of an AE). If no trigger is present, the review is deemed complete. When present, a trigger prompts an in-depth review for an AE. Any event identified is assigned a level of harm using the Medication Event Reporting and Prevention (MERP) Index, ranging from a near miss (A) to patient death (I). Events are noted as present on arrival or in the ED, an act of commission or omission, and are assigned one of four event categories. A second-level physician performs a confirmatory review of all AEs and independently reviews 10% of cases to estimate the false-negative rate. All AEs or potential AEs were reviewed in monthly group calls for consensus on findings. The primary outcome is the proportion of visits in which an AE is identified, overall and by site. Secondary outcomes include categories of events, distribution of harm ratings, and association of AEs with sociodemographic and clinical factors and triggers. We present sociodemographic data and details about AEs and results of logistic regression for associations of AEs with of triggers, sociodemographics, and clinical variables.ResultsWe captured 2594 visits that are representative, within site, of their patient population. Overall, the sample is 64% white, 54% female, and with a mean age of 51. Variability is observed between sites for age, race, and insurance, but not sex. A total of 240 events were identified in 228 visits (8.8%) of which 53.3% were present on arrival, 19.7% were acts of omission, and 44.6% were medication-related, with some variability across sites. A MERP F score (contributing to need for admission, higher level of care, or prolonged hospitalization) was the most common severity level (35.4% of events). Overall, 185 (77.1%) of 240 events involved patient harm (MERP level ≥ E), affecting 175 visits (6.7%). Triggers were present in 951 visits (36.6%). Presence of any trigger was strongly associated with an AE (adjusted odds ratio = 4.6, 95% confidence interval = 3.2-6.6). Ten triggers were individually associated with AEs (adjusted odds ratio = 2.1-7.7). Variability was observed across sites in individual trigger associations, event rates, and categories, but not in severity ratings of events. The overall false-negative rate was 6.1%.ConclusionsThe trigger tool approach was successful in identifying meaningful events. The ED trigger tool seems to be a promising approach for identifying all-cause harm in the ED.

Project description:ObjectiveProlonged Emergency Department (ED) wait times lead to diminished healthcare quality. Utilizing machine learning (ML) to predict patient wait times could aid in ED operational management. Our aim is to perform a comprehensive analysis of ML models for ED wait time prediction, identify key feature importance and associations with prolonged wait times, and interpret prediction model clinical relevance among ED patients.MethodsThis is a single-centered retrospective study. We included ED patients assigned an Emergency Severity Index (ESI) level of 3 at triage. Patient wait times were categorized as <30 minutes and ≥30 minutes (prolonged wait time). We employed five ML algorithms - cross-validation logistic regression (CVLR), random forest (RF), extreme gradient boosting (XGBoost), artificial neural network (ANN), and support vector machine (SVM) - for predicting patient prolonged wait times. Performance assessment utilized accuracy, recall, precision, F1 score, false positive rate (FPR), and false negative rate (FNR). Furthermore, using XGBoost as an example, model key features and partial dependency plots (PDP) of these key features were illustrated. Shapley additive explanations (SHAP) were employed to interpret model outputs. Additionally, a top key feature interaction analysis was conducted.ResultsAmong total 177,665 patients, nearly half of them (48.20%, 85,632) experienced prolonged ED wait times. Though all five ML models exhibited similar performance, minimizing FNR is associated with the most clinical relevance for wait time predictions. The top features influencing patient wait times and gaining the top ranked interactions were ED crowding condition and patient mode of arrival.ConclusionsNearly half of the patients experienced prolonged wait times in the ED. ML models demonstrated acceptable performance, particularly in minimizing FNR when predicting ED wait times. The prediction of prolonged wait times was influenced by multiple interacting factors. Proper application of ML models to clinical practice requires interpreting their predictions of prolonged wait times in the context of clinical significance.

Project description:BackgroundDevelopment of emergency department (ED) triage systems that accurately differentiate and prioritize critically ill from stable patients remains challenging. We used machine learning models to predict clinical outcomes, and then compared their performance with that of a conventional approach-the Emergency Severity Index (ESI).MethodsUsing National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, from 2007 through 2015, we identified all adult patients (aged ≥ 18 years). In the randomly sampled training set (70%), using routinely available triage data as predictors (e.g., demographics, triage vital signs, chief complaints, comorbidities), we developed four machine learning models: Lasso regression, random forest, gradient boosted decision tree, and deep neural network. As the reference model, we constructed a logistic regression model using the five-level ESI data. The clinical outcomes were critical care (admission to intensive care unit or in-hospital death) and hospitalization (direct hospital admission or transfer). In the test set (the remaining 30%), we measured the predictive performance, including area under the receiver-operating-characteristics curve (AUC) and net benefit (decision curves) for each model.ResultsOf 135,470 eligible ED visits, 2.1% had critical care outcome and 16.2% had hospitalization outcome. In the critical care outcome prediction, all four machine learning models outperformed the reference model (e.g., AUC, 0.86 [95%CI 0.85-0.87] in the deep neural network vs 0.74 [95%CI 0.72-0.75] in the reference model), with less under-triaged patients in ESI triage levels 3 to 5 (urgent to non-urgent). Likewise, in the hospitalization outcome prediction, all machine learning models outperformed the reference model (e.g., AUC, 0.82 [95%CI 0.82-0.83] in the deep neural network vs 0.69 [95%CI 0.68-0.69] in the reference model) with less over-triages in ESI triage levels 1 to 3 (immediate to urgent). In the decision curve analysis, all machine learning models consistently achieved a greater net benefit-a larger number of appropriate triages considering a trade-off with over-triages-across the range of clinical thresholds.ConclusionsCompared to the conventional approach, the machine learning models demonstrated a superior performance to predict critical care and hospitalization outcomes. The application of modern machine learning models may enhance clinicians' triage decision making, thereby achieving better clinical care and optimal resource utilization.

Project description:BackgroundApproximately 10% of emergency department (ED) visits among dialysis patients are for conditions that could potentially be managed in outpatient settings, such as hyperkalemia.ObjectiveUsing population-based data, we derived and internally validated a risk score to identify hemodialysis patients at increased risk of hyperkalemia-related ED events.DesignRetrospective cohort study.SettingTen in-center hemodialysis sites in southern Alberta, Canada.PatientsAll maintenance hemodialysis patients (≥18 years) between March 2009 and March 2017.MeasurementsPredictors of hyperkalemia-related ED events included patient demographics, comorbidities, health-system use, laboratory measurements, and dialysis information. The outcome of interest (hyperkalemia-related ED events) was defined by International Classification of Diseases (10th Revision; ICD-10) codes and/or serum potassium [K+] ≥6 mmol/L.MethodsBootstrapped logistic regression was used to derive and internally validate a model of important predictors of hyperkalemia-related ED events. A point system was created based on regression coefficients. Model discrimination was assessed by an optimism-adjusted C-statistic and calibration by deciles of risk and calibration slope.ResultsOf the 1533 maintenance hemodialysis patients in our cohort, 331 (21.6%) presented to the ED with 615 hyperkalemia-related ED events. A 9-point scale for risk of a hyperkalemia-related ED event was created with points assigned to 5 strong predictors based on their regression coefficients: ≥1 laboratory measurement of serum K+ ≥6 mmol/L in the prior 6 months (3 points); ≥1 Hemoglobin A1C [HbA1C] measurement ≥8% in the prior 12 months (1 point); mean ultrafiltration of ≥10 mL/kg/h over the preceding 2 weeks (2 points); ≥25 hours of cumulative time dialyzing over the preceding 2 weeks (1 point); and dialysis vintage of ≥2 years (2 points). Model discrimination (C-statistic: 0.75) and calibration were good.LimitationsMeasures related to health behaviors, social determinants of health, and residual kidney function were not available for inclusion as potential predictors.ConclusionsWhile this tool requires external validation, it may help identify high-risk patients and allow for preventative strategies to avoid unnecessary ED visits and improve patient quality of life.Trial registrationNot applicable-observational study design.

Project description:BackgroundSepsis is a significant health burden on a global scale. Timely identification and treatment of sepsis can greatly improve patient outcomes, including survival rates. However, time-consuming laboratory results are often needed for screening sepsis. We aimed to develop a quick sepsis screening tool (qSepsis) based on patients' non-laboratory clinical data at the emergency department (ED) using machine learning (ML), and compare its performance with established clinical scores.MethodsThis retrospective study included patients admitted to the ED of Zhongnan Hospital of Wuhan University (Wuhan, China) from 1/1/2015 to 5/31/2022. Patients who were under 18 years of age, had cardiopulmonary arrest upon arrival at the ED, or had missing and abnormal medical record data were excluded. The qSepsis was derived by three ML algorithms, including logistic regression (LR), random forest (RF), and extreme gradient boosting (XGB). To benchmark the existing clinical tools for assessing the risk of sepsis in the ED, qSepsis was compared with the Systemic Inflammatory Response Syndrome (SIRS), the Quick Sepsis-Related Organ Failure Assessment (qSOFA), and the Modified Early Warning Score (MEWS). The external validation was performed with the Medical Information Mart for Intensive Care IV ED database (United States), and adopted the same inclusion and exclusion criteria. The predictive power of qSepsis and other clinical scores was measured using the area under the receiver operating characteristic curve (AUROC). The primary outcome of the study was the diagnosis of sepsis in the ED based on the Sepsis 3.0 criteria, which served as the basis for developing the qSepsis tool.FindingsA total of 414,864 patients were finally included in the cohort (median ([IQR]) patient age, 43 (29, 60) years; 202,730 (48.87%) females, 212,134 (51.13%) males), and 200,089 in the external testing cohort (median (SD) patient age, 57 (39, 71) years; 107,427 (53.69%) females, 92,663 (46.31%) males). For internal testing, LR outperformed RF and XGB with an AUROC of 0.862 (95% CI, 0.855-0.869). In external testing, the AUROC decreased to 0.766 (95% CI, 0.758-0.774) for LR, 0.725 (95% CI, 0.717-0.733) for RF, and 0.735 (95% CI, 0.728-0.742) for XGB. In addition, the AUROC for the qSOFA, MEWS, and SIRS scores in external validation cohort were 0.579 (95% CI, 0.563-0.596), 0.600 (95% CI, 0.578-0.622), and 0.704 (95% CI, 0.683-0.725), respectively. The area under the precision-recall curve (AUPRC) for the qSepsis model was 0.213 (95% CI: 0.204-0.222). The AUPRC values for the other scores were as follows: SIRS, 0.071 (95% CI: 0.013-0.099); qSOFA, 0.096 (95% CI: 0.003-0.186); and MEWS, 0.083 (95% CI: 0.063-0.111).InterpretationThis retrospective study demonstrated that qSepsis had better predictive performance in terms of AUROC and area under the precision-recall curve (AUPRC) compared to existing assessment scores. It has the potential to be used in pre-hospital settings with limited access to laboratory tests and in the ED for quick screening of patients with sepsis. However, due to its low positive predictive value (PPV), the false alarms may increase in actual clinical practice.FundingTransformation of Scientific and Technological Achievements Fund Project of Zhongnan Hospital of Wuhan University.

Project description:BackgroundConventional diagnosis of COVID-19 with reverse transcription polymerase chain reaction (RT-PCR) testing (hereafter, PCR) is associated with prolonged time to diagnosis and significant costs to run the test. The SARS-CoV-2 virus might lead to characteristic patterns in the results of widely available, routine blood tests that could be identified with machine learning methodologies. Machine learning modalities integrating findings from these common laboratory test results might accelerate ruling out COVID-19 in emergency department patients.ObjectiveWe sought to develop (ie, train and internally validate with cross-validation techniques) and externally validate a machine learning model to rule out COVID 19 using only routine blood tests among adults in emergency departments.MethodsUsing clinical data from emergency departments (EDs) from 66 US hospitals before the pandemic (before the end of December 2019) or during the pandemic (March-July 2020), we included patients aged ≥20 years in the study time frame. We excluded those with missing laboratory results. Model training used 2183 PCR-confirmed cases from 43 hospitals during the pandemic; negative controls were 10,000 prepandemic patients from the same hospitals. External validation used 23 hospitals with 1020 PCR-confirmed cases and 171,734 prepandemic negative controls. The main outcome was COVID 19 status predicted using same-day routine laboratory results. Model performance was assessed with area under the receiver operating characteristic (AUROC) curve as well as sensitivity, specificity, and negative predictive value (NPV).ResultsOf 192,779 patients included in the training, external validation, and sensitivity data sets (median age decile 50 [IQR 30-60] years, 40.5% male [78,249/192,779]), AUROC for training and external validation was 0.91 (95% CI 0.90-0.92). Using a risk score cutoff of 1.0 (out of 100) in the external validation data set, the model achieved sensitivity of 95.9% and specificity of 41.7%; with a cutoff of 2.0, sensitivity was 92.6% and specificity was 59.9%. At the cutoff of 2.0, the NPVs at a prevalence of 1%, 10%, and 20% were 99.9%, 98.6%, and 97%, respectively.ConclusionsA machine learning model developed with multicenter clinical data integrating commonly collected ED laboratory data demonstrated high rule-out accuracy for COVID-19 status, and might inform selective use of PCR-based testing.

Dataset Information

Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool.

Background

Methods

Findings

Interpretation

Funding

Publications

Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets