Benchmarking emergency department prediction models with machine learning and public electronic health records.
Ontology highlight
ABSTRACT:
SUBMITTER: Xie F
PROVIDER: S-EPMC9610299 | biostudies-literature | 2022 Oct
REPOSITORIES: biostudies-literature
Publications
Benchmarking emergency department prediction models with machine learning and public electronic health records.
Xie Feng F Zhou Jun J Lee Jin Wee JW Tan Mingrui M Li Siqi S Rajnthern Logasan S/O LS Chee Marcel Lucas ML Chakraborty Bibhas B Wong An-Kwok Ian AI Dagan Alon A Ong Marcus Eng Hock MEH Gao Fei F Liu Nan N
Scientific data 20221027 1
The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision supp ...[more]
Similar Datasets
Project description:ObjectiveProlonged Emergency Department (ED) wait times lead to diminished healthcare quality. Utilizing machine learning (ML) to predict patient wait times could aid in ED operational management. Our aim is to perform a comprehensive analysis of ML models for ED wait time prediction, identify key feature importance and associations with prolonged wait times, and interpret prediction model clinical relevance among ED patients.MethodsThis is a single-centered retrospective study. We included ED patients assigned an Emergency Severity Index (ESI) level of 3 at triage. Patient wait times were categorized as <30 minutes and ≥30 minutes (prolonged wait time). We employed five ML algorithms - cross-validation logistic regression (CVLR), random forest (RF), extreme gradient boosting (XGBoost), artificial neural network (ANN), and support vector machine (SVM) - for predicting patient prolonged wait times. Performance assessment utilized accuracy, recall, precision, F1 score, false positive rate (FPR), and false negative rate (FNR). Furthermore, using XGBoost as an example, model key features and partial dependency plots (PDP) of these key features were illustrated. Shapley additive explanations (SHAP) were employed to interpret model outputs. Additionally, a top key feature interaction analysis was conducted.ResultsAmong total 177,665 patients, nearly half of them (48.20%, 85,632) experienced prolonged ED wait times. Though all five ML models exhibited similar performance, minimizing FNR is associated with the most clinical relevance for wait time predictions. The top features influencing patient wait times and gaining the top ranked interactions were ED crowding condition and patient mode of arrival.ConclusionsNearly half of the patients experienced prolonged wait times in the ED. ML models demonstrated acceptable performance, particularly in minimizing FNR when predicting ED wait times. The prediction of prolonged wait times was influenced by multiple interacting factors. Proper application of ML models to clinical practice requires interpreting their predictions of prolonged wait times in the context of clinical significance.
| S-EPMC11917090 | biostudies-literature
Project description:BackgroundDevelopment of emergency department (ED) triage systems that accurately differentiate and prioritize critically ill from stable patients remains challenging. We used machine learning models to predict clinical outcomes, and then compared their performance with that of a conventional approach-the Emergency Severity Index (ESI).MethodsUsing National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, from 2007 through 2015, we identified all adult patients (aged ≥ 18 years). In the randomly sampled training set (70%), using routinely available triage data as predictors (e.g., demographics, triage vital signs, chief complaints, comorbidities), we developed four machine learning models: Lasso regression, random forest, gradient boosted decision tree, and deep neural network. As the reference model, we constructed a logistic regression model using the five-level ESI data. The clinical outcomes were critical care (admission to intensive care unit or in-hospital death) and hospitalization (direct hospital admission or transfer). In the test set (the remaining 30%), we measured the predictive performance, including area under the receiver-operating-characteristics curve (AUC) and net benefit (decision curves) for each model.ResultsOf 135,470 eligible ED visits, 2.1% had critical care outcome and 16.2% had hospitalization outcome. In the critical care outcome prediction, all four machine learning models outperformed the reference model (e.g., AUC, 0.86 [95%CI 0.85-0.87] in the deep neural network vs 0.74 [95%CI 0.72-0.75] in the reference model), with less under-triaged patients in ESI triage levels 3 to 5 (urgent to non-urgent). Likewise, in the hospitalization outcome prediction, all machine learning models outperformed the reference model (e.g., AUC, 0.82 [95%CI 0.82-0.83] in the deep neural network vs 0.69 [95%CI 0.68-0.69] in the reference model) with less over-triages in ESI triage levels 1 to 3 (immediate to urgent). In the decision curve analysis, all machine learning models consistently achieved a greater net benefit-a larger number of appropriate triages considering a trade-off with over-triages-across the range of clinical thresholds.ConclusionsCompared to the conventional approach, the machine learning models demonstrated a superior performance to predict critical care and hospitalization outcomes. The application of modern machine learning models may enhance clinicians' triage decision making, thereby achieving better clinical care and optimal resource utilization.
| S-EPMC6387562 | biostudies-literature
Project description:ObjectiveHospital emergency department (ED) visits by asthmatics differ based on race and season. The objectives of this study were to investigate season- and race-specific disparities for asthma risk, and to identify environmental exposure variables associated with ED visits among more than 42,000 individuals of African American (AA) and European American (EA) descent identified through electronic health records (EHRs).MethodsWe examined data from 42,375 individuals (AAs = 14,491, EAs = 27,884) identified in EHRs. We considered associated demographic (race, age, gender, insurance), clinical (smoking status, ED visits, FEV1%), and environmental exposures data (mold, pollen, and pollutants). Machine learning techniques, including random forest (RF), extreme gradient boosting (XGB), and decision tree (DT) were used to build and identify race- and -season-specific predictive models for asthma ED visits.ResultsSignificant differences in ED visits and FEV1% among AAs and EAs were identified. ED visits by AAs was 32.0% higher than EAs and AAs had 6.4% lower FEV1% value than EAs. XGB model was used to accurately classify asthma patients visiting ED into AAs and EAs. Pollen factor and pollution (PM2.5, PM10) were the key variables for asthma in AAs and EAs, respectively. Age and cigarette smoking increase asthma risk independent of seasons.ConclusionsIn this study, we observed racial and season-specific disparities between AAs and EAs asthmatics for ED visit and FEV1% severity, suggesting the need to address asthma disparities through key predictors including socio-economic status, particulate matter, and mold.
| S-EPMC8221365 | biostudies-literature
Project description:The usefulness of genomic prediction in crop and livestock breeding programs has prompted efforts to develop new and improved genomic prediction algorithms, such as artificial neural networks and gradient tree boosting. However, the performance of these algorithms has not been compared in a systematic manner using a wide range of datasets and models. Using data of 18 traits across six plant species with different marker densities and training population sizes, we compared the performance of six linear and six non-linear algorithms. First, we found that hyperparameter selection was necessary for all non-linear algorithms and that feature selection prior to model training was critical for artificial neural networks when the markers greatly outnumbered the number of training lines. Across all species and trait combinations, no one algorithm performed best, however predictions based on a combination of results from multiple algorithms (i.e., ensemble predictions) performed consistently well. While linear and non-linear algorithms performed best for a similar number of traits, the performance of non-linear algorithms vary more between traits. Although artificial neural networks did not perform best for any trait, we identified strategies (i.e., feature selection, seeded starting weights) that boosted their performance to near the level of other algorithms. Our results highlight the importance of algorithm selection for the prediction of trait values.
| S-EPMC6829122 | biostudies-literature
Project description:BackgroundScreening carotid B-mode ultrasonography is a frequently used method to detect subjects with carotid atherosclerosis (CAS). Due to the asymptomatic progression of most CAS patients, early identification is challenging for clinicians, and it may trigger ischemic stroke. Recently, machine learning has shown a strong ability to classify data and a potential for prediction in the medical field. The combined use of machine learning and the electronic health records of patients could provide clinicians with a more convenient and precise method to identify asymptomatic CAS.MethodsRetrospective cohort study using routine clinical data of medical check-up subjects from April 19, 2010 to November 15, 2019. Six machine learning models (logistic regression [LR], random forest [RF], decision tree [DT], eXtreme Gradient Boosting [XGB], Gaussian Naïve Bayes [GNB], and K-Nearest Neighbour [KNN]) were used to predict asymptomatic CAS and compared their predictability in terms of the area under the receiver operating characteristic curve (AUCROC), accuracy (ACC), and F1 score (F1).ResultsOf the 18,441 subjects, 6553 were diagnosed with asymptomatic CAS. Compared to DT (AUCROC 0.628, ACC 65.4%, and F1 52.5%), the other five models improved prediction: KNN + 7.6% (0.704, 68.8%, and 50.9%, respectively), GNB + 12.5% (0.753, 67.0%, and 46.8%, respectively), XGB + 16.0% (0.788, 73.4%, and 55.7%, respectively), RF + 16.6% (0.794, 74.5%, and 56.8%, respectively) and LR + 18.1% (0.809, 74.7%, and 59.9%, respectively). The highest achieving model, LR predicted 1045/1966 cases (sensitivity 53.2%) and 3088/3566 non-cases (specificity 86.6%). A tenfold cross-validation scheme further verified the predictive ability of the LR.ConclusionsAmong machine learning models, LR showed optimal performance in predicting asymptomatic CAS. Our findings set the stage for an early automatic alarming system, allowing a more precise allocation of CAS prevention measures to individuals probably to benefit most.
| S-EPMC8020544 | biostudies-literature
Project description:BackgroundLate-Onset Alzheimer's Disease (LOAD) is a leading form of dementia. There is no effective cure for LOAD, leaving the treatment efforts to depend on preventive cognitive therapies, which stand to benefit from the timely estimation of the risk of developing the disease. Fortunately, a growing number of Machine Learning methods that are well positioned to address this challenge are becoming available.ResultsWe conducted systematic comparisons of representative Machine Learning models for predicting LOAD from genetic variation data provided by the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort. Our experimental results demonstrate that the classification performance of the best models tested yielded ∼72% of area under the ROC curve.ConclusionsMachine learning models are promising alternatives for estimating the genetic risk of LOAD. Systematic machine learning model selection also provides the opportunity to identify new genetic markers potentially associated with the disease.
| S-EPMC6915925 | biostudies-literature
Project description:Cardiovascular disease (CVD) remains a major cause of mortality in the UK, prompting the need for improved risk predictive models for primary prevention. Machine learning (ML) models utilizing electronic health records (EHRs) offer potential enhancements over traditional risk scores like QRISK3 and ASCVD. To systematically evaluate and compare the efficacy of ML models against conventional CVD risk prediction algorithms using EHR data for medium to long-term (5-10 years) CVD risk prediction. A systematic review and random-effect meta-analysis were conducted according to preferred reporting items for systematic reviews and meta-analyses guidelines, assessing studies from 2010 to 2024. We retrieved 32 ML models and 26 conventional statistical models from 20 selected studies, focusing on performance metrics such as area under the curve (AUC) and heterogeneity across models. ML models, particularly random forest and deep learning, demonstrated superior performance, with the highest recorded pooled AUCs of 0.865 (95% CI: 0.812-0.917) and 0.847 (95% CI: 0.766-0.927), respectively. These significantly outperformed the conventional risk score of 0.765 (95% CI: 0.734-0.796). However, significant heterogeneity (I² > 99%) and potential publication bias were noted across the studies. While ML models show enhanced calibration for CVD risk, substantial variability and methodological concerns limit their current clinical applicability. Future research should address these issues by enhancing methodological transparency and standardization to improve the reliability and utility of these models in clinical settings. This study highlights the advanced capabilities of ML models in CVD risk prediction and emphasizes the need for rigorous validation to facilitate their integration into clinical practice.
| S-EPMC11750195 | biostudies-literature
Project description:Urinary tract infections (UTIs) are a major cause of emergency hospital admissions, but it remains challenging to diagnose them reliably. Application of machine learning (ML) to routine patient data could support clinical decision-making. We developed a ML model predicting bacteriuria in the ED and evaluated its performance in key patient groups to determine scope for its future use to improve UTI diagnosis and thus guide antibiotic prescribing decisions in clinical practice. We used retrospective electronic health records from a large UK hospital (2011-2019). Non-pregnant adults who attended the ED and had a urine sample cultured were eligible for inclusion. The primary outcome was predominant bacterial growth ≥104 cfu/mL in urine. Predictors included demography, medical history, ED diagnoses, blood tests, and urine flow cytometry. Linear and tree-based models were trained via repeated cross-validation, re-calibrated, and validated on data from 2018/19. Changes in performance were investigated by age, sex, ethnicity, and suspected ED diagnosis, and compared to clinical judgement. Among 12,680 included samples, 4,677 (36.9%) showed bacterial growth. Relying primarily on flow cytometry parameters, our best model achieved an area under the ROC curve (AUC) of 0.813 (95% CI 0.792-0.834) in the test data, and achieved both higher sensitivity and specificity compared to proxies of clinician's judgement. Performance remained stable for white and non-white patients but was lower during a period of laboratory procedure change in 2015, in patients ≥65 years (AUC 0.783, 95% CI 0.752-0.815), and in men (AUC 0.758, 95% CI 0.717-0.798). Performance was also slightly reduced in patients with recorded suspicion of UTI (AUC 0.797, 95% CI 0.765-0.828). Our results suggest scope for use of ML to inform antibiotic prescribing decisions by improving diagnosis of suspected UTI in the ED, but performance varied with patient characteristics. Clinical utility of predictive models for UTI is therefore likely to differ for important patient subgroups including women <65 years, women ≥65 years, and men. Tailored models and decision thresholds may be required that account for differences in achievable performance, background incidence, and risks of infectious complications in these groups.
| S-EPMC10263340 | biostudies-literature
Project description:ObjectivesTriage is an essential emergency department (ED) process designed to provide timely management depending on acuity and severity; however, the process may be inconsistent with clinical and hospitalization outcomes. Therefore, studies have attempted to augment this process with machine learning models, showing advantages in predicting critical conditions and hospitalization outcomes. The aim of this study was to utilize nationwide registry data to develop a machine learning-based classification model to predict the clinical course of pediatric ED visits.MethodsThis cross-sectional observational study used data from the National Emergency Department Information System on emergency visits of children under 15 years of age from January 1, 2016, to December 31, 2017. The primary and secondary outcomes were to identify critically ill children and predict hospitalization from triage data, respectively. We developed and tested a random forest model with the under sampled dataset and validated the model using the entire dataset. We compared the model's performance with that of the conventional triage system.ResultsA total of 2,621,710 children were eligible for the analysis and included 12,951 (0.5%) critical outcomes and 303,808 (11.6%) hospitalizations. After validation, the area under the receiver operating characteristic curve was 0.991 (95% confidence interval [CI] 0.991-0.992) for critical outcomes and 0.943 (95% CI 0.943-0.944) for hospitalization, which were higher than those of the conventional triage system.ConclusionsThe machine learning-based model using structured triage data from a nationwide database can effectively predict critical illness and hospitalizations among children visiting the ED.
| S-EPMC8853514 | biostudies-literature
Project description:ImportanceWhile machine learning approaches may enhance prediction ability, little is known about their utility in emergency department (ED) triage.ObjectivesTo examine the performance of machine learning approaches to predict clinical outcomes and disposition in children in the ED and to compare their performance with conventional triage approaches.Design, setting, and participantsPrognostic study of ED data from the National Hospital Ambulatory Medical Care Survey from January 1, 2007, through December 31, 2015. A nationally representative sample of 52 037 children aged 18 years or younger who presented to the ED were included. Data analysis was performed in August 2018.Main outcomes and measuresThe outcomes were critical care (admission to an intensive care unit and/or in-hospital death) and hospitalization (direct hospital admission or transfer). In the training set (70% random sample), using routinely available triage data as predictors (eg, demographic characteristics and vital signs), we derived 4 machine learning-based models: lasso regression, random forest, gradient-boosted decision tree, and deep neural network. In the test set (the remaining 30% of the sample), we measured the models' prediction performance by computing C statistics, prospective prediction results, and decision curves. These machine learning models were built for each outcome and compared with the reference model using the conventional triage classification information.ResultsOf 52 037 eligible ED visits by children (median [interquartile range] age, 6 [2-14] years; 24 929 [48.0%] female), 163 (0.3%) had the critical care outcome and 2352 (4.5%) had the hospitalization outcome. For the critical care prediction, all machine learning approaches had higher discriminative ability compared with the reference model, although the difference was not statistically significant (eg, C statistics of 0.85 [95% CI, 0.78-0.92] for the deep neural network vs 0.78 [95% CI, 0.71-0.85] for the reference; P = .16), and lower number of undertriaged critically ill children in the conventional triage levels 3 to 5 (urgent to nonurgent). For the hospitalization prediction, all machine learning approaches had significantly higher discrimination ability (eg, C statistic, 0.80 [95% CI, 0.78-0.81] for the deep neural network vs 0.73 [95% CI, 0.71-0.75] for the reference; P < .001) and fewer overtriaged children who did not require inpatient management in the conventional triage levels 1 to 3 (immediate to urgent). The decision curve analysis demonstrated a greater net benefit of machine learning models over ranges of clinical thresholds.Conclusions and relevanceMachine learning-based triage had better discrimination ability to predict clinical outcomes and disposition, with reduction in undertriaging critically ill children and overtriaging children who are less ill.
| S-EPMC6484561 | biostudies-literature