Dataset Information

Development and internal validation of machine learning-based models and external validation of existing risk scores for outcome prediction in patients with ischaemic stroke.

ABSTRACT:

Aims

We developed new machine learning (ML) models and externally validated existing statistical models [ischaemic stroke predictive risk score (iScore) and totalled health risks in vascular events (THRIVE) scores] for predicting the composite of recurrent stroke or all-cause mortality at 90 days and at 3 years after hospitalization for first acute ischaemic stroke (AIS).

Methods and results

In adults hospitalized with AIS from January 2005 to November 2016, with follow-up until November 2019, we developed three ML models [random forest (RF), support vector machine (SVM), and extreme gradient boosting (XGBOOST)] and externally validated the iScore and THRIVE scores for predicting the composite outcomes after AIS hospitalization, using data from 721 patients and 90 potential predictor variables. At 90 days and 3 years, 11 and 34% of patients, respectively, reached the composite outcome. For the 90-day prediction, the area under the receiver operating characteristic curve (AUC) was 0.779 for RF, 0.771 for SVM, 0.772 for XGBOOST, 0.720 for iScore, and 0.664 for THRIVE. For 3-year prediction, the AUC was 0.743 for RF, 0.777 for SVM, 0.773 for XGBOOST, 0.710 for iScore, and 0.675 for THRIVE.

Conclusion

The study provided three ML-based predictive models that achieved good discrimination and clinical usefulness in outcome prediction after AIS and broadened the application of the iScore and THRIVE scoring system for long-term outcome prediction. Our findings warrant comparative analyses of ML and existing statistical method-based risk prediction tools for outcome prediction after AIS in new data sets.

SUBMITTER: Axford D

PROVIDER: S-EPMC10944684 | biostudies-literature | 2024 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and internal validation of machine learning-based models and external validation of existing risk scores for outcome prediction in patients with ischaemic stroke.

Axford Daniel D Sohel Ferdous F Abedi Vida V Zhu Ye Y Zand Ramin R Barkoudah Ebrahim E Krupica Troy T Iheasirim Kingsley K Sharma Umesh M UM Dugani Sagar B SB Takahashi Paul Y PY Bhagra Sumit S Murad Mohammad H MH Saposnik Gustavo G Yousufuddin Mohammed M

European heart journal. Digital health 20231122 2

<h4>Aims</h4>We developed new machine learning (ML) models and externally validated existing statistical models [ischaemic stroke predictive risk score (iScore) and totalled health risks in vascular events (THRIVE) scores] for predicting the composite of recurrent stroke or all-cause mortality at 90 days and at 3 years after hospitalization for first acute ischaemic stroke (AIS).<h4>Methods and results</h4>In adults hospitalized with AIS from January 2005 to November 2016, with follow-up until N ...[more]

PMID: 38505491

Similar Datasets

Project description:Background and purposePrediction models for outcome of patients with acute ischemic stroke who will undergo endovascular treatment have been developed to improve patient management. The aim of the current study is to provide an overview of preintervention models for functional outcome after endovascular treatment and to validate these models with data from daily clinical practice.MethodsWe systematically searched within Medline, Embase, Cochrane, Web of Science, to include prediction models. Models identified from the search were validated in the MR CLEAN (Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands) registry, which includes all patients treated with endovascular treatment within 6.5 hours after stroke onset in the Netherlands between March 2014 and November 2017. Predictive performance was evaluated according to discrimination (area under the curve) and calibration (slope and intercept of the calibration curve). Good functional outcome was defined as a score of 0-2 or 0-3 on the modified Rankin Scale depending on the model.ResultsAfter screening 3468 publications, 19 models were included in this validation. Variables included in the models mainly addressed clinical and imaging characteristics at baseline. In the validation cohort of 3156 patients, discriminative performance ranged from 0.61 (SPAN-100 [Stroke Prognostication Using Age and NIH Stroke Scale]) to 0.80 (MR PREDICTS). Best-calibrated models were THRIVE (The Totaled Health Risks in Vascular Events; intercept -0.06 [95% CI, -0.14 to 0.02]; slope 0.84 [95% CI, 0.75-0.95]), THRIVE-c (intercept 0.08 [95% CI, -0.02 to 0.17]; slope 0.71 [95% CI, 0.65-0.77]), Stroke Checkerboard score (intercept -0.05 [95% CI, -0.13 to 0.03]; slope 0.97 [95% CI, 0.88-1.08]), and MR PREDICTS (intercept 0.43 [95% CI, 0.33-0.52]; slope 0.93 [95% CI, 0.85-1.01]).ConclusionsThe THRIVE-c score and MR PREDICTS both showed a good combination of discrimination and calibration and were, therefore, superior in predicting functional outcome for patients with ischemic stroke after endovascular treatment within 6.5 hours. Since models used different predictors and several models had relatively good predictive performance, the decision on which model to use in practice may also depend on simplicity of the model, data availability, and the comparability of the population and setting.

Project description:IntroductionThe Early Prediction of Functional Outcome after Stroke (EPOS) model for independent gait is a tool to predict between days 2 and 9 poststroke whether patients will regain independent gait 6 months after stroke. External validation of the model is important to determine its clinical applicability and generalizability by testing its performance in an independent cohort. Therefore, this study aimed to perform a temporal and geographical external validation of the EPOS prediction model for independent gait after stroke but with the endpoint being 3 months instead of the original 6 months poststroke.MethodsTwo prospective longitudinal cohort studies consisting of patients with first-ever stroke admitted to a Swiss hospital stroke unit. Sitting balance and strength of the paretic leg were tested at days 1 and 8 post-stroke in Cohort I and at days 3 and 9 in Cohort II. Independent gait was assessed 3 months after symptom onset. The performance of the model in terms of discrimination (area under the receiver operator characteristic (ROC) curve; AUC), classification, and calibration was assessed.ResultsIn Cohort I [N = 39, median age: 74 years, 33% women, median National Institutes of Health Stroke Scale (NIHSS) 9], the AUC (95% confidence interval (CI)] was 0.675 (0.510, 0.841) on day 1 and 0.921 (0.811, 1.000) on day 8. For Cohort II (N = 78, median age: 69 years, 37% women, median NIHSS 8), this was 0.801 (0.684, 0.918) on day 3 and 0.846 (0.741, 0.951) on day 9.Discussion and conclusionExternal validation of the EPOS prediction model for independent gait 3 months after stroke resulted in an acceptable performance from day 3 onward in mild-to-moderately affected patients with first-ever stroke without severe prestroke disability. The impact of applying this model in clinical practice should be investigated within this subgroup of patients with stroke. To improve the generalizability of patients with recurrent stroke and those with more severe, neurological comorbidities, the performance of the EPOS model within these patients should be determined across different geographical areas.

Project description:BackgroundMany dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records.MethodsWe identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters' Dementia Risk Score, Mehta's RxDx-Dementia Risk Index, and Nori's ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models.ResultsWe reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67-0.76 c-statistic). The Mehta model (development c-statistic: 0.81) transported well to some of the external databases (0.69-0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62-0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard.ConclusionWe observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.

Project description:BackgroundTwo clinical scoring systems, the status epilepticus severity score (STESS) and the epidemiology-based mortality score in status epilepticus (EMSE), are used to predict mortality in patients with status epilepticus (SE). The aim of this study was to compare the outcome-prediction function of the two scoring systems regarding in-hospital mortality using a multicenter large cohort of adult patients with SE. Moreover, we studied the potential role of these two scoring systems in predicting the functional outcome in patients with SE.MethodsThe SE cohort consisted of patients from the epilepsy centers of eight academic tertiary medical centers in South Korea. The clinical and electroencephalography data for all adult patients with SE from January 2013 to December 2014 were derived from a prospective SE database. The primary outcome variable was defined as in-hospital death. The secondary outcome variable was defined as a poor functional outcome, i.e., a score of 1-3 on the Glasgow Outcome Scale, at discharge.ResultsAmong the 120 non-hypoxic patients with SE recruited into the study, 16 (13.3%) died in the hospital and 64 (53.3%) were discharged with a poor functional outcome. The receiver-operating characteristic (ROC) curve for prediction of in-hospital death based on the STESS had an area under the curve of 0.673 with an optimal cutoff value for discrimination (best match for both sensitivity (0.56) and specificity (0.70)) that was ≥ 4 points. The two combinations of elements of the EMSE system (EMSE-ALDEg and EMSE-ECLEg) predicted not only in-hospital mortality with the best match for sensitivity (more than 0.6) and specificity (more than 0.6), but also a poor functional outcome with the best match for both sensitivity (>0.7) and specificity (>0.6). STESS did not predict a poor functional outcome (area under the ROC, 0.581; P = 0.23).ConclusionAlthough the EMSE is a clinical scoring system that focuses on individual mortality, we did not find differences between the EMSE and STESS in the prediction of in-hospital death. The EMSE was useful in predicting poor functional outcome, as it was significantly better than STESS.

Project description:AimsThe increasing prevalence of ischaemic stroke (IS) can partly be explained by the likewise growing number of patients with chronic kidney disease (CKD). Risk scores have been developed to identify high-risk patients, allowing for personalized anticoagulation therapy. However, predictive performance in CKD is unclear. The aim of this study is to validate six commonly used risk scores for IS in atrial fibrillation (AF) patients across the spectrum of kidney function.Methods and resultsOverall, 36 004 subjects with newly diagnosed AF from SCREAM (Stockholm CREAtinine Measurements), a healthcare utilization cohort of Stockholm residents, were included. Predictive performance of the AFI, CHADS2, Modified CHADS2, CHA2DS2-VASc, ATRIA, and GARFIELD-AF risk scores was evaluated across three strata of kidney function: normal kidney function [estimated glomerular filtration rate (eGFR) >60 mL/min/1.73 m2], mild CKD (eGFR 30-60 mL/min/1.73 m2), and advanced CKD (eGFR <30 mL/min/1.73 m2). Predictive performance was assessed by discrimination and calibration. During 1.9 years, 3069 (8.5%) patients suffered an IS. Discrimination was dependent on eGFR: the median c-statistic in normal eGFR was 0.75 (range 0.68-0.78), but decreased to 0.68 (0.58-0.73) and 0.68 (0.55-0.74) for mild and advanced CKD, respectively. Calibration was reasonable and largely independent of eGFR. The Modified CHADS2 score showed good performance across kidney function strata, both for discrimination [c-statistic: 0.78 (95% confidence interval 0.77-0.79), 0.73 (0.71-0.74) and 0.74 (0.69-0.79), respectively] and calibration.ConclusionIn the most clinically relevant stages of CKD, predictive performance of the majority of risk scores was poor, increasing the risk of misclassification and thus of over- or undertreatment. The Modified CHADS2 score performed good and consistently across all kidney function strata, and should therefore be preferred for risk estimation in AF patients.

Project description:Background and purposeMechanical thrombectomy greatly improves stroke outcomes. Nonetheless, some patients fall short of full recovery despite good reperfusion. The purpose of this study was to develop machine learning (ML) models for the pre-interventional prediction of functional outcome at 3 months of thrombectomy in acute ischemic stroke (AIS), using clinical and auto-extractable radiological information consistently available upon first emergency evaluation.Materials and methodsA two-center retrospective cohort of 293 patients with AIS who underwent thrombectomy was analyzed. ML models were developed to predict dichotomized modified Rankin score at 90 days (mRS-90) using clinical and imaging features, both separately and combined. Conventional and experimental imaging biomarkers were quantified using automated image-processing software from non-contract computed tomography (CT) and computed tomography angiography (CTA). Shapley Additive Explanation (SHAP) was applied for model interpretability and predictor importance analysis of the optimal model.ResultsMerging clinical and imaging features returned the best results for mRS-90 prediction. The best performing classifier was Extreme Gradient Boosting (XGB) with an area under the receiver operating characteristic curve (AUC) = 84% using selected features. The most important classifying features were age, baseline National Institutes of Health Stroke Scale (NIHSS), occlusion side, degree of brain atrophy [primarily represented by cortical cerebrospinal fluid (CSF) volume and lateral ventricle volume], early ischemic core [primarily represented by e-Alberta Stroke Program Early CT Score (ASPECTS)], and collateral circulation deficit volume on CTA.ConclusionMachine learning that is applied to quantifiable image features from CT and CTA alongside basic clinical characteristics constitutes a promising automated method in the pre-interventional prediction of stroke prognosis. Interpretable models allow for exploring which initial features contribute the most to post-thrombectomy outcome prediction overall and for each individual patient outcome.

Project description:ImportanceAcute kidney injury (AKI) is associated with increased morbidity and mortality in hospitalized patients. Current methods to identify patients at high risk of AKI are limited, and few prediction models have been externally validated.ObjectiveTo internally and externally validate a machine learning risk score to detect AKI in hospitalized patients.Design, setting, and participantsThis diagnostic study included 495 971 adult hospital admissions at the University of Chicago (UC) from 2008 to 2016 (n = 48 463), at Loyola University Medical Center (LUMC) from 2007 to 2017 (n = 200 613), and at NorthShore University Health System (NUS) from 2006 to 2016 (n = 246 895) with serum creatinine (SCr) measurements. Patients with an SCr concentration at admission greater than 3.0 mg/dL, with a prior diagnostic code for chronic kidney disease stage 4 or higher, or who received kidney replacement therapy within 48 hours of admission were excluded. A simplified version of a previously published gradient boosted machine AKI prediction algorithm was used; it was validated internally among patients at UC and externally among patients at NUS and LUMC.Main outcomes and measuresPrediction of Kidney Disease Improving Global Outcomes SCr-defined stage 2 AKI within a 48-hour interval was the primary outcome. Discrimination was assessed by the area under the receiver operating characteristic curve (AUC).ResultsThe study included 495 971 adult admissions (mean [SD] age, 63 [18] years; 87 689 [17.7%] African American; and 266 866 [53.8%] women) across 3 health systems. The development of stage 2 or higher AKI occurred in 15 664 of 48 463 patients (3.4%) in the UC cohort, 5711 of 200 613 (2.8%) in the LUMC cohort, and 3499 of 246 895 (1.4%) in the NUS cohort. In the UC cohort, 332 patients (0.7%) required kidney replacement therapy compared with 672 patients (0.3%) in the LUMC cohort and 440 patients (0.2%) in the NUS cohort. The AUCs for predicting at least stage 2 AKI in the next 48 hours were 0.86 (95% CI, 0.86-0.86) in the UC cohort, 0.85 (95% CI, 0.84-0.85) in the LUMC cohort, and 0.86 (95% CI, 0.86-0.86) in the NUS cohort. The AUCs for receipt of kidney replacement therapy within 48 hours were 0.96 (95% CI, 0.96-0.96) in the UC cohort, 0.95 (95% CI, 0.94-0.95) in the LUMC cohort, and 0.95 (95% CI, 0.94-0.95) in the NUS cohort. In time-to-event analysis, a probability cutoff of at least 0.057 predicted the onset of stage 2 AKI a median (IQR) of 27 (6.5-93) hours before the eventual doubling in SCr concentrations in the UC cohort, 34.5 (19-85) hours in the NUS cohort, and 39 (19-108) hours in the LUMC cohort.Conclusions and relevanceIn this study, the machine learning algorithm demonstrated excellent discrimination in both internal and external validation, supporting its generalizability and potential as a clinical decision support tool to improve AKI detection and outcomes.

Dataset Information

Development and internal validation of machine learning-based models and external validation of existing risk scores for outcome prediction in patients with ischaemic stroke.

Aims

Methods and results

Conclusion

Publications

Development and internal validation of machine learning-based models and external validation of existing risk scores for outcome prediction in patients with ischaemic stroke.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets