Dataset Information

Validity of prognostic models of critical COVID-19 is variable. A systematic review with external validation.

ABSTRACT:

Objectives

To identify prognostic models which estimate the risk of critical COVID-19 in hospitalized patients and to assess their validation properties.

Study design and setting

We conducted a systematic review in Medline (up to January 2021) of studies developing or updating a model that estimated the risk of critical COVID-19, defined as death, admission to intensive care unit, and/or use of mechanical ventilation during admission. Models were validated in two datasets with different backgrounds (HM [private Spanish hospital network], n = 1,753, and ICS [public Catalan health system], n = 1,104), by assessing discrimination (area under the curve [AUC]) and calibration (plots).

Results

We validated 18 prognostic models. Discrimination was good in nine of them (AUCs ≥ 80%) and higher in those predicting mortality (AUCs 65%-87%) than those predicting intensive care unit admission or a composite outcome (AUCs 53%-78%). Calibration was poor in all models providing outcome's probabilities and good in four models providing a point-based score. These four models used mortality as outcome and included age, oxygen saturation, and C-reactive protein among their predictors.

Conclusion

The validity of models predicting critical COVID-19 by using only routinely collected predictors is variable. Four models showed good discrimination and calibration when externally validated and are recommended for their use.

SUBMITTER: Cardenas-Fuentes G

PROVIDER: S-EPMC10152752 | biostudies-literature | 2023 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Validity of prognostic models of critical COVID-19 is variable. A systematic review with external validation.

Cárdenas-Fuentes Gabriela G Bosch de Basea Magda M Cobo Inés I Subirana Isaac I Ceresa Mario M Famada Ernest E Gimeno-Santos Elena E Delgado-Ortiz Laura L Faner Rosa R Molina-Molina María M Agustí Àlvar À Muñoz Xavier X Sibila Oriol O Gea Joaquim J Garcia-Aymerich Judith J

Journal of clinical epidemiology 20230502

<h4>Objectives</h4>To identify prognostic models which estimate the risk of critical COVID-19 in hospitalized patients and to assess their validation properties.<h4>Study design and setting</h4>We conducted a systematic review in Medline (up to January 2021) of studies developing or updating a model that estimated the risk of critical COVID-19, defined as death, admission to intensive care unit, and/or use of mechanical ventilation during admission. Models were validated in two datasets with dif ...[more]

PMID: 37142168

Similar Datasets

Project description:Objectives backgroundTo externally validate clinical prediction models that aim to predict progression to invasive ventilation or death on the ICU in patients admitted with confirmed COVID-19 pneumonitis.DesignSingle-center retrospective external validation study.Data sourcesRoutinely collected healthcare data in the ICU electronic patient record. Curated data recorded for each ICU admission for the purposes of the U.K. Intensive Care National Audit and Research Centre (ICNARC).SettingThe ICU at Manchester Royal Infirmary, Manchester, United Kingdom.PatientsThree hundred forty-nine patients admitted to ICU with confirmed COVID-19 Pneumonitis, older than 18 years, from March 1, 2020, to February 28, 2022. Three hundred two met the inclusion criteria for at least one model. Fifty-five of the 349 patients were admitted before the widespread adoption of dexamethasone for the treatment of severe COVID-19 (pre-dexamethasone patients).OutcomesAbility to be externally validated, discriminate, and calibrate.MethodsArticles meeting the inclusion criteria were identified, and those that gave sufficient details on predictors used and methods to generate predictions were tested in our cohort of patients, which matched the original publications' inclusion/exclusion criteria and endpoint.ResultsThirteen clinical prediction articles were identified. There was insufficient information available to validate models in five of the articles; a further three contained predictors that were not routinely measured in our ICU cohort and were not validated; three had performance that was substantially lower than previously published (range C-statistic = 0.483-0.605 in pre-dexamethasone patients and C = 0.494-0.564 among all patients). One model retained its discriminative ability in our cohort compared with previously published results (C = 0.672 and 0.686), and one retained performance among pre-dexamethasone patients but was poor in all patients (C = 0.793 and 0.596). One model could be calibrated but with poor performance.ConclusionsOur findings, albeit from a single center, suggest that the published performance of COVID-19 prediction models may not be replicated when translated to other institutions. In light of this, we would encourage bedside intensivists to reflect on the role of clinical prediction models in their own clinical decision-making.

Project description:BackgroundThe rapid spread of coronavirus disease 2019 (COVID-19) revealed significant constraints in critical care capacity. In anticipation of subsequent waves, reliable prediction of disease severity is essential for critical care capacity management and may enable earlier targeted interventions to improve patient outcomes. The purpose of this study is to develop and externally validate a prognostic model/clinical tool for predicting COVID-19 critical disease at presentation to medical care.MethodsThis is a retrospective study of a prognostic model for the prediction of COVID-19 critical disease where critical disease was defined as ICU admission, ventilation, and/or death. The derivation cohort was used to develop a multivariable logistic regression model. Covariates included patient comorbidities, presenting vital signs, and laboratory values. Model performance was assessed on the validation cohort by concordance statistics. The model was developed with consecutive patients with COVID-19 who presented to University of California Irvine Medical Center in Orange County, California. External validation was performed with a random sample of patients with COVID-19 at Emory Healthcare in Atlanta, Georgia.ResultsOf a total 3208 patients tested in the derivation cohort, 9% (299/3028) were positive for COVID-19. Clinical data including past medical history and presenting laboratory values were available for 29% (87/299) of patients (median age, 48 years [range, 21-88 years]; 64% [36/55] male). The most common comorbidities included obesity (37%, 31/87), hypertension (37%, 32/87), and diabetes (24%, 24/87). Critical disease was present in 24% (21/87). After backward stepwise selection, the following factors were associated with greatest increased risk of critical disease: number of comorbidities, body mass index, respiratory rate, white blood cell count, % lymphocytes, serum creatinine, lactate dehydrogenase, high sensitivity troponin I, ferritin, procalcitonin, and C-reactive protein. Of a total of 40 patients in the validation cohort (median age, 60 years [range, 27-88 years]; 55% [22/40] male), critical disease was present in 65% (26/40). Model discrimination in the validation cohort was high (concordance statistic: 0.94, 95% confidence interval 0.87-1.01). A web-based tool was developed to enable clinicians to input patient data and view likelihood of critical disease.Conclusions and relevanceWe present a model which accurately predicted COVID-19 critical disease risk using comorbidities and presenting vital signs and laboratory values, on derivation and validation cohorts from two different institutions. If further validated on additional cohorts of patients, this model/clinical tool may provide useful prognostication of critical care needs.

Project description:ObjectivesExisting UK prognostic models for patients admitted to the hospital with COVID-19 are limited by reliance on comorbidities, which are under-recorded in secondary care, and lack of imaging data among the candidate predictors. Our aims were to develop and externally validate novel prognostic models for adverse outcomes (death and intensive therapy unit (ITU) admission) in UK secondary care and externally validate the existing 4C score.DesignCandidate predictors included demographic variables, symptoms, physiological measures, imaging and laboratory tests. Final models used logistic regression with stepwise selection.SettingModel development was performed in data from University Hospitals Birmingham (UHB). External validation was performed in the CovidCollab dataset.ParticipantsPatients with COVID-19 admitted to UHB January-August 2020 were included.Main outcome measuresDeath and ITU admission within 28 days of admission.Results1040 patients with COVID-19 were included in the derivation cohort; 288 (28%) died and 183 (18%) were admitted to ITU within 28 days of admission. Area under the receiver operating characteristic curve (AUROC) for mortality was 0.791 (95% CI 0.761 to 0.822) in UHB and 0.767 (95% CI 0.754 to 0.780) in CovidCollab; AUROC for ITU admission was 0.906 (95% CI 0.883 to 0.929) in UHB and 0.811 (95% CI 0.795 to 0.828) in CovidCollab. Models showed good calibration. Addition of comorbidities to candidate predictors did not improve model performance. AUROC for the International Severe Acute Respiratory and Emerging Infection Consortium 4C score in the UHB dataset was 0.753 (95% CI 0.720 to 0.785).ConclusionsThe novel prognostic models showed good discrimination and calibration in derivation and external validation datasets, and performed at least as well as the existing 4C score using only routinely collected patient information. The models can be integrated into electronic medical records systems to calculate each individual patient's probability of death or ITU admission at the time of hospital admission. Implementation of the models and clinical utility should be evaluated.

Project description:BackgroundSeveral prognostic models for outcomes after chronic subdural hematoma (CSDH) treatment have been published in recent years. However, these models are not sufficiently validated for use in daily clinical practice. We aimed to assess the performance of existing prediction models for outcomes in patients diagnosed with CSDH.MethodsWe systematically searched relevant literature databases up to February 2021 to identify prognostic models for outcome prediction in patients diagnosed with CSDH. For the external validation of prognostic models, we used a retrospective database, containing data of 2384 patients from three Dutch regions. Prognostic models were included if they predicted either mortality, hematoma recurrence, functional outcome, or quality of life. Models were excluded when predictors were absent in our database or available for < 150 patients in our database. We assessed calibration, and discrimination (quantified by the concordance index C) of the included prognostic models in our retrospective database.ResultsWe identified 1680 original publications of which 1656 were excluded based on title or abstract, mostly because they did not concern CSDH or did not define a prognostic model. Out of 18 identified models, three could be externally validated in our retrospective database: a model for 30-day mortality in 1656 patients, a model for 2 months, and another for 3-month hematoma recurrence both in 1733 patients. The models overestimated the proportion of patients with these outcomes by 11% (15% predicted vs. 4% observed), 1% (10% vs. 9%), and 2% (11% vs. 9%), respectively. Their discriminative ability was poor to modest (C of 0.70 [0.63-0.77]; 0.46 [0.35-0.56]; 0.59 [0.51-0.66], respectively).ConclusionsNone of the examined models showed good predictive performance for outcomes after CSDH treatment in our dataset. This study confirms the difficulty in predicting outcomes after CSDH and emphasizes the heterogeneity of CSDH patients. The importance of developing high-quality models by using unified predictors and relevant outcome measures and appropriate modeling strategies is warranted.

Project description:Current evidence on COVID-19 prognostic models is inconsistent and clinical applicability remains controversial. We performed a systematic review to summarize and critically appraise the available studies that have developed, assessed and/or validated prognostic models of COVID-19 predicting health outcomes. We searched six bibliographic databases to identify published articles that investigated univariable and multivariable prognostic models predicting adverse outcomes in adult COVID-19 patients, including intensive care unit (ICU) admission, intubation, high-flow nasal therapy (HFNT), extracorporeal membrane oxygenation (ECMO) and mortality. We identified and assessed 314 eligible articles from more than 40 countries, with 152 of these studies presenting mortality, 66 progression to severe or critical illness, 35 mortality and ICU admission combined, 17 ICU admission only, while the remaining 44 studies reported prediction models for mechanical ventilation (MV) or a combination of multiple outcomes. The sample size of included studies varied from 11 to 7,704,171 participants, with a mean age ranging from 18 to 93 years. There were 353 prognostic models investigated, with area under the curve (AUC) ranging from 0.44 to 0.99. A great proportion of studies (61.5%, 193 out of 314) performed internal or external validation or replication. In 312 (99.4%) studies, prognostic models were reported to be at high risk of bias due to uncertainties and challenges surrounding methodological rigor, sampling, handling of missing data, failure to deal with overfitting and heterogeneous definitions of COVID-19 and severity outcomes. While several clinical prognostic models for COVID-19 have been described in the literature, they are limited in generalizability and/or applicability due to deficiencies in addressing fundamental statistical and methodological concerns. Future large, multi-centric and well-designed prognostic prospective studies are needed to clarify remaining uncertainties.

Project description:BackgroundPre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting.MethodsIPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis.ResultsOf 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model's calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%.ConclusionsThe evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice.Trial registrationPROSPERO ID: CRD42015029349 .

Dataset Information

Validity of prognostic models of critical COVID-19 is variable. A systematic review with external validation.

Objectives

Study design and setting

Results

Conclusion

Publications

Validity of prognostic models of critical COVID-19 is variable. A systematic review with external validation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets