Dataset Information

External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients.

ABSTRACT:

Objective

To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6-12 hours, >12-18 hours, >18-24 hours, and >24-30 hours in the future.

Design

Prognostic study comparing multi-institutional CI-D models' performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset.

Participants

All pediatric inpatients admitted from January 1st 2018 -February 29th 2020 through the emergency department.

Main outcome(s) and measure(s)

The main outcome was inpatient care in routine or ICU care locations.

Results

A total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15-157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493-0.545 and area under the precision-recall curve (AUPRC) was 0.262-0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906-0.944 and AUPRC range from 0.754-0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care.

Conclusion and relevance

Models developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.

SUBMITTER: Patel AK

PROVIDER: S-EPMC10824440 | biostudies-literature | 2024

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients.

Patel Anita K AK Trujillo-Rivera Eduardo E Chamberlain James M JM Morizono Hiroki H Pollack Murray M MM

PloS one 20240129 1

<h4>Objective</h4>To assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6-12 hours, >12-18 hours, >18-24 hours, and >24-30 hours in the future.<h4>Design ...[more]

PMID: 38285704

Similar Datasets

Project description:BackgroundThe Criticality Index-Mortality uses physiology, therapy, and intensity of care to compute mortality risk for pediatric ICU patients. If the frequency of mortality risk computations were increased to every 3 h with model performance that could improve the assessment of severity of illness, it could be utilized to monitor patients for significant mortality risk change.ObjectivesTo assess the performance of a dynamic method of updating mortality risk every 3 h using the Criticality Index-Mortality methodology and identify variables that are significant contributors to mortality risk predictions.PopulationThere were 8,399 pediatric ICU admissions with 312 (3.7%) deaths from January 1, 2018 to February 29, 2020. We randomly selected 75% of patients for training, 13% for validation, and 12% for testing.ModelA neural network was trained to predict hospital survival or death during or following an ICU admission. Variables included age, gender, laboratory tests, vital signs, medications categories, and mechanical ventilation variables. The neural network was calibrated to mortality risk using nonparametric logistic regression.ResultsDiscrimination assessed across all time periods found an AUROC of 0.851 (0.841-0.862) and an AUPRC was 0.443 (0.417-0.467). When assessed for performance every 3 h, the AUROCs had a minimum value of 0.778 (0.689-0.867) and a maximum value of 0.885 (0.841,0.862); the AUPRCs had a minimum value 0.148 (0.058-0.328) and a maximum value of 0.499 (0.229-0.769). The calibration plot had an intercept of 0.011, a slope of 0.956, and the R2 was 0.814. Comparison of observed vs. expected proportion of deaths revealed that 95.8% of the 543 risk intervals were not statistically significantly different. Construct validity assessed by death and survivor risk trajectories analyzed by mortality risk quartiles and 7 high and low risk diseases confirmed a priori clinical expectations about the trajectories of death and survivors.ConclusionsThe Criticality Index-Mortality computing mortality risk every 3 h for pediatric ICU patients has model performance that could enhance the clinical assessment of severity of illness. The overall Criticality Index-Mortality framework was effectively applied to develop an institutionally specific, and clinically relevant model for dynamic risk assessment of pediatric ICU patients.

Project description:ObjectivesTo assess severity of illness trajectories described by the Criticality Index for survivors and deaths in five patient groups defined by the sequence of patient care in ICU and routine patient care locations.DesignThe Criticality Index developed using a calibrated, deep neural network, measures severity of illness using physiology, therapies, and therapeutic intensity. Criticality Index values in sequential 6-hour time periods described severity trajectories.SettingHospitals with pediatric inpatient and ICU care.PatientsPediatric patients never cared for in an ICU (n = 20,091), patients only cared for in the ICU (n = 2,096) and patients cared for in both ICU and non-ICU care locations (n = 17,023) from 2009 to 2016 Health Facts database (Cerner Corporation, Kansas City, MO).InterventionsNone.Measurements and main resultsCriticality Index values were consistent with clinical experience. The median (25-75th percentile) ICU Criticality Index values (0.878 [0.696-0.966]) were more than 80-fold higher than the non-ICU values (0.010 [0.002-0.099]). Non-ICU Criticality Index values for patients transferred to the ICU were 40-fold higher than those never transferred to the ICU (0.164 vs 0.004). The median for ICU deaths was higher than ICU survivors (0.983 vs 0.875) (p < 0.001). The severity trajectories for the five groups met expectations based on clinical experience. Survivors had increasing Criticality Index values in non-ICU locations prior to ICU admission, decreasing Criticality Index values in the ICU, and decreasing Criticality Index values until hospital discharge. Deaths had higher Criticality Index values than survivors, steeper increases prior to the ICU, and worsening values in the ICU. Deaths had a variable course, especially those who died in non-ICU care locations, consistent with deaths associated with both active therapies and withdrawals/limitations of care.ConclusionsSeverity trajectories measured by the Criticality Index showed strong validity, reflecting the expected clinical course for five diverse patient groups.

Dataset Information

External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients.

Objective

Design

Participants

Main outcome(s) and measure(s)

Results

Conclusion and relevance

Publications

External evaluation of the Dynamic Criticality Index: A machine learning model to predict future need for ICU care in hospitalized pediatric patients.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets