Dataset Information

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models.

ABSTRACT:

Objective

Health care providers increasingly rely upon predictive algorithms when making important treatment decisions, however, evidence indicates that these tools can lead to inequitable outcomes across racial and socio-economic groups. In this study, we introduce a bias evaluation checklist that allows model developers and health care providers a means to systematically appraise a model's potential to introduce bias.

Materials and methods

Our methods include developing a bias evaluation checklist, a scoping literature review to identify 30-day hospital readmission prediction models, and assessing the selected models using the checklist.

Results

We selected 4 models for evaluation: LACE, HOSPITAL, Johns Hopkins ACG, and HATRIX. Our assessment identified critical ways in which these algorithms can perpetuate health care inequalities. We found that LACE and HOSPITAL have the greatest potential for introducing bias, Johns Hopkins ACG has the most areas of uncertainty, and HATRIX has the fewest causes for concern.

Discussion

Our approach gives model developers and health care providers a practical and systematic method for evaluating bias in predictive models. Traditional bias identification methods do not elucidate sources of bias and are thus insufficient for mitigation efforts. With our checklist, bias can be addressed and eliminated before a model is fully developed or deployed.

Conclusion

The potential for algorithms to perpetuate biased outcomes is not isolated to readmission prediction models; rather, we believe our results have implications for predictive models across health care. We offer a systematic method for evaluating potential bias with sufficient flexibility to be utilized across models and applications.

SUBMITTER: Wang HE

PROVIDER: S-EPMC9277650 | biostudies-literature | 2022 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models.

Wang H Echo HE Landers Matthew M Adams Roy R Subbaswamy Adarsh A Kharrazi Hadi H Gaskin Darrell J DJ Saria Suchi S

Journal of the American Medical Informatics Association : JAMIA 20220701 8

<h4>Objective</h4>Health care providers increasingly rely upon predictive algorithms when making important treatment decisions, however, evidence indicates that these tools can lead to inequitable outcomes across racial and socio-economic groups. In this study, we introduce a bias evaluation checklist that allows model developers and health care providers a means to systematically appraise a model's potential to introduce bias.<h4>Materials and methods</h4>Our methods include developing a bias e ...[more]

PMID: 35579328

Similar Datasets

Project description:ObjectivesTo summarise multivariable predictive models for 30-day unplanned hospital readmissions (UHRs) in paediatrics, describe their performance and completeness in reporting, and determine their potential for application in practice.DesignSystematic review.Data sourceCINAHL, Embase and PubMed up to 7 October 2021.Eligibility criteriaEnglish or German language studies aiming to develop or validate a multivariable predictive model for 30-day paediatric UHRs related to all-cause, surgical conditions or general medical conditions were included.Data extraction and synthesisStudy characteristics, risk factors significant for predicting readmissions and information about performance measures (eg, c-statistic) were extracted. Reporting quality was addressed by the 'Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis' (TRIPOD) adherence form. The study quality was assessed by applying six domains of potential biases. Due to expected heterogeneity among the studies, the data were qualitatively synthesised.ResultsBased on 28 studies, 37 predictive models were identified, which could potentially be used for determining individual 30-day UHR risk in paediatrics. The number of study participants ranged from 190 children to 1.4 million encounters. The two most common significant risk factors were comorbidity and (postoperative) length of stay. 23 models showed a c-statistic above 0.7 and are primarily applicable at discharge. The median TRIPOD adherence of the models was 59% (P25-P75, 55%-69%), ranging from a minimum of 33% to a maximum of 81%. Overall, the quality of many studies was moderate to low in all six domains.ConclusionPredictive models may be useful in identifying paediatric patients at increased risk of readmission. To support the application of predictive models, more attention should be placed on completeness in reporting, particularly for those items that may be relevant for implementation in practice.

Project description:ImportanceAccurate identification of hospital readmission risk during a current hospitalization may enhance decision-making, facilitate targeted systems-level interventions, and avoid preventable readmissions.ObjectiveTo temporally and externally validate a suite of readmission risk prediction models across 48 children's hospitals to assess their generalizability and feasibility for future clinical implementation.Design, setting, and participantsThis prognostic study analyzed data from the Pediatric Health Information System (PHIS) database, which contains billing and resource use data from 48 US children's hospitals, including the derivation hospital (DH) and 47 hospitals participating in the PHIS database (hereafter other PHIS hospitals). Children aged 18 years or younger discharged from these hospitals between January 1, 2016, to December 31, 2019, were included. This cohort was divided as specified into the 3 prediction models at the DH: 6 months or older with no recent hospitalizations (new admission model [NAM]), 6 months or older with 1 or more prior hospitalizations within the last 6 months (recent admission model [RAM]), and 6 months or younger (young infant model [YIM]). Data were analyzed from August 9 to December 1, 2023.Main outcomes and measuresThe primary validation outcome was hospital-level discrimination measured with area under the receiver operating characteristic curve (AUROC). Predictors included demographic, clinical, and utilization variables. All-cause 30-day readmission was modeled for each hospital using logistic regression and parameter estimates from the DH. Calibration plots examined observed vs predicted outcome frequencies for each hospital.ResultsIn external validation, a total of 851 499 children were discharged from 48 hospitals (16 330 DH discharges and 835 169 other PHIS hospital discharges). The largest group of children was aged 5 to 14 years (281 193 [33.0%]). In temporal validation, the DH PHIS 2016-2018 cohort included 45 682 discharges. All-cause 30-day readmission rates were 7.2% for NAM, 35.5% for RAM, and 11.7% for YIM. The 2019 DH PHIS cohort included 16 330 discharges. All cause 30-day readmision rates were 7.2% for NAM, 35.1% for RAM, and 11.1% for YIM. Temporal validation demonstrated reduced discrimination across all 3 models (median AUROC, 0.65 [95% CI 0.62-0.67] for the NAM; 0.73 [95% CI 0.72-0.75) for RAM; 0.67 [95% CI 0.63-0.70) for the YIM compared with the original estimates (median AUROC 0.76 [95% CI 0.85-0.78] for the NAM; 0.84 [95% CI 0.83-0.84] for the RAM; 0.79 [95% CI 0.77-0.80] for the YIM). Overall readmission rates were 5.9% for NAM, 30.1% for RAM, and 7.6% for YIM. External validation yielded similiar findings as the temporal validation, although with demonstrable variation in performance across hospitals (median [range] AUROC, 0.64 [0.60-0.68] for the NAM; 0.73 [0.64-0.80] for the RAM; 0.65 [0.53-0.74] for the YIM). Most hospitals were poorly calibrated, with both significant overestimation and underestimation of observed risk. Of 47 other PHIS hospitals, only 3 for the RAM (6.4%) and 9 for both the NAM and YIM (19.1%) were adequately calibrated.Conclusions and relevanceThis prognostic study found that the readmission risk prediction models had reduced predictive accuracy across time and variability in hospital-level performance. These findings stress the importance of local validation prior to clinical implementation and suggest opportunities to improve generalizability, including multicenter derivation and expansion of candidate predictors.

Project description:BackgroundLowering hospital readmission rates has become a primary target for the Centers for Medicare & Medicaid Services, but studies of the relationship between adherence to the recommended hospital care processes and readmission rates have provided inconsistent and inconclusive results.ObjectiveTo examine the association between hospital performance on Medicare's Hospital Compare process quality measures and 30-day readmission rates for patients with acute myocardial infarction (AMI), heart failure and pneumonia, and for those undergoing major surgery.Design, setting and participantsWe assessed hospital performance on process measures using the 2007 Hospital Inpatient Quality Reporting Program. The process measures for each condition were aggregated in two separate measures: Overall Measure (OM) and Appropriate Care Measure (ACM) scores. Readmission rates were calculated using Medicare claims.Main outcome measureRisk-standardized 30-day all-cause readmission rate was calculated as the ratio of predicted to expected rate standardized by the overall mean readmission rate. We calculated predicted readmission rate using hierarchical generalized linear models and adjusting for patient-level factors.ResultsAmong patients aged ≥ 66 years, the median OM score ranged from 79.4 % for abdominal surgery to 95.7 % for AMI, and the median ACM scores ranged from 45.8 % for abdominal surgery to 87.9 % for AMI. We observed a statistically significant, but weak, correlation between performance scores and readmission rates for pneumonia (correlation coefficient R = 0.07), AMI (R = 0.10), and orthopedic surgery (R = 0.06). The difference in the mean readmission rate between hospitals in the 1st and 4th quartiles of process measure performance was statistically significant only for AMI (0.25 percentage points) and pneumonia (0.31 percentage points). Performance on process measures explained less than 1 % of hospital-level variation in readmission rates.ConclusionsHospitals with greater adherence to recommended care processes did not achieve meaningfully better 30-day hospital readmission rates compared to those with lower levels of performance.

Project description:ObjectivesEarly hospital readmissions or deaths are key healthcare quality measures in pay-for-performance programs. Predictive models could identify patients at higher risk of readmission or death and target interventions. However, existing models usually do not incorporate social determinants of health (SDH) information, although this information is of great importance to address health disparities related to social risk factors. The objective of this study is to examine the impact of social determinants of health on predictive models for potentially avoidable 30-day readmission.MethodsWe extracted electronic health record data for 19,941 hospital admissions between January 2015 and November 2017 at an academic medical center in New York City. We applied the Simplified HOSPITAL score model to predict potentially avoidable 30-day readmission or death and examined if incorporating individual- and community-level SDH could improve the prediction using cross-validation. We calculated the C-statistic for discrimination, Brier score for accuracy, and Hosmer-Lemeshow test for calibration for each model using logistic regression. Analysis was conducted for all patients and three subgroups that may be disproportionately affected by social risk factors, namely Medicaid patients, patients who are 65 or older, and obese patients.ResultsThe Simplified HOSPITAL score model achieved similar performance in our sample compared to previous studies. Adding SDH did not improve the prediction among all patients. However, adding individual- and community-level SDH at the US census tract level significantly improved the prediction for all three subgroups. Specifically, C-statistics improved from 0.70 to 0.73 for Medicaid patients, from 0.66 to 0.68 for patients 65 or older, and from 0.70 to 0.73 for obese patients.ConclusionsPatients from certain subgroups may be more likely to be affected by social risk factors. Incorporating SDH into predictive models may be helpful to identify these patients and reduce health disparities associated with vulnerable social conditions.

Dataset Information

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models.

Objective

Materials and methods

Results

Discussion

Conclusion

Publications

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets