Dataset Information

External validation of a model to predict future chronic opioid use among hospitalized patients.

ABSTRACT:

Background

Previous research demonstrates an association between opioid prescribing at hospital discharge and future chronic opioid use. Various opioid guidelines and policies contributed to changes in opioid prescribing practices. How this affected hospitalized patients remains unknown.

Objective

Externally validate a prediction model to identify hospitalized patients at the highest risk for future chronic opioid therapy (COT).

Designs

Retrospective analysis of health record data from 2011 to 2022 using logistic regression.

Participants

Hospitalized adults with limited to no opioid use 1-year prior to hospitalization.

Settings

A statewide healthcare system.

Main measurements

Used variables associated with progression to COT in a derivation cohort from a different healthcare system to predict expected outcomes in the validation cohort.

Key results

The derivation cohort included 17,060 patients, of whom 9653 (56.6%) progressed to COT 1 year after discharge. Compared to the derivation cohort, in the validation cohort, patients who received indigent care (odds ratio [OR] = 0.40, 95% confidence interval [CI] = 0.27-0.59, p < .001) were least likely to progress to COT. Among variables assessed, opioid receipt at discharge was most strongly associated with progression to COT (OR = 3.74, 95% CI = 3.06-4.61, p < .001). The receiver operating characteristic curve for the validation set using coefficients from the derivation cohort performed slightly better than chance (AUC = 0.55).

Conclusions

Our results highlight the importance of externally validating a prediction model prior to use outside of the derivation population. Periodic updates to models are necessary as policy changes and clinical practice recommendations may affect model performance.

SUBMITTER: Calcaterra SL

PROVIDER: S-EPMC9899308 | biostudies-literature | 2023 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

External validation of a model to predict future chronic opioid use among hospitalized patients.

Calcaterra Susan L SL Grimm Eric E Keniston Angela A

Journal of hospital medicine 20221216 2

<h4>Background</h4>Previous research demonstrates an association between opioid prescribing at hospital discharge and future chronic opioid use. Various opioid guidelines and policies contributed to changes in opioid prescribing practices. How this affected hospitalized patients remains unknown.<h4>Objective</h4>Externally validate a prediction model to identify hospitalized patients at the highest risk for future chronic opioid therapy (COT).<h4>Designs</h4>Retrospective analysis of health reco ...[more]

PMID: 36524583

Similar Datasets

Project description:BACKGROUND:Opioids are commonly prescribed in the hospital; yet, little is known about which patients will progress to chronic opioid therapy (COT) following discharge. We defined COT as receipt of ≥ 90-day supply of opioids with < 30-day gap in supply over a 180-day period or receipt of ≥ 10 opioid prescriptions over 1 year. Predictive tools to identify hospitalized patients at risk for future chronic opioid use could have clinical utility to improve pain management strategies and patient education during hospitalization and discharge. OBJECTIVE:The objective of this study was to identify a parsimonious statistical model for predicting future COT among hospitalized patients not on COT before hospitalization. DESIGN:Retrospective analysis electronic health record (EHR) data from 2008 to 2014 using logistic regression. PATIENTS:Hospitalized patients at an urban, safety net hospital. MAIN MEASUREMENTS:Independent variables included medical and mental health diagnoses, substance and tobacco use disorder, chronic or acute pain, surgical intervention during hospitalization, past year receipt of opioid or non-opioid analgesics or benzodiazepines, opioid receipt at hospital discharge, milligrams of morphine equivalents prescribed per hospital day, and others. KEY RESULTS:Model prediction performance was estimated using area under the receiver operator curve, accuracy, sensitivity, and specificity. A model with 13 covariates was chosen using stepwise logistic regression on a randomly down-sampled subset of the data. Sensitivity and specificity were optimized using the Youden's index. This model predicted correctly COT in 79% of the patients and no COT correctly in 78% of the patients. CONCLUSIONS:Our model accessed EHR data to predict 79% of the future COT among hospitalized patients. Application of such a predictive model within the EHR could identify patients at high risk for future chronic opioid use to allow clinicians to provide early patient education about pain management strategies and, when able, to wean opioids prior to discharge while incorporating alternative therapies for pain into discharge planning.

Project description:BackgroundOpioid misuse screening in hospitals is resource-intensive and rarely done. Many hospitalized patients are never offered opioid treatment. An automated approach leveraging routinely captured electronic health record (EHR) data may be easier for hospitals to institute. We previously derived and internally validated an opioid classifier in a separate hospital setting. The aim is to externally validate our previously published and open-source machine-learning classifier at a different hospital for identifying cases of opioid misuse.MethodsAn observational cohort of 56,227 adult hospitalizations was examined between October 2017 and December 2019 during a hospital-wide substance use screening program with manual screening. Manually completed Drug Abuse Screening Test served as the reference standard to validate a convolutional neural network (CNN) classifier with coded word embedding features from the clinical notes of the EHR. The opioid classifier utilized all notes in the EHR and sensitivity analysis was also performed on the first 24 h of notes. Calibration was performed to account for the lower prevalence than in the original cohort.ResultsManual screening for substance misuse was completed in 67.8% (n = 56,227) with 1.1% (n = 628) identified with opioid misuse. The data for external validation included 2,482,900 notes with 67,969 unique clinical concept features. The opioid classifier had an AUC of 0.99 (95% CI 0.99-0.99) across the encounter and 0.98 (95% CI 0.98-0.99) using only the first 24 h of notes. In the calibrated classifier, the sensitivity and positive predictive value were 0.81 (95% CI 0.77-0.84) and 0.72 (95% CI 0.68-0.75). For the first 24 h, they were 0.75 (95% CI 0.71-0.78) and 0.61 (95% CI 0.57-0.64).ConclusionsOur opioid misuse classifier had good discrimination during external validation. Our model may provide a comprehensive and automated approach to opioid misuse identification that augments current workflows and overcomes manual screening barriers.

Project description:ImportanceThe Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM's ability to identify patients with sepsis has not been adequately evaluated despite widespread use.ObjectiveTo externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care.Design, setting, and participantsThis retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019.ExposureThe ESM score, calculated every 15 minutes.Main outcomes and measuresSepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies.ResultsWe identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.Conclusions and relevanceThis external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

Project description:ObjectiveTo assess the single site performance of the Dynamic Criticality Index (CI-D) models developed from a multi-institutional database to predict future care. Secondarily, to assess future care-location predictions in a single institution when CI-D models are re-developed using single-site data with identical variables and modeling methods. Four CI-D models were assessed for predicting care locations >6-12 hours, >12-18 hours, >18-24 hours, and >24-30 hours in the future.DesignPrognostic study comparing multi-institutional CI-D models' performance in a single-site electronic health record dataset to an institution-specific CI-D model developed using identical variables and modelling methods. The institution did not participate in the multi-institutional dataset.ParticipantsAll pediatric inpatients admitted from January 1st 2018 -February 29th 2020 through the emergency department.Main outcome(s) and measure(s)The main outcome was inpatient care in routine or ICU care locations.ResultsA total of 29,037 pediatric hospital admissions were included, with 5,563 (19.2%) admitted directly to the ICU, 869 (3.0%) transferred from routine to ICU care, and 5,023 (17.3%) transferred from ICU to routine care. Patients had a median [IQR] age 68 months (15-157), 47.5% were female and 43.4% were black. The area under the receiver operating characteristic curve (AUROC) for the multi-institutional CI-D models applied to a single-site test dataset was 0.493-0.545 and area under the precision-recall curve (AUPRC) was 0.262-0.299. The single-site CI-D models applied to an independent single-site test dataset had an AUROC 0.906-0.944 and AUPRC range from 0.754-0.824. Accuracy at 0.95 sensitivity for those transferred from routine to ICU care was 72.6%-81.0%. Accuracy at 0.95 specificity was 58.2%-76.4% for patients who transferred from ICU to routine care.Conclusion and relevanceModels developed from multi-institutional datasets and intended for application to individual institutions should be assessed locally and may benefit from re-development with site-specific data prior to deployment.

Project description:Background and aimsUnhealthy alcohol use (UAU) is one of the leading causes of global morbidity. A machine learning approach to alcohol screening could accelerate best practices when integrated into electronic health record (EHR) systems. This study aimed to validate externally a natural language processing (NLP) classifier developed at an independent medical center.DesignRetrospective cohort study.SettingThe site for validation was a midwestern United States tertiary-care, urban medical center that has an inpatient structured universal screening model for unhealthy substance use and an active addiction consult service.Participants/casesUnplanned admissions of adult patients between October 23, 2017 and December 31, 2019, with EHR documentation of manual alcohol screening were included in the cohort (n = 57 605).MeasurementsThe Alcohol Use Disorders Identification Test (AUDIT) served as the reference standard. AUDIT scores ≥5 for females and ≥8 for males served as cases for UAU. To examine error in manual screening or under-reporting, a post hoc error analysis was conducted, reviewing discordance between the NLP classifier and AUDIT-derived reference. All clinical notes excluding the manual screening and AUDIT documentation from the EHR were included in the NLP analysis.FindingsUsing clinical notes from the first 24 hours of each encounter, the NLP classifier demonstrated an area under the receiver operating characteristic curve (AUCROC) and precision-recall area under the curve (PRAUC) of 0.91 (95% CI = 0.89-0.92) and 0.56 (95% CI = 0.53-0.60), respectively. At the optimal cut point of 0.5, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 0.66 (95% CI = 0.62-0.69), 0.98 (95% CI = 0.98-0.98), 0.35 (95% CI = 0.33-0.38), and 1.0 (95% CI = 1.0-1.0), respectively.ConclusionsExternal validation of a publicly available alcohol misuse classifier demonstrates adequate sensitivity and specificity for routine clinical use as an automated screening tool for identifying at-risk patients.

Dataset Information

External validation of a model to predict future chronic opioid use among hospitalized patients.

Background

Objective

Designs

Participants

Settings

Main measurements

Key results

Conclusions

Publications

External validation of a model to predict future chronic opioid use among hospitalized patients.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets