How to Develop and Validate Prediction Models for Orthopedic Outcomes.
Ontology highlight
ABSTRACT:
SUBMITTER: Zaniletti I
PROVIDER: S-EPMC10023373 | biostudies-literature | 2023 Apr
REPOSITORIES: biostudies-literature
Publications
How to Develop and Validate Prediction Models for Orthopedic Outcomes.
Zaniletti Isabella I Larson Dirk R DR Lewallen David G DG Berry Daniel J DJ Maradit Kremers Hilal H
The Journal of arthroplasty 20221224 4
Prediction models are common in medicine for predicting outcomes such as mortality, complications, or response to treatment. Despite the growing interest in these models in arthroplasty (and orthopaedics in general), few have been adopted in clinical practice. If robustly built and validated, prediction models can be excellent tools to support surgical decision making. In this paper, we provide an overview of the statistical concepts surrounding prediction models and outline practical steps for ...[more]
Similar Datasets
Project description:ObjectiveTo determine whether readily available patient, ultrasound and treatment outcome data can be used to develop, validate and externally test two machine learning (ML) models for predicting the success of expectant and medical management of miscarriage respectively.MethodsA retrospective, multi-site study of patients opting for expectant or medical management of miscarriage was undertaken. A total of 1075 patients across two hospital early pregnancy units were eligible for inclusion. Data pre-processing derived 14 features for predictive modelling. A combination of eight linear, Bayesian, neural-net and tree-based machine learning algorithms were applied to ten different feature sets. The area under the receiver operating characteristic curve (AUC) scores were the metrics used to demonstrate the performance of the best performing model and feature selection combination for the training, validation and external data set for expectant and medical management separately.ResultsParameters were in the majority well matched across training, validation and external test sets. The respective optimum training, validation and external test set AUC scores were as follows in the expectant management cohort: 0.72 (95% CI 0.67,0.77), 0.63 (95% CI 0.53,0.73) and 0.70 (95% CI 0.60,0.79) (Logistic Regression in combination with Least Absolute Shrinkage and Selection Operator (LASSO)). The AUC scores in the medical management cohort were 0.64 (95% CI 0.56,0.72), 0.62 (95% CI 0.45,0.77) and 0.71 (95% CI 0.58,0.83) (Logistic Regression in combination with Recursive Feature Elimination (RFE)).ConclusionsPerformance of our expectant and medical miscarriage management ML models demonstrate consistency across validation and external test sets. These ML methods, validated and externally tested, have the potential to offer personalised prediction outcome of expectant and medical management of miscarriage.
| S-EPMC11869538 | biostudies-literature
Project description:BackgroundNon-arrivals to scheduled ambulatory visits are common and lead to a discontinuity of care, poor health outcomes, and increased subsequent healthcare utilization. Reducing non-arrivals is important given their association with poorer health outcomes and cost to health systems.ObjectiveTo develop and validate a prediction model for ambulatory non-arrivals.DesignRetrospective cohort study.Patients or subjectsPatients at an integrated health system who had an outpatient visit scheduled from January 1, 2020, to February 28, 2022.Main measuresNon-arrivals to scheduled appointments.Key resultsThere were over 4.3 million ambulatory appointments from 1.2 million adult patients. Patients with appointment non-arrivals were more likely to be single, racial/ethnic minorities, and not having an established primary care provider compared to those who arrived at their appointments. A prediction model using the XGBoost machine learning algorithm had the highest AUC value (0.768 [0.767-0.770]). Using SHAP values, the most impactful features in the model include rescheduled appointments, lead time (number of days from scheduled to appointment date), appointment provider, number of days since last appointment with the same department, and a patient's prior appointment status within the same department. Scheduling visits close to an appointment date is predicted to be less likely to result in a non-arrival. Overall, the prediction model calibrated well for each department, especially over the operationally relevant probability range of 0 to 40%. Departments with fewer observations and lower non-arrival rates generally had a worse calibration.ConclusionsUsing a machine learning algorithm, we developed a prediction model for non-arrivals to scheduled ambulatory appointments usable for all medical specialties. The proposed prediction model can be deployed within an electronic health system or integrated into other dashboards to reduce non-arrivals. Future work will focus on the implementation and application of the model to reduce non-arrivals.
| S-EPMC9910253 | biostudies-literature
Project description:Background and purpose - External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines.Material and methods - We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting.Results - We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43-89), with 6 items being reported in less than 4/18 of the studies.Interpretation - Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.
| S-EPMC8436968 | biostudies-literature
Project description:BackgroundEarly recognition of sepsis and the prediction of mortality in patients with infection are important. This multi-center, ED-based study aimed to develop and validate a 28-day mortality prediction model for patients with infection using various machine learning (ML) algorithms.MethodsPatients with acute infection requiring intravenous antibiotic treatment during the first 24 h of admission were prospectively recruited. Patient demographics, comorbidities, clinical signs and symptoms, laboratory test data, selected sepsis-related novel biomarkers, and 28-day mortality were collected and divided into training (70%) and testing (30%) datasets. Logistic regression and seven ML algorithms were used to develop the prediction models. The area under the receiver operating characteristic curve (AUROC) was used to compare different models.ResultsA total of 555 patients were recruited with a full panel of biomarker tests. Among them, 18% fulfilled Sepsis-3 criteria, with a 28-day mortality rate of 8%. The wrapper algorithm selected 30 features, including disease severity scores, biochemical parameters, and conventional and few sepsis-related biomarkers. Random forest outperformed other ML models (AUROC: 0.96; 95% confidence interval: 0.93-0.98) and SOFA and early warning scores (AUROC: 0.64-0.84) in the prediction of 28-day mortality in patients with infection. Additionally, random forest remained the best-performing model, with an AUROC of 0.95 (95% CI: 0.91-0.98, p = 0.725) after removing five sepsis-related novel biomarkers.ConclusionsOur results demonstrated that ML models provide a more accurate prediction of 28-day mortality with an enhanced ability in dealing with multi-dimensional data than the logistic regression model.
| S-EPMC9030924 | biostudies-literature
Project description:BackgroundCognitive impairment and depressive symptoms are prevalent and closely interrelated mental health issues in the elderly. Traditional methods for identifying depressive symptoms in this population often lack effectiveness. Machine learning provides a promising alternative for developing predictive models that can facilitate early identification and intervention.MethodsThis study utilized data from 945 participants aged 60 years and older with cognitive impairment, sourced from National Health and Nutrition Examination Surveys (2011-2014). Depressive symptoms were assessed using the Patient Health Questionnaire-9. Lasso regression was applied for feature selection, ensuring consistency across models. Several machine learning models, including XGBoost, Logistic Regression, Random Forest, and SVM, were trained and evaluated. Model performance was assessed using accuracy, precision, recall, F1 score, and AUC.ResultsThe incidence of depressive symptoms in older adults with cognitive impairment was 14.07%. Key predictors identified by lasso included general health, memory difficulties, and age, among others. Notably, general health emerged as a novel and significant predictor in this population, underscoring the interplay between physical and mental health. XGBoost was the best model for comprehensively comparing discrimination, calibration, and clinical utility.ConclusionsMachine learning models, particularly XGBoost, effectively predict depressive symptoms in cognitively impaired older adults. The findings highlight the importance of physical, cognitive, and social factors in depressive symptoms risk. These models have the potential to assist in early screening and intervention, improving patient outcomes. Future research should explore ways to enhance model generalizability, including the use of clinically diagnosed depressive symptoms data and alternative feature selection approaches.
| S-EPMC11895390 | biostudies-literature
Project description:Machine learning (ML) studies are becoming increasingly popular in orthopedics but lack a critically appraisal of their adherence to peer-reviewed guidelines. The objective of this review was to (1) evaluate quality and transparent reporting of ML prediction models in orthopedic surgery based on the transparent reporting of multivariable prediction models for individual prognosis or diagnosis (TRIPOD), and (2) assess risk of bias with the Prediction model Risk Of Bias ASsessment Tool. A systematic review was performed to identify all ML prediction studies published in orthopedic surgery through June 18th, 2020. After screening 7138 studies, 59 studies met the study criteria and were included. Two reviewers independently extracted data and discrepancies were resolved by discussion with at least two additional reviewers present. Across all studies, the overall median completeness for the TRIPOD checklist was 53% (interquartile range 47%-60%). The overall risk of bias was low in 44% (n = 26), high in 41% (n = 24), and unclear in 15% (n = 9). High overall risk of bias was driven by incomplete reporting of performance measures, inadequate handling of missing data, and use of small datasets with inadequate outcome numbers. Although the number of ML studies in orthopedic surgery is increasing rapidly, over 40% of the existing models are at high risk of bias. Furthermore, over half incompletely reported their methods and/or performance measures. Until these issues are adequately addressed to give patients and providers trust in ML models, a considerable gap remains between the development of ML prediction models and their implementation in orthopedic practice.
| S-EPMC9290012 | biostudies-literature
Project description:IntroductionType 2 diabetes mellitus (T2DM) is a major cause of blindness, kidney failure, myocardial infarction, stroke and lower limb amputation. We are still unable, however, to accurately predict or identify which patients are at a higher risk of deterioration. Most risk stratification tools do not account for novel factors such as sociodemographic determinants, self-management ability or access to healthcare. Additionally, most tools are based in clinical trials, with limited external generalisability.ObjectiveThe aim of this work is to design and validate a machine learning-based tool to identify patients with T2DM at high risk of clinical deterioration, based on a comprehensive set of patient-level characteristics retrieved from a population health linked dataset.Sample and designRetrospective cohort study of patients with diagnosis of T2DM on 1 January 2015, with a 5-year follow-up. Anonymised electronic healthcare records from the Whole System Integrated Care (WSIC) database will be used.Preliminary outcomesOutcome variables of clinical deterioration will include retinopathy, chronic renal disease, myocardial infarction, stroke, peripheral arterial disease or death. Predictor variables will include sociodemographic and geographic data, patients' ability to self-manage disease, clinical and metabolic parameters and healthcare service usage. Prognostic models will be defined using multidependence Bayesian networks. The derivation cohort, comprising 80% of the patients, will be used to define the prognostic models. Model parameters will be internally validated by comparing the area under the receiver operating characteristic curve in the derivation cohort with those calculated from a leave-one-out and a 10 times twofold cross-validation.Ethics and disseminationThe study has received approvals from the Information Governance Committee at the WSIC. Results will be made available to people with T2DM, their caregivers, the funders, diabetes care societies and other researchers.
| S-EPMC8327849 | biostudies-literature
Project description:Background and purpose - Advancements in software and hardware have enabled the rise of clinical prediction models based on machine learning (ML) in orthopedic surgery. Given their growing popularity and their likely implementation in clinical practice we evaluated which outcomes these new models have focused on and what methodologies are being employed.Material and methods - We performed a systematic search in PubMed, Embase, and Cochrane Library for studies published up to June 18, 2020. Studies reporting on non-ML prediction models or non-orthopedic outcomes were excluded. After screening 7,138 studies, 59 studies reporting on 77 prediction models were included. We extracted data regarding outcome, study design, and reported performance metrics.Results - Of the 77 identified ML prediction models the most commonly reported outcome domain was medical management (17/77). Spinal surgery was the most commonly involved orthopedic subspecialty (28/77). The most frequently employed algorithm was neural networks (42/77). Median size of datasets was 5,507 (IQR 635-26,364). The median area under the curve (AUC) was 0.80 (IQR 0.73-0.86). Calibration was reported for 26 of the models and 14 provided decision-curve analysis.Interpretation - ML prediction models have been developed for a wide variety of topics in orthopedics. Topics regarding medical management were the most commonly studied. Heterogeneity between studies is based on study size, algorithm, and time-point of outcome. Calibration and decision-curve analysis were generally poorly reported.
| S-EPMC8519550 | biostudies-literature
Project description:SARS-CoV-2 has caused over 6.9 million deaths and continues to produce lasting health consequences. COVID-19 manifests broadly from no symptoms to death. In a retrospective cross-sectional study, we developed personalized risk assessment models that predict clinical outcomes for individuals with COVID-19 and inform targeted interventions. We sequenced viruses from SARS-CoV-2-positive nasopharyngeal swab samples between July 2020 and July 2022 from 4450 individuals in Missouri and retrieved associated disease courses, clinical history, and urban-rural classification. We integrated this data to develop machine learning-based predictive models to predict hospitalization, ICU admission, and long COVID.The mean age was 38.3 years (standard deviation = 21.4) with 55.2% (N = 2453) females and 44.8% (N = 1994) males (not reported, N = 4). Our analyses revealed a comprehensive set of predictors for each outcome, encompassing human, environment, and virus genome-wide genetic markers. Immunosuppression, cardiovascular disease, older age, cardiac, gastrointestinal, and constitutional symptoms, rural residence, and specific amino acid substitutions were associated with hospitalization. ICU admission was associated with acute respiratory distress syndrome, ventilation, bacterial co-infection, rural residence, and non-wild type SARS-CoV-2 variants. Finally, long COVID was associated with hospital admission, ventilation, and female sex.Overall, we developed risk assessment models that offer the capability to identify patients with COVID-19 necessitating enhanced monitoring or early interventions. Of importance, we demonstrate the value of including key elements of virus, host, and environmental factors to predict patient outcomes, serving as a valuable platform in the field of personalized medicine with the potential for adaptation to other infectious diseases.
| S-EPMC11182058 | biostudies-literature
Project description:PurposeThe purpose of this study was to develop and validate a prediction model for 90-day mortality following a total knee replacement (TKR). TKR is a safe and cost-effective surgical procedure for treating severe knee osteoarthritis (OA). Although complications following surgery are rare, prediction tools could help identify high-risk patients who could be targeted with preventative interventions. The aim was to develop and validate a simple model to help inform treatment choices.MethodsA mortality prediction model for knee OA patients following TKR was developed and externally validated using a US claims database and a UK general practice database. The target population consisted of patients undergoing a primary TKR for knee OA, aged ≥ 40 years and registered for ≥ 1 year before surgery. LASSO logistic regression models were developed for post-operative (90-day) mortality. A second mortality model was developed with a reduced feature set to increase interpretability and usability.ResultsA total of 193,615 patients were included, with 40,950 in The Health Improvement Network (THIN) database and 152,665 in Optum. The full model predicting 90-day mortality yielded AUROC of 0.78 when trained in OPTUM and 0.70 when externally validated on THIN. The 12 variable model achieved internal AUROC of 0.77 and external AUROC of 0.71 in THIN.ConclusionsA simple prediction model based on sex, age, and 10 comorbidities that can identify patients at high risk of short-term mortality following TKR was developed that demonstrated good, robust performance. The 12-feature mortality model is easily implemented and the performance suggests it could be used to inform evidence based shared decision-making prior to surgery and targeting prophylaxis for those at high risk.Level of evidenceIII.
| S-EPMC9418076 | biostudies-literature