Dataset Information

Machine learning prediction models in orthopedic surgery: A systematic review in transparent reporting.

ABSTRACT: Machine learning (ML) studies are becoming increasingly popular in orthopedics but lack a critically appraisal of their adherence to peer-reviewed guidelines. The objective of this review was to (1) evaluate quality and transparent reporting of ML prediction models in orthopedic surgery based on the transparent reporting of multivariable prediction models for individual prognosis or diagnosis (TRIPOD), and (2) assess risk of bias with the Prediction model Risk Of Bias ASsessment Tool. A systematic review was performed to identify all ML prediction studies published in orthopedic surgery through June 18th, 2020. After screening 7138 studies, 59 studies met the study criteria and were included. Two reviewers independently extracted data and discrepancies were resolved by discussion with at least two additional reviewers present. Across all studies, the overall median completeness for the TRIPOD checklist was 53% (interquartile range 47%-60%). The overall risk of bias was low in 44% (n = 26), high in 41% (n = 24), and unclear in 15% (n = 9). High overall risk of bias was driven by incomplete reporting of performance measures, inadequate handling of missing data, and use of small datasets with inadequate outcome numbers. Although the number of ML studies in orthopedic surgery is increasing rapidly, over 40% of the existing models are at high risk of bias. Furthermore, over half incompletely reported their methods and/or performance measures. Until these issues are adequately addressed to give patients and providers trust in ML models, a considerable gap remains between the development of ML prediction models and their implementation in orthopedic practice.

SUBMITTER: Groot OQ

PROVIDER: S-EPMC9290012 | biostudies-literature | 2022 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Machine learning prediction models in orthopedic surgery: A systematic review in transparent reporting.

Groot Olivier Q OQ Ogink Paul T PT Lans Amanda A Twining Peter K PK Kapoor Neal D ND DiGiovanni William W Bindels Bas J J BJJ Bongers Michiel E R MER Oosterhoff Jacobien H F JHF Karhade Aditya V AV Oner F C FC Verlaan Jorrit-Jan JJ Schwab Joseph H JH

Journal of orthopaedic research : official publication of the Orthopaedic Research Society 20210329 2

Machine learning (ML) studies are becoming increasingly popular in orthopedics but lack a critically appraisal of their adherence to peer-reviewed guidelines. The objective of this review was to (1) evaluate quality and transparent reporting of ML prediction models in orthopedic surgery based on the transparent reporting of multivariable prediction models for individual prognosis or diagnosis (TRIPOD), and (2) assess risk of bias with the Prediction model Risk Of Bias ASsessment Tool. A systemat ...[more]

PMID: 33734466

Similar Datasets

Project description:BackgroundWhile many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.MethodsWe included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item.ResultsOur search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).ConclusionSimilar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.Systematic review registrationPROSPERO, CRD42019161764.

Project description:IntroductionThere was limited evidence on the quality of reporting and methodological quality of prediction models using machine learning methods in preterm birth. This systematic review aimed to assess the reporting quality and risk of bias of a machine learning-based prediction model in preterm birth.Material and methodsWe conducted a systematic review, searching the PubMed, Embase, the Cochrane Library, China National Knowledge Infrastructure, China Biology Medicine disk, VIP Database, and WanFang Data from inception to September 27, 2021. Studies that developed (validated) a prediction model using machine learning methods in preterm birth were included. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement and Prediction model Risk of Bias Assessment Tool (PROBAST) to evaluate the reporting quality and the risk of bias of included studies, respectively. Findings were summarized using descriptive statistics and visual plots. The protocol was registered in PROSPERO (no. CRD 42022301623).ResultsTwenty-nine studies met the inclusion criteria, with 24 development-only studies and 5 development-with-validation studies. Overall, TRIPOD adherence per study ranged from 17% to 79%, with a median adherence of 49%. The reporting of title, abstract, blinding of predictors, sample size justification, explanation of model, and model performance were mostly poor, with TRIPOD adherence ranging from 4% to 17%. For all included studies, 79% had a high overall risk of bias, and 21% had an unclear overall risk of bias. The analysis domain was most commonly rated as high risk of bias in included studies, mainly as a result of small effective sample size, selection of predictors based on univariable analysis, and lack of calibration evaluation.ConclusionsReporting and methodological quality of machine learning-based prediction models in preterm birth were poor. It is urgent to improve the design, conduct, and reporting of such studies to boost the application of machine learning-based prediction models in preterm birth in clinical practice.

Project description:AimsWe conducted a systematic review assessing the reporting quality of studies validating models based on machine learning (ML) for clinical diagnosis, with a specific focus on the reporting of information concerning the participants on which the diagnostic task was evaluated on.MethodMedline Core Clinical Journals were searched for studies published between July 2015 and July 2018. Two reviewers independently screened the retrieved articles, a third reviewer resolved any discrepancies. An extraction list was developed from the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guideline. Two reviewers independently extracted the data from the eligible articles. Third and fourth reviewers checked, verified the extracted data as well as resolved any discrepancies between the reviewers.ResultsThe search results yielded 161 papers, of which 28 conformed to the eligibility criteria. Detail of data source was reported in 24 of the 28 papers. For all of the papers, the set of patients on which the ML-based diagnostic system was evaluated was partitioned from a larger dataset, and the method for deriving such set was always reported. Information on the diagnostic/non-diagnostic classification was reported well (23/28). The least reported items were the use of reporting guideline (0/28), distribution of disease severity (8/28 patient flow diagram (10/28) and distribution of alternative diagnosis (10/28). A large proportion of studies (23/28) had a delay between the conduct of the reference standard and ML tests, while one study did not and four studies were unclear. For 15 studies, it was unclear whether the evaluation group corresponded to the setting in which the ML test will be applied to.ConclusionAll studies in this review failed to use reporting guidelines, and a large proportion of them lacked adequate detail on participants, making it difficult to replicate, assess and interpret study findings.Prospero registration numberCRD42018099167.

Project description:ObjectivePostoperative red blood cell (RBC) transfusion is widely used during the perioperative period but is often associated with a high risk of infection and complications. However, prediction models for RBC transfusion in patients with orthopedic surgery have not yet been developed. We aimed to identify predictors and constructed prediction models for RBC transfusion after orthopedic surgery using interpretable machine learning algorithms.MethodsThis retrospective cohort study reviewed a total of 59,605 patients undergoing orthopedic surgery from June 2013 to January 2019 across 7 tertiary hospitals in China. Patients were randomly split into training (80%) and test subsets (20%). The feature selection method of recursive feature elimination (RFE) was used to identify an optimal feature subset from thirty preoperative variables, and six machine learning algorithms were applied to develop prediction models. The Shapley Additive exPlanations (SHAP) value was employed to evaluate the contribution of each predictor towards the prediction of postoperative RBC transfusion. For simplicity of the clinical utility, a risk score system was further established using the top risk factors identified by machine learning models.ResultsOf the 59,605 patients with orthopedic surgery, 19,921 (33.40%) underwent postoperative RBC transfusion. The CatBoost model exhibited an AUC of 0.831 (95% CI: 0.824-0.836) on the test subset, which significantly outperformed five other prediction models. The risk of RBC transfusion was associated with old age (>60 years) and low RBC count (<4.0 × 1012/L) with clear threshold effects. Extremes of BMI, low albumin, prolonged activated partial thromboplastin time, repair and plastic operations on joint structures were additional top predictors for RBC transfusion. The risk score system derived from six risk factors performed well with an AUC of 0.801 (95% CI: 0.794-0.807) on the test subset.ConclusionBy applying an interpretable machine learning framework in a large-scale multicenter retrospective cohort, we identified novel modifiable risk factors and developed prediction models with good performance for postoperative RBC transfusion in patients undergoing orthopedic surgery. Our findings may allow more precise identification of high-risk patients for optimal control of risk factors and achieve personalized RBC transfusion for orthopedic patients.

Project description:BackgroundWith the growing excitement of the potential benefits of using machine learning and artificial intelligence in medicine, the number of published clinical prediction models that use these approaches has increased. However, there is evidence (albeit limited) that suggests that the reporting of machine learning-specific aspects in these studies is poor. Further, there are no reviews assessing the reporting quality or broadly accepted reporting guidelines for these aspects.ObjectiveThis paper presents the protocol for a systematic review that will assess the reporting quality of machine learning-specific aspects in studies that use machine learning to develop clinical prediction models.MethodsWe will include studies that use a supervised machine learning algorithm to develop a prediction model for use in clinical practice (ie, for diagnosis or prognosis of a condition or identification of candidates for health care interventions). We will search MEDLINE for studies published in 2019, pseudorandomly sort the records, and screen until we obtain 100 studies that meet our inclusion criteria. We will assess reporting quality with a novel checklist developed in parallel with this review, which includes content derived from existing reporting guidelines, textbooks, and consultations with experts. The checklist will cover 4 key areas where the reporting of machine learning studies is unique: modelling steps (order and data used for each step), model performance (eg, reporting the performance of each model compared), statistical methods (eg, describing the tuning approach), and presentation of models (eg, specifying the predictors that contributed to the final model).ResultsWe completed data analysis in August 2021 and are writing the manuscript. We expect to submit the results to a peer-reviewed journal in early 2022.ConclusionsThis review will contribute to more standardized and complete reporting in the field by identifying areas where reporting is poor and can be improved.Trial registrationPROSPERO International Prospective Register of Systematic Reviews CRD42020206167; https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=206167.International registered report identifier (irrid)RR1-10.2196/30956.

Dataset Information

Machine learning prediction models in orthopedic surgery: A systematic review in transparent reporting.

Publications

Machine learning prediction models in orthopedic surgery: A systematic review in transparent reporting.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets