Appraising the Quality of Development and Reporting in Surgical Prediction Models.
Ontology highlight
ABSTRACT:
SUBMITTER: Marwaha JS
PROVIDER: S-EPMC9713676 | biostudies-literature | 2023 Feb
REPOSITORIES: biostudies-literature
Publications
Appraising the Quality of Development and Reporting in Surgical Prediction Models.
Marwaha Jayson S JS Chen Hao Wei HW Habashy Karl K Choi Jeff J Spain David A DA Brat Gabriel A GA
JAMA surgery 20230201 2
Similar Datasets
Project description:Background and purpose - External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines.Material and methods - We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting.Results - We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43-89), with 6 items being reported in less than 4/18 of the studies.Interpretation - Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.
| S-EPMC8436968 | biostudies-literature
Project description:BackgroundTo evaluate the reporting quality and methodological quality of dynamic prediction model (DPM) studies on cancer prognosis.MethodsExtensive search for DPM studies on cancer prognosis was conducted in MEDLINE, EMBASE, and the Cochrane Library databases. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) and the Prediction model Risk of Bias Assessment Tool (PROBAST) were used to assess reporting quality and methodological quality, respectively.ResultsA total of 34 DPM studies were identified since the first publication in 2005, the main modeling methods for DPMs included the landmark model and the joint model. Regarding the reporting quality, the median overall TRIPOD adherence score was 75%. The TRIPOD items were poorly reported, especially the title (23.53%), model specification, including presentation (55.88%) and interpretation (50%) of the DPM usage, and implications for clinical use and future research (29.41%). Concerning methodological quality, most studies were of low quality (n = 30) or unclear (n = 3), mainly due to statistical analysis issues.ConclusionsThe Landmark model and joint model show potential in DPM. The suboptimal reporting and methodological qualities of current DPM studies should be improved to facilitate clinical application.
| S-EPMC11872325 | biostudies-literature
Project description:Transparent and accurate reporting is essential to evaluate the validity and applicability of risk prediction models. Our aim was to evaluate the reporting quality of studies developing and validating risk prediction models for melanoma according to the TRIPOD (Transparent Reporting of a multivariate prediction model for Individual Prognosis Or Diagnosis) checklist. We included studies that were identified by a recent systematic review and updated the literature search to ensure that our TRIPOD rating included all relevant studies. Six reviewers assessed compliance with all 37 TRIPOD components for each study using the published "TRIPOD Adherence Assessment Form". We further examined a potential temporal effect of the reporting quality. Altogether 42 studies were assessed including 35 studies reporting the development of a prediction model and seven studies reporting both development and validation. The median adherence to TRIPOD was 57% (range 29% to 78%). Study components that were least likely to be fully reported were related to model specification, title and abstract. Although the reporting quality has slightly increased over the past 35 years, there is still much room for improvement. Adherence to reporting guidelines such as TRIPOD in the publication of study results must be adopted as a matter of course to achieve a sufficient level of reporting quality necessary to foster the use of the prediction models in applications.
| S-EPMC8871554 | biostudies-literature
Project description:IntroductionThread-embedding acupuncture (TEA) is a special type of acupuncture treatment in which medical threads are inserted into subcutaneous tissues or muscles at therapeutic points. TEA is a medical practice that combines acupuncture and medical threads. As such, it is necessary to evaluate the safety of TEA. This systematic review and meta-analysis aimed to assess the safety of TEA and reporting quality of studies regarding TEA.Methods and analysisThe systematic review will be conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and the Cochrane Handbook for Systematic Reviews of Interventions. Searching strategies will be systemically conducted using the following databases from their inception date to September 2022: MEDLINE, Embase, Cochrane Library, Web of Science, China National Knowledge Infrastructure (CNKI), CiNii, J-STAGE, Korean Medical Database, Korean Studies Information Service System (KISS), ScienceON and Oriental Medicine Advanced Searching Integrated System (OASIS). The search strategies will be adjusted for each database as appropriate. The risk of bias will be assessed using the McMaster tool to identify the quality of harm assessment and reporting in study reports (McHarm). A meta-analysis will be used to synthesise the frequency and incidence of adverse events.Ethics and disseminationNo ethical approval and consent is required for this systematic review. The results of this systematic review will be disseminated through peer-reviewed publications and conference presentations.Prospero registration numberCRD42022297123.
| S-EPMC9516149 | biostudies-literature
Project description:IntroductionWhile there are guidelines for reporting on observational studies (eg, Strengthening the Reporting of Observational Studies in Epidemiology, Reporting of Studies Conducted Using Observational Routinely Collected Health Data Statement), estimation of causal effects from both observational data and randomised experiments (eg, A Guideline for Reporting Mediation Analyses of Randomised Trials and Observational Studies, Consolidated Standards of Reporting Trials, PATH) and on prediction modelling (eg, Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis), none is purposely made for deriving and validating models from observational data to predict counterfactuals for individuals on one or more possible interventions, on the basis of given (or inferred) causal structures. This paper describes methods and processes that will be used to develop a Reporting Guideline for Causal and Counterfactual Prediction Models (PRECOG).Methods and analysisPRECOG will be developed following published guidance from the Enhancing the Quality and Transparency of Health Research (EQUATOR) network and will comprise five stages. Stage 1 will be meetings of a working group every other week with rotating external advisors (active until stage 5). Stage 2 will comprise a systematic review of literature on counterfactual prediction modelling for biomedical sciences (registered in Prospective Register of Systematic Reviews). In stage 3, a computer-based, real-time Delphi survey will be performed to consolidate the PRECOG checklist, involving experts in causal inference, epidemiology, statistics, machine learning, informatics and protocols/standards. Stage 4 will involve the write-up of the PRECOG guideline based on the results from the prior stages. Stage 5 will seek the peer-reviewed publication of the guideline, the scoping/systematic review and dissemination.Ethics and disseminationThe study will follow the principles of the Declaration of Helsinki. The study has been registered in EQUATOR and approved by the University of Florida's Institutional Review Board (#202200495). Informed consent will be obtained from the working groups and the Delphi survey participants. The dissemination of PRECOG and its products will be done through journal publications, conferences, websites and social media.
| S-EPMC9214357 | biostudies-literature
Project description:BackgroundCurrent understanding of the onset of inflammatory bowel diseases relies heavily on data derived from animal models of colitis. However, the omission of information concerning the method used makes the interpretation of studies difficult or impossible. We assessed the current quality of methods reporting in 4 animal models of colitis that are used to inform clinical research into inflammatory bowel disease: dextran sulfate sodium, interleukin-10, CD45RB T cell transfer, and 2,4,6-trinitrobenzene sulfonic acid (TNBS).MethodsWe performed a systematic review based on PRISMA guidelines, using a PubMed search (2000-2014) to obtain publications that used a microarray to describe gene expression in colitic tissue. Methods reporting quality was scored against a checklist of essential and desirable criteria.ResultsFifty-eight articles were identified and included in this review (29 dextran sulfate sodium, 15 interleukin-10, 5 T cell transfer, and 16 TNBS; some articles use more than 1 colitis model). A mean of 81.7% (SD = ±7.038) of criteria were reported across all models. Only 1 of the 58 articles reported all essential criteria on our checklist. Animal age, gender, housing conditions, and mortality/morbidity were all poorly reported.ConclusionsFailure to include all essential criteria is a cause for concern; this failure can have large impact on the quality and replicability of published colitis experiments. We recommend adoption of our checklist as a requirement for publication to improve the quality, comparability, and standardization of colitis studies and will make interpretation and translation of data to human disease more reliable.
| S-EPMC4450905 | biostudies-literature
Project description:BackgroundStandardized methods for reporting surgical quality have been described for all the major urological procedures apart from radical nephroureterectomy (RNU).ObjectiveTo propose a tetrafecta criterion for assessing the quality of RNU based on a consensus panel within the Young Association of Urology (YAU) Urothelial Group, and to test the impact of this tetrafecta in a multicenter, large contemporary cohort of patients treated with RNU for upper tract urothelial carcinoma (UTUC).Design setting and participantsThis was a retrospective analysis of 1765 patients with UTUC treated between 2000 and 2021.Outcome measurements and statistical analysisWe interviewed the YAU Urothelial Group to propose and score a list of items to be included in the "RNU-fecta." A ranking was generated for the criteria with the highest sum score. These criteria were applied to a large multicenter cohort of patients. Kaplan-Meier curves were built to evaluate differences in overall survival (OS) rates between groups, and a multivariable logistic regression model was used to find the predictors of achieving the RNU tetrafecta.Results and limitationsThe criteria with the highest score included three surgical items such as negative soft tissue surgical margins, bladder cuff excision, lymph node dissection according to guideline recommendations, and one oncological item defined by the absence of any recurrence in ≤12 mo. These items formed the RNU tetrafecta. Within a median follow-up of 30 mo, 52.6% of patients achieved the RNU tetrafecta. The 5-yr OS rates were significantly higher for patients achieving tetrafecta than for their counterparts (76% vs 51%). Younger age, lower body mass index, and robotic approach were found to be independent predictors of tetrafecta achievement. Conversely, a higher Eastern Cooperative Oncology Group score, higher clinical stage, and bladder cancer history were inversely associated with tetrafecta.ConclusionsHerein, we present a "tetrafecta" composite endpoint that may serve as a potential tool to assess the overall quality of the RNU procedure. Pending external validation, this tool could allow a comparison between surgical series and may be useful for assessing the learning curve of the procedure as well as for evaluating the impact of new technologies in the field.Patient summaryIn this study, a tetrafecta criterion was developed for assessing the surgical quality of radical nephroureterectomy in patients with upper tract urothelial carcinoma. Patients who achieved tetrafecta had higher 5-yr overall survival rates than those who did not.
| S-EPMC9334825 | biostudies-literature
Project description:Prediction models aim to use available data to predict a health state or outcome that has not yet been observed. Prediction is primarily relevant to clinical practice, but is also used in research, and administration. While prediction modeling involves estimating the relationship between patient factors and outcomes, it is distinct from casual inference. Prediction modeling thus requires unique considerations for development, validation, and updating. This document represents an effort from editors at 31 respiratory, sleep, and critical care medicine journals to consolidate contemporary best practices and recommendations related to prediction study design, conduct, and reporting. Herein, we address issues commonly encountered in submissions to our various journals. Key topics include considerations for selecting predictor variables, operationalizing variables, dealing with missing data, the importance of appropriate validation, model performance measures and their interpretation, and good reporting practices. Supplemental discussion covers emerging topics such as model fairness, competing risks, pitfalls of "modifiable risk factors", measurement error, and risk for bias. This guidance is not meant to be overly prescriptive; we acknowledge that every study is different, and no set of rules will fit all cases. Additional best practices can be found in the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines, to which we refer readers for further details.
| S-EPMC7161722 | biostudies-literature
Project description:Hospital quality measures are a vital component of a learning health system, yet they can be costly to report, statistically underpowered, and inconsistent due to poor interrater reliability. Large language models (LLMs) have recently demonstrated impressive performance on health care-related tasks and offer a promising way to provide accurate abstraction of complete charts at scale. To evaluate this approach, we deployed an LLM-based system that ingests Fast Healthcare Interoperability Resources data and outputs a completed Severe Sepsis and Septic Shock Management Bundle (SEP-1) abstraction. We tested the system on a sample of 100 manual SEP-1 abstractions that University of California San Diego Health reported to the Centers for Medicare & Medicaid Services in 2022. The LLM system achieved agreement with manual abstractors on the measure category assignment in 90 of the abstractions (90%; κ=0.82; 95% confidence interval, 0.71 to 0.92). Expert review of the 10 discordant cases identified four that were mistakes introduced by manual abstraction. This pilot study suggests that LLMs using interoperable electronic health record data may perform accurate abstractions for complex quality measures. (Funded by the National Institute of Allergy and Infectious Diseases [1R42AI177108-1] and others.).
| S-EPMC11658346 | biostudies-literature
Project description:BackgroundIndividual-based modeling is a growing technique in the HIV transmission and prevention literature, but insufficient attention has been paid to formally evaluate the quality of reporting in this field. We present reporting recommendations for individual-based models for HIV treatment and prevention, assess the quality of reporting in the existing literature, and comment on the contribution of this model type to HIV policy and prediction.MethodsWe developed reporting recommendations for individual-based HIV transmission mathematical models, and through a systematic search, used them to evaluate the reporting in the existing literature. We identified papers that employed individual-based simulation models and were published in English prior to December 31, 2012. Articles were included if the models they employed simulated and tracked individuals, simulated HIV transmission between individuals in a particular population, and considered a particular treatment or prevention intervention. The papers were assessed with the reporting recommendations.FindingsOf 214 full text articles examined, 32 were included in the evaluation, representing 20 independent individual-based HIV treatment and prevention mathematical models. Manuscripts universally reported the objectives, context, and modeling conclusions in the context of the modeling assumptions and the model's predictive capabilities, but the reporting of individual-based modeling methods, parameterization and calibration was variable. Six papers discussed the time step used and one discussed efforts to maintain internal validity in coding.ConclusionIndividual-based models represent detailed HIV transmission processes with the potential to contribute to inference and policy making for many different regions and populations. The rigor in reporting of assumptions, methods, and calibration of individual-based models focused on HIV transmission and prevention varies greatly. Higher standards for reporting of statistically rigorous calibration and model assumption testing need to be implemented to increase confidence in existing and future modeling results.
| S-EPMC3787035 | biostudies-literature