Dataset Information

Incorporating published univariable associations in diagnostic and prognostic modeling.

ABSTRACT:

Background

Diagnostic and prognostic literature is overwhelmed with studies reporting univariable predictor-outcome associations. Currently, methods to incorporate such information in the construction of a prediction model are underdeveloped and unfamiliar to many researchers.

Methods

This article aims to improve upon an adaptation method originally proposed by Greenland (1987) and Steyerberg (2000) to incorporate previously published univariable associations in the construction of a novel prediction model. The proposed method improves upon the variance estimation component by reconfiguring the adaptation process in established theory and making it more robust. Different variants of the proposed method were tested in a simulation study, where performance was measured by comparing estimated associations with their predefined values according to the Mean Squared Error and coverage of the 90% confidence intervals.

Results

Results demonstrate that performance of estimated multivariable associations considerably improves for small datasets where external evidence is included. Although the error of estimated associations decreases with increasing amount of individual participant data, it does not disappear completely, even in very large datasets.

Conclusions

The proposed method to aggregate previously published univariable associations with individual participant data in the construction of a novel prediction models outperforms established approaches and is especially worthwhile when relatively limited individual participant data are available.

SUBMITTER: Debray TP

PROVIDER: S-EPMC3548751 | biostudies-literature | 2012 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Incorporating published univariable associations in diagnostic and prognostic modeling.

Debray Thomas P A TP Koffijberg Hendrik H Lu Difei D Vergouwe Yvonne Y Steyerberg Ewout W EW Moons Karel G M KG

BMC medical research methodology 20120810

<h4>Background</h4>Diagnostic and prognostic literature is overwhelmed with studies reporting univariable predictor-outcome associations. Currently, methods to incorporate such information in the construction of a prediction model are underdeveloped and unfamiliar to many researchers.<h4>Methods</h4>This article aims to improve upon an adaptation method originally proposed by Greenland (1987) and Steyerberg (2000) to incorporate previously published univariable associations in the construction o ...[more]

PMID: 22883206

Similar Datasets

Project description:ObjectiveWe aimed to improve prediction of outcome for patients with colorectal liver metastases, via prognostic models incorporating PET-derived measures, including radiomic features that move beyond conventional standard uptake value (SUV) measures.Patients and methodsA range of parameters including volumetric and heterogeneity measures were derived from FDG PET images of 52 patients with colorectal intrahepatic-only metastases (29 males and 23 females; mean age 62.9 years [SD 9.8; range 32-82]). The patients underwent PET/CT imaging as part of the clinical workup prior to final decision on treatment. Univariate and multivariate models were implemented, which included statistical considerations (to discourage false discovery and overfitting), to predict overall survival (OS), progression-free survival (PFS) and event-free survival (EFS). Kaplan-Meier survival analyses were performed, where the subjects were divided into high-risk and low-risk groups, from which the hazard ratios (HR) were computed via Cox proportional hazards regression.ResultsCommonly-invoked SUV metrics performed relatively poorly for different prediction tasks (SUVmax HR = 1.48, 0.83 and 1.16; SUVpeak HR = 2.05, 1.93, and 1.64, for OS, PFS and EFS, respectively). By contrast, the number of liver metastases and metabolic tumor volume (MTV) each performed well (with respective HR values of 2.71, 2.61 and 2.42, and 2.62, 1.96 and 2.29, for OS, PFS and EFS). Total lesion glycolysis (TLG) also resulted in similar performance as MTV. Multivariate prognostic modeling incorporating different features (including those quantifying intra-tumor heterogeneity) resulted in further enhanced prediction. Specifically, HR values of 4.29, 4.02 and 3.20 (p-values = 0.00004, 0.0019 and 0.0002) were obtained for OS, PFS and EFS, respectively.ConclusionsPET-derived measures beyond commonly invoked SUV parameters hold significant potential towards improved prediction of clinical outcome in patients with liver metastases, especially when utilizing multivariate models.

Project description:ObjectivesPrognostic and diagnostic models must work in their intended clinical setting, proven via "external evaluation", preferably by authors uninvolved with model development. By systematic review, we determined the proportion of models published in high-impact radiological journals that are evaluated subsequently.MethodsWe hand-searched three radiological journals for multivariable diagnostic/prognostic models 2013-2015 inclusive, developed using regression. We assessed completeness of data presentation to allow subsequent external evaluation. We then searched literature to August 2022 to identify external evaluations of these index models.ResultsWe identified 98 index studies (73 prognostic; 25 diagnostic) describing 145 models. Only 15 (15%) index studies presented an evaluation (two external). No model was updated. Only 20 (20%) studies presented a model equation. Just 7 (15%) studies developing Cox models presented a risk table, and just 4 (9%) presented the baseline hazard. Two (4%) studies developing non-Cox models presented the intercept. Just 20 (20%) articles presented a Kaplan-Meier curve of the final model. The 98 index studies attracted 4224 citations (including 559 self-citations), median 28 per study. We identified just six (6%) subsequent external evaluations of an index model, five of which were external evaluations by researchers uninvolved with model development, and from a different institution.ConclusionsVery few prognostic or diagnostic models published in radiological literature are evaluated externally, suggesting wasted research effort and resources. Authors' published models should present data sufficient to allow external evaluation by others. To achieve clinical utility, researchers should concentrate on model evaluation and updating rather than continual redevelopment.Clinical relevance statementThe large majority of prognostic and diagnostic models published in high-impact radiological journals are never evaluated. It would be more efficient for researchers to evaluate existing models rather than practice continual redevelopment.Key points• Systematic review of highly cited radiological literature identified few diagnostic or prognostic models that were evaluated subsequently by researchers uninvolved with the original model. • Published radiological models frequently omit important information necessary for others to perform an external evaluation: Only 20% of studies presented a model equation or nomogram. • A large proportion of research citing published models focuses on redevelopment and ignores evaluation and updating, which would be a more efficient use of research resources.

Project description:BackgroundObservational studies show that liver enzymes are diabetes risk factors. However, previous observational investigations on the relationship between liver enzymes and diabetic microvascular complications produced contradictory results. The purpose of this research is to examine the independent causal effects of liver enzymes on diabetic microvascular complications.MethodsUnivariable Mendelian randomization (UVMR) and multivariable Mendelian randomization (MVMR) were utilized to disentangle the causal effects. The genome-wide association study (GWAS) summary-level statistics were collected from the UK biobank and the FinnGen consortium. Single nucleotide polymorphisms (SNPs) were selected as genetic instruments with genome-wide significance (p < 5 ×10-8). Five UVMR approaches, including inverse variance weighted (IVW), Bayesian weighted Mendelian randomization, MR-Pleiotropy Residual Sum and Outlier (MR-PRESSO), weighted median, and MR-Egger, and three MVMR approaches, including the extended versions of IVW, MR-Egger, and the Q-minimization methods, were performed to evaluate the causal effects. The robustness of the MR results was further confirmed using several sensitivity analyses.ResultsUVMR revealed that a genetically predisposed per standard deviation increase in serum alanine aminotransferase (ALT) level increased the risk of diabetic retinopathy (DR) in type 2 diabetes mellitus (T2DM) (IVW OR = 1.489, 95% CI = 1.206-1.772, p = 0.006). Likewise, serum aspartate aminotransferase (AST) levels showed similar results (IVW OR = 1.376, 95% CI = 1.115-1.638, p = 0.017). Furthermore, these effects were consistent after controlling for glycemia and blood pressure using MVMR analysis. Additionally, sensitivity analyses further strengthened the causality. However, no significant associations were found between alkaline phosphatase (ALP), gamma-glutamyl transferase (GGT), and diabetic microvascular complications.ConclusionsRobust evidence was demonstrated for an independent causal effect of serum ALT or AST concentration on the risk of DR in T2DM. Further investigations are necessary to elucidate the potential biological mechanisms and confirm their clinical significance for early prevention and intervention.

Project description:BackgroundThe reporting of machine learning (ML) prognostic and diagnostic modeling studies is often inadequate, making it difficult to understand and replicate such studies. To address this issue, multiple consensus and expert reporting guidelines for ML studies have been published. However, these guidelines cover different parts of the analytics lifecycle, and individually, none of them provide a complete set of reporting requirements.ObjectiveWe aimed to consolidate the ML reporting guidelines and checklists in the literature to provide reporting items for prognostic and diagnostic ML in in-silico and shadow mode studies.MethodsWe conducted a literature search that identified 192 unique peer-reviewed English articles that provide guidance and checklists for reporting ML studies. The articles were screened by their title and abstract against a set of 9 inclusion and exclusion criteria. Articles that were filtered through had their quality evaluated by 2 raters using a 9-point checklist constructed from guideline development good practices. The average κ was 0.71 across all quality criteria. The resulting 17 high-quality source papers were defined as having a quality score equal to or higher than the median. The reporting items in these 17 articles were consolidated and screened against a set of 6 inclusion and exclusion criteria. The resulting reporting items were sent to an external group of 11 ML experts for review and updated accordingly. The updated checklist was used to assess the reporting in 6 recent modeling papers in JMIR AI. Feedback from the external review and initial validation efforts was used to improve the reporting items.ResultsIn total, 37 reporting items were identified and grouped into 5 categories based on the stage of the ML project: defining the study details, defining and collecting the data, modeling methodology, model evaluation, and explainability. None of the 17 source articles covered all the reporting items. The study details and data description reporting items were the most common in the source literature, with explainability and methodology guidance (ie, data preparation and model training) having the least coverage. For instance, a median of 75% of the data description reporting items appeared in each of the 17 high-quality source guidelines, but only a median of 33% of the data explainability reporting items appeared. The highest-quality source articles tended to have more items on reporting study details. Other categories of reporting items were not related to the source article quality. We converted the reporting items into a checklist to support more complete reporting.ConclusionsOur findings supported the need for a set of consolidated reporting items, given that existing high-quality guidelines and checklists do not individually provide complete coverage. The consolidated set of reporting items is expected to improve the quality and reproducibility of ML modeling studies.

Project description:BackgroundAccumulating evidence suggests that the gut microbiota and its metabolites may be involved in autoimmune hypothyroidism. However, the causal association between gut microbiota, metabolites and autoimmune hypothyroidism remains to be determined.MethodsInstrumental variables were screened from the GWAS datasets of 211 gut microbiota taxonomic groups, gut microbiota-derived metabolites, and autoimmune hypothyroidism. Univariable Mendelian randomization (MR) and multivariable Mendelian randomization (MVMR) were used to analyse the potential causal relationship between autoimmune hypothyroidism, these metabolites, or these microbiota. During the MR analysis, we alternated multiple MR methods with different model assumptions to assess the consistency and robustness of the findings: inverse variance weighted (IVW), weighted median, MR pleiotropy residual sum and outlier (MRPRESSO) and MR-Egger methods. Reverse MR analysis was performed to assess the possibility of reverse causality. Finally, enrichment analyses were used to investigate potential biofunctions.ResultsThe IVW results of univariable MR showed that the phyla Actinobacteria, genus DefluviitaleaceaeUCG011, genus Eggerthella, family Defluviitaleaceae, genus Subdoligranulum, genus RuminococcaceaeUCG011, and genus Intestinimonas were associated with autoimmune hypothyroidism. After FDR adjustment, the absence of a causal relationship between gut microbiota and autoimmune hypothyroidism (PFDR > 0.05) suggested a possible marginal association. The results on gut metabolites showed that N-(3-furoyl)glycine, pipecolate, phenylalanine, allantoin, indololactate and alanine were associated with autoimmune hypothyroidism. After FDR correction, only indololactate was associated with hypothyroidism (OR=1.592; 95% CI, 1.228-2.065; PFDR= 0.036). Family Defluviitaleaceae and genus DefluviitaleaceaeUCG011 were suggestively significant in the MVMR. The results of reverse MR analysis showed no reverse causality between autoimmune hypothyroidism and the identified gut microbiota. Enrichment analysis revealed that several key regulatory pathways were significantly enriched.ConclusionThis study supported that there were beneficial or detrimental causal effects of gut microbiota and its metabolites on autoimmune hypothyroidism risk, which provides more theoretical support for mechanistic research on the "thyroid-gut" axis.

Dataset Information

Incorporating published univariable associations in diagnostic and prognostic modeling.

Background

Methods

Results

Conclusions

Publications

Incorporating published univariable associations in diagnostic and prognostic modeling.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets