Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods.
ABSTRACT: Thresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.Methodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.We propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (? the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.If followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.
Project description:INTRODUCTION:Joint arthroplasty is a particularly complex orthopaedic surgical procedure performed on joints, including the hip, knee, shoulder, ankle, elbow, wrist and even digit joints. Increasing evidence from volume-outcomes research supports the finding that patients undergoing joint arthroplasty in high-volume hospitals or by high-volume surgeons achieve better outcomes, and minimum case load requirements have been established in some areas. However, the relationships between hospital/surgeon volume and outcomes in patients undergoing arthroplasty are not fully understood. Furthermore, whether elective arthroplasty should be restricted to high-volume hospitals or surgeons remains in dispute, and little is known regarding where the thresholds should be set for different types of joint arthroplasties. METHODS AND ANALYSES:This is a protocol for a suite of systematic reviews and dose-response meta-analyses, which will be amended and updated in conjunction with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols. Electronic databases, including PubMed and Embase, will be searched for observational studies examining the relationship between the hospital or surgeon volume and clinical outcomes in adult patients undergoing primary or revision of joint arthroplasty. We will use records management software for study selection and a predefined standardised file for data extraction and management. Quality will be assessed using the Newcastle-Ottawa Scale, and the meta-analysis, subgroup analysis and sensitivity analysis will be performed using Stata statistical software. Once the volume-outcome relationships are established, we will examine the potential non-linear relationships between hospital/surgeon volume and outcomes and detect whether thresholds or turning points exist. ETHICS AND DISSEMINATION:Ethical approval is not required, because these studies are based on aggregated published data. The results of this suite of systematic reviews and meta-analyses will be submitted to peer-reviewed journals for publication. PROSPERO REGISTRATION NUMBER:CRD42017056639.
Project description:BACKGROUND:There are no overviews of systematic reviews investigating haemoglobin thresholds for transfusion. This is important as the literature on transfusion thresholds has grown considerably in recent years. Our aim was to synthesise evidence from systematic reviews and meta-analyses of the effects of restrictive and liberal transfusion strategies on mortality. METHODS:This was a systematic review of systematic reviews (overview). We searched MEDLINE, Embase, Web of Science Core Collection, PubMed, Google Scholar, and the Joanna Briggs Institute EBP Database, from 2008 to 2018. We included systematic reviews and meta-analyses of randomised controlled trials comparing mortality in patients assigned to red cell transfusion strategies based on haemoglobin thresholds. Two independent reviewers extracted data and assessed methodological quality. We assessed the methodological quality of included reviews using AMSTAR 2 and the quality of evidence pooled using an algorithm to assign GRADE levels. RESULTS:We included 19 systematic reviews reporting 33 meta-analyses of mortality outcomes from 53 unique randomised controlled trials. Of the 33 meta-analyses, one was graded as high quality, 15 were moderate, and 17 were low. Of the meta-analyses presenting high- to moderate-quality evidence, 12 (75.0%) reported no statistically significant difference in mortality between restrictive and liberal transfusion groups and four (25.0%) reported significantly lower mortality for patients assigned to a restrictive transfusion strategy. We found few systematic reviews addressed clinical differences between included studies: variation was observed in haemoglobin threshold concentrations, the absolute between group difference in haemoglobin threshold concentration, time to randomisation (resulting in transfusions administered prior to randomisation), and transfusion dosing regimens. CONCLUSIONS:Meta-analyses graded as high to moderate quality indicate that in most patient populations no difference in mortality exists between patients assigned to a restrictive or liberal transfusion strategy. TRIAL REGISTRATION:PROSPERO CRD42019120503.
Project description:BACKGROUND: Thresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probability of getting a result assuming an alternative hypothesis to the null hypothesis is true. Second, a confidence interval or a P-value showing significance may be caused by multiplicity. Third, statistical significance does not necessarily result in clinical significance. Therefore, assessment of intervention effects in randomised clinical trials deserves more rigour in order to become more valid. METHODS: Several methodologies for assessing the statistical and clinical significance of intervention effects in randomised clinical trials were considered. Balancing simplicity and comprehensiveness, a simple five-step procedure was developed. RESULTS: For a more valid assessment of results from a randomised clinical trial we propose the following five-steps: (1) report the confidence intervals and the exact P-values; (2) report Bayes factor for the primary outcome, being the ratio of the probability that a given trial result is compatible with a 'null' effect (corresponding to the P-value) divided by the probability that the trial result is compatible with the intervention effect hypothesised in the sample size calculation; (3) adjust the confidence intervals and the statistical significance threshold if the trial is stopped early or if interim analyses have been conducted; (4) adjust the confidence intervals and the P-values for multiplicity due to number of outcome comparisons; and (5) assess clinical significance of the trial results. CONCLUSIONS: If the proposed five-step procedure is followed, this may increase the validity of assessments of intervention effects in randomised clinical trials.
Project description:OBJECTIVE:To assess the effect of publication bias on the results and conclusions of systematic reviews and meta-analyses. DESIGN:Analysis of published meta-analyses by trim and fill method. STUDIES:48 reviews in Cochrane Database of Systematic Reviews that considered a binary endpoint and contained 10 or more individual studies. MAIN OUTCOME MEASURES:Number of reviews with missing studies and effect on conclusions of meta-analyses. RESULTS:The trim and fill fixed effects analysis method estimated that 26 (54%) of reviews had missing studies and in 10 the number missing was significant. The corresponding figures with a random effects model were 23 (48%) and eight. In four cases, statistical inferences regarding the effect of the intervention were changed after the overall estimate for publication bias was adjusted for. CONCLUSIONS:Publication or related biases were common within the sample of meta-analyses assessed. In most cases these biases did not affect the conclusions. Nevertheless, researchers should check routinely whether conclusions of systematic reviews are robust to possible non-random selection mechanisms.
Project description:Background: Pancreatic ductal adenocarcinoma (PDAC) leads to the majority of cancer-related deaths due to its morbidity with similar mortality. Lack of effective prognostic biomarkers are the main reason for belated post-operative intervention of recurrence which causes high mortality. Numerous systematic reviews and meta-analyses have explored the prognostic value of biomarkers in PDAC so far. In this article, we performed an umbrella review analyzing these studies to provide an overview of associations between prognostic biomarkers and PDAC survival outcome and synthesized these results to guide better clinical practice. Methods: Systematic reviews and meta-analyses investigating the associations between PDAC survival outcomes and prognostic biomarkers were acquired via the PubMed and Embase databases from inception till February 1, 2020. Associations supported by nominally statistically significant results were classified into strong, highly suggestive, suggestive, and weak based on several critical factors such as the statistical significance of summary estimates, the number of events, the estimate of the largest study included, interstudy heterogeneity, small-study effects, 95% predictive interval (PI), excess significance bias, and the results of credibility ceiling sensitivity analyses. Results: We included 41 meta-analyses containing 63 associations between PDAC survival outcomes and prognostic biomarkers. Although, none was supported by strong evidence among these associations, an association between C-reactive protein to albumin ratio (CAR) and PDAC overall survival (OS) and an association between neutrophil-lymphocyte ratio (NLR) and PDAC OS were supported by highly suggestive evidence. Otherwise, the association between lactate dehydrogenase (LDH) and PDAC OS was supported by suggestive evidence. The remaining 60 associations were supported by weak or not suggestive evidence. Conclusion: Associations between CAR or NLR and PDAC OS were supported by highly suggestive evidence. And the association between LDH and PDAC OS was supported by suggestive evidence. Although the methodological quality of the included systematic reviews and meta-analyses which were evaluated by AMSTAR2.0 is generally poor, the identification of the relatively robust prognostic biomarkers of PDAC may guide better post-operative intervention and follow-up to prolong patients' survival.
Project description:BACKGROUND: Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews. METHODS/PRINCIPAL FINDINGS: We selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0-19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI -8 to 36). The issue of multiplicity was addressed in 6% of all the reviews. CONCLUSION/SIGNIFICANCE: Statistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.
Project description:BACKGROUND:There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies. METHODS AND FINDINGS:Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93-1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance. CONCLUSIONS:Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors' Summary.
Project description:OBJECTIVE:To simulate possible changes in systematic review results if rapid review methods were used. STUDY DESIGN AND SETTING:We recalculated meta-analyses for binary primary outcomes in Cochrane systematic reviews, simulating rapid review methods. We simulated searching only PubMed, excluding older articles (5, 7, 10, 15, and 20 years before the search date), excluding smaller trials (<50, <100, and <200 participants), and using the largest trial only. We examined percentage changes in pooled odds ratios (ORs) (classed as no important change [<5%], small [<20%], moderate [<30%], or large [?30%]), statistical significance, and biases observed using rapid methods. RESULTS:Two thousand five hundred and twelve systematic reviews (16,088 studies) were included. Rapid methods resulted in the loss of all data in 3.7-44.7% of meta-analyses. Searching only PubMed had the smallest risk of changed ORs (19% [477/2,512] were small changes or greater; 10% [260/2,512] were moderate or greater). Changes in ORs varied substantially with each rapid review method; 8.4-21.3% were small, 1.9-8.8% were moderate, and 4.7-34.1% were large. Changes in statistical significance occurred in 6.5-38.6% of meta-analyses. Changes from significant to nonsignificant were most common (2.1-13.7% meta-analyses). We found no evidence of bias with any rapid review method. CONCLUSION:Searching PubMed only might be considered where a ?10% risk of the primary outcome OR changing by >20% could be tolerated. This could be the case in scoping reviews, resource limitation, or where syntheses are needed urgently. Other situations, such as clinical guidelines and regulatory decisions, favor more comprehensive systematic review methods.
Project description:INTRODUCTION:There has been a significant increase in the number of systematic reviews and meta-analyses of randomised controlled trials investigating thresholds for red blood cell transfusion. To systematically collate, appraise and synthesise the results of these systematic reviews and meta-analyses, we will conduct an overview of systematic reviews. METHODS AND ANALYSIS:This is a protocol for an overview of systematic reviews. We will search five databases: MEDLINE, Embase, Web of Science Core Collection, PubMed (for prepublication, in process and non-Medline records) and Google Scholar. We will consider systematic reviews and meta-analyses of randomised controlled trials evaluating the effect of haemoglobin thresholds for red blood cell transfusion on mortality. Two authors will independently screen titles and abstracts retrieved in the literature search and select studies meeting the eligibility criteria for full-text review. We will extract data onto a predefined form designed to summarise the key characteristics of each review. We will assess the methodological quality of included reviews and the quality of evidence in included reviews. ETHICS AND DISSEMINATION:Formal ethics approval is not required for this overview as we will only analyse published literature. The findings of this study will be presented at relevant conferences and submitted for peer-review publication. The results are likely to be used by clinicians, policy makers and developers of clinical guidelines and will inform suggestions for future systematic reviews and randomised controlled trials. PROSPERO REGISTRATION NUMBER:CRD42019120503.
Project description:BACKGROUND:COVID-19 is a rapidly spreading virus infection that has quickly caused extensive burden to individual, families, countries, and the globe. No intervention has yet been proven effective for the treatment of COVID-19. Some randomized clinical trials assessing the effects of different drugs have been published, and more are currently underway. There is an urgent need for a living, dynamic systematic review that continuously evaluates the beneficial and harmful effects of all available interventions for COVID-19. METHODS/DESIGN:We will conduct a living systematic review based on searches of major medical databases (e.g., MEDLINE, EMBASE, CENTRAL) and clinical trial registries from their inception onwards to identify relevant randomized clinical trials. We will update the literature search once a week to continuously assess if new evidence is available. Two review authors will independently extract data and perform risk of bias assessment. We will include randomized clinical trials comparing any intervention for the treatment of COVID-19 (e.g., pharmacological interventions, fluid therapy, invasive or noninvasive ventilation, or similar interventions) with any comparator (e.g., an "active" comparator, standard care, placebo, no intervention, or "active placebo") for participants in all age groups with a diagnosis of COVID-19. Primary outcomes will be all-cause mortality and serious adverse events. Secondary outcomes will be admission to intensive care, mechanical ventilation, renal replacement therapy, quality of life, and non-serious adverse events. The living systematic review will include aggregate data meta-analyses, Trial Sequential Analyses, network meta-analysis, and individual patient data meta-analyses. Risk of bias will be assessed with domains, an eight-step procedure will be used to assess if the thresholds for clinical significance are crossed, and the certainty of the evidence will be assessed by Grading of Recommendations, Assessment, Development and Evaluations (GRADE). DISCUSSION:COVID-19 has become a pandemic with substantial mortality. A living systematic review evaluating the beneficial and harmful effects of pharmacological and other interventions is urgently needed. This review will continuously inform best practice in treatment and clinical research of this highly prevalent disease. SYSTEMATIC REVIEW REGISTRATION:PROSPERO CRD42020178787.