Dataset Information

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods.

ABSTRACT:

Background

Thresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.

Methods

Methodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.

Results

We propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (≈ the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.

Conclusions

If followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.

SUBMITTER: Jakobsen JC

PROVIDER: S-EPMC4251848 | biostudies-literature | 2014 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods.

Jakobsen Janus Christian JC Wetterslev Jørn J Winkel Per P Lange Theis T Gluud Christian C

BMC medical research methodology 20141121

<h4>Background</h4>Thresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.<h4>Methods</h4>Methodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedur ...[more]

PMID: 25416419

Similar Datasets

Project description:BackgroundThere are no overviews of systematic reviews investigating haemoglobin thresholds for transfusion. This is important as the literature on transfusion thresholds has grown considerably in recent years. Our aim was to synthesise evidence from systematic reviews and meta-analyses of the effects of restrictive and liberal transfusion strategies on mortality.MethodsThis was a systematic review of systematic reviews (overview). We searched MEDLINE, Embase, Web of Science Core Collection, PubMed, Google Scholar, and the Joanna Briggs Institute EBP Database, from 2008 to 2018. We included systematic reviews and meta-analyses of randomised controlled trials comparing mortality in patients assigned to red cell transfusion strategies based on haemoglobin thresholds. Two independent reviewers extracted data and assessed methodological quality. We assessed the methodological quality of included reviews using AMSTAR 2 and the quality of evidence pooled using an algorithm to assign GRADE levels.ResultsWe included 19 systematic reviews reporting 33 meta-analyses of mortality outcomes from 53 unique randomised controlled trials. Of the 33 meta-analyses, one was graded as high quality, 15 were moderate, and 17 were low. Of the meta-analyses presenting high- to moderate-quality evidence, 12 (75.0%) reported no statistically significant difference in mortality between restrictive and liberal transfusion groups and four (25.0%) reported significantly lower mortality for patients assigned to a restrictive transfusion strategy. We found few systematic reviews addressed clinical differences between included studies: variation was observed in haemoglobin threshold concentrations, the absolute between group difference in haemoglobin threshold concentration, time to randomisation (resulting in transfusions administered prior to randomisation), and transfusion dosing regimens.ConclusionsMeta-analyses graded as high to moderate quality indicate that in most patient populations no difference in mortality exists between patients assigned to a restrictive or liberal transfusion strategy.Trial registrationPROSPERO CRD42019120503.

Project description:BackgroundPolicy makers and others often require synthesis of knowledge in an area within six months or less. Traditional systematic reviews typically take at least 12 months to conduct. Rapid reviews streamline traditional systematic review methods in order to synthesize evidence within a shortened timeframe. There is great variation in the process of conducting rapid reviews. This review sought to examine methods used for rapid reviews, as well as implications of methodological streamlining in terms of rigour, bias, and results.MethodsA comprehensive search strategy--including five electronic databases, grey literature, hand searching of relevant journals, and contacting key informants--was undertaken. All titles and abstracts (n = 1,989) were reviewed independently by two reviewers. Relevance criteria included articles published between 1995 and 2009 about conducting rapid reviews or addressing comparisons of rapid reviews versus traditional reviews. Full articles were retrieved for any titles deemed relevant by either reviewer (n = 70). Data were extracted from all relevant methodological articles (n = 45) and from exemplars of rapid review methods (n = 25).ResultsRapid reviews varied from three weeks to six months; various methods for speeding up the process were employed. Some limited searching by years, databases, language, and sources beyond electronic searches. Several employed one reviewer for title and abstract reviewing, full text review, methodological quality assessment, and/or data extraction phases. Within rapid review studies, accelerating the data extraction process may lead to missing some relevant information. Biases may be introduced due to shortened timeframes for literature searching, article retrieval, and appraisal.ConclusionsThis review examined the continuum between diverse rapid review methods and traditional systematic reviews. It also examines potential implications of streamlined review methods. More of these rapid reviews need to be published in the peer-reviewed literature with an emphasis on articulating methods employed. While one consistent methodological approach may not be optimal or appropriate, it is important that researchers undertaking reviews within the rapid to systematic continuum provide detailed descriptions of methods used and discuss the implications of their chosen methods in terms of potential bias introduced. Further research comparing full systematic reviews with rapid reviews will enhance understanding of the limitations of these methods.

Project description:Quality of healthcare can be improved when the best external evidence available is integrated in clinical decision-making in a systematic explicit manner. With the rapid expansion of clinical evidence, the opportunities for evidence-based high-quality healthcare increase. Paradoxically, the likelihood of any one person to get a complete and balanced picture of the evidence available decreases. This is especially true for rehabilitation interventions that are complex in nature and where clinical research is rather diverse. Given the complex nature of the evidence, there is a substantial risk of misinterpreting the complex information both at the level of individual sources (e.g., reports of clinical trials) and for aggregated data syntheses (e.g., systematic reviews and meta-analyses). These risks are inherent in these sources themselves and are in addition related to the methodological expertise necessary to make valid use of the evidence for clinical decision-making. Taken together, there is a great demand for systematic structured guidance from evidence to clinical decision. This methodology paper describes a structured process for the development and report of evidence-based clinical practice recommendations that uses systematic reviews and meta-analyses as evidence source. It provides a comprehensive framework with specific requirements for the development group, the formulation of the healthcare question addressed, the systematic search for the evidence, its critical appraisal, the extraction and the outcome-centered presentation of the evidence, the rating of its quality, strengths and weaknesses, any further considerations relevant for decision-making, and an explicit recommendation statement along with its justification, implementation, and resource aspects. The suggested methodology uses international standards in evidence synthesis, critical appraisal of systematic reviews, rating the quality of evidence, characteristics of recommendations, and guideline development as developed by Cochrane, GRADE (Grading of Recommendations Assessment, Development and Evaluation), AMSTAR (A MeaSurement Tool to Assess systematic Reviews), and AGREE (Appraisal of Guidelines for REsearch & Evaluation). An added distinctive feature of the methodology is to focus on the most up-to-date, most valid evidence and hence to support the development of valid practice recommendations in an efficient way. Practice recommendations generated by such a valid methodology would be generally applicable and promote evidence-based clinical practice globally.

Project description:BackgroundThe benefits of stroke unit care in terms of reducing death, dependency and institutional care were demonstrated in a 2009 Cochrane review carried out by the Stroke Unit Trialists' Collaboration.MethodsAs requested by the Belgian health authorities, a systematic review and meta-analysis of the effect of acute stroke units was performed. Clinical trials mentioned in the original Cochrane review were included. In addition, an electronic database search on Medline, Embase, the Cochrane Central Register of Controlled Trials, and Physiotherapy Evidence Database (PEDro) was conducted to identify trials published since 2006. Trials investigating acute stroke units compared to alternative care were eligible for inclusion. Study quality was appraised according to the criteria recommended by Scottish Intercollegiate Guidelines Network (SIGN) and the GRADE system. In the meta-analysis, dichotomous outcomes were estimated by calculating odds ratios (OR) and continuous outcomes were estimated by calculating standardized mean differences. The weight of a study was calculated based on inverse variance.ResultsEvidence from eight trials comparing acute stroke unit and conventional care (general medical ward) were retained for the main synthesis and analysis. The findings from this study were broadly in line with the original Cochrane review: acute stroke units can improve survival and independency, as well as reduce the chance of hospitalization and the length of inpatient stay. The improvement with stroke unit care on mortality was less conclusive and only reached borderline level of significance (OR 0.84, 95% CI 0.70 to 1.00, P = 0.05). This improvement became statistically non-significant (OR 0.87, 95% CI 0.74 to 1.03, P = 0.12) when data from two unpublished trials (Goteborg-Ostra and Svendborg) were added to the analysis. After further also adding two additional trials (Beijing, Stockholm) with very short observation periods (until discharge), the difference between acute stroke units and general medical wards on death remained statistically non-significant (OR 0.86, 95% CI 0.74 to 1.01, P = 0.06). Furthermore, based on figures reported by the clinical trials included in this study, a slightly higher proportion of patients became dependent after receiving care in stroke units than those treated in general medical wards - although the difference was not statistically significant. This result could have an impact on the future demand for healthcare services for individuals that survive a stroke but became dependent on their care-givers.ConclusionsThese findings demonstrate that a well-conducted meta-analysis can produce results that can be of value to policymakers but the choice of inclusion/exclusion criteria and outcomes in this context needs careful consideration. The financing of interventions such as stroke units that increase independency and reduce inpatient stays are worthwhile in a context of an ageing population with increasing care needs. One limitation of this study was the selection of trials published in only four languages: English, French, Dutch and German. This choice was pragmatic in the context of this study, where the objective was to support health authorities in their decision processes.

Dataset Information

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods.

Background

Methods

Results

Conclusions

Publications

Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets