Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study.
ABSTRACT: BACKGROUND: There is no consensus on how authors conducting meta-analysis should deal with trial participants with missing outcome data. The objectives of this study are to assess in Cochrane and non-Cochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having missing participant data (MPD), (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their meta-analyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in meta-analyses of dichotomous outcomes. METHODS/DESIGN: We will conduct a methodological study of Cochrane and non-Cochrane systematic reviews. Eligible systematic reviews will include a group-level meta-analysis of a patient-important dichotomous efficacy outcome, with a statistically significant effect estimate. Teams of two reviewers will determine eligibility and subsequently extract information from each eligible systematic review in duplicate and independently, using standardized, pre-piloted forms. The teams will then use a similar process to extract information from the trials included in the meta-analyses of interest. We will assess first which categories of trial participants the systematic reviewers consider as having MPD. Second, we will assess how trialists reported on participants with missing outcome data in trials. Third, we will compare what systematic reviewers report having done, and what they actually did, in dealing with MPD in their meta-analysis. Fourth, we will conduct imputation studies to assess the effects of different methods of dealing with MPD on the pooled effect estimates of meta-analyses. We will specifically calculate for each method (1) the percentage of systematic reviews that lose statistical significance and (2) the mean change of effect estimates across systematic reviews. DISCUSSION: The impact of different methods of dealing with MPD on pooled effect estimates will help judge the associated risk of bias in systematic reviews. Our findings will inform recommendations regarding what assumptions for MPD should be used to test the robustness of meta-analytical results.
Project description:BACKGROUND: Systematic reviewers may encounter a multiplicity of outcome data in the reports of randomised controlled trials included in the review (for example, multiple measurement instruments measuring the same outcome, multiple time points, and final and change from baseline values). The primary objectives of this study are to investigate in a cohort of systematic reviews of randomised controlled trials of interventions for rheumatoid arthritis, osteoarthritis, depressive disorders and anxiety disorders: (i) how often there is multiplicity of outcome data in trial reports; (ii) the association between selection of trial outcome data included in a meta-analysis and the magnitude and statistical significance of the trial result, and; (iii) the impact of the selection of outcome data on meta-analytic results. METHODS/DESIGN: Forty systematic reviews (20 Cochrane, 20 non-Cochrane) of RCTs published from January 2010 to January 2012 and indexed in the Cochrane Database of Systematic Reviews (CDSR) or PubMed will be randomly sampled. The first meta-analysis of a continuous outcome within each review will be included. From each review protocol (where available) and published review we will extract information regarding which types of outcome data were eligible for inclusion in the meta-analysis (for example, measurement instruments, time points, analyses). From the trial reports we will extract all outcome data that are compatible with the meta-analysis outcome as it is defined in the review and with the outcome data eligibility criteria and hierarchies in the review protocol. The association between selection of trial outcome data included in a meta-analysis and the magnitude and statistical significance of the trial result will be investigated. We will also investigate the impact of the selected trial result on the magnitude of the resulting meta-analytic effect estimates. DISCUSSION: The strengths of this empirical study are that our objectives and methods are pre-specified and transparent. The results may inform methods guidance for systematic review conduct and reporting, particularly for dealing with multiplicity of randomised controlled trial outcome data.
Project description:<h4>Objective</h4>To quantify bias related to specific methodological characteristics in child-relevant randomized controlled trials (RCTs).<h4>Design</h4>Meta-epidemiological study.<h4>Data sources</h4>We identified systematic reviews containing a meta-analysis with 10-40 RCTs that were relevant to child health in the Cochrane Database of Systematic Reviews.<h4>Data extraction</h4>Two reviewers independently assessed RCTs using items in the Cochrane Risk of Bias tool and other study factors. We used meta-epidemiological methods to assess for differences in effect estimates between studies classified as high/unclear vs. low risk of bias.<h4>Results</h4>We included 287 RCTs from 17 meta-analyses. The proportion of studies at high/unclear risk of bias was: 79% sequence generation, 83% allocation concealment, 67% blinding of participants, 47% blinding of outcome assessment, 49% incomplete outcome data, 32% selective outcome reporting, 44% other sources of bias, 97% overall risk of bias, 56% funding, 35% baseline imbalance, 13% blocked randomization in unblinded trials, and 1% early stopping for benefit. We found no significant differences in effect estimates for studies that were high/unclear vs. low risk of bias for any of the risk of bias domains, overall risk of bias, or other study factors.<h4>Conclusions</h4>We found no differences in effect estimates between studies based on risk of bias. A potential explanation is the number of trials included, in particular the small number of studies with low risk of bias. Until further evidence is available, reviewers should not exclude RCTs from systematic reviews and meta-analyses based solely on risk of bias particularly in the area of child health.
Project description:BACKGROUND:To provide empirical evidence about prevalence, reporting and handling of missing outcome data in systematic reviews with network meta-analysis and acknowledgement of their impact on the conclusions. METHODS:We conducted a systematic survey including all published systematic reviews of randomized controlled trials comparing at least three interventions from January 1, 2009 until March 31, 2017. RESULTS:We retrieved 387 systematic reviews with network meta-analysis. Description of missing outcome data was available in 63 reviews. Intention-to-treat analysis was the most prevalent method (71%), followed by missing outcome data investigated as secondary outcome (e.g., acceptability) (40%). Bias due to missing outcome data was evaluated in half the reviews with explicit judgments in 18 (10%) reviews. Only 88 reviews interpreted their results acknowledging the implications of missing outcome data and mostly using the network meta-analysis results on missing outcome data as secondary outcome. We were unable to judge the actual strategy applied to deal with missing outcome data in 65% of the reviews due to insufficient information. Six percent of network meta-analyses were re-analyzed in sensitivity analysis considering missing outcome data, while 4% explicitly justified the strategy for dealing with missing outcome data. CONCLUSIONS:The description and handling of missing outcome data as well as the acknowledgment of their implications for the conclusions from network meta-analysis are deemed underreported.
Project description:The main objective was to assess the credibility of the evidence using Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) in oral health systematic reviews on the Cochrane Database of Systematic Reviews (CDSR) and elsewhere.Systematic Reviews or meta-analyses (January 2008-December 2013) from 14 high impact general dental and specialty dental journals and the Cochrane Database of Systematic Reviews were screened for meta-analyses. Data was collected at the systematic review, meta-analysis and trial level. Two reviewers applied and agreed on the GRADE rating for the selected meta-analyses.From the 510 systematic reviews initially identified 91 reviews (41 Cochrane and 50 non-Cochrane) were eligible for inclusion. The quality of evidence was high in 2% and moderate in 18% of the included meta-analyses with no difference between Cochrane and non-Cochrane reviews, journal impact factor or year of publication. The most common domains prompting downgrading of the evidence were study limitations (risk of bias) and imprecision (risk of play of chance).The quality of the evidence in oral health assessed using GRADE is predominantly low or very low suggesting a pressing need for more randomised clinical trials and other studies of higher quality in order to inform clinical decisions thereby reducing the risk of instituting potentially ineffective and/or harmful therapies.
Project description:Pharmacological thromboprophylaxis in the peri-operative period involves a trade-off between reduction in venous thromboembolism (VTE) and an increase in bleeding. Baseline risks, in the absence of prophylaxis, for VTE and bleeding are known to vary widely between urological procedures, but their magnitude is highly uncertain. Systematic reviews and meta-analyses addressing baseline risks are uncommon, needed, and require methodological innovation. In this article, we describe the rationale and methods for a series of systematic reviews of the risks of symptomatic VTE and bleeding requiring reoperation in urological surgery.We searched MEDLINE from January 1, 2000 until April 10, 2014 for observational studies reporting on symptomatic VTE or bleeding after urological procedures. Additional studies known to experts and studies cited in relevant review articles were added. Teams of two reviewers, independently assessed articles for eligibility, evaluated risk of bias, and abstracted data. We derived best estimates of risk from the median estimates among studies rated at the lowest risk of bias. The primary endpoints were 30-day post-operative risk estimates of symptomatic VTE and bleeding requiring reoperation, stratified by procedure and patient risk factors.This series of systematic reviews will inform clinicians and patients regarding the trade-off between VTE prevention and bleeding. Our work advances standards in systematic reviews of surgical complications, including assessment of risk of bias, criteria for arriving at best estimates of risk (including modeling of timing of events and dealing with suboptimal data reporting), dealing with subgroups at higher and lower risk of bias, and use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to rate certainty in estimates of risk. The results will be incorporated in the upcoming European Association Urology Guideline on Thromboprophylaxis.PROSPERO CRD42014010342.
Project description:OBJECTIVE:To assess the risk of bias associated with missing outcome data in systematic reviews. DESIGN:Imputation study. SETTING:Systematic reviews. POPULATION:100 systematic reviews that included a group level meta-analysis with a statistically significant effect on a patient important dichotomous efficacy outcome. MAIN OUTCOME MEASURES:Median percentage change in the relative effect estimate when applying each of the following assumption (four commonly discussed but implausible assumptions (best case scenario, none had the event, all had the event, and worst case scenario) and four plausible assumptions for missing data based on the informative missingness odds ratio (IMOR) approach (IMOR 1.5 (least stringent), IMOR 2, IMOR 3, IMOR 5 (most stringent)); percentage of meta-analyses that crossed the threshold of the null effect for each method; and percentage of meta-analyses that qualitatively changed direction of effect for each method. Sensitivity analyses based on the eight different methods of handling missing data were conducted. RESULTS:100 systematic reviews with 653 randomised controlled trials were included. When applying the implausible but commonly discussed assumptions, the median change in the relative effect estimate varied from 0% to 30.4%. The percentage of meta-analyses crossing the threshold of the null effect varied from 1% (best case scenario) to 60% (worst case scenario), and 26% changed direction with the worst case scenario. When applying the plausible assumptions, the median percentage change in relative effect estimate varied from 1.4% to 7.0%. The percentage of meta-analyses crossing the threshold of the null effect varied from 6% (IMOR 1.5) to 22% (IMOR 5) of meta-analyses, and 2% changed direction with the most stringent (IMOR 5). CONCLUSION:Even when applying plausible assumptions to the outcomes of participants with definite missing data, the average change in pooled relative effect estimate is substantive, and almost a quarter (22%) of meta-analyses crossed the threshold of the null effect. Systematic review authors should present the potential impact of missing outcome data on their effect estimates and use this to inform their overall GRADE (grading of recommendations assessment, development, and evaluation) ratings of risk of bias and their interpretation of the results.
Project description:<h4>Background</h4>Well-constructed systematic reviews and meta-analyses are key tools in evidenced-based healthcare. However, a common problem with performing a meta-analysis is missing data, such as standard deviations (SD). An increasing number of methods are utilised to calculate or impute missing SDs, allowing these studies to be included in analyses. The aim of this review was to investigate the methods reported and utilised for handling missing change SDs in meta-analyses, using the topic of exercise therapy in heart failure patients as a model.<h4>Methods</h4>A systematic search of PubMed, EMBASE and Cochrane Library from 1 January 2014 to 31st March 2018 was conducted for meta-analyses of exercise based trials in heart failure. Studies were eligible to be included if they performed a meta-analysis of change in exercise capacity in heart failure patients after a training intervention.<h4>Results</h4>Twenty two publications performed a meta-analysis on the effect of exercise therapy on exercise capacity in heart failure patients. Eleven (50%) publications did not directly report the approach for dealing with missing change SDs. Approaches reported and utilised to deal with missing change SDs included imputation, actual and approximate algebraic recalculation using study level summary statistics and exclusion of studies.<h4>Conclusion</h4>Change SDs are often not reported in trial papers and while in the first instance meta-analysts should attempt to obtain missing data from trial authors, this information is frequently not forthcoming. Meta-analysts are then forced to make a decision on how these trials and missing data are to be handled. Whilst not one approach is favoured for dealing with this matter, authors need to clearly report the approach to be utilised for missing change SDs. Where change SDs are imputed meta-analyst are encouraged to explore several options and have a sound rationale as to the choice, and where data is imputed, sensitivity analysis should be conducted.
Project description:OBJECTIVES:To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS:Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (?). RESULTS:In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (? ?=? 0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median ? ?= 0.19), ranging from ? ?=? -0.04 ("Other bias") to ? ?=? 0.62 ("Sequence generation"). There was also no agreement (? ?=? -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS:Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.
Project description:Systematic reviews of treatment interventions in stable or chronic conditions often require the synthesis of clinical trials with a cross-over design. Previous work has indicated that methodology for analysing cross-over data is inadequate in trial reports and in systematic reviews assessing trials with this design.We assessed systematic review methodology for synthesising cross-over trials among Cochrane Cystic Fibrosis and Genetic Disorders Group reviews published to July 2015, and assessed the quality of reporting among the cross-over trials included in these reviews.We performed data extraction of methodology and reporting in reviews, trials identified and trials included within reviews.We reviewed a total of 142 Cochrane systematic reviews including 53 reviews which synthesised evidence from 218 cross-over trials. Thirty-three (63%) Cochrane reviews described a clear and appropriate method for the inclusion of cross-over data, and of these 19 (56%) used the same method to analyse results. 145 cross-over trials were described narratively or treated as parallel trials in reviews but in 30 (21%) of these trials data existed in the trial reports to account for the cross-over design. At the trial level, the analysis and presentation of results were often inappropriate or unclear, with only 69 (32%) trials presenting results that could be included in meta-analysis.Despite development of accessible, technical guidance and training for Cochrane systematic reviewers, statistical analysis and reporting of cross-over data is inadequate at both the systematic review and the trial level. Plain language and practical guidance for the inclusion of cross-over data in meta-analysis would benefit systematic reviewers, who come from a wide range of health specialties. Minimum reporting standards for cross-over trials are needed.
Project description:BACKGROUND:Stringent requirements exist regarding the transparency of the study selection process and the reliability of results. A 2-step selection process is generally recommended; this is conducted by 2 reviewers independently of each other (conventional double-screening). However, the approach is resource intensive, which can be a problem, as systematic reviews generally need to be completed within a defined period with a limited budget. The aim of the following methodological systematic review was to analyse the evidence available on whether single screening is equivalent to double screening in the screening process conducted in systematic reviews. METHODS:We searched Medline, PubMed and the Cochrane Methodology Register (last search 10/2018). We also used supplementary search techniques and sources ("similar articles" function in PubMed, conference abstracts and reference lists). We included all evaluations comparing single with double screening. Data were summarized in a structured, narrative way. RESULTS:The 4 evaluations included investigated a total of 23 single screenings (12 sets for screening involving 9 reviewers). The median proportion of missed studies was 5% (range 0 to 58%). The median proportion of missed studies was 3% for the 6 experienced reviewers (range: 0 to 21%) and 13% for the 3 reviewers with less experience (range: 0 to 58%). The impact of missing studies on the findings of meta-analyses had been reported in 2 evaluations for 7 single screenings including a total of 18,148 references. In 3 of these 7 single screenings - all conducted by the same reviewer (with less experience) - the findings would have changed substantially. The remaining 4 of these 7 screenings were conducted by experienced reviewers and the missing studies had no impact or a negligible on the findings of the meta-analyses. CONCLUSIONS:Single screening of the titles and abstracts of studies retrieved in bibliographic searches is not equivalent to double screening, as substantially more studies are missed. However, in our opinion such an approach could still represent an appropriate methodological shortcut in rapid reviews, as long as it is conducted by an experienced reviewer. Further research on single screening is required, for instance, regarding factors influencing the number of studies missed.