Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews.
ABSTRACT: BACKGROUND AND OBJECTIVE:Randomized trials may be designed to provide evidence more strongly related to efficacy or effectiveness of an intervention. When systematic reviews are used to inform clinical or policy decisions, it is important to know the efficacy-effectiveness nature of the included trials. The objective of this study was to develop a tool to characterize randomized trials included in a systematic review on an efficacy-effectiveness continuum. METHODS:We extracted rating domains and descriptors from existing tools and used a modified Delphi procedure to condense the domains and develop a new tool. The feasibility and interrater reliability of the tool was tested on trials from four systematic reviews. RESULTS:The Rating of Included Trials on the Efficacy-Effectiveness Spectrum (RITES) tool rates clinical trials on a five-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions, and (4) clinical relevance of interventions. When RITES was piloted on trials from three reviews by unaffiliated raters, ratings were variable (intraclass correlation coefficient [ICC] 0.25-0.66 for the four domains); but, when RITES was used on one review by the review authors with expertise on the topic, the ratings were consistent (ICCs > 0.80. CONCLUSION:RITES may help to characterize the efficacy-effectiveness nature of trials included in systematic reviews.
Project description:OBJECTIVES:To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS:Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (?). RESULTS:In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (? ?=? 0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median ? ?= 0.19), ranging from ? ?=? -0.04 ("Other bias") to ? ?=? 0.62 ("Sequence generation"). There was also no agreement (? ?=? -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS:Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.
Project description:This study tests the feasibility of applying the pragmatic-explanatory continuum indicator summary (version "PRECIS-2") tool to randomized controlled trials of Chinese herbal medicine. A search was conducted to identify potentially eligible randomized controlled trials. Using the PRECIS-2 tool, assessment of trials was performed independently by 2 evaluators using a scale of 1-5 for each criterion (1 = maximal efficacy, 5 = maximal effectiveness). A total of 7,166 reports were retrieved from databases and 159 were included in the full text. Though PRECIS-2 describes quantitative scoring in detail, evaluators were uncertain about several specific operationalizations and found high evaluator variation in the first independent ratings. After discussion and reaching consensus, inter-evaluator reliability improved. For PRECIS-2 ratings over time, there was no evidence that the design and performance of RCTs of CHM paid more attention to "efficacy" criteria after the implementation of PRECIS (all P > 0.05). More research is needed to establish the easiest and most useful tool to distinguish between effectiveness and efficacy results.
Project description:BACKGROUND:In the long term, smoking cessation can decrease the risk of cancer, stroke, and heart attacks and improve overall survival. The aim of the proposed umbrella review is to summarize existing systematic reviews that assessed the effects of pharmacological interventions for smoking cessation and to evaluate the methodological quality of previously conducted systematic reviews. METHODS:Databases such as the Cochrane Library, PubMed, MEDLINE, EMBASE, CINAHIL PsychINFO Web of Science, Conference Papers Index, Scopus, and Google Scholar will be used to retrieve reviews. Systematic reviews which included only randomized control trials will be considered in this review. The primary outcome will be prolonged abstinence from smoking for a minimum of 6 months follow-up, and the secondary outcome will be point abstinence rate from smoking of less than 6 months follow-up but more than 7?days. Methodological quality of the included reviews will be assessed using the Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) tool, which contains 16 domains. Two authors will screen the titles and abstracts of all reviews obtained by the search strategy, assess the full text of selected articles for inclusion, and extract data independently. The quality appraisal will be also assessed by two authors (AM, CC) independently, and Cohen's Kappa statistic will be used to assess inter-ratter agreement. The findings of the study will be narrated qualitatively to describe the effect of different pharmacotherapy on smoking cessation. DISCUSSION:The World Health Organization recommends treatment of tobacco dependence as one approach in its comprehensive tobacco control policy. To date, many trials and systematic reviews have been conducted to assess the effectiveness of pharmacotherapy for smoking cessation. Therefore, the findings of the umbrella review will improve clinical decision-making and be used as a baseline for future studies. SYSTEMATIC REVIEW REGISTRATION:PROSPERO CRD42017080906.
Project description:Cognition-oriented treatments - commonly categorized as cognitive training, cognitive rehabilitation and cognitive stimulation - are promising approaches for the prevention of cognitive and functional decline in older adults. We conducted a systematic overview of meta-analyses investigating the efficacy of cognition-oriented treatments on cognitive and non-cognitive outcomes in older adults with or without cognitive impairment. Review quality was assessed by A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR). We identified 51 eligible reviews, 46 of which were included in the quantitative synthesis. The confidence ratings were "moderate" for 9 (20%), "low" for 13 (28%) and "critically low" for 24 (52%) of the 46 reviews. While most reviews provided pooled effect estimates for objective cognition, non-cognitive outcomes of potential relevance were more sparsely reported. The mean effect estimate on cognition was small for cognitive training in healthy older adults (mean Hedges' g?=?0.32, range 0.13-0.64, 19 reviews), mild cognitive impairment (mean Hedges' g?=?0.40, range 0.32-0.60, five reviews), and dementia (mean Hedges' g?=?0.38, range 0.09-1.16, seven reviews), and small for cognitive stimulation in dementia (mean Hedges' g?=?0.36, range 0.26-0.44, five reviews). Meta-regression revealed that higher AMSTAR score was associated with larger effect estimates for cognitive outcomes. The available evidence supports the efficacy of cognition-oriented treatments improving cognitive performance in older adults. The extent to which such effects are of clinical value remains unclear, due to the scarcity of high-quality evidence and heterogeneity in reported findings. An important avenue for future trials is to include relevant non-cognitive outcomes in a more consistent way and, for meta-analyses in the field, there is a need for better adherence to methodological standards. PROSPERO registration number: CRD42018084490.
Project description:This systematic review updated and extended the "UK evidence report" by Bronfort et al. (Chiropr Osteopath 18:3, 2010) with respect to conditions/interventions that received an 'inconclusive' or 'negative' evidence rating or were not covered in the report.A literature search of more than 10 general medical and specialised databases was conducted in August 2011 and updated in March 2013. Systematic reviews, primary comparative studies and qualitative studies of patients with musculoskeletal or non-musculoskeletal conditions treated with manual therapy and reporting clinical outcomes were included. Study quality was assessed using standardised instruments, studies were summarised, and the results were compared against the evidence ratings of Bronfort. These were either confirmed, updated, or new categories not assessed by Bronfort were added.25,539 records were found; 178 new and additional studies were identified, of which 72 were systematic reviews, 96 were randomised controlled trials, and 10 were non-randomised primary studies. Most 'inconclusive' or 'moderate' evidence ratings of the UK evidence report were confirmed. Evidence ratings changed in a positive direction from inconclusive to moderate evidence ratings in only three cases (manipulation/mobilisation [with exercise] for rotator cuff disorder; spinal mobilisation for cervicogenic headache; and mobilisation for miscellaneous headache). In addition, evidence was identified on a large number of non-musculoskeletal conditions not previously considered; most of this evidence was rated as inconclusive.Overall, there was limited high quality evidence for the effectiveness of manual therapy. Most reviewed evidence was of low to moderate quality and inconsistent due to substantial methodological and clinical diversity. Areas requiring further research are highlighted.
Project description:OBJECTIVES:To determine the breadth of outcome domains used in the reporting of adult cochlear implant surgery for the purpose of registry and quality measure development. DATA SOURCES:Systematic review of randomized controlled trials. REVIEW METHODS:In consultation with a medical librarian, search strategies were constructed to identify randomized controlled trials studying adults undergoing cochlear implantation. MEDLINE, EMBASE, Scopus, CINAHL, Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, and the Database of Abstracts of Reviews of Effects were searched from database inception to July 2015. Studies were evaluated for level of evidence and risk of bias with the Cochrane Collaboration's Risk of Bias Tool, and outcome domains were extracted from each study. RESULTS:Of 4473 unique citations found, 8 studies were included in this review. All 8 trials were evidence level 1B. Risk of bias was low in 2 trials and high in the other 6. Reported outcome domains included speech perception in quiet and noise, speech tracking, quality of life, timbre perception, hearing preservation, vestibular function, electrode insertion technique, functional measures, functional imaging, fitting time, and tinnitus. CONCLUSIONS:An analysis of randomized controlled trials studying cochlear implantation in adults yielded a wide spectrum of outcome domains. This is the first study to comprehensively describe the breadth of outcome domains in adult cochlear implantation. Validated instruments from these domains could be considered for potential inclusion as quality measures and registry use.
Project description:BACKGROUND:Many systematic reviews of clinical trials on acupuncture were performed within the Cochrane Collaboration, the evidence-based medicine (EBM) most recognized organization. Objective of the article was to systematically collect and identify systematic reviews of acupuncture published in the Cochrane Library and assess their quality from a methodological perspective. METHODS:A comprehensive literature search was performed in the Cochrane Database of Systematic Reviews to identify the reviews of acupuncture conducted until June 2019. The methodological quality of the included reviews was assessed using the AMSTAR 2 checklist, an evaluation tool for systematic reviews. RESULTS:Out of a total of 126 eligible reviews, 50 systematic reviews were included. According to the AMSTAR 2, 52% of Cochrane Systematic Reviews (CSRs) were of low quality, due to the presence of one or more weaknesses in at least one of the domains defined as critical for the methodological quality assessment. The less satisfied critical domain was inadequate investigation and discussion of publication bias. Declaration of potential sources of conflict of interest, and funding of the authors of the review and of the included studies were other important weaknesses. CONCLUSIONS:The main methodological flaws in the included CSRs were related to topics of relatively new concern in the conduction of systematic reviews of the literature. However, both, lack of attention about retrieval of negative studies, and statements about conflict of interests are crucial point for the evaluation of therapeutic interventions according to EBM methodology.
Project description:OBJECTIVES:Multiple systematic reviews have reported on the impact of rib fracture fixation in the presence of flail chest and multiple rib fractures, however this practice remains controversial. Our aim is to synthesise the effectiveness of surgical rib fracture fixation as evidenced by systematic reviews. DESIGN:A systematic search identified systematic reviews comparing effectiveness of rib fracture fixation with non-operative management of adults with flail chest or unifocal non-flail rib fractures. MEDLINE, EMBASE, Cochrane Database of Systematic Reviews and Science Citation Index were last searched 17 March 2017. Risk of bias was assessed using the Risk Of Bias In Systematic reviews (ROBIS) tool. The primary outcome was duration of mechanical ventilation. RESULTS:Twelve systematic reviews were included, consisting of 3 unique randomised controlled trials and 19 non-randomised studies. Length of mechanical ventilation was shorter in the fixation group compared with the non-operative group in flail chest; pooled estimates ranged from -4.52 days, 95% CI (-5.54 to -3.5) to -7.5 days, 95% CI (-9.9 to -5.5). Pneumonia, length of hospital and intensive care unit stay all showed a statistically significant improvement in favour of fixation for flail chest; however, all outcomes in favour of fixation had substantial heterogeneity. There was no statistically significant difference between groups in mortality. Two systematic reviews included one non-randomised studies of unifocal non-flail rib fracture population; due to limited evidence the benefits with surgery are uncertain. CONCLUSIONS:Synthesis of the reviews has shown some potential improvement in patient outcomes with flail chest after fixation. For future review updates, meta-analysis for effectiveness may need to take into account indications and timing of surgery as a subgroup analysis to address clinical heterogeneity between primary studies. Further robust evidence is required before conclusions can be drawn of the effectiveness of surgical fixation for flail chest and in particular, unifocal non-flail rib fractures. PROSPERO REGISTRATION NUMBER:CRD42016053494.
Project description:BACKGROUND:The World Health Organization (WHO) and the International Labour Organization (ILO) are developing joint estimates of the work-related burden of disease and injury (WHO/ILO Joint Estimates). For this, systematic reviews of studies estimating the prevalence of exposure to selected occupational risk factors will be conducted to provide input data for estimations of the number of exposed workers. A critical part of systematic review methods is to assess risk of bias (RoB) of individual studies. In this article, we present and describe the development of such a tool, called the Risk of Bias in Studies estimating Prevalence of Exposure to Occupational risk factors (RoB-SPEO) tool; report results from RoB-SPEO's pilot testing; note RoB-SPEO's limitations; and suggest how the tool might be tested and developed further. METHODS:Selected existing RoB tools used in environmental and occupational health systematic reviews were reviewed and analysed. From existing tools, we identified domains for the new tool and, if necessary, added new domains. For each domain, we then identified and integrated components from the existing tools (i.e. instructions, domains, guiding questions, considerations, ratings and rating criteria), and, if necessary, we developed new components. Finally, we elicited feedback from other systematic review methodologists and exposure scientists and agreed upon RoB-SPEO. Nine experts pilot tested RoB-SPEO, and we calculated a raw measure of inter-rater agreement (Pi) for each of its domain, rating Pi?<?0.4 as poor, 0.4???Pi???0.8 as substantial and Pi?>?0.80 as almost perfect agreement. RESULTS:Our review found no standard tool for assessing RoB in prevalence studies of exposure to occupational risk factors. We identified six existing tools for environmental and occupational health systematic reviews and found that their components for assessing RoB differ considerably. With the new RoB-SPEO tool, assessors judge RoB for each of eight domains: (1) bias in selection of participants into the study; (2) bias due to a lack of blinding of study personnel; (3) bias due to exposure misclassification; (4) bias due to incomplete exposure data; (5) bias due to conflict of interest; (6) bias due to selective reporting of exposures; (7) bias due to difference in numerator and denominator; and (8) other bias. The RoB-SPEO's ratings are low, probably low, probably high, high or no information. Pilot testing of the RoB-SPEO tool found substantial inter-rater agreement for six domains (range of Pi for these domains: 0.51-0.80), but poor agreement for two domains (i.e. Pi of 0.31 and 0.33 for biases due to incomplete exposure data and in selection of participants into the study, respectively). Limitations of RoB-SPEO include that it has not yet been fully performance-tested. CONCLUSIONS:We developed the RoB-SPEO tool for assessing RoB in prevalence studies of exposure to occupational risk factors. The tool will be applied and its performance tested in the ongoing systematic reviews for the WHO/ILO Joint Estimates.
Project description:Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P?<?0.001), while the difference was not significant for the RoB domain regarding randomization (P?=?0.124) and allocation concealment (P?=?0.039, ??<?0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.