ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
Ontology highlight
ABSTRACT:
SUBMITTER: Sterne JA
PROVIDER: S-EPMC5062054 | biostudies-literature | 2016 Oct
REPOSITORIES: biostudies-literature
Publications
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
Sterne Jonathan Ac JA Hernán Miguel A MA Reeves Barnaby C BC Savović Jelena J Berkman Nancy D ND Viswanathan Meera M Henry David D Altman Douglas G DG Ansari Mohammed T MT Boutron Isabelle I Carpenter James R JR Chan An-Wen AW Churchill Rachel R Deeks Jonathan J JJ Hróbjartsson Asbjørn A Kirkham Jamie J Jüni Peter P Loke Yoon K YK Pigott Theresa D TD Ramsay Craig R CR Regidor Deborah D Rothstein Hannah R HR Sandhu Lakhbir L Santaguida Pasqualina L PL Schünemann Holger J HJ Shea Beverly B Shrier Ian I Tugwell Peter P Turner Lucy L Valentine Jeffrey C JC Waddington Hugh H Waters Elizabeth E Wells George A GA Whiting Penny F PF Higgins Julian Pt JP
BMJ (Clinical research ed.) 20161012
Similar Datasets
Project description:ObjectivesWe aimed to review how 'Risk of Bias In Non-randomized Studies-of Interventions' (ROBINS-I), a Cochrane risk of bias assessment tool, has been used in recent systematic reviews.Study design and settingDatabase and citation searches were conducted in March 2020 to identify recently published reviews using ROBINS-I. Reported ROBINS-I assessments and data on how ROBINS-I was used were extracted from each review. Methodological quality of reviews was assessed using AMSTAR 2 ('A MeaSurement Tool to Assess systematic Reviews').ResultsOf 181 hits, 124 reviews were included. Risk of bias was serious/critical in 54% of assessments on average, most commonly due to confounding. Quality of reviews was mostly low, and modifications and incorrect use of ROBINS-I were common, with 20% reviews modifying the rating scale, 20% understating overall risk of bias, and 19% including critical-risk of bias studies in evidence synthesis. Poorly conducted reviews were more likely to report low/moderate risk of bias (predicted probability 57% [95% CI: 47-67] in critically low-quality reviews, 31% [19-46] in high/moderate-quality reviews).ConclusionLow-quality reviews frequently apply ROBINS-I incorrectly, and may thus inappropriately include or give too much weight to uncertain evidence. Readers should be aware that such problems can lead to incorrect conclusions in reviews.
| S-EPMC8809341 | biostudies-literature
Project description:Assessing the risk of bias (RoB) of individual studies is a critical part in determining the certainty of a body of evidence from non-randomized studies (NRS) that evaluate potential health effects due to environmental exposures. The recently released RoB in NRS of Interventions (ROBINS-I) instrument has undergone careful development for health interventions. Using the fundamental design of ROBINS-I, which includes evaluating RoB against an ideal target trial, we explored developing a version of the instrument to evaluate RoB in exposure studies. During three sequential rounds of assessment, two or three raters (evaluators) independently applied ROBINS-I to studies from two systematic reviews and one case-study protocol that evaluated the relationship between environmental exposures and health outcomes. Feedback from raters, methodologists, and topic-specific experts informed important modifications to tailor the instrument to exposure studies. We identified the following areas of distinction for the modified instrument: terminology, formulation of the ideal target randomized experiment, guidance for cross-sectional studies and exposure assessment (both quality of measurement method and concern for potential exposure misclassification), and evaluation of issues related to study sensitivity. Using the target experiment approach significantly impacts the process for how environmental and occupational health studies are considered in the Grading of Recommendations Assessment, Development and Evaluation (GRADE) evidence-synthesis framework.
| S-EPMC9581061 | biostudies-literature
Project description:Aggregated species occurrence and abundance data from disparate sources are increasingly accessible to ecologists for the analysis of temporal trends in biodiversity. However, sampling biases relevant to any given research question are often poorly explored and infrequently reported; this can undermine statistical inference. In other disciplines, it is common for researchers to complete 'risk-of-bias' assessments to expose and document the potential for biases to undermine conclusions. The huge growth in available data, and recent controversies surrounding their use to infer temporal trends, indicate that similar assessments are urgently needed in ecology.We introduce ROBITT, a structured tool for assessing the 'Risk-Of-Bias In studies of Temporal Trends in ecology'. ROBITT has a similar format to its counterparts in other disciplines: it comprises signalling questions designed to elicit information on the potential for bias in key study domains. In answering these, users will define study inferential goal(s) and relevant statistical target populations. This information is used to assess potential sampling biases across domains relevant to the research question (e.g. geography, taxonomy, environment), and how these vary through time. If assessments indicate biases, then users must clearly describe them and/or explain what mitigating action will be taken.Everything that users need to complete a ROBITT assessment is provided: the tool, a guidance document and a worked example. Following other disciplines, the tool and guidance document were developed through a consensus-forming process across experts working in relevant areas of ecology and evidence synthesis.We propose that researchers should be strongly encouraged to include a ROBITT assessment when publishing studies of biodiversity trends, especially when using aggregated data. This will help researchers to structure their thinking, clearly acknowledge potential sampling issues, highlight where expert consultation is required and provide an opportunity to describe data checks that might go unreported. ROBITT will also enable reviewers, editors and readers to establish how well research conclusions are supported given a dataset combined with some analytical approach. In turn, it should strengthen evidence-based policy and practice, reduce differing interpretations of data and provide a clearer picture of the uncertainties associated with our understanding of reality.
| S-EPMC9541136 | biostudies-literature
Project description:BackgroundSystematic Reviews (SRs) of experimental animal studies are not yet common practice, but awareness of the merits of conducting such SRs is steadily increasing. As animal intervention studies differ from randomized clinical trials (RCT) in many aspects, the methodology for SRs of clinical trials needs to be adapted and optimized for animal intervention studies. The Cochrane Collaboration developed a Risk of Bias (RoB) tool to establish consistency and avoid discrepancies in assessing the methodological quality of RCTs. A similar initiative is warranted in the field of animal experimentation.MethodsWe provide an RoB tool for animal intervention studies (SYRCLE's RoB tool). This tool is based on the Cochrane RoB tool and has been adjusted for aspects of bias that play a specific role in animal intervention studies. To enhance transparency and applicability, we formulated signalling questions to facilitate judgment.ResultsThe resulting RoB tool for animal studies contains 10 entries. These entries are related to selection bias, performance bias, detection bias, attrition bias, reporting bias and other biases. Half these items are in agreement with the items in the Cochrane RoB tool. Most of the variations between the two tools are due to differences in design between RCTs and animal studies. Shortcomings in, or unfamiliarity with, specific aspects of experimental design of animal studies compared to clinical studies also play a role.ConclusionsSYRCLE's RoB tool is an adapted version of the Cochrane RoB tool. Widespread adoption and implementation of this tool will facilitate and improve critical appraisal of evidence from animal studies. This may subsequently enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.
| S-EPMC4230647 | biostudies-literature
Project description: Not available
| S-EPMC5833365 | biostudies-literature
Project description:BackgroundSystematic reviews of the effects of healthcare interventions frequently include non-randomized studies. These are subject to confounding and a range of other biases that are seldom considered in detail when synthesizing and interpreting the results. Our aims were to assess the reliability and usability of a new Cochrane risk of bias (RoB) tool for non-randomized studies of interventions and to determine whether restricting analysis to studies with low or moderate RoB made a material difference to the results of the reviews.Methods and findingsWe selected two systematic reviews of population-based, controlled non-randomized studies of the relationship between the use of thiazolidinediones (TZDs) and cyclooxygenase-2 (COX-2) inhibitors and major cardiovascular events. Two epidemiologists applied the Cochrane RoB tool and made assessments across the seven specified domains of bias for each of 37 component studies. Inter-rater agreement was measured using the weighted Kappa statistic. We grouped studies according to overall RoB and performed statistical pooling for (a) all studies and (b) only studies with low or moderate RoB. Kappa scores across the seven bias domains ranged from 0.50 to 1.0. In the COX-2 inhibitor review, two studies had low overall RoB, 14 had moderate RoB, and five had serious RoB. In the TZD review, six studies had low RoB, four had moderate RoB, four had serious RoB, and two had critical RoB. The pooled odds ratios for myocardial infarction, heart failure, and death for rosiglitazone versus pioglitazone remained significantly elevated when analyses were confined to studies with low or moderate RoB. However, the estimate for myocardial infarction declined from 1.14 (95% CI 1.07-1.24) to 1.06 (95% CI 0.99-1.13) when analysis was confined to studies with low RoB. Estimates of pooled relative risks of cardiovascular events with COX-2 inhibitors compared with no nonsteroidal anti-inflammatory drug changed little when analyses were confined to studies with low or moderate RoB. The exception was a rise in the relative risk associated with ibuprofen from 1.07 (95% CI 0.97-1.18) to 1.14 (95% CI 1.03-1.26). The main limitation of our study was testing the instrument on a narrow range of pharmacoepidemiological studies; we cannot assume our findings extend to a broader range of interventions and settings.ConclusionsThe Cochrane RoB tool highlighted a wide range of risks of bias in studies included in two widely cited reviews and had the potential to change the conclusions of the reviews. Systematic reviews that incorporate non-randomized studies of medical interventions should include a detailed assessment of RoB for each included study.
| S-EPMC4821619 | biostudies-literature
Project description:Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet's AC1 was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC1 ranging from 0.071 to 0.535). After training, the multi-rater AC1 ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC1: 0.405, 95%-CI 0.149-0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.
| S-EPMC10003882 | biostudies-literature
Project description:BackgroundProton-pump inhibitors (PPIs) are widely prescribed as acid-suppression therapy. Some observational studies suggest that long-term use of PPIs is potentially associated with certain adverse kidney outcomes. We conducted a systematic literature review to assess potential bias in non-randomized studies reporting on putative associations between PPIs and adverse kidney outcomes (acute kidney injury, acute interstitial nephritis, chronic interstitial nephritis, acute tubular necrosis, chronic kidney disease, and end-stage renal disease).MethodsWe searched the medical literature within 10 years of 17 December 2020. Pre-specified criteria guided identification of relevant English language articles for assessment. Risk of bias on an outcome-specific basis was evaluated using the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) tool by two independent reviewers.ResultsOf 620 initially identified records, 26 studies met a priori eligibility criteria and underwent risk of bias assessment. Nineteen studies were judged as having a moderate risk of bias for reported adverse kidney outcomes, while six studies were judged as having a serious risk of bias (mainly due to inadequate control of confounders and selection bias). We were unable to determine the overall risk of bias in two studies (one of which was assessed as having a moderate risk of bias for a different adverse kidney outcome) due to insufficient information presented. Effect estimates for PPIs in relation to adverse kidney outcomes varied widely (0.24-7.34) but associations mostly showed increased risk.ConclusionUsing ROBINS-I, we found that non-randomized observational studies suggesting kidney harm by PPIs have moderate to serious risk of bias, making it challenging to establish causality. Additional high-quality, real-world evidence among generalizable populations are needed to better understand the relation between PPI treatment and acute and chronic kidney outcomes, accounting for the effects of varying durations of PPI treatment, self-treatment with over-the-counter PPIs, and potential critical confounders.
| S-EPMC8841917 | biostudies-literature
Project description:BackgroundThe use of Mendelian randomization (MR) in epidemiology has increased considerably in recent years, with a subsequent increase in systematic reviews of MR studies. We conducted a systematic review of tools designed for assessing risk of bias and/or quality of evidence in MR studies and a review of systematic reviews of MR studies.MethodsWe systematically searched MEDLINE, Embase, the Web of Science, preprints servers and Google Scholar for articles containing tools for assessing, conducting and/or reporting MR studies. We also searched for systematic reviews and protocols of systematic reviews of MR studies. From eligible articles we collected data on tool characteristics and content, as well as details of narrative description of bias assessment.ResultsOur searches retrieved 2464 records to screen, from which 14 tools, 35 systematic reviews and 38 protocols were included in our review. Seven tools were designed for assessing risk of bias/quality of evidence in MR studies and evaluation of their content revealed that all seven tools addressed the three core assumptions of instrumental variable analysis, violation of which can potentially introduce bias in MR analysis estimates.ConclusionWe present an overview of tools and methods to assess risk of bias/quality of evidence in MR analysis. Issues commonly addressed relate to the three standard assumptions of instrumental variables analyses, the choice of genetic instrument(s) and features of the population(s) from which the data are collected (particularly in two-sample MR), in addition to more traditional non-MR-specific epidemiological biases. The identified tools should be tested and validated for general use before recommendations can be made on their widespread use. Our findings should raise awareness about the importance of bias related to MR analysis and provide information that is useful for assessment of MR studies in the context of systematic reviews.
| S-EPMC9908059 | biostudies-literature
Project description:PurposeThere is limited knowledge on the reliability of risk of bias (ROB) tools for assessing internal validity in systematic reviews of exposure and frequency studies. We aimed to identify and then compare the inter-rater reliability (IRR) of six commonly used tools for frequency (Loney scale, Gyorkos checklist, American Academy of Neurology [AAN] tool) and exposure (Newcastle-Ottawa scale, SIGN50 checklist, AAN tool) studies.MethodsSix raters independently assessed the ROB of 30 frequency and 30 exposure studies using the three respective ROB tools. Articles were rated as low, intermediate, or high ROB. We calculated an intraclass correlation coefficient (ICC) for each tool and category of ROB tool. We compared the IRR between ROB tools and tool type by inspection of overlapping ICC 95% CIs and by comparing their coefficients after transformation to Fisher's Z values. We assessed the criterion validity of the AAN ROB tools by calculating an ICC for each rater in comparison with the original ratings from the AAN.ResultsAll individual ROB tools had an IRR in the substantial range or higher (ICC point estimates between 0.61 and 0.80). The IRR was almost perfect (ICC point estimate > 0.80) for the AAN frequency tool and the SIGN50 checklist. All tools were comparable in IRR, except for the AAN frequency tool which had a significantly higher ICC than the Gyorkos checklist (p = 0.021) and trended towards a higher ICC when compared to the Loney scale (p = 0.085). When examined by category of ROB tool, scales, and checklists had a substantial IRR, whereas the AAN tools had an almost perfect IRR. For the criterion validity of the AAN ROB tools, the average agreement between our raters and the original AAN ratings was moderate.ConclusionAll tools had substantial IRRs except for the AAN frequency tool and the SIGN50 checklist, which both had an almost perfect IRR. The AAN ROB tools were the only category of ROB tools to demonstrate an almost perfect IRR. This category of ROB tools had fewer and simpler criteria. Overall, parsimonious tools with clear instructions, such as those from the AAN, may provide more reliable ROB assessments.
| S-EPMC10702000 | biostudies-literature