Optimism bias leads to inconclusive results-an empirical study.
ABSTRACT: Optimism bias refers to unwarranted belief in the efficacy of new therapies. We assessed the impact of optimism bias on a proportion of trials that did not answer their research question successfully and explored whether poor accrual or optimism bias is responsible for inconclusive results.Systematic review.Retrospective analysis of a consecutive-series phase III randomized controlled trials (RCTs) performed under the aegis of National Cancer Institute Cooperative groups.Three hundred fifty-nine trials (374 comparisons) enrolling 150,232 patients were analyzed. Seventy percent (262 of 374) of the trials generated conclusive results according to the statistical criteria. Investigators made definitive statements related to the treatment preference in 73% (273 of 374) of studies. Investigators' judgments and statistical inferences were concordant in 75% (279 of 374) of trials. Investigators consistently overestimated their expected treatment effects but to a significantly larger extent for inconclusive trials. The median ratio of expected and observed hazard ratio or odds ratio was 1.34 (range: 0.19-15.40) in conclusive trials compared with 1.86 (range: 1.09-12.00) in inconclusive studies (P<0.0001). Only 17% of the trials had treatment effects that matched original researchers' expectations.Formal statistical inference is sufficient to answer the research question in 75% of RCTs. The answers to the other 25% depend mostly on subjective judgments, which at times are in conflict with statistical inference. Optimism bias significantly contributes to inconclusive results.
Project description:Oral mucositis has been an extremely serious complication resulted from cytotoxic effects of the chemotherapy among cancer patients. Several randomized controlled trials investigated the efficacy of zinc sulfate in prevention of this morbid condition among cancer patients undergoing chemotherapy, however conclusive findings has not yet been generated. This systematic review will assess the efficacy and safety of oral zinc sulfate for chemotherapy-induced oral mucositis.We will electronically search all potential citations in PubMed, EMBASE, and Cochrane Central Register of Controlled Trials (CENTRAL) from their inception to April 2018. The randomized controlled trials, which investigated the efficacy of oral zinc sulfate for chemotherapy-induced oral mucositis will be considered. We will assign 2 independent investigators to perform search, screen citations, extract data, and appraise risk of bias. And then, the primary investigator will adopt RevMan 5.3 software to complete all statistical analyses.The findings from this systematic review and meta-analysis will be submitted to a peer-reviewed journal for publication. Moreover, we will disseminate all results in any topic-related conference.This protocol has been registered with a number of CRD42018093605.
Project description:One of the most accepted findings across psychology is that people are unrealistically optimistic in their judgments of comparative risk concerning future life events-they judge negative events as less likely to happen to themselves than to the average person. Harris and Hahn (2011), however, demonstrated how unbiased (non-optimistic) responses can result in data patterns commonly interpreted as indicative of optimism due to statistical artifacts. In the current paper, we report the results of 5 studies that control for these statistical confounds and observe no evidence for residual unrealistic optimism, even observing a 'severity effect' whereby severe outcomes were overestimated relative to neutral ones (Studies 3 & 4). We conclude that there is no evidence supporting an optimism interpretation of previous results using the prevalent comparison method.
Project description:Previous research reviewed treatment success and whether the collective uncertainty principle is met in RCTs in the US National Cancer Institute portfolio. This paper classifies clinical trials funded by the UK HTA programme by results using the method applied to the US Cancer Institute trials, and compares the two portfolios.Data on all completed randomised controlled trials funded by the HTA programme 1993-2008 were extracted. Each trial's primary results was classified into six categories; 1) statistically significant in favour of the new treatment, 2) statistically significant in favour of the control treatment 3) true negative, 4) truly inconclusive, 5) inconclusive in favour of new treatment or 6) inconclusive in favour of control treatment. Trials were classified by comparing the 95% confidence interval for the difference in primary outcome to the difference specified in the sample size calculation. The results were compared with Djulbegovic's analysis of NCI trials.Data from 51 superiority trials were included, involving over 48,000 participants and a range of diseases and interventions. 85 primary comparisons were available because some trials had more than two randomised arms or had several primary outcomes. The new treatment had superior results (whether significant or not) in 61% of the comparisons (52/85 95% CI 49.9% to 71.6%). The results were conclusive in 46% of the comparisons (19% statistically significant in favour of the new treatment, 5% statistically significant in favour of the control and 22% true negative). The results were classified as truly inconclusive (i.e. failed to answer the question asked) for 24% of comparisons (20/85). HTA trials included fewer truly inconclusive and statistically significant results and more results rated as true negative than NCI trials.The pattern of results in HTA trials is similar to that of the National Cancer Institute portfolio. Differences that existed were plausible given the differences in the types of trials -HTA trials are more pragmatic. The results indicate HTA trials are compatible with equipoise. This classification usefully summarises the results from clinical trials and enables comparisons of different portfolios of trials.
Project description:Protective effects of calcium supplementation against colorectal adenomas have been documented in systematic reviews; however, the results have not been conclusive. Our objective was to update and systematically evaluate the evidence for calcium supplementation taking into consideration the risks of systematic and random error and to GRADE the evidence.The study comprised a systematic review with meta-analysis and trial sequential analysis (TSA) of randomized controlled trials (RCTs). We searched for RCTs published up until September 2016. Retrieved trials were evaluated using risk of bias. Primary outcome measures were the incidences of any recurrent adenomas and of advanced adenomas. Meta-analytic estimates were calculated with the random-effects model and random errors were evaluated with trial sequential analyses (TSAs).Five randomized trials (2234 patients with a history of adenomas) were included. Two of the 5 trials showed either unclear or high risks of bias in most criteria. Meta-analysis of good quality RCTs suggest a moderate protective effect of calcium supplementation on recurrence of adenomas (relative risk [RR], 0.88 [95% CI 0.79-0.99]); however, its effects on advanced adenomas did not show statistical significance (RR, 1.02 [95% CI 0.67-1.55]). Subgroup analyses demonstrated a greater protective effect on recurrence of adenomas with elemental calcium dose ?1600?mg/day (RR, 0.74 [95% CI 0.56-0.97]) compared to ?1200?mg/day (RR, 0.84 [95% CI 0.73-0.97]). No major serious adverse events were associated with the use of calcium, but there was an increase in the incidence of hypercalcemia (P?=?.0095). TSA indicated a lack of firm evidence for a beneficial effect. Concerns with directness and imprecision rated down the quality of the evidence to "low."The available good quality RCTs suggests a possible beneficial effect of calcium supplementation on the recurrence of adenomas; however, TSA indicated that the accumulated evidence is still inconclusive. Using GRADE-methodology, we conclude that the quality of evidence is low. Large well-designed randomized trials with low risk of bias are needed.
Project description:Results from clinical trials can be susceptible to bias if investigators choose their analysis approach after seeing trial data, as this can allow them to perform multiple analyses and then choose the method that provides the most favourable result (commonly referred to as 'p-hacking'). Pre-specification of the planned analysis approach is essential to help reduce such bias, as it ensures analytical methods are chosen in advance of seeing the trial data. For this reason, guidelines such as SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and ICH-E9 (International Conference for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use) require the statistical methods for a trial's primary outcome be pre-specified in the trial protocol. However, pre-specification is only effective if done in a way that does not allow p-hacking. For example, investigators may pre-specify a certain statistical method such as multiple imputation, but give little detail on how it will be implemented. Because there are many different ways to perform multiple imputation, this approach to pre-specification is ineffective, as it still allows investigators to analyse the data in different ways before deciding on a final approach. In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking. This framework was designed based on the principles in the SPIRIT and ICH-E9 guidelines and is intended to be used in conjunction with these guidelines to help investigators design the statistical analysis strategy for the trial's primary outcome in the trial protocol.
Project description:Favors from a sender to a receiver are known to bias decisions made by the recipient, especially when the decision relates to the sender, a feature of social exchange known as reciprocity. Using an art-viewing paradigm possessing no objectively correct answer for preferring one piece of art over another, we show that sponsorship of the experiment by a company endows the logo of the company with the capacity to bias revealed preference for art displayed next to the logo. Merely offering to sponsor the experiment similarly endowed the gesturing logo of the company with the capacity to bias revealed preferences. These effects do not depend upon the size of the displayed art or the proximity of the sponsoring logo to the piece of art. We used functional magnetic resonance imaging to show that such monetary favors do not modulate a special collection of brain responses but instead modulate responses in neural networks normally activated by a wide range of preference judgments. The results raise the important possibility that monetary favors bias judgments in domains seemingly unrelated to the favor but nevertheless act in an implicit way through neural networks that underlie normal, ongoing preference judgments.
Project description:BACKGROUND:We systematically reviewed the evidence on the efficacy and safety of dorsal root ganglion (DRG) targeted pulsed radiofrequency (PRF) versus any comparator for treatment of non-neuropathic pain. METHODS:We searched MEDLINE, CINAHL, Embase, PsycINFO, clinicaltrials.gov and WHO clinical trial register until January 8, 2019. All study designs were eligible. Two authors independently conducted literature screening. Primary outcomes were pain intensity and serious adverse events (SAEs). Secondary outcomes were any other pain-related outcome and any other safety outcome that was reported. We assessed the risk of bias using the Cochrane tool and Risk of Bias In Non-randomized Studies of Interventions (ROBINS-I). We conducted narrative evidence synthesis and assessed the conclusiveness of included studies regarding efficacy and safety. RESULTS:We included 17 studies with 599 participants, which analyzed various pain syndromes. Two studies were randomized controlled trials; both included participants with low back pain (LBP). Non-randomized studies included patients with the following indications: LBP, postsurgical pain, pain associated with herpes zoster, cervicogenic headache, complex regional pain syndrome type 1, intractable vertebral metastatic pain, chronic scrotal and inguinal pain, occipital radiating pain in rheumatoid arthritis and chronic migraine. In these studies, the PRF was usually initiated after other treatments have failed. Eleven studies had positive conclusive statements (11/17) about efficacy; the remaining had positive inconclusive statements. Only three studies provided conclusiveness of evidence statements regarding safety - two indicated that the evidence was positive conclusive, and one positive inconclusive. The risk of bias was predominantly unclear in randomized and serious in non-randomized studies. CONCLUSION:Poor quality and few participants characterize evidence about benefits and harms of DRG PRF in patients with non-neuropathic pain. Results from available studies should only be considered preliminary. Not all studies have reported data regarding the safety of the intervention, but those that did, indicate that the intervention is relatively safe. As the procedure is non-destructive and early results are promising, further comparative studies about PRF in non-neuropathic pain syndromes would be welcomed.
Project description:<h4>Background</h4>An important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews are considered golden standard regarding systematic review methodology, but Cochrane's instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors' assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews (CDSR).<h4>Methods</h4>We analyzed Cochrane reviews published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many Cochrane reviews had different judgments for the same supporting explanations.<h4>Results</h4>In the main analysis we included 10,292 judgments and supporting explanations for attrition bias from 729 Cochrane reviews. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of Cochrane review authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same Cochrane review.<h4>Conclusion</h4>We found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane's risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.
Project description:A prospective first-in-human Phase 1 CRISPR gene editing trial in the United States for patients with melanoma, synovial sarcoma, and multiple myeloma offers hope that gene editing tools may usefully treat human disease. An overarching ethical challenge with first-in-human Phase 1 clinical trials, however, is knowing when it is ethically acceptable to initiate such trials on the basis of safety and efficacy data obtained from pre-clinical studies. If the pre-clinical studies that inform trial design are themselves poorly designed - as a result of which the quality of pre-clinical evidence is deficient - then the ethical requirement of scientific validity for clinical research may not be satisfied. In turn, this could mean that the Phase 1 clinical trial will be unsafe and that trial participants will be exposed to risk for no potential benefit. To assist sponsors, researchers, clinical investigators and reviewers in deciding when it is ethically acceptable to initiate first-in-human Phase 1 CRISPR gene editing clinical trials, structured processes have been developed to assess and minimize translational distance between pre-clinical and clinical research. These processes draw attention to various features of internal validity, construct validity, and external validity. As well, the credibility of supporting evidence is to be critically assessed with particular attention to optimism bias, financial conflicts of interest and publication bias. We critically examine the pre-clinical evidence used to justify the first-inhuman Phase 1 CRISPR gene editing cancer trial in the United States using these tools. We conclude that the proposed trial cannot satisfy the ethical requirement of scientific validity because the supporting pre-clinical evidence used to inform trial design is deficient.
Project description:AIM:To assess the use of extracorporeal cardiopulmonary resuscitation (ECPR), compared with manual or mechanical cardiopulmonary resuscitation (CPR), for out-of-hospital cardiac arrest (OHCA) and in-hospital cardiac arrest (IHCA) in adults and children. METHODS:The PRISMA guidelines were followed. We searched Medline, Embase, and Evidence-Based Medicine Reviews for randomized clinical trials and observational studies published before May 22, 2018. The population included adult and pediatric patients with OHCA and IHCA of any origin. Two investigators reviewed studies for relevance, extracted data, and assessed risk of bias using the ROBINS-I tool. Outcomes included short-term and long-term survival and favorable neurological outcome. RESULTS:We included 25 observational studies, of which 15 studies were in adult OHCA, 7 studies were in adult IHCA, and 3 studies were in pediatric IHCA. There were no studies in pediatric OHCA. No randomized trials were included. Results from individual studies were largely inconsistent, although several studies in adult and pediatric IHCA were in favor of ECPR. The risk of bias for individual studies was overall assessed to be critical, with confounding being the primary source of bias. The overall quality of evidence was assessed to be very low. Heterogeneity across studies precluded any meaningful meta-analyses. CONCLUSIONS:There is inconclusive evidence to either support or refute the use of ECPR for OHCA and IHCA in adults and children. The quality of evidence across studies is very low.