Dataset Information

Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation.

ABSTRACT:

Objective

Abstracts of scientific reports are sometimes criticized for exaggerating significant results when compared to the corresponding full texts. Such abstracts can mislead the readers. We aimed to conduct a systematic review of overstatements in abstract conclusions in psychiatry trials.

Methods

We searched for randomized controlled trials published in 2014 that explicitly claimed effectiveness of any intervention for mental disorders in their abstract conclusion, using the Cochrane Register of Controlled Trials. Claims of effectiveness in abstract conclusion were categorized into three types: superiority (stating superiority of intervention to control), limited superiority (intervention has limited superiority), and equal efficactiveness (claiming equal effectiveness of intervention with standard treatment control), and full text results into three types: significant (all primary outcomes were statistically significant in favor of the intervention), mixed (primary outcomes included both significant and non-significant results), or all results non-significant. By comparing these classifications, we assessed whether each abstract was overstated. Our primary outcome was the proportion of overstated abstract conclusions.

Results

We identified and included 60 relevant trials. 20 out of 60 studies (33.3%) showed overstatements. Nine reports reported only significant results although none of their primary outcomes were significant. Large sample size (>300) and publication in high impact factor (IF>10) journals were associated with low occurrence of overstatements.

Conclusions

We found that one in three psychiatry studies claiming effectiveness in their abstract conclusion, either superior to control or equal to standard treatment, for any mental disorders were overstated in comparison with the full text results. Readers of the psychiatry literature are advised to scrutinize the full text results regardless of the claims in the abstract.

Trial registration

University hospital Medical Information Network (UMIN) Clinical Trials Registry (UMIN000018668).

SUBMITTER: Shinohara K

PROVIDER: S-EPMC5597227 | biostudies-literature | 2017

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation.

Shinohara Kiyomi K Suganuma Aya M AM Imai Hissei H Takeshima Nozomi N Hayasaka Yu Y Furukawa Toshi A TA

PloS one 20170913 9

<h4>Objective</h4>Abstracts of scientific reports are sometimes criticized for exaggerating significant results when compared to the corresponding full texts. Such abstracts can mislead the readers. We aimed to conduct a systematic review of overstatements in abstract conclusions in psychiatry trials.<h4>Methods</h4>We searched for randomized controlled trials published in 2014 that explicitly claimed effectiveness of any intervention for mental disorders in their abstract conclusion, using the ...[more]

PMID: 28902885

Similar Datasets

Project description:OBJECTIVES:To investigate whether overstatements in abstract conclusions influence primary care physicians' evaluations when they read reports of randomised controlled trials (RCTs) DESIGN: RCT setting: This study was a parallel-group randomised controlled survey, conducted online while masking the study hypothesis. PARTICIPANTS:Volunteers were recruited from members of the Japan Primary Care Association in January 2017. We sent email invitations to 7040 primary care physicians. Among the 787 individuals who accessed the website, 622 were eligible and automatically randomised into 'without overstatement' (n=307) and 'with overstatement' (n=315) groups. INTERVENTIONS:We selected five abstracts from published RCTs with at least one non-significant primary outcome and overstatement in the abstract conclusion. To construct a version without overstatement, we rewrote the conclusion sections. The methods and results sections were standardised to provide the necessary information of primary outcome information when it was missing in the original abstract. Participants were randomly assigned to read an abstract either with or without overstatements and asked to evaluate the benefit of the intervention. OUTCOME MEASURES:The primary outcome was the participants' evaluation of the benefit of the intervention discussed in the abstract, on a scale from 0 to 10. A secondary outcome was the validity of the conclusion. RESULTS:There was no significant difference between the groups with respect to their evaluation of the benefit of the intervention (mean difference: 0.07, 95% CI -0.28 to 0.42, p=0.69). Participants in the 'without' group considered the study conclusion to be more valid than those in the 'with' group (mean difference: 0.97, 95% CI 0.59 to 1.36, P<0.001). CONCLUSION:The overstatements in abstract conclusions did not significantly influence the primary care physicians' evaluations of the intervention effect when necessary information about the primary outcomes was distinctly reported. TRIAL REGISTRATION NUMBER:UMIN000025317; Pre-results.

Project description:BackgroundThis meta-epidemiological study aimed to assess methodological quality of a sample of contemporary non-randomised clinical studies of clinical interventions.MethodsThis was a cross-sectional study of observational studies published between January 1, 2012 and December 31, 2018. Studies were identified in PubMed using search terms 'association', 'observational,' 'non-randomised' 'comparative effectiveness' within titles or abstracts. Each study was appraised against 35 quality criteria by two authors independently, with each criterion rated fully, partially or not satisfied. These quality criteria were grouped into 6 categories: justification for observational design (n = 2); minimisation of bias in study design and data collection (n = 11); use of appropriate methods to create comparable groups (n = 6); appropriate adjustment of observed effects (n = 5); validation of observed effects (n = 9); and authors interpretations (n = 2).ResultsOf 50 unique studies, 49 (98%) were published in two US general medical journals. No study fully satisfied all applicable criteria; the mean (+/-SD) proportion of applicable criteria fully satisfied across all studies was 72% (+/- 10%). The categories of quality criteria demonstrating the lowest proportions of fully satisfied criteria were measures used to adjust observed effects (criteria 20, 23, 24) and validate observed effects (criteria 25, 27, 33). Criteria associated with ≤50% of full satisfaction across studies, where applicable, comprised: imputation methods to account for missing data (50%); justification for not performing an RCT (42%); interaction analyses in identifying independent prognostic factors potentially influencing intervention effects (42%); use of statistical correction to minimise type 1 error in multiple outcome analyses (33%); clinically significant effect sizes (30%); residual bias analyses for unmeasured or unknown confounders (14%); and falsification tests for residual confounding (8%). The proportions of fully satisfied criteria did not change over time.ConclusionsRecently published observational studies fail to fully satisfy more than one in four quality criteria. Criteria that were not or only partially satisfied were identified which serve as remediable targets for researchers and journal editors.

Project description:ImportanceDelirium, an acute disorder with high morbidity and mortality, is often preventable through multicomponent nonpharmacological strategies. The efficacy of these strategies for preventing subsequent adverse outcomes has been limited to small studies to date.ObjectiveTo evaluate available evidence on multicomponent nonpharmacological delirium interventions in reducing incident delirium and preventing poor outcomes associated with delirium.Data sourcesPubMed, Google Scholar, ScienceDirect, and the Cochrane Database of Systematic Reviews from January 1, 1999, to December 31, 2013.Study selectionStudies examining the following outcomes were included: delirium incidence, falls, length of stay, rate of discharge to a long-term care institution (institutionalization), and change in functional or cognitive status.Data extraction and synthesisTwo experienced physician reviewers independently and blindly abstracted data on outcome measures using a standardized approach. The reviewers conducted quality ratings based on the Cochrane risk-of-bias criteria for each study.Main outcomes and measuresWe identified 14 interventional studies. The results for outcomes of delirium incidence, falls, length of stay, and institutionalization were pooled for the meta-analysis, but heterogeneity limited our meta-analysis of the results for change in functional or cognitive status. Overall, 11 studies demonstrated significant reductions in delirium incidence (odds ratio [OR], 0.47; 95% CI, 0.38-0.58). Four randomized or matched trials reduced delirium incidence by 44% (OR, 0.56; 95% CI, 0.42-0.76). The rate of falls decreased significantly among intervention patients in 4 studies (OR, 0.38; 95% CI, 0.25-0.60); in 2 randomized or matched trials, the rate of falls was reduced by 64% (OR, 0.36; 95% CI, 0.22-0.61). Length of stay and institutionalization also trended toward decreases in the intervention groups, with a mean difference of -0.16 (95% CI, -0.97 to 0.64) day shorter and the odds of institutionalization 5% lower (OR, 0.95; 95% CI, 0.71-1.26). Among higher-quality randomized or matched trials, length of stay trended -0.33 (95% CI, -1.38 to 0.72) day shorter, and the odds of institutionalization trended 6% lower (OR, 0.94; 95% CI, 0.69-1.30).Conclusions and relevanceMulticomponent nonpharmacological delirium prevention interventions are effective in reducing delirium incidence and preventing falls, with a trend toward decreasing length of stay and avoiding institutionalization. Given the current focus on prevention of hospital-based complications and improved cost-effectiveness of care, this meta-analysis supports the use of these interventions to advance acute care for older persons.

Project description:BackgroundPilot/feasibility or studies with small sample sizes may be associated with inflated effects. This study explores the vibration of effect sizes (VoE) in meta-analyses when considering different inclusion criteria based upon sample size or pilot/feasibility status.MethodsSearches were to identify systematic reviews that conducted meta-analyses of behavioral interventions on topics related to the prevention/treatment of childhood obesity from January 2016 to October 2019. The computed summary effect sizes (ES) were extracted from each meta-analysis. Individual studies included in the meta-analyses were classified into one of the following four categories: self-identified pilot/feasibility studies or based upon sample size but not a pilot/feasibility study (N ≤ 100, N > 100, and N > 370 the upper 75th of sample size). The VoE was defined as the absolute difference (ABS) between the re-estimations of summary ES restricted to study classifications compared to the originally reported summary ES. Concordance (kappa) of statistical significance of summary ES between the four categories of studies was assessed. Fixed and random effects models and meta-regressions were estimated. Three case studies are presented to illustrate the impact of including pilot/feasibility and N ≤ 100 studies on the estimated summary ES.ResultsA total of 1602 effect sizes, representing 145 reported summary ES, were extracted from 48 meta-analyses containing 603 unique studies (avg. 22 studies per meta-analysis, range 2-108) and included 227,217 participants. Pilot/feasibility and N ≤ 100 studies comprised 22% (0-58%) and 21% (0-83%) of studies included in the meta-analyses. Meta-regression indicated the ABS between the re-estimated and original summary ES where summary ES ranged from 0.20 to 0.46 depending on the proportion of studies comprising the original ES were either mostly small (e.g., N ≤ 100) or mostly large (N > 370). Concordance was low when removing both pilot/feasibility and N ≤ 100 studies (kappa = 0.53) and restricting analyses only to the largest studies (N > 370, kappa = 0.35), with 20% and 26% of the originally reported statistically significant ES rendered non-significant. Reanalysis of the three case study meta-analyses resulted in the re-estimated ES rendered either non-significant or half of the originally reported ES.ConclusionsWhen meta-analyses of behavioral interventions include a substantial proportion of both pilot/feasibility and N ≤ 100 studies, summary ES can be affected markedly and should be interpreted with caution.

Dataset Information

Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation.

Objective

Methods

Results

Conclusions

Trial registration

Publications

Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets