Dataset Information

Heterogeneity in effect size estimates.

ABSTRACT: A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.

SUBMITTER: Holzmeister F

PROVIDER: S-EPMC11317577 | biostudies-literature | 2024 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Heterogeneity in effect size estimates.

Holzmeister Felix F Johannesson Magnus M Böhm Robert R Dreber Anna A Huber Jürgen J Kirchler Michael M

Proceedings of the National Academy of Sciences of the United States of America 20240730 32

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity ...[more]

PMID: 39078672

Similar Datasets

Project description:ObjectiveTo assess the influence of trial sample size on treatment effect estimates within meta-analyses.DesignMeta-epidemiological study.Data sources93 meta-analyses (735 randomised controlled trials) assessing therapeutic interventions with binary outcomes, published in the 10 leading journals of each medical subject category of the Journal Citation Reports or in the Cochrane Database of Systematic Reviews.Data extractionSample size, outcome data, and risk of bias extracted from each trial.Data synthesisTrials within each meta-analysis were sorted by their sample size: using quarters within each meta-analysis (from quarter 1 with 25% of the smallest trials, to quarter 4 with 25% of the largest trials), and using size groups across meta-analyses (ranging from <50 to ≥ 1000 patients). Treatment effects were compared within each meta-analysis between quarters or between size groups by average ratios of odds ratios (where a ratio of odds ratios less than 1 indicates larger effects in smaller trials).ResultsTreatment effect estimates were significantly larger in smaller trials, regardless of sample size. Compared with quarter 4 (which included the largest trials), treatment effects were, on average, 32% larger in trials in quarter 1 (which included the smallest trials; ratio of odds ratios 0.68, 95% confidence interval 0.57 to 0.82), 17% larger in trials in quarter 2 (0.83, 0.75 to 0.91), and 12% larger in trials in quarter 3 (0.88, 0.82 to 0.95). Similar results were obtained when comparing treatment effect estimates between different size groups. Compared with trials of 1000 patients or more, treatment effects were, on average, 48% larger in trials with fewer than 50 patients (0.52, 0.41 to 0.66) and 10% larger in trials with 500-999 patients (0.90, 0.82 to 1.00).ConclusionsTreatment effect estimates differed within meta-analyses solely based on trial sample size, with stronger effect estimates seen in small to moderately sized trials than in the largest trials.

Project description:Eradication or local extinction of the human parasite Schistosoma mansoni is a goal for many control programs. Population genetic analyses are helping to evaluate and guide these efforts, yet what to sample, how to sample and how densely to sample is not well established. We determined the S. mansoni allele frequency profile of nearly all infected inhabitants in two small Brazilian communities and created sub-samples representing 5-50% of all detected human infections (infrapopulations). Samples were selected at random with replacement, and each size class was replicated 100 times. Mean pairwise differentiation for all infrapopulations (Di) and the variance effective population size (Ne) were calculated for each sample. Prior to community-wide treatment, the true mean Di was moderate (0.095-0.123) and Ne large (>30,000). Most samples of <50% of those infected produced estimates outside of 5% of the true value. For estimates within 10%, sample sizes of >15% of all infrapopulations were required. At the 3 year follow-up after treatment, the Di increased and Ne was reduced by >15 fold. At this time sampling of >30-45% was needed to achieve the same accuracy. Following a second treatment and 4 years from baseline, the Di further increased and Ne decreased with little change in the sampling effort required. Extensive sampling is required for accurate estimates of these important population parameters. Characteristics such as population census size, infection prevalence, the community's treatment history and the degree of infrapopulation differentiation should be taken into account. The intensity of infection was weakly correlated with the ability of a single infrapopulation to represent the component population (Dic), indicating a tendency toward random acquisition of parasite genotypes. This also suggests that targeted sampling from those most heavily infected will better represent the genetic diversity of the whole community than a random sample of infrapopulations.

Dataset Information

Heterogeneity in effect size estimates.

Publications

Heterogeneity in effect size estimates.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets