Dataset Information

Big data, observational research and P-value: a recipe for false-positive findings? A study of simulated and real prospective cohorts.

ABSTRACT:

Background

An increasing number of observational studies combine large sample sizes with low participation rates, which could lead to standard inference failing to control the false-discovery rate. We investigated if the 'empirical calibration of P-value' method (EPCV), reliant on negative controls, can preserve type I error in the context of survival analysis.

Methods

We used simulated cohort studies with 50% participation rate and two different selection bias mechanisms, and a real-life application on predictors of cancer mortality using data from four population-based cohorts in Northern Italy (n = 6976 men and women aged 25-74 years at baseline and 17 years of median follow-up).

Results

Type I error for the standard Cox model was above the 5% nominal level in 15 out of 16 simulated settings; for n = 10 000, the chances of a null association with hazard ratio = 1.05 having a P-value < 0.05 were 42.5%. Conversely, EPCV with 10 negative controls preserved the 5% nominal level in all the simulation settings, reducing bias in the point estimate by 80-90% when its main assumption was verified. In the real case, 15 out of 21 (71%) blood markers with no association with cancer mortality according to literature had a P-value < 0.05 in age- and gender-adjusted Cox models. After calibration, only 1 (4.8%) remained statistically significant.

Conclusions

In the analyses of large observational studies prone to selection bias, the use of empirical distribution to calibrate P-values can substantially reduce the number of trivial results needing further screening for relevance and external validity.

SUBMITTER: Veronesi G

PROVIDER: S-EPMC7394945 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Big data, observational research and P-value: a recipe for false-positive findings? A study of simulated and real prospective cohorts.

Veronesi Giovanni G Grassi Guido G Savelli Giordano G Quatto Piero P Zambon Antonella A

International journal of epidemiology 20200601 3

<h4>Background</h4>An increasing number of observational studies combine large sample sizes with low participation rates, which could lead to standard inference failing to control the false-discovery rate. We investigated if the 'empirical calibration of P-value' method (EPCV), reliant on negative controls, can preserve type I error in the context of survival analysis.<h4>Methods</h4>We used simulated cohort studies with 50% participation rate and two different selection bias mechanisms, and a r ...[more]

PMID: 31620789

Similar Datasets

Project description:BackgroundSingle Nucleotide Polymorphisms (SNPs) are widely used molecular markers, and their use has increased massively since the inception of Next Generation Sequencing (NGS) technologies, which allow detection of large numbers of SNPs at low cost. However, both NGS data and their analysis are error-prone, which can lead to the generation of false positive (FP) SNPs. We explored the relationship between FP SNPs and seven factors involved in mapping-based variant calling - quality of the reference sequence, read length, choice of mapper and variant caller, mapping stringency and filtering of SNPs by read mapping quality and read depth. This resulted in 576 possible factor level combinations. We used error- and variant-free simulated reads to ensure that every SNP found was indeed a false positive.ResultsThe variation in the number of FP SNPs generated ranged from 0 to 36,621 for the 120 million base pairs (Mbp) genome. All of the experimental factors tested had statistically significant effects on the number of FP SNPs generated and there was a considerable amount of interaction between the different factors. Using a fragmented reference sequence led to a dramatic increase in the number of FP SNPs generated, as did relaxed read mapping and a lack of SNP filtering. The choice of reference assembler, mapper and variant caller also significantly affected the outcome. The effect of read length was more complex and suggests a possible interaction between mapping specificity and the potential for contributing more false positives as read length increases.ConclusionsThe choice of tools and parameters involved in variant calling can have a dramatic effect on the number of FP SNPs produced, with particularly poor combinations of software and/or parameter settings yielding tens of thousands in this experiment. Between-factor interactions make simple recommendations difficult for a SNP discovery pipeline but the quality of the reference sequence is clearly of paramount importance. Our findings are also a stark reminder that it can be unwise to use the relaxed mismatch settings provided as defaults by some read mappers when reads are being mapped to a relatively unfinished reference sequence from e.g. a non-model organism in its early stages of genomic exploration.

Project description:PurposeTo evaluate the risk of cancer (positive predictive value [PPV]) associated with specific findings (mass, calcifications, architectural distortion, asymmetry) in mammographic examinations with abnormal results, to determine the distribution of these findings in examinations in which the patients received a diagnosis of cancer and examinations in which the patients did not, and to analyze PPV variation according to radiologist and patient factors.Materials and methodsHIPAA-compliant institutional review board approval was obtained. PPV of mammographic findings was evaluated in a prospective cohort of 10,262 women who underwent 10,641 screening or diagnostic mammographic examinations with abnormal results between January 1998 and December 2002 in the San Francisco Mammography Registry. The cohort was linked with the Surveillance Epidemiology and End Results program to determine cancer status among these women. PPVs were calculated for each finding and were stratified according to patient characteristics, cancer type, and radiologist reader.ResultsCases of breast cancer (n = 1552) were identified (invasive, n = 1287; ductal carcinoma in situ, n = 270); in five, both kinds of breast cancer were recorded. Overall, of the number of interpretations, masses were most frequently noted in 56%, followed by calcifications in 29%, asymmetry in 12%, and architectural distortion in 4%. Masses, calcifications, architectural distortion, and developing asymmetry demonstrated similar PPVs in screening examinations (9.7%, 12.7%, 10.2%, and 7.4%, respectively), whereas one-view-only and focal asymmetry demonstrated lower PPVs (3.6% and 3.7%, respectively) and were a frequent reason for an abnormal result (42%). Overall, one (5%) in 20 invasive cancers was identified with asymmetry, one (6%) in 16 invasive cancers was identified with architectural distortion, one (21%) in five invasive cancers was identified with calcifications, and two (68%) in three invasive cancers were identified with a mass.ConclusionFive percent of invasive cancers were identified with asymmetry, and asymmetry is more weakly associated with cancer in screening examinations than are mass, calcifications, and architectural distortion.

Dataset Information

Big data, observational research and P-value: a recipe for false-positive findings? A study of simulated and real prospective cohorts.

Background

Methods

Results

Conclusions

Publications

Big data, observational research and P-value: a recipe for false-positive findings? A study of simulated and real prospective cohorts.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets