Sampling for computational efficiency when conducting analyses in big data.

ABSTRACT: A challenge to research in big data is the inherent computational intensity of analyses, particularly when using rigorous methods to address biases. We demonstrate the use of sampling methods in big data to estimate parameters using fewer resources. Our motivating question was whether lung cancer incidence differs by baseline HIV status, using a cohort of nearly 30 million Medicaid beneficiaries. We targeted three parameters (with listed estimator): incidence rate ratio (IRR, Poisson model), hazard ratio (HR, Cox model), and risk ratio (RR, Kaplan-Meier). We controlled for confounders using inverse probability weighting. We ran analyses using the full sample and several sampling schemes: divide-and-recombine (10, 20, 50 samples), sub-cohort, and case-cohort. We compared point estimates, st

SUBMITTER: Rudolph JE 

PROVIDER: S-EPMC12862598 | biostudies-literature | 2025 Dec

REPOSITORIES: biostudies-literature