Dataset Information

Bag of little bootstraps for massive and distributed longitudinal data.

ABSTRACT: Linear mixed models are widely used for analyzing longitudinal datasets, and the inference for variance component parameters relies on the bootstrap method. However, health systems and technology companies routinely generate massive longitudinal datasets that make the traditional bootstrap method infeasible. To solve this problem, we extend the highly scalable bag of little bootstraps method for independent data to longitudinal data and develop a highly efficient Julia package MixedModelsBLB.jl. Simulation experiments and real data analysis demonstrate the favorable statistical performance and computational advantages of our method compared to the traditional bootstrap method. For the statistical inference of variance components, it achieves 200 times speedup on the scale of 1 million subjects (20 million total observations), and is the only currently available tool that can handle more than 10 million subjects (200 million total observations) using desktop computers.

SUBMITTER: Zhou X

PROVIDER: S-EPMC9159544 | biostudies-literature | 2022 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Bag of little bootstraps for massive and distributed longitudinal data.

Zhou Xinkai X Zhou Jin J JJ Zhou Hua H

Statistical analysis and data mining 20211122 3

Linear mixed models are widely used for analyzing longitudinal datasets, and the inference for variance component parameters relies on the bootstrap method. However, health systems and technology companies routinely generate massive longitudinal datasets that make the traditional bootstrap method infeasible. To solve this problem, we extend the highly scalable bag of little bootstraps method for independent data to longitudinal data and develop a highly efficient Julia package MixedModelsBLB.jl. ...[more]

PMID: 35656342

Similar Datasets

Project description:BACKGROUND:The advent of next-generation sequencing (NGS) has made whole-genome sequencing of cohorts of individuals a reality. Primary datasets of raw or aligned reads of this sort can get very large. For scientific questions where curated called variants are not sufficient, the sheer size of the datasets makes analysis prohibitively expensive. In order to make re-analysis of such data feasible without the need to have access to a large-scale computing facility, we have developed a highly scalable, storage-agnostic framework, an associated API and an easy-to-use web user interface to execute custom filters on large genomic datasets. RESULTS:We present BAMSI, a Software as-a Service (SaaS) solution for filtering of the 1000 Genomes phase 3 set of aligned reads, with the possibility of extension and customization to other sets of files. Unique to our solution is the capability of simultaneously utilizing many different mirrors of the data to increase the speed of the analysis. In particular, if the data is available in private or public clouds - an increasingly common scenario for both academic and commercial cloud providers - our framework allows for seamless deployment of filtering workers close to data. We show results indicating that such a setup improves the horizontal scalability of the system, and present a possible use case of the framework by performing an analysis of structural variation in the 1000 Genomes data set. CONCLUSIONS:BAMSI constitutes a framework for efficient filtering of large genomic data sets that is flexible in the use of compute as well as storage resources. The data resulting from the filter is assumed to be greatly reduced in size, and can easily be downloaded or routed into e.g. a Hadoop cluster for subsequent interactive analysis using Hive, Spark or similar tools. In this respect, our framework also suggests a general model for making very large datasets of high scientific value more accessible by offering the possibility for organizations to share the cost of hosting data on hot storage, without compromising the scalability of downstream analysis.

Project description:The biogeographic distribution of diversity among populations of threatened mammalian species is generally investigated using population genetics. However, intraspecific phenotypic diversity is rarely assessed beyond taxonomy-focused linear measurements or qualitative descriptions. Here, we use a technique widely used in the evolutionary sciences-geometric morphometrics-to characterize shape diversity in the skull of an endangered marsupial, the northern quoll, across its 5,000 km distribution range along Northern Australia. Skull shape is a proxy for feeding, behavior, and phenotypic differentiation, allowing us to ask whether populations can be distinguished and whether patterns of variation indicate adaptability to changing environmental conditions. We analyzed skull shape in 101 individuals across four mainland populations and several islands. We assessed the contribution of population, size, sex, rainfall, temperature, and geography to skull shape variation using principal component analysis, Procrustes ANOVA, and variation partitioning analyses. The populations harbor similar amounts of broadly overlapping skull shape variation, with relatively low geographic effects. Size predicted skull shape best, coinciding with braincase size variation and differences in zygomatic arches. Size-adjusted differences in populations explained less variation with far smaller effect sizes, relating to changes in the insertion areas of masticatory muscles, as well as the upper muzzle and incisor region. Climatic and geographic variables contributed little. Strikingly, the vast majority of shape variation-76%-remained unexplained. Our results suggest a uniform intraspecific scope for shape variation, possibly due to allometric constraints or phenotypic plasticity beyond the relatively strong allometric effect. The lack of local adaptation indicates that cross-breeding between populations will not reduce local morphological skull (and probably general musculoskeletal) adaptation because none exists. However, the potential for heritable morphological variation (e.g., specialization to local diets) seems exceedingly limited. We conclude that 3D geometric morphometrics can provide a comprehensive, statistically rigorous phenomic contribution to genetic-based conservation studies.

Project description:BackgroundNon-specific feature selection is a dimension reduction procedure performed prior to cluster analysis of high dimensional molecular data. Not all measured features are expected to show biological variation, so only the most varying are selected for analysis. In DNA methylation studies, DNA methylation is measured as a proportion, bounded between 0 and 1, with variance a function of the mean. Filtering on standard deviation biases the selection of probes to those with mean values near 0.5. We explore the effect this has on clustering, and develop alternate filter methods that utilize a variance stabilizing transformation for Beta distributed data and do not share this bias.ResultsWe compared results for 11 different non-specific filters on eight Infinium HumanMethylation data sets, selected to span a variety of biological conditions. We found that for data sets having a small fraction of samples showing abnormal methylation of a subset of normally unmethylated CpGs, a characteristic of the CpG island methylator phenotype in cancer, a novel filter statistic that utilized a variance-stabilizing transformation for Beta distributed data outperformed the common filter of using standard deviation of the DNA methylation proportion, or its log-transformed M-value, in its ability to detect the cancer subtype in a cluster analysis. However, the standard deviation filter always performed among the best for distinguishing subgroups of normal tissue. The novel filter and standard deviation filter tended to favour features in different genome contexts; for the same data set, the novel filter always selected more features from CpG island promoters and the standard deviation filter always selected more features from non-CpG island intergenic regions. Interestingly, despite selecting largely non-overlapping sets of features, the two filters did find sample subsets that overlapped for some real data sets.ConclusionsWe found two different filter statistics that tended to prioritize features with different characteristics, each performed well for identifying clusters of cancer and non-cancer tissue, and identifying a cancer CpG island hypermethylation phenotype. Since cluster analysis is for discovery, we would suggest trying both filters on any new data sets, evaluating the overlap of features selected and clusters discovered.

Dataset Information

Bag of little bootstraps for massive and distributed longitudinal data.

Publications

Bag of little bootstraps for massive and distributed longitudinal data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets