Dataset Information

Estimation of site frequency spectra from low-coverage sequencing data using stochastic EM reduces overfitting, runtime, and memory usage.

ABSTRACT: The site frequency spectrum is an important summary statistic in population genetics used for inference on demographic history and selection. However, estimation of the site frequency spectrum from called genotypes introduces bias when working with low-coverage sequencing data. Methods exist for addressing this issue but sometimes suffer from 2 problems. First, they can have very high computational demands, to the point that it may not be possible to run estimation for genome-scale data. Second, existing methods are prone to overfitting, especially for multidimensional site frequency spectrum estimation. In this article, we present a stochastic expectation-maximization algorithm for inferring the site frequency spectrum from NGS data that address these challenges. We show that this algorithm greatly reduces runtime and enables estimation with constant, trivial RAM usage. Furthermore, the algorithm reduces overfitting and thereby improves downstream inference. An implementation is available at github.com/malthesr/winsfs.

SUBMITTER: Rasmussen MS

PROVIDER: S-EPMC9713400 | biostudies-literature | 2022 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Estimation of site frequency spectra from low-coverage sequencing data using stochastic EM reduces overfitting, runtime, and memory usage.

Rasmussen Malthe Sebro MS Garcia-Erill Genís G Korneliussen Thorfinn Sand TS Wiuf Carsten C Albrechtsen Anders A

Genetics 20221101 4

The site frequency spectrum is an important summary statistic in population genetics used for inference on demographic history and selection. However, estimation of the site frequency spectrum from called genotypes introduces bias when working with low-coverage sequencing data. Methods exist for addressing this issue but sometimes suffer from 2 problems. First, they can have very high computational demands, to the point that it may not be possible to run estimation for genome-scale data. Second, ...[more]

PMID: 36173322

Similar Datasets

Project description:BackgroundRandom forests have become popular for clinical risk prediction modeling. In a case study on predicting ovarian malignancy, we observed training AUCs close to 1. Although this suggests overfitting, performance was competitive on test data. We aimed to understand the behavior of random forests for probability estimation by (1) visualizing data space in three real-world case studies and (2) a simulation study.MethodsFor the case studies, multinomial risk estimates were visualized using heatmaps in a 2-dimensional subspace. The simulation study included 48 logistic data-generating mechanisms (DGM), varying the predictor distribution, the number of predictors, the correlation between predictors, the true AUC, and the strength of true predictors. For each DGM, 1000 training datasets of size 200 or 4000 with binary outcomes were simulated, and random forest models were trained with minimum node size 2 or 20 using the ranger R package, resulting in 192 scenarios in total. Model performance was evaluated on large test datasets (N = 100,000).ResultsThe visualizations suggested that the model learned "spikes of probability" around events in the training set. A cluster of events created a bigger peak or plateau (signal), isolated events local peaks (noise). In the simulation study, median training AUCs were between 0.97 and 1 unless there were 4 binary predictors or 16 binary predictors with a minimum node size of 20. The median discrimination loss, i.e., the difference between the median test AUC and the true AUC, was 0.025 (range 0.00 to 0.13). Median training AUCs had Spearman correlations of around 0.70 with discrimination loss. Median test AUCs were higher with higher events per variable, higher minimum node size, and binary predictors. Median training calibration slopes were always above 1 and were not correlated with median test slopes across scenarios (Spearman correlation - 0.11). Median test slopes were higher with higher true AUC, higher minimum node size, and higher sample size.ConclusionsRandom forests learn local probability peaks that often yield near perfect training AUCs without strongly affecting AUCs on test data. When the aim is probability estimation, the simulation results go against the common recommendation to use fully grown trees in random forest models.

Project description:BACKGROUND:Recent research has found that abnormal functioning of Microtubules (MTs) could be linked to fatal diseases such as Alzheimer's. Hence, there is an imminent need to understand the implications of MTs for disease- diagnosis. However, studies of cellular processes like MTs are often constrained by physical limitations of their data acquisition systems such as optical microscopes and are vulnerable to either destruction of the specimen or the probe. In addition, study of MTs is challenged with non-uniform sampling of the MT dynamic instability phenomenon relative to its time-lapse observation of the cellular processes. Thus, the above caveats limit the overall period of time that the MT data can be collected, thereby causing limited data availability scenario. RESULTS:In this work, two novel superresolution frameworks based on Expectation Maximization (EM) based Maximum Likelihood (ML) estimation using Kalman filters (MLK) technique are proposed to address the issues of non-uniform sampling and limited data availability of MT signals. The proposed MLK methods optimizes prediction of missing observations in the MT signal through information extraction using correlation-based patch processing and principal component analysis -based mutual information. Experimental results prove that the proposed MLK-based superresolution methods outperformed nonlinear interpolation and compressed sensing methods. CONCLUSIONS:This work aims to address limited data availability and data/observation loss incurred due to non-uniform sampling of biological signals such as MTs. For this purpose, statistical modelling of stochastic MT signals using EM based ML driven Kalman estimation (MLK) is considered as a fundamental framework for prediction of missing MT observations. It was experimentally validated that the proposed superresolution methods provided superior overall performance, better MT signal estimation using fewer samples, high SNR, low errors, and better MT parameter estimation than other methods.

Dataset Information

Estimation of site frequency spectra from low-coverage sequencing data using stochastic EM reduces overfitting, runtime, and memory usage.

Publications

Estimation of site frequency spectra from low-coverage sequencing data using stochastic EM reduces overfitting, runtime, and memory usage.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets