Dataset Information

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks.

ABSTRACT: Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

SUBMITTER: Koehler Leman J

PROVIDER: S-EPMC8630030 | biostudies-literature | 2021 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks.

Koehler Leman Julia J Lyskov Sergey S Lewis Steven M SM Adolf-Bryfogle Jared J Alford Rebecca F RF Barlow Kyle K Ben-Aharon Ziv Z Farrell Daniel D Fell Jason J Hansen William A WA Harmalkar Ameya A Jeliazkov Jeliazko J Kuenze Georg G Krys Justyna D JD Ljubetič Ajasja A Loshbaugh Amanda L AL Maguire Jack J Moretti Rocco R Mulligan Vikram Khipple VK Nance Morgan L ML Nguyen Phuong T PT Ó Conchúir Shane S Roy Burman Shourya S SS Samanta Rituparna R Smith Shannon T ST Teets Frank F Tiemann Johanna K S JKS Watkins Andrew A Woods Hope H Yachnin Brahm J BJ Bahl Christopher D CD Bailey-Kellogg Chris C Baker David D Das Rhiju R DiMaio Frank F Khare Sagar D SD Kortemme Tanja T Labonte Jason W JW Lindorff-Larsen Kresten K Meiler Jens J Schief William W Schueler-Furman Ora O Siegel Justin B JB Stein Amelie A Yarov-Yarovoy Vladimir V Kuhlman Brian B Leaver-Fay Andrew A Gront Dominik D Gray Jeffrey J JJ Bonneau Richard R

Nature communications 20211129 1

Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchm ...[more]

PMID: 34845212

Similar Datasets

Project description:BackgroundDeveloping and evaluating new technology that enables researchers to recover gene-expression levels of colonic cells from fecal samples could be key to a non-invasive screening tool for early detection of colon cancer. The current study, to the best of our knowledge, is the first to investigate and report the reproducibility of fecal microarray data. Using the intraclass correlation coefficient (ICC) as a measure of reproducibility and the preliminary analysis of fecal and mucosal data, we assessed the reliability of mixture density estimation and the reproducibility of fecal microarray data. Using Monte Carlo-based methods, we explored whether ICC values should be modeled as a beta-mixture or transformed first and fitted with a normal-mixture. We used outcomes from bootstrapped goodness-of-fit tests to determine which approach is less sensitive toward potential violation of distributional assumptions.ResultsThe graphical examination of both the distributions of ICC and probit-transformed ICC (PT-ICC) clearly shows that there are two components in the distributions. For ICC measurements, which are between 0 and 1, the practice in literature has been to assume that the data points are from a beta-mixture distribution. Nevertheless, in our study we show that the use of a normal-mixture modeling approach on PT-ICC could provide superior performance.ConclusionsWhen modeling ICC values of gene expression levels, using mixture of normals in the probit-transformed (PT) scale is less sensitive toward model mis-specification than using mixture of betas. We show that a biased conclusion could be made if we follow the traditional approach and model the two sets of ICC values using the mixture of betas directly. The problematic estimation arises from the sensitivity of beta-mixtures toward model mis-specification, particularly when there are observations in the neighborhood of the the boundary points, 0 or 1. Since beta-mixture modeling is commonly used in approximating the distribution of measurements between 0 and 1, our findings have important implications beyond the findings of the current study. By using the normal-mixture approach on PT-ICC, we observed the quality of reproducible genes in fecal array data to be comparable to those in mucosal arrays.

Dataset Information

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks.

Publications

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets