Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

First steps into the cloud: Using Amazon data storage and computing with Python notebooks.

ABSTRACT: With the oncoming age of big data, biologists are encountering more use cases for cloud-based computing to streamline data processing and storage. Unfortunately, cloud platforms are difficult to learn, and there are few resources for biologists to demystify them. We have developed a guide for experimental biologists to set up cloud processing on Amazon Web Services to cheaply outsource data processing and storage. Here we provide a guide for setting up a computing environment in the cloud and showcase examples of using Python and Julia programming languages. We present example calcium imaging data in the zebrafish brain and corresponding analysis using suite2p software. Tools for budget and user management are further discussed in the attached protocol. Using this guide, researchers with limited coding experience can get started with cloud-based computing or move existing coding infrastructure into the cloud environment.

SUBMITTER: Pollak DJ

PROVIDER: S-EPMC9910747 | biostudies-literature | 2023

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

First steps into the cloud: Using Amazon data storage and computing with Python notebooks.

Pollak Daniel J DJ Chawla Gautam G Andreev Andrey A Prober David A DA

PloS one 20230209 2

With the oncoming age of big data, biologists are encountering more use cases for cloud-based computing to streamline data processing and storage. Unfortunately, cloud platforms are difficult to learn, and there are few resources for biologists to demystify them. We have developed a guide for experimental biologists to set up cloud processing on Amazon Web Services to cheaply outsource data processing and storage. Here we provide a guide for setting up a computing environment in the cloud and sh ...[more]

PMID: 36757918

Similar Datasets

Toward collaborative open data science in metabolomics using Jupyter Notebooks and cloud computing.

Project description:BackgroundA lack of transparency and reporting standards in the scientific community has led to increasing and widespread concerns relating to reproduction and integrity of results. As an omics science, which generates vast amounts of data and relies heavily on data science for deriving biological meaning, metabolomics is highly vulnerable to irreproducibility. The metabolomics community has made substantial efforts to align with FAIR data standards by promoting open data formats, data repositories, online spectral libraries, and metabolite databases. Open data analysis platforms also exist; however, they tend to be inflexible and rely on the user to adequately report their methods and results. To enable FAIR data science in metabolomics, methods and results need to be transparently disseminated in a manner that is rapid, reusable, and fully integrated with the published work. To ensure broad use within the community such a framework also needs to be inclusive and intuitive for both computational novices and experts alike.Aim of reviewTo encourage metabolomics researchers from all backgrounds to take control of their own data science, mould it to their personal requirements, and enthusiastically share resources through open science.Key scientific concepts of reviewThis tutorial introduces the concept of interactive web-based computational laboratory notebooks. The reader is guided through a set of experiential tutorials specifically targeted at metabolomics researchers, based around the Jupyter Notebook web application, GitHub data repository, and Binder cloud computing platform.

| S-EPMC6745024 | biostudies-literature

Nanopore sequencing data analysis using Microsoft Azure cloud computing service.

Project description:Genetic information provides insights into the exome, genome, epigenetics and structural organisation of the organism. Given the enormous amount of genetic information, scientists are able to perform mammoth tasks to improve the standard of health care such as determining genetic influences on outcome of allogeneic transplantation. Cloud based computing has increasingly become a key choice for many scientists, engineers and institutions as it offers on-demand network access and users can conveniently rent rather than buy all required computing resources. With the positive advancements of cloud computing and nanopore sequencing data output, we were motivated to develop an automated and scalable analysis pipeline utilizing cloud infrastructure in Microsoft Azure to accelerate HLA genotyping service and improve the efficiency of the workflow at lower cost. In this study, we describe (i) the selection process for suitable virtual machine sizes for computing resources to balance between the best performance versus cost effectiveness; (ii) the building of Docker containers to include all tools in the cloud computational environment; (iii) the comparison of HLA genotype concordance between the in-house manual method and the automated cloud-based pipeline to assess data accuracy. In conclusion, the Microsoft Azure cloud based data analysis pipeline was shown to meet all the key imperatives for performance, cost, usability, simplicity and accuracy. Importantly, the pipeline allows for the on-going maintenance and testing of version changes before implementation. This pipeline is suitable for the data analysis from MinION sequencing platform and could be adopted for other data analysis application processes.

| S-EPMC9718390 | biostudies-literature

CloudProteoAnalyzer: scalable processing of big data from proteomics using cloud computing

Project description: Not available

| S-EPMC10942798 | biostudies-literature

Cloud Computing Enabled Big Multi-Omics Data Analytics.

Project description:High-throughput experiments enable researchers to explore complex multifactorial diseases through large-scale analysis of omics data. Challenges for such high-dimensional data sets include storage, analyses, and sharing. Recent innovations in computational technologies and approaches, especially in cloud computing, offer a promising, low-cost, and highly flexible solution in the bioinformatics domain. Cloud computing is rapidly proving increasingly useful in molecular modeling, omics data analytics (eg, RNA sequencing, metabolomics, or proteomics data sets), and for the integration, analysis, and interpretation of phenotypic data. We review the adoption of advanced cloud-based and big data technologies for processing and analyzing omics data and provide insights into state-of-the-art cloud bioinformatics applications.

| S-EPMC8323418 | biostudies-literature

BioJupies: Automated Generation of Interactive Notebooks for RNA-Seq Data Analysis in the Cloud.

Project description:BioJupies is a web application that enables the automated creation, storage, and deployment of Jupyter Notebooks containing RNA-seq data analyses. Through an intuitive interface, novice users can rapidly generate tailored reports to analyze and visualize their own raw sequencing files, gene expression tables, or fetch data from >9,000 published studies containing >300,000 preprocessed RNA-seq samples. Generated notebooks have the executable code of the entire pipeline, rich narrative text, interactive data visualizations, differential expression, and enrichment analyses. The notebooks are permanently stored in the cloud and made available online through a persistent URL. The notebooks are downloadable, customizable, and can run within a Docker container. By providing an intuitive user interface for notebook generation for RNA-seq data analysis, starting from the raw reads all the way to a complete interactive and reproducible report, BioJupies is a useful resource for experimental and computational biologists. BioJupies is freely available as a web-based application from http://biojupies.cloud.

| S-EPMC6265050 | biostudies-literature

Creating Ion Channel Kinetic Models Using Cloud Computing.

Project description:Computational modeling of ion channels provides key insight into experimental electrophysiology results and can be used to connect channel dynamics to emergent phenomena observed at the tissue and organ levels. However, creation of these models requires substantial mathematical and computational background. This tutorial seeks to lower the barrier to creating these models by providing an automated pipeline for creating Markov models of an ion channel kinetics dataset. We start by detailing how to encode sample voltage-clamp protocols and experimental data into the program and its implementation in a cloud computing environment. We guide the reader on how to build a containerized instance, push the machine image, and finally run the routine on cluster nodes. While providing open-source code has become more standard in computational studies, this tutorial provides unprecedented detail on the use of the program and the creation of channel models, starting from inputting the raw experimental data. © 2022 Wiley Periodicals LLC. Basic Protocol: Creation of ion channel kinetic models with a cloud computing environment Alternate Protocol: Instructions for use in a standard high-performance compute cluster.

| S-EPMC9006544 | biostudies-literature

Harnessing cloud computing with Galaxy Cloud.

Project description: Not available

| S-EPMC3868438 | biostudies-literature

Cloud computing for voxel-wise SEM analysis of MRI data.

Project description:As data collection costs fall and vast quantities of data are collected, data analysis time can become a bottleneck. For massively parallel analyses, cloud computing offers the short-term rental of ample processing power. Recent software innovations have reduced the offort needed to take advantage of cloud computing. To demonstrate, we replicate a voxel-wise examination of the genetic contributions to cortical development by age using evidence from 1,748 MRI scans. Specifically, we employ off-the-shelf Kubernetes software that permits us to re-run our analyses using almost the same computer code as was published in the original article. Large, well funded institutions may continue to maintain their own computing clusters. However, the modest cost of renting and ease of utilizing cloud computing services makes unprecedented compute power available to all researchers, whether or not affliated with a research institution. We expect this to spur innovation in the sophisticated modeling of large datasets.

| S-EPMC6534137 | biostudies-literature

A novel data storage logic in the cloud.

Project description:Databases which store and manage long-term scientific information related to life science are used to store huge amount of quantitative attributes. Introduction of a new entity attribute requires modification of the existing data tables and the programs that use these data tables. The solution is increasing the virtual data tables while the number of screens remains the same. The main objective of the present study was to introduce a logic called Joker Tao (JT) which provides universal data storage for cloud-based databases. It means all types of input data can be interpreted as an entity and attribute at the same time, in the same data table.

| S-EPMC5583739 | biostudies-literature

Integrative web cloud computing and analytics using MiPair for design-based comparative analysis with paired microbiome data.

Project description:Pairing (or blocking) is a design technique that is widely used in comparative microbiome studies to efficiently control for the effects of potential confounders (e.g., genetic, environmental, or behavioral factors). Some typical paired (block) designs for human microbiome studies are repeated measures designs that profile each subject's microbiome twice (or more than twice) (1) for pre and post treatments to see the effects of a treatment on microbiome, or (2) for different organs of the body (e.g., gut, mouth, skin) to see the disparity in microbiome between (or across) body sites. Researchers have developed a sheer number of web-based tools for user-friendly microbiome data processing and analytics, though there is no web-based tool currently available for such paired microbiome studies. In this paper, we thus introduce an integrative web-based tool, named MiPair, for design-based comparative analysis with paired microbiome data. MiPair is a user-friendly web cloud service that is built with step-by-step data processing and analytic procedures for comparative analysis between (or across) groups or between baseline and other groups. MiPair employs parametric and non-parametric tests for complete or incomplete block designs to perform comparative analyses with respect to microbial ecology (alpha- and beta-diversity) and taxonomy (e.g., phylum, class, order, family, genus, species). We demonstrate its usage through an example clinical trial on the effects of antibiotics on gut microbiome. MiPair is an open-source software that can be run on our web server ( http://mipair.micloud.kr ) or on user's computer ( https://github.com/yj7599/mipairgit ).

| S-EPMC9705534 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data