MaRe: Processing Big Data with application containers on Apache Spark.

ABSTRACT:

Background

Life science is increasingly driven by Big Data analytics, and the MapReduce programming model has been proven successful for data-intensive analyses. However, current MapReduce frameworks offer poor support for reusing existing processing tools in bioinformatics pipelines. Furthermore, these frameworks do not have native support for application containers, which are becoming popular in scientific data processing.

Results

Here we present MaRe, an open source programming library that introduces support for Docker containers in Apache Spark. Apache Spark and Docker are the MapReduce framework and container engine that have collected the largest open source community; thus, MaRe provides interoperability with the cutting-edge software ecosystem. We demonstrate MaRe o

SUBMITTER: Capuccini M 

PROVIDER: S-EPMC7199472 | biostudies-literature | 2020 May

REPOSITORIES: biostudies-literature