Field of genes: using Apache Kafka as a bioinformatic data repository.

ABSTRACT:

Background

Bioinformatic research is increasingly dependent on large-scale datasets, accessed either from private or public repositories. An example of a public repository is National Center for Biotechnology Information's (NCBI's) Reference Sequence (RefSeq). These repositories must decide in what form to make their data available. Unstructured data can be put to almost any use but are limited in how access to them can be scaled. Highly structured data offer improved performance for specific algorithms but limit the wider usefulness of the data. We present an alternative: lightly structured data stored in Apache Kafka in a way that is amenable to parallel access and streamed processing, including subsequent transformations into more highly structured representations. We contend th

SUBMITTER: Lawlor B 

PROVIDER: S-EPMC5906921 | biostudies-literature | 2018 Apr

REPOSITORIES: biostudies-literature