Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit.

ABSTRACT:

Background

Large streamed datasets, characteristic of life science applications, are often resource-intensive to process, transport and store. We propose a pipeline model, a design pattern for scientific pipelines, where an incoming stream of scientific data is organized into a tiered or ordered "data hierarchy". We introduce the HASTE Toolkit, a proof-of-concept cloud-native software toolkit based on this pipeline model, to partition and prioritize data streams to optimize use of limited computing resources.

Findings

In our pipeline model, an "interestingness function" assigns an interestingness score to data objects in the stream, inducing a data hierarchy. From this score, a "policy" guides decisions on how to prioritize computational resource use for a given object. The

SUBMITTER: Blamey B 

PROVIDER: S-EPMC7976223 | biostudies-literature | 2021 Mar

REPOSITORIES: biostudies-literature