FDup: a framework for general-purpose and efficient entity deduplication of record collections.

ABSTRACT: Deduplication is a technique aiming at identifying and resolving duplicate metadata records in a collection. This article describes FDup (Flat Collections Deduper), a general-purpose software framework supporting a complete deduplication workflow to manage big data record collections: metadata record data model definition, identification of candidate duplicates, identification of duplicates. FDup brings two main innovations: first, it delivers a full deduplication framework in a single easy-to-use software package based on Apache Spark Hadoop framework, where developers can customize the optimal and parallel workflow steps of blocking, sliding windows, and similarity matching function via an intuitive configuration file; second, it introduces a novel approach to improve performance, beyond

SUBMITTER: De Bonis M 

PROVIDER: S-EPMC9575841 | biostudies-literature | 2022

REPOSITORIES: biostudies-literature