Intrinsic evaluation of text mining tools may not predict performance on realistic tasks.

ABSTRACT: Biomedical text mining and other automated techniques are beginning to achieve performance which suggests that they could be applied to aid database curators. However, few studies have evaluated how these systems might work in practice. In this article we focus on the problem of annotating mutations in Protein Data Bank (PDB) entries, and evaluate the relationship between performance of two automated techniques, a text-mining-based approach (MutationFinder) and an alignment-based approach, in intrinsic versus extrinsic evaluations. We find that high performance on gold standard data (an intrinsic evaluation) does not necessarily translate to high performance for database annotation (an extrinsic evaluation). We show that this is in part a result of lack of access to the full text of journa

SUBMITTER: Caporaso JG 

PROVIDER: S-EPMC2517250 | biostudies-literature | 2008

REPOSITORIES: biostudies-literature