Dataset Information

Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites.

ABSTRACT:

Background

The increasing complexity of data streams and computational processes in modern clinical health information systems makes reproducibility challenging. Clinical natural language processing (NLP) pipelines are routinely leveraged for the secondary use of data. Workflow management systems (WMS) have been widely used in bioinformatics to handle the reproducibility bottleneck.

Objective

To evaluate if WMS and other bioinformatics practices could impact the reproducibility of clinical NLP frameworks.

Materials and methods

Based on the literature across multiple researcho fields (NLP, bioinformatics and clinical informatics) we selected articles which (1) review reproducibility practices and (2) highlight a set of rules or guidelines to ensure tool or pipeline reproducibility. We aggregate insight from the literature to define reproducibility recommendations. Finally, we assess the compliance of 7 NLP frameworks to the recommendations.

Results

We identified 40 reproducibility features from 8 selected articles. Frameworks based on WMS match more than 50% of features (26 features for LAPPS Grid, 22 features for OpenMinted) compared to 18 features for current clinical NLP framework (cTakes, CLAMP) and 17 features for GATE, ScispaCy, and Textflows.

Discussion

34 recommendations are endorsed by at least 2 articles from our selection. Overall, 15 features were adopted by every NLP Framework. Nevertheless, frameworks based on WMS had a better compliance with the features.

Conclusion

NLP frameworks could benefit from lessons learned from the bioinformatics field (eg, public repositories of curated tools and workflows or use of containers for shareability) to enhance the reproducibility in a clinical setting.

SUBMITTER: Digan W

PROVIDER: S-EPMC7936396 | biostudies-literature | 2021 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites.

Digan William W Névéol Aurélie A Neuraz Antoine A Wack Maxime M Baudoin David D Burgun Anita A Rance Bastien B

Journal of the American Medical Informatics Association : JAMIA 20210301 3

<h4>Background</h4>The increasing complexity of data streams and computational processes in modern clinical health information systems makes reproducibility challenging. Clinical natural language processing (NLP) pipelines are routinely leveraged for the secondary use of data. Workflow management systems (WMS) have been widely used in bioinformatics to handle the reproducibility bottleneck.<h4>Objective</h4>To evaluate if WMS and other bioinformatics practices could impact the reproducibility of ...[more]

PMID: 33319904

Similar Datasets

Project description:BackgroundSymptoms are a core concept of nursing interest. Large-scale secondary data reuse of notes in electronic health records (EHRs) has the potential to increase the quantity and quality of symptom research. However, the symptom language used in clinical notes is complex. A need exists for methods designed specifically to identify and study symptom information from EHR notes.ObjectivesWe aim to describe a method that combines standardized vocabularies, clinical expertise, and natural language processing to generate comprehensive symptom vocabularies and identify symptom information in EHR notes. We piloted this method with five diverse symptom concepts: constipation, depressed mood, disturbed sleep, fatigue, and palpitations.MethodsFirst, we obtained synonym lists for each pilot symptom concept from the Unified Medical Language System. Then, we used two large bodies of text (clinical notes from Columbia University Irving Medical Center and PubMed abstracts containing Medical Subject Headings or key words related to the pilot symptoms) to further expand our initial vocabulary of synonyms for each pilot symptom concept. We used NimbleMiner, an open-source natural language processing tool, to accomplish these tasks and evaluated NimbleMiner symptom identification performance by comparison to a manually annotated set of nurse- and physician-authored common EHR note types.ResultsCompared to the baseline Unified Medical Language System synonym lists, we identified up to 11 times more additional synonym words or expressions, including abbreviations, misspellings, and unique multiword combinations, for each symptom concept. Natural language processing system symptom identification performance was excellent.DiscussionUsing our comprehensive symptom vocabularies and NimbleMiner to label symptoms in clinical notes produced excellent performance metrics. The ability to extract symptom information from EHR notes in an accurate and scalable manner has the potential to greatly facilitate symptom science research.

Project description:ObjectivesTo identify childhood respiratory tract-related illness presentation rates and service utilisation in primary care by interrogating free text and coded data from electronic medical records.DesignRetrospective cohort study. Data interrogation used a natural language processing software inference algorithm.Setting36 primary care practices in New Zealand. Data analysed from January 2008 to December 2013.ParticipantsThe records from 77 582 children enrolled were reviewed over a 6-year period to estimate the presentation of childhood respiratory illness and service utilisation. This cohort represents 268 919 person-years of data and over 650 000 unique consultations.Main outcome measureChildhood respiratory illness presentation rate to primary care practice, with description of seasonal and yearly variation.ResultsRespiratory conditions constituted 46% of all child-general practitioner consultations with a stable year-on-year pattern of seasonal peaks. Upper respiratory tract infection was the most common respiratory category accounting for 21.0% of all childhood consultations, followed by otitis media (12.2%), wheeze-related illness (9.7%), throat infection (7.4%) and lower respiratory tract infection (4.4%). Almost 70% of children presented to their general practitioner with at least one respiratory condition in their first year of life; this reduced to approximately 25% for children aged 10-17.ConclusionThis is the first study to assess the primary care incidence and service utilisation of childhood respiratory illness in a large primary care cohort by interrogating electronic medical record free text. The study identified the very high primary care workload related to childhood respiratory illness, especially during the first 2 years of life. These data can enable more effective planning of health service delivery. The findings and methodology have relevance to many countries, and the use of primary care 'big data' in this way can be applied to other health conditions.

Project description:BackgroundNovel approaches that complement and go beyond evidence-based medicine are required in the domain of chronic diseases, given the growing incidence of such conditions on the worldwide population. A promising avenue is the secondary use of electronic health records (EHRs), where patient data are analyzed to conduct clinical and translational research. Methods based on machine learning to process EHRs are resulting in improved understanding of patient clinical trajectories and chronic disease risk prediction, creating a unique opportunity to derive previously unknown clinical insights. However, a wealth of clinical histories remains locked behind clinical narratives in free-form text. Consequently, unlocking the full potential of EHR data is contingent on the development of natural language processing (NLP) methods to automatically transform clinical text into structured clinical data that can guide clinical decisions and potentially delay or prevent disease onset.ObjectiveThe goal of the research was to provide a comprehensive overview of the development and uptake of NLP methods applied to free-text clinical notes related to chronic diseases, including the investigation of challenges faced by NLP methodologies in understanding clinical narratives.MethodsPreferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed and searches were conducted in 5 databases using "clinical notes," "natural language processing," and "chronic disease" and their variations as keywords to maximize coverage of the articles.ResultsOf the 2652 articles considered, 106 met the inclusion criteria. Review of the included papers resulted in identification of 43 chronic diseases, which were then further classified into 10 disease categories using the International Classification of Diseases, 10th Revision. The majority of studies focused on diseases of the circulatory system (n=38) while endocrine and metabolic diseases were fewest (n=14). This was due to the structure of clinical records related to metabolic diseases, which typically contain much more structured data, compared with medical records for diseases of the circulatory system, which focus more on unstructured data and consequently have seen a stronger focus of NLP. The review has shown that there is a significant increase in the use of machine learning methods compared to rule-based approaches; however, deep learning methods remain emergent (n=3). Consequently, the majority of works focus on classification of disease phenotype with only a handful of papers addressing extraction of comorbidities from the free text or integration of clinical notes with structured data. There is a notable use of relatively simple methods, such as shallow classifiers (or combination with rule-based methods), due to the interpretability of predictions, which still represents a significant issue for more complex methods. Finally, scarcity of publicly available data may also have contributed to insufficient development of more advanced methods, such as extraction of word embeddings from clinical notes.ConclusionsEfforts are still required to improve (1) progression of clinical NLP methods from extraction toward understanding; (2) recognition of relations among entities rather than entities in isolation; (3) temporal extraction to understand past, current, and future clinical events; (4) exploitation of alternative sources of clinical knowledge; and (5) availability of large-scale, de-identified clinical corpora.

Dataset Information

Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites.

Background

Objective

Materials and methods

Results

Discussion

Conclusion

Publications

Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets