Dataset Information

Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network.

ABSTRACT: The electronic Medical Records and Genomics (eMERGE) Network assessed the feasibility of deploying portable phenotype rule-based algorithms with natural language processing (NLP) components added to improve performance of existing algorithms using electronic health records (EHRs). Based on scientific merit and predicted difficulty, eMERGE selected six existing phenotypes to enhance with NLP. We assessed performance, portability, and ease of use. We summarized lessons learned by: (1) challenges; (2) best practices to address challenges based on existing evidence and/or eMERGE experience; and (3) opportunities for future research. Adding NLP resulted in improved, or the same, precision and/or recall for all but one algorithm. Portability, phenotyping workflow/process, and technology were major themes. With NLP, development and validation took longer. Besides portability of NLP technology and algorithm replicability, factors to ensure success include privacy protection, technical infrastructure setup, intellectual property agreement, and efficient communication. Workflow improvements can improve communication and reduce implementation time. NLP performance varied mainly due to clinical document heterogeneity; therefore, we suggest using semi-structured notes, comprehensive documentation, and customization options. NLP portability is possible with improved phenotype algorithm performance, but careful planning and architecture of the algorithms is essential to support local customizations.

SUBMITTER: Pacheco JA

PROVIDER: S-EPMC9898520 | biostudies-literature | 2023 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network.

Pacheco Jennifer A JA Rasmussen Luke V LV Wiley Ken K Person Thomas Nate TN Cronkite David J DJ Sohn Sunghwan S Murphy Shawn S Gundelach Justin H JH Gainer Vivian V Castro Victor M VM Liu Cong C Mentch Frank F Lingren Todd T Sundaresan Agnes S AS Eickelberg Garrett G Willis Valerie V Furmanchuk Al'ona A Patel Roshan R Carrell David S DS Deng Yu Y Walton Nephi N Satterfield Benjamin A BA Kullo Iftikhar J IJ Dikilitas Ozan O Smith Joshua C JC Peterson Josh F JF Shang Ning N Kiryluk Krzysztof K Ni Yizhao Y Li Yikuan Y Nadkarni Girish N GN Rosenthal Elisabeth A EA Walunas Theresa L TL Williams Marc S MS Karlson Elizabeth W EW Linder Jodell E JE Luo Yuan Y Weng Chunhua C Wei WeiQi W

Scientific reports 20230203 1

The electronic Medical Records and Genomics (eMERGE) Network assessed the feasibility of deploying portable phenotype rule-based algorithms with natural language processing (NLP) components added to improve performance of existing algorithms using electronic health records (EHRs). Based on scientific merit and predicted difficulty, eMERGE selected six existing phenotypes to enhance with NLP. We assessed performance, portability, and ease of use. We summarized lessons learned by: (1) challenges; ...[more]

PMID: 36737471

Similar Datasets

Project description:OBJECTIVES:This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance. METHODS:We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis. RESULTS:On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser). CONCLUSION:In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.

Project description:Natural language descriptions of organismal phenotypes, a principal object of study in biology, are abundant in the biological literature. Expressing these phenotypes as logical statements using ontologies would enable large-scale analysis on phenotypic information from diverse systems. However, considerable human effort is required to make these phenotype descriptions amenable to machine reasoning. Natural language processing tools have been developed to facilitate this task, and the training and evaluation of these tools depend on the availability of high quality, manually annotated gold standard data sets. We describe the development of an expert-curated gold standard data set of annotated phenotypes for evolutionary biology. The gold standard was developed for the curation of complex comparative phenotypes for the Phenoscape project. It was created by consensus among three curators and consists of entity-quality expressions of varying complexity. We use the gold standard to evaluate annotations created by human curators and those generated by the Semantic CharaParser tool. Using four annotation accuracy metrics that can account for any level of relationship between terms from two phenotype annotations, we found that machine-human consistency, or similarity, was significantly lower than inter-curator (human-human) consistency. Surprisingly, allowing curatorsaccess to external information did not significantly increase the similarity of their annotations to the gold standard or have a significant effect on inter-curator consistency. We found that the similarity of machine annotations to the gold standard increased after new relevant ontology terms had been added. Evaluation by the original authors of the character descriptions indicated that the gold standard annotations came closer to representing their intended meaning than did either the curator or machine annotations. These findings point toward ways to better design software to augment human curators and the use of the gold standard corpus will allow training and assessment of new tools to improve phenotype annotation accuracy at scale.

Project description:BackgroundThroughout the COVID-19 pandemic, many hospitals conducted routine testing of hospitalized patients for SARS-CoV-2 infection upon admission. Some of these patients are admitted for reasons unrelated to COVID-19 and incidentally test positive for the virus. Because COVID-19-related hospitalizations have become a critical public health indicator, it is important to identify patients who are hospitalized because of COVID-19 as opposed to those who are admitted for other indications.ObjectiveWe compared the performance of different computable phenotype definitions for COVID-19 hospitalizations that use different types of data from electronic health records (EHRs), including structured EHR data elements, clinical notes, or a combination of both data types.MethodsWe conducted a retrospective data analysis, using clinician chart review-based validation at a large academic medical center. We reviewed and analyzed the charts of 586 hospitalized individuals who tested positive for SARS-CoV-2 in January 2022. We used LASSO (least absolute shrinkage and selection operator) regression and random forests to fit classification algorithms that incorporated structured EHR data elements, clinical notes, or a combination of structured data and clinical notes. We used natural language processing to incorporate data from clinical notes. The performance of each model was evaluated based on the area under the receiver operator characteristic curve (AUROC) and an associated decision rule based on sensitivity and positive predictive value. We also identified top words and clinical indicators of COVID-19-specific hospitalization and assessed the impact of different phenotyping strategies on estimated hospital outcome metrics.ResultsBased on a chart review, 38.2% (224/586) of patients were determined to have been hospitalized for reasons other than COVID-19, despite having tested positive for SARS-CoV-2. A computable phenotype that used clinical notes had significantly better discrimination than one that used structured EHR data elements (AUROC: 0.894 vs 0.841; P<.001) and performed similarly to a model that combined clinical notes with structured data elements (AUROC: 0.894 vs 0.893; P=.91). Assessments of hospital outcome metrics significantly differed based on whether the population included all hospitalized patients who tested positive for SARS-CoV-2 or those who were determined to have been hospitalized due to COVID-19.ConclusionsThese findings highlight the importance of cause-specific phenotyping for COVID-19 hospitalizations. More generally, this work demonstrates the utility of natural language processing approaches for deriving information related to patient hospitalizations in cases where there may be multiple conditions that could serve as the primary indication for hospitalization.

Project description:IntroductionFrontotemporal dementia (FTD) encompasses a clinically and pathologically diverse group of neurodegenerative disorders, yet little work has quantified the unique phenotypic clinical presentations of FTD among post-9/11 era veterans. To identify phenotypes of FTD using natural language processing (NLP) aided medical chart reviews of post-9/11 era U.S. military Veterans diagnosed with FTD in Veterans Health Administration care.MethodsA medical record chart review of clinician/provider notes was conducted using a Natural Language Processing (NLP) tool, which extracted features related to cognitive dysfunction. NLP features were further organized into seven Research Domain Criteria Initiative (RDoC) domains, which were clustered to identify distinct phenotypes.ResultsVeterans with FTD were more likely to have notes that reflected the RDoC domains, with cognitive and positive valence domains showing the greatest difference across groups. Clustering of domains identified three symptom phenotypes agnostic to time of an individual having FTD, categorized as Low (16.4%), Moderate (69.2%), and High (14.5%) distress. Comparison across distress groups showed significant differences in physical and psychological characteristics, particularly prior history of head injury, insomnia, cardiac issues, anxiety, and alcohol misuse. The clustering result within the FTD group demonstrated a phenotype variant that exhibited a combination of language and behavioral symptoms. This phenotype presented with manifestations indicative of both language-related impairments and behavioral changes, showcasing the coexistence of features from both domains within the same individual.DiscussionThis study suggests FTD also presents across a continuum of severity and symptom distress, both within and across variants. The intensity of distress evident in clinical notes tends to cluster with more co-occurring conditions. This examination of phenotypic heterogeneity in clinical notes indicates that sensitivity to FTD diagnosis may be correlated to overall symptom distress, and future work incorporating NLP and phenotyping may help promote strategies for early detection of FTD.

Dataset Information

Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network.

Publications

Evaluation of the portability of computable phenotypes with natural language processing in the eMERGE network.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets