Dataset Information

Knowledge-enhanced visual-language pre-training on chest radiology images.

ABSTRACT: While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose an approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and radiology reports. We evaluate KAD on four external X-ray datasets and demonstrate that its zero-shot performance is not only comparable to that of fully supervised models but also superior to the average of three expert radiologists for three (out of five) pathologies with statistical significance. Moreover, when few-shot annotation is available, KAD outperforms all existing approaches in fine-tuning settings, demonstrating its potential for application in different clinical scenarios.

SUBMITTER: Zhang X

PROVIDER: S-EPMC10382552 | biostudies-literature | 2023 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Knowledge-enhanced visual-language pre-training on chest radiology images.

Zhang Xiaoman X Wu Chaoyi C Zhang Ya Y Xie Weidi W Wang Yanfeng Y

Nature communications 20230728 1

While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose an approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and ra ...[more]

PMID: 37507376

Similar Datasets

Project description:ObjectivesWe sought to investigate if artificial medical images can blend with original ones and whether they adhere to the variable anatomical constraints provided.MethodsArtificial images were generated with a generative model trained on publicly available standard and low-dose chest CT images (805 scans; 39,803 2D images), of which 17% contained evidence of pathological formations (lung nodules). The test set (90 scans; 5121 2D images) was used to assess if artificial images (512 × 512 primary and control image sets) blended in with original images, using both quantitative metrics and expert opinion. We further assessed if pathology characteristics in the artificial images can be manipulated.ResultsPrimary and control artificial images attained an average objective similarity of 0.78 ± 0.04 (ranging from 0 [entirely dissimilar] to 1[identical]) and 0.76 ± 0.06, respectively. Five radiologists with experience in chest and thoracic imaging provided a subjective measure of image quality; they rated artificial images as 3.13 ± 0.46 (range of 1 [unrealistic] to 4 [almost indistinguishable to the original image]), close to their rating of the original images (3.73 ± 0.31). Radiologists clearly distinguished images in the control sets (2.32 ± 0.48 and 1.07 ± 0.19). In almost a quarter of the scenarios, they were not able to distinguish primary artificial images from the original ones.ConclusionArtificial images can be generated in a way such that they blend in with original images and adhere to anatomical constraints, which can be manipulated to augment the variability of cases.Critical relevance statementArtificial medical images can be used to enhance the availability and variety of medical training images by creating new but comparable images that can blend in with original images.Key points• Artificial images, similar to original ones, can be created using generative networks. • Pathological features of artificial images can be adjusted through guiding the network. • Artificial images proved viable to augment the depth and broadening of diagnostic training.

Project description:Radiology reports contain a radiologist's interpretations of images, and these images frequently describe spatial relations. Important radiographic findings are mostly described in reference to an anatomical location through spatial prepositions. Such spatial relationships are also linked to various differential diagnoses and often described through uncertainty phrases. Structured representation of this clinically significant spatial information has the potential to be used in a variety of downstream clinical informatics applications. Our focus is to extract these spatial representations from the reports. For this, we first define a representation framework based on the Spatial Role Labeling (SpRL) scheme, which we refer to as Rad-SpRL. In Rad-SpRL, common radiological entities tied to spatial relations are encoded through four spatial roles: Trajector, Landmark, Diagnosis, and Hedge, all identified in relation to a spatial preposition (or Spatial Indicator). We annotated a total of 2,000 chest X-ray reports following Rad-SpRL. We then propose a deep learning-based natural language processing (NLP) method involving word and character-level encodings to first extract the Spatial Indicators followed by identifying the corresponding spatial roles. Specifically, we use a bidirectional long short-term memory (Bi-LSTM) conditional random field (CRF) neural network as the baseline model. Additionally, we incorporate contextualized word representations from pre-trained language models (BERT and XLNet) for extracting the spatial information. We evaluate both gold and predicted Spatial Indicators to extract the four types of spatial roles. The results are promising, with the highest average F1 measure for Spatial Indicator extraction being 91.29 (XLNet); the highest average overall F1 measure considering all the four spatial roles being 92.9 using gold Indicators (XLNet); and 85.6 using predicted Indicators (BERT pre-trained on MIMIC notes). The corpus is available in Mendeley at http://dx.doi.org/10.17632/yhb26hfz8n.1 and https://github.com/krobertslab/datasets/blob/master/Rad-SpRL.xml.

Project description:Background The integration of large language models (LLMs) into medical education has received increasing attention as a potential tool to enhance learning experiences. However, there remains a need to explore radiology postgraduate students' engagement with LLMs and their perceptions of their utility in medical education. Hence, we conducted this study to investigate radiology postgraduate students' knowledge, attitudes, and practices regarding LLMs in medical education. Materials and Methods A cross-sectional quantitative survey was conducted online on Google Forms. Participants from all over India were recruited via social media platforms and snowball sampling techniques. A previously validated questionnaire was used to assess knowledge, attitude, and practices regarding LLMs. Descriptive statistical analysis was employed to summarize participants' responses. Results A total of 252 (139 [55.16%] males and 113 [44.84%] females) radiology postgraduate students with a mean age of 28.33 ± 3.32 years participated in the study. The majority of the participants (47.62%) were familiar with LLMs with their potential incorporation with traditional teaching-learning tools (71.82%). They are open to including LLMs as a learning tool (71.03%) and think that it would provide comprehensive medical information (62.7%). Residents take the help of LLMs when they do not get the desired information from books (46.43%) or Internet search engines (59.13%). The overall score of knowledge (3.52 ± 0.58), attitude (3.75 ± 0.51), and practice (3.15 ± 0.57) were statistically significantly different (analysis of variance [ANOVA], p < 0.0001), with the highest score in attitude and lowest in practice. However, no significant differences were found in the scores for knowledge ( p = 0.64), attitude ( p = 0.99), and practice ( p = 0.25) depending on the year of training. Conclusion Radiology postgraduate students are familiar with LLM and recognize the potential benefits of LLMs in postgraduate radiology education. Although they have a positive attitude toward the use of LLMs, they are concerned about its limitations and use it only in limited situations for educational purposes.

Dataset Information

Knowledge-enhanced visual-language pre-training on chest radiology images.

Publications

Knowledge-enhanced visual-language pre-training on chest radiology images.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets