RATING: Medical knowledge-guided rheumatoid arthritis assessment from multimodal ultrasound images via deep learning.
Ontology highlight
ABSTRACT:
SUBMITTER: Zhou Z
PROVIDER: S-EPMC9583187 | biostudies-literature | 2022 Oct
REPOSITORIES: biostudies-literature
Publications
RATING: Medical knowledge-guided rheumatoid arthritis assessment from multimodal ultrasound images via deep learning.
Zhou Zhanping Z Zhao Chenyang C Qiao Hui H Wang Ming M Guo Yuchen Y Wang Qian Q Zhang Rui R Wu Huaiyu H Dong Fajin F Qi Zhenhong Z Li Jianchu J Tian Xinping X Zeng Xiaofeng X Jiang Yuxin Y Xu Feng F Dai Qionghai Q Yang Meng M
Patterns (New York, N.Y.) 20220929 10
Multimodal ultrasound has demonstrated its power in the clinical assessment of rheumatoid arthritis (RA). However, for radiologists, it requires strong experience. In this paper, we propose a rheumatoid arthritis knowledge guided (RATING) system that automatically scores the RA activity and generates interpretable features to assist radiologists' decision-making based on deep learning. RATING leverages the complementary advantages of multimodal ultrasound images and solves the limited training d ...[more]
Similar Datasets
Project description:BackgroundKnowledge graphs are a powerful tool for organizing knowledge, processing information and integrating scattered information, effectively visualizing the relationships among entities and supporting further intelligent applications. One of the critical tasks in building knowledge graphs is knowledge extraction. The existing knowledge extraction models in the Chinese medical domain usually require high-quality and large-scale manually labeled corpora for model training. In this study, we investigate rheumatoid arthritis (RA)-related Chinese electronic medical records (CEMRs) and address the automatic knowledge extraction task with a small number of annotated samples from CEMRs, from which an authoritative RA knowledge graph is constructed.MethodsAfter constructing the domain ontology of RA and completing manual labeling, we propose the MC-bidirectional encoder representation from transformers-bidirectional long short-term memory-conditional random field (BERT-BiLSTM-CRF) model for the named entity recognition (NER) task and the MC-BERT + feedforward neural network (FFNN) model for the entity extraction task. The pretrained language model (MC-BERT) is trained with many unlabeled medical data and fine-tuned using other medical domain datasets. We apply the established model to automatically label the remaining CEMRs, and then an RA knowledge graph is constructed based on the entities and entity relations, a preliminary assessment is conducted, and an intelligent application is presented.ResultsThe proposed model achieved better performance than that of other widely used models in knowledge extraction tasks, with mean F1 scores of 92.96% in entity recognition and 95.29% in relation extraction. This study preliminarily confirmed that using a pretrained medical language model could solve the problem that knowledge extraction from CEMRs requires a large number of manual annotations. An RA knowledge graph based on the above identified entities and extracted relations from 1,986 CEMRs was constructed. Experts verified the effectiveness of the constructed RA knowledge graph.ConclusionsIn this paper, an RA knowledge graph based on CEMRs was established, the processes of data annotation, automatic knowledge extraction, and knowledge graph construction were described, and a preliminary assessment and an application were presented. The study demonstrated the viability of a pretrained language model combined with a deep neural network for knowledge extraction tasks from CEMRs based on a small number of manually annotated samples.
| S-EPMC10240026 | biostudies-literature
Project description:ObjectivesThis study aimed to differentially diagnose thyroid nodules (TNs) of Thyroid Imaging Reporting and Data System (TI-RADS) 3-5 categories using a deep learning (DL) model based on multimodal ultrasound (US) images and explore its auxiliary role for radiologists with varying degrees of experience.MethodsPreoperative multimodal US images of 1,138 TNs of TI-RADS 3-5 categories were randomly divided into a training set (n = 728), a validation set (n = 182), and a test set (n = 228) in a 4:1:1.25 ratio. Grayscale US (GSU), color Doppler flow imaging (CDFI), strain elastography (SE), and region of interest mask (Mask) images were acquired in both transverse and longitudinal sections, all of which were confirmed by pathology. In this study, fivefold cross-validation was used to evaluate the performance of the proposed DL model. The diagnostic performance of the mature DL model and radiologists in the test set was compared, and whether DL could assist radiologists in improving diagnostic performance was verified. Specificity, sensitivity, accuracy, positive predictive value, negative predictive value, and area under the receiver operating characteristics curves (AUC) were obtained.ResultsThe AUCs of DL in the differentiation of TNs were 0.858 based on (GSU + SE), 0.909 based on (GSU + CDFI), 0.906 based on (GSU + CDFI + SE), and 0.881 based (GSU + Mask), which were superior to that of 0.825-based single GSU (p = 0.014, p< 0.001, p< 0.001, and p = 0.002, respectively). The highest AUC of 0.928 was achieved by DL based on (G + C + E + M)US, the highest specificity of 89.5% was achieved by (G + C + E)US, and the highest accuracy of 86.2% and sensitivity of 86.9% were achieved by DL based on (G + C + M)US. With DL assistance, the AUC of junior radiologists increased from 0.720 to 0.796 (p< 0.001), which was slightly higher than that of senior radiologists without DL assistance (0.796 vs. 0.794, p > 0.05). Senior radiologists with DL assistance exhibited higher accuracy and comparable AUC than that of DL based on GSU (83.4% vs. 78.9%, p = 0.041; 0.822 vs. 0.825, p = 0.512). However, the AUC of DL based on multimodal US images was significantly higher than that based on visual diagnosis by radiologists (p< 0.05).ConclusionThe DL models based on multimodal US images showed exceptional performance in the differential diagnosis of suspicious TNs, effectively increased the diagnostic efficacy of TN evaluations by junior radiologists, and provided an objective assessment for the clinical and surgical management phases that follow.
| S-EPMC9680169 | biostudies-literature
Project description:ObjectiveCurrently, there are no reliable biomarkers for predicting therapeutic response in patients with rheumatoid arthritis (RA). The synovium may unlock critical information for determining efficacy, since a reduction in the numbers of sublining synovial macrophages remains the most reproducible biomarker. Thus, a clinically actionable method for the collection of synovial tissue, which can be analyzed using high-throughput strategies, must become a reality. This study was undertaken to assess the feasibility of utilizing synovial biopsies as a precision medicine-based approach for patients with RA.MethodsRheumatologists at 6 US academic sites were trained in minimally invasive ultrasound-guided synovial tissue biopsy. Biopsy specimens obtained from patients with RA and synovial tissue from patients with osteoarthritis (OA) were subjected to histologic analysis, fluorescence-activated cell sorting, and RNA sequencing (RNA-seq). An optimized protocol for digesting synovial tissue was developed to generate high-quality RNA-seq libraries from isolated macrophage populations. Associations were determined between macrophage transcriptional profiles and clinical parameters in RA patients.ResultsPatients with RA reported minimal adverse effects in response to synovial biopsy. Comparable RNA quality was observed from synovial tissue and isolated macrophages between patients with RA and patients with OA. Whole tissue samples from patients with RA demonstrated a high degree of transcriptional heterogeneity. In contrast, the transcriptional profile of isolated RA synovial macrophages highlighted different subpopulations of patients and identified 6 novel transcriptional modules that were associated with disease activity and therapy.ConclusionPerformance of synovial tissue biopsies by rheumatologists in the US is feasible and generates high-quality samples for research. Through the use of cutting-edge technologies to analyze synovial biopsy specimens in conjunction with corresponding clinical information, a precision medicine-based approach for patients with RA is attainable.
| S-EPMC5984677 | biostudies-literature
Project description:ObjectiveTo study the classification performance of a pre-trained convolutional neural network (CNN) with transfer learning by artificial joint ultrasonography images in rheumatoid arthritis (RA).MethodsThis retrospective study focused on abnormal synovial vascularity and created 870 artificial joint ultrasound images based on the European League Against Rheumatism/Outcome Measure in Rheumatology scoring system. One CNN, the Visual Geometry Group (VGG)-16, was trained with transfer learning using the 870 artificial images for initial training and the original plus five additional images for second training. The models were then tested for the ability to classify joints using real joint ultrasound images obtained from patients with RA. The study was registered in UMIN Clinical Trials Registry (UMIN000054321).ResultsA total of 156 clinical joint ultrasound images from 74 patients with RA were included. The initial model showed moderate classification performance, but the area under curve (AUC) for grade 1 synovitis was particularly low (0.59). The second model showed improvement in classifying grade 1 synovitis (AUC 0.73).ConclusionsArtificial images may be useful for training VGG-16. The present novel approach of using artificial images as an alternative to actual images for training a CNN has the potential to be applied in medical imaging fields that face difficulties in collecting real clinical images.
| S-EPMC11795604 | biostudies-literature
Project description:The high level of reactive oxygen species (ROS) in the rheumatoid arthritis (RA) microenvironment (RAM) and its persistent inflammatory nature can promote damage to joints, bones, and the synovium. Targeting strategies that integrate effective RAM regulation with imaging-based monitoring could lead to improvements in the diagnosis and treatment of RA. Here, we report the combined use of small interfering RNAs (siRNAsT/I) and Prussian blue nanoparticles (PBNPs) to silence the expression of proinflammatory cytokines TNF-α/IL-6 and scavenge the ROS associated with RAM. To enhance the in vitro and in vivo biological stability, biocompatibility, and targeting capability of the siRNAsT/I and PBNPs, macrophage membrane vesicles were used to prepare biomimetic nanoparticles, M@P-siRNAsT/I. The resulting constructs were found to suppress tumor necrosis factor-α/interleukin-6 expression and overcome the hypoxic nature of RAM, thus alleviating RA-induced joint damage in a mouse model. The M@P-siRNAsT/I of this study could be monitored via near-infrared photoacoustic (PA) imaging. Moreover, multispectral PA imaging without the need for labeling permitted the real-time evaluation of M@P-siRNAsT/I as a putative RA treatment. Clinical microcomputed tomography and histological analysis confirmed the effectiveness of the treatment. We thus suggest that macrophage-biomimetic M@P-siRNAsT/I and their analogs assisted by PA imaging could provide a new strategy for RA diagnosis, treatment, and monitoring.
| S-EPMC9618076 | biostudies-literature
Project description:Ultrasonography is one of the key medical imaging modalities for evaluating breast lesions. For differentiating benign from malignant lesions, computer-aided diagnosis (CAD) systems have greatly assisted radiologists by automatically segmenting and identifying features of lesions. Here, we present deep learning (DL)-based methods to segment the lesions and then classify benign from malignant, utilizing both B-mode and strain elastography (SE-mode) images. We propose a weighted multimodal U-Net (W-MM-U-Net) model for segmenting lesions where optimum weight is assigned on different imaging modalities using a weighted-skip connection method to emphasize its importance. We design a multimodal fusion framework (MFF) on cropped B-mode and SE-mode ultrasound (US) lesion images to classify benign and malignant lesions. The MFF consists of an integrated feature network (IFN) and a decision network (DN). Unlike other recent fusion methods, the proposed MFF method can simultaneously learn complementary information from convolutional neural networks (CNNs) trained using B-mode and SE-mode US images. The features from the CNNs are ensembled using the multimodal EmbraceNet model and DN classifies the images using those features. The experimental results (sensitivity of 100 ± 0.00% and specificity of 94.28 ± 7.00%) on the real-world clinical data showed that the proposed method outperforms the existing single- and multimodal methods. The proposed method predicts seven benign patients as benign three times out of five trials and six malignant patients as malignant five out of five trials. The proposed method would potentially enhance the classification accuracy of radiologists for breast cancer detection in US images.
| S-EPMC10658476 | biostudies-literature
Project description:BackgroundAnalyses of lymphoid organs are required to further elucidate the pathogenesis of inflammatory diseases like rheumatoid arthritis (RA). Yet, invasive tissue collection methods are scarcely applied, because they are often considered burdensome, although patients do not always consider invasive methods as a high burden. We aimed to investigate the perspectives of study participants undergoing ultrasound-guided inguinal lymph node (LN) needle biopsy sampling and determine the molecular and cellular quantity and quality of LN biopsies.MethodsTogether with patient research partners, questionnaires were developed to evaluate the motives, expectations, and experiences of participants undergoing a LN biopsy. Healthy controls and RA(-risk) patients were asked to complete these questionnaires before and after the procedure. RNA and lymphocyte yields from obtained LN biopsies were also calculated.ResultsWe included 50 individuals, of which 43 (86%) reported their pre- and post-procedure experiences. The median reported pain on a 5-point Likert scale (1 not to 5 very painful) was 1. Interestingly, almost all (n = 32; 74%) study participants would undergo a second procedure and more than half (n = 23; 54%) would encourage others to take part in the LN biopsy study. Motives for current and future participation were mostly altruistic. Inguinal hematoma occurred frequently, but no other significant or unexpected complications ensued. The LN biopsies yielded sufficient and high-quality RNA and lymphocyte numbers.ConclusionsUltrasound-guided inguinal LN biopsy sampling is well-tolerated, safe, and provides sufficient material for further molecular and cellular analyses. Our participants' positive experiences endorse the application of this research tool to further elucidate the pathogenesis of RA and other inflammatory diseases.
| S-EPMC8812012 | biostudies-literature
Project description:BackgroundRecent advances in artificial intelligence and digital image processing have inspired the use of deep neural networks for segmentation tasks in multimodal medical imaging. Unlike natural images, multimodal medical images contain much richer information regarding different modal properties and therefore present more challenges for semantic segmentation. However, there is no report on systematic research that integrates multi-scaled and structured analysis of single-modal and multimodal medical images.MethodsWe propose a deep neural network, named as Modality Preserving U-Net (MPU-Net), for modality-preserving analysis and segmentation of medical targets from multimodal medical images. The proposed MPU-Net consists of a modality preservation encoder (MPE) module that preserves the feature independency among the modalities and a modality fusion decoder (MFD) module that performs a multiscale feature fusion analysis for each modality in order to provide a rich feature representation for the final task. The effectiveness of such a single-modal preservation and multimodal fusion feature extraction approach is verified by multimodal segmentation experiments and an ablation study using brain tumor and prostate datasets from Medical Segmentation Decathlon (MSD).ResultsThe segmentation experiments demonstrated the superiority of MPU-Net over other methods in the segmentation tasks for multimodal medical images. In the brain tumor segmentation tasks, the Dice scores (DSCs) for the whole tumor (WT), the tumor core (TC) and the enhancing tumor (ET) regions were 89.42%, 86.92%, and 84.59%, respectively. In the meanwhile, the 95% Hausdorff distance (HD95) results were 3.530, 4.899 and 2.555, respectively. In the prostate segmentation tasks, the DSCs for the peripheral zone (PZ) and the transitional zone (TZ) of the prostate were 71.20% and 90.38%, respectively. In the meanwhile, the 95% HD95 results were 6.367 and 4.766, respectively. The ablation study showed that the combination of single-modal preservation and multimodal fusion methods improved the performance of multimodal medical image feature analysis.ConclusionsIn the segmentation tasks using brain tumor and prostate datasets, the MPU-Net method has achieved the improved performance in comparison with the conventional methods, indicating its potential application for other segmentation tasks in multimodal medical images.
| S-EPMC10423364 | biostudies-literature
Project description:ObjectiveTo assess the determinants of patients' (PTGL) and physicians' (MDGL) global assessment of rheumatoid arthritis (RA) activity and factors associated with discordance among them.MethodsA total of 7,028 patients in the Quantitative Standard Monitoring of Patients with RA study had PTGL and MDGL assessed at the same clinic visit on a 0-10-cm visual analog scale (VAS). Three patient groups were defined: concordant rating group (PTGL and MDGL within ±2 cm), higher patient rating group (PTGL exceeding MDGL by >2 cm), and lower patient rating group (PTGL less than MDGL by >2 cm). Multivariable regression analysis was used to identify determinants of PTGL and MDGL and their discordance.ResultsThe mean ± SD VAS scores for PTGL and MDGL were 4.01 ± 2.70 and 2.91 ± 2.37, respectively. Pain was overwhelmingly the single most important determinant of PTGL, followed by fatigue. In contrast, MDGL was most influenced by swollen joint count (SJC), followed by erythrocyte sedimentation rate (ESR) and tender joint count (TJC). A total of 4,454 (63.4%), 2,106 (30%), and 468 (6.6%) patients were in the concordant, higher, and lower patient rating groups, respectively. Odds of higher patient rating increased with higher pain, fatigue, psychological distress, age, and morning stiffness, and decreased with higher SJC, TJC, and ESR. Lower patient rating odds increased with higher SJC, TJC, and ESR, and decreased with lower fatigue levels.ConclusionNearly 36% of patients had discordance in RA activity assessment from their physicians. Sensitivity to the "disease experience" of patients, particularly pain and fatigue, is warranted for effective care of RA.
| S-EPMC3703925 | biostudies-literature
Project description:MotivationRecent brain mapping efforts are producing large-scale whole-brain images using different imaging modalities. Accurate alignment and delineation of anatomical structures in these images are essential for numerous studies. These requirements are typically modeled as two distinct tasks: registration and segmentation. However, prevailing methods, fail to fully explore and utilize the inherent correlation and complementarity between the two tasks. Furthermore, variations in brain anatomy, brightness, and texture pose another formidable challenge in designing multi-modal similarity metrics. A high-throughput approach capable of overcoming the bottleneck of multi-modal similarity metric design, while effective leveraging the highly correlated and complementary nature of two tasks is highly desirable.ResultsWe introduce a deep learning framework for joint registration and segmentation of multi-modal brain images. Under this framework, registration and segmentation tasks are deeply coupled and collaborated at two hierarchical layers. In the inner layer, we establish a strong feature-level coupling between the two tasks by learning a unified common latent feature representation. In the outer layer, we introduce a mutually supervised dual-branch network to decouple latent features and facilitate task-level collaboration between registration and segmentation. Since the latent features we designed are also modality-independent, the bottleneck of designing multi-modal similarity metric is essentially addressed. Another merit offered by this framework is the interpretability of latent features, which allows intuitive manipulation of feature learning, thereby further enhancing network training efficiency and the performance of both tasks. Extensive experiments conducted on both multi-modal and mono-modal datasets of mouse and human brains demonstrate the superiority of our method.Availability and implementationThe code is available at https://github.com/tingtingup/DCRS.
| S-EPMC11543610 | biostudies-literature