Sleep CLIP: A Multimodal Sleep Staging Model Based on Sleep Signals and Sleep Staging Labels.
Ontology highlight
ABSTRACT:
SUBMITTER: Yang W
PROVIDER: S-EPMC10490238 | biostudies-literature | 2023 Aug
REPOSITORIES: biostudies-literature
Publications
Sleep CLIP: A Multimodal Sleep Staging Model Based on Sleep Signals and Sleep Staging Labels.
Yang Weijia W Wang Yuxian Y Hu Jiancheng J Yuan Tuming T
Sensors (Basel, Switzerland) 20230823 17
Since the release of the contrastive language-image pre-training (CLIP) model designed by the OpenAI team, it has been applied in several fields owing to its high accuracy. Sleep staging is an important method of diagnosing sleep disorders, and the completion of sleep staging tasks with high accuracy has always remained the main goal of sleep staging algorithm designers. This study is aimed at designing a multimodal model based on the CLIP model that is more suitable for sleep staging tasks usin ...[more]
Similar Datasets
Project description:Polysomnography (PSG) is crucial for diagnosing sleep disorders, but manual scoring of PSG is time-consuming and subjective, leading to high variability. While machine-learning models have improved PSG scoring, their clinical use is hindered by the 'black-box' nature. In this study, we present SleepXViT, an automatic sleep staging system using Vision Transformer (ViT) that provides intuitive, consistent explanations by mimicking human 'visual scoring'. Tested on KISS-a PSG image dataset from 7745 patients across four hospitals-SleepXViT achieved a Macro F1 score of 81.94%, outperforming baseline models and showing robust performances on public datasets SHHS1 and SHHS2. Furthermore, SleepXViT offers well-calibrated confidence scores, enabling expert review for low-confidence predictions, alongside high-resolution heatmaps highlighting essential features and relevance scores for adjacent epochs' influence on sleep stage predictions. Together, these explanations reinforce the scoring consistency of SleepXViT, making it both reliable and interpretable, thereby facilitating the synergy between the AI model and human scorers in clinical settings.
| S-EPMC11762271 | biostudies-literature
Project description:BackgroundSurgical action recognition is an essential technology in context-aware-based autonomous surgery, whereas the accuracy is limited by clinical dataset scale. Leveraging surgical videos from virtual reality (VR) simulations to research algorithms for the clinical domain application, also known as domain adaptation, can effectively reduce the cost of data acquisition and annotation, and protect patient privacy.MethodsWe introduced a surgical domain adaptation method based on the contrastive language-image pretraining model (SDA-CLIP) to recognize cross-domain surgical action. Specifically, we utilized the Vision Transformer (ViT) and Transformer to extract video and text embeddings, respectively. Text embedding was developed as a bridge between VR and clinical domains. Inter- and intra-modality loss functions were employed to enhance the consistency of embeddings of the same class. Further, we evaluated our method on the MICCAI 2020 EndoVis Challenge SurgVisDom dataset.ResultsOur SDA-CLIP achieved a weighted F1-score of 65.9% (+18.9%) on the hard domain adaptation task (trained only with VR data) and 84.4% (+4.4%) on the soft domain adaptation task (trained with VR and clinical-like data), which outperformed the first place team of the challenge by a significant margin.ConclusionsThe proposed SDA-CLIP model can effectively extract video scene information and textual semantic information, which greatly improves the performance of cross-domain surgical action recognition. The code is available at https://github.com/Lycus99/SDA-CLIP.
| S-EPMC10585553 | biostudies-literature
Project description:Different biological signals are recorded in sleep labs during sleep for the diagnosis and treatment of human sleep problems. Classification of sleep stages with electroencephalography (EEG) is preferred to other biological signals due to its advantages such as providing clinical information, cost-effectiveness, comfort, and ease of use. The evaluation of EEG signals taken during sleep by clinicians is a tiring, time-consuming, and error-prone method. Therefore, it is clinically mandatory to determine sleep stages by using software-supported systems. Like all classification problems, the accuracy rate is used to compare the performance of studies in this domain, but this metric can be accurate when the number of observations is equal in classes. However, since there is not an equal number of observations in sleep stages, this metric is insufficient in the evaluation of such systems. For this purpose, in recent years, Cohen's kappa coefficient and even the sensitivity of NREM1 have been used for comparing the performance of these systems. Still, none of them examine the system from all dimensions. Therefore, in this study, two new metrics based on the polygon area metric, called the normalized area of sensitivity polygon and normalized area of the general polygon, are proposed for the performance evaluation of sleep staging systems. In addition, a new sleep staging system is introduced using the applications offered by the MATLAB program. The existing systems discussed in the literature were examined with the proposed metrics, and the best systems were compared with the proposed sleep staging system. According to the results, the proposed system excels in comparison with the most advanced machine learning methods. The single-channel method introduced based on the proposed metrics can be used for robust and reliable sleep stage classification from all dimensions required for real-time applications.
| S-EPMC8131449 | biostudies-literature
Project description:High-throughput sequencing of RNAs isolated by crosslinking immunoprecipitation (HITS-CLIP, also called CLIP-Seq) has been used to map global RNA-protein interactions. However, a critical caveat of HITS-CLIP results is that they contain non-linear background noise-different extent of non-specific interactions caused by individual transcript abundance-that has been inconsiderately normalized, resulting in sacrifice of sensitivity. To properly deconvolute RNA-protein interactions, we have implemented CLIPick, a flexible peak calling pipeline for analyzing HITS-CLIP data, which statistically determines the signal-to-noise ratio for each transcript based on the expression-dependent background simulation. Comprising of streamlined Python modules with an easy-to-use standalone graphical user interface, CLIPick robustly identifies significant peaks and quantitatively defines footprint regions within which RNA-protein interactions were occurred. CLIPick outperforms other peak callers in accuracy and sensitivity, selecting the largest number of peaks particularly in lowly expressed transcripts where such marginal signals are hard to discriminate. Specifically, the application of CLIPick to Argonaute (Ago) HITS-CLIP data were sensitive enough to uncover extended features of microRNA target sites, and these sites were experimentally validated. CLIPick enables to resolve critical interactions in a wide spectrum of transcript levels and extends the scope of HITS-CLIP analysis. CLIPick is available at: http://clip.korea.ac.kr/clipick/.
| S-EPMC6265468 | biostudies-literature
Project description:ObjectivesTo propose a transfer learning (TL) radiomics model that efficiently combines the information from gray scale and elastogram ultrasound images for accurate liver fibrosis grading.MethodsTotally 466 patients undergoing partial hepatectomy were enrolled, including 401 with chronic hepatitis B and 65 without fibrosis pathologically. All patients received elastography and got liver stiffness measurement (LSM) 2-3 days before surgery. We proposed a deep convolutional neural network by TL to analyze images of gray scale modality (GM) and elastogram modality (EM). The TL process was used for liver fibrosis classification by Inception-V3 network which pretrained on ImageNet. The diagnostic performance of TL and non-TL was compared. The value of single modalities, including GM and EM alone, and multimodalities, including GM + LSM and GM + EM, was evaluated and compared with that of LSM and serological indexes. Receiver operating characteristic curve analysis was performed to calculate the optimal area under the curve (AUC) for classifying fibrosis of S4, ≥ S3, and ≥ S2.ResultsTL in GM and EM demonstrated higher diagnostic accuracy than non-TL, with significantly higher AUCs (all p < .01). Single-modal GM and EM both performed better than LSM and serum indexes (all p < .001). Multimodal GM + EM was the most accurate prediction model (AUCs are 0.950, 0.932, and 0.930 for classifying S4, ≥ S3, and ≥ S2, respectively) compared with GM + LSM, GM and EM alone, LSM, and biomarkers (all p < .05).ConclusionsLiver fibrosis can be staged by a transfer learning modal based on the combination of gray scale and elastogram ultrasound images, with excellent performance.Key points• Transfer learning consists in applying to a specific deep learning algorithm that pretrained on another relevant problem, expected to reduce the risk of overfitting due to insufficient medical images. • Liver fibrosis can be staged by transfer learning radiomics with excellent performance. • The most accurate prediction model of transfer learning by Inception-V3 network is the combination of gray scale and elastogram ultrasound images.
| S-EPMC7160214 | biostudies-literature
Project description:Sleep disorders affect a large portion of the global population and are strong predictors of morbidity and all-cause mortality. Sleep staging segments a period of sleep into a sequence of phases providing the basis for most clinical decisions in sleep medicine. Manual sleep staging is difficult and time-consuming as experts must evaluate hours of polysomnography (PSG) recordings with electroencephalography (EEG) and electrooculography (EOG) data for each patient. Here, we present U-Sleep, a publicly available, ready-to-use deep-learning-based system for automated sleep staging ( sleep.ai.ku.dk ). U-Sleep is a fully convolutional neural network, which was trained and evaluated on PSG recordings from 15,660 participants of 16 clinical studies. It provides accurate segmentations across a wide range of patient cohorts and PSG protocols not considered when building the system. U-Sleep works for arbitrary combinations of typical EEG and EOG channels, and its special deep learning architecture can label sleep stages at shorter intervals than the typical 30 s periods used during training. We show that these labels can provide additional diagnostic information and lead to new ways of analyzing sleep. U-Sleep performs on par with state-of-the-art automatic sleep staging systems on multiple clinical datasets, even if the other systems were built specifically for the particular data. A comparison with consensus-scores from a previously unseen clinic shows that U-Sleep performs as accurately as the best of the human experts. U-Sleep can support the sleep staging workflow of medical experts, which decreases healthcare costs, and can provide highly accurate segmentations when human expertize is lacking.
| S-EPMC8050216 | biostudies-literature
Project description:BackgroundForecasting the patient's response to neoadjuvant chemoradiotherapy (nCRT) is crucial for managing locally advanced rectal cancer (LARC). This study investigates whether a predictive model using image-text features extracted from endorectal ultrasound (ERUS) via Contrastive Language-Image Pretraining (CLIP) can predict tumor regression grade (TRG) before nCRT.MethodsA retrospective analysis of 577 LARC patients who received nCRT followed by surgery was conducted from January 2018 to December 2023. ERUS scans and TRG were used to assess nCRT response, categorizing patients into good (TRG 0) and poor (TRG 1-3) responders. Image and text features were extracted using the ResNet50+RBT3 (RN50) and ViT-B/16+RoBERTa-wwm (VB16) components of the Chinese-CLIP model. LightGBM was used for model construction and comparison. A subset of 100 patients from each responder group was used to compare the CLIP method with manual radiomics methods (logistic regression, support vector machines, and random forest). SHapley Additive exPlanations (SHAP) technique was used to analyze feature contributions.ResultsThe RN50 and VB16 models achieved AUROC scores of 0.928 (95% CI: 0.90-0.96) and 0.900 (95% CI: 0.86-0.93), respectively, outperforming manual radiomics methods. SHAP analysis indicated that image features dominated the RN50 model, while both image and text features were significant in the VB16 model.ConclusionsThe CLIP-based predictive model using ERUS image-text features and LightGBM showed potential for improving personalized treatment strategies. However, this study is limited by its retrospective design and single-center data.
| S-EPMC11633952 | biostudies-literature
Project description:Costly signaling theory provides an explanation for why humans are willing to a pay a premium for conspicuous products such as luxury brand-labeled clothing or conspicuous environmentally friendly cars. According to the theory, the extra cost of such products is a signal of social status and wealth and leads to advantages in social interactions for the signaler. A previous study found positive evidence for the case of luxury brand labels. However, an issue of this study was that some of the experiments were not conducted in a perfectly double-blind manner. I resolved this by replicating variations of the original design in a double-blind procedure. Additionally, besides the luxury label condition, I introduced a "green" label condition. Thus, the hypothesis that signaling theory is able to explain pro-environmental behavior was tested for the first time in a natural field setting. Further, I conducted experiments in both average and below-average socioeconomic neighborhoods, where, according to signaling theory, the effects of luxury signals should be even stronger. In contrast to the original study, I did not find positive effects of the luxury brand label in any of the five experiments. Nor did I find evidence for a green-signaling effect. Moreover, in poor neighborhoods a negative tendency of the luxury label actually became evident. This suggests that a signaling theory explanation of costly labels must take into account the characteristics of the observers, e.g. their social status.
| S-EPMC5295666 | biostudies-literature
Project description:Sleep is a fundamental biological process with profound implications for physical and mental health, yet our understanding of its complex patterns and their relationships to a broad spectrum of diseases remains limited. While polysomnography (PSG), the gold standard for sleep analysis, captures rich multimodal physiological data, analyzing these measurements has been challenging due to limited flexibility across recording environments, poor generalizability across cohorts, and difficulty in leveraging information from multiple signals simultaneously. To address this gap, we curated over 585,000 hours of high-quality sleep recordings from approximately 65,000 participants across multiple cohorts and developed SleepFM, a multimodal sleep foundation model trained with a novel contrastive learning approach, designed to accommodate any PSG montage. SleepFM produces informative sleep embeddings that enable predictions of future diseases. We systematically demonstrate that SleepFM embeddings can predict 130 future diseases, as modeled by Phecodes, with C-Index and AUROC of at least 0.75 on held-out participants (Bonferroni-corrected p < 0.01). This includes accurate predictions for death (C-Index: 0.84 [95% CI: 0.81-0.87]), heart failure (C-Index: 0.80 [95% CI: 0.77-0.83]), chronic kidney disease (C-Index: 0.79 [95% CI: 0.77-0.81]), dementia (C-Index: 0.85 [95% CI: 0.82-0.87]), stroke (C-Index: 0.78 [95% CI: 0.76-0.81]), atrial fibrillation (C-Index: 0.78 [95% CI: 0.75-0.81]), and myocardial infarction (C-Index: 0.81 [95% CI: 0.78-0.84]). The model's generalizability was further validated through strong performance on the Sleep Heart Health Study (SHHS), a dataset unseen during pre-training. Additionally, SleepFM demonstrates strong performance on traditional sleep analysis tasks, achieving competitive results in both sleep staging (mean F1 scores: 0.70-0.78) and sleep apnea diagnosis (AUROC: 0.90-0.94). Beyond these standard applications, our analysis reveals that specific sleep stages and physiological signals carry distinct predictive power for different diseases. This work demonstrates how foundation models can leverage sleep polysomnography data to uncover the extensive relationship between sleep physiology and future disease risk.
| S-EPMC11838666 | biostudies-literature
Project description:Automated sleep staging has been previously limited by a combination of clinical and physiological heterogeneity. Both factors are in principle addressable with large data sets that enable robust calibration. However, the impact of sample size remains uncertain. The objectives are to investigate the extent to which machine learning methods can approximate the performance of human scorers when supplied with sufficient training cases and to investigate how staging performance depends on the number of training patients, contextual information, model complexity, and imbalance between sleep stage proportions. A total of 102 features were extracted from six electroencephalography (EEG) channels in routine polysomnography. Two thousand nights were partitioned into equal (n = 1000) training and testing sets for validation. We used epoch-by-epoch Cohen's kappa statistics to measure the agreement between classifier output and human scorer according to American Academy of Sleep Medicine scoring criteria. Epoch-by-epoch Cohen's kappa improved with increasing training EEG recordings until saturation occurred (n = ~300). The kappa value was further improved by accounting for contextual (temporal) information, increasing model complexity, and adjusting the model training procedure to account for the imbalance of stage proportions. The final kappa on the testing set was 0.68. Testing on more EEG recordings leads to kappa estimates with lower variance. Training with a large data set enables automated sleep staging that compares favorably with human scorers. Because testing was performed on a large and heterogeneous data set, the performance estimate has low variance and is likely to generalize broadly.
| S-EPMC6251659 | biostudies-literature