Dataset Information

Quantifying social roles in multi-animal videos using subject-aware deep-learning.

ABSTRACT: Analyzing social behaviors is critical for many fields, including neuroscience, psychology, and ecology. While computational tools have been developed to analyze videos containing animals engaging in limited social interactions under specific experimental conditions, automated identification of the social roles of freely moving individuals in a multi-animal group remains unresolved. Here we describe a deep-learning-based system - named LabGym2 - for identifying and quantifying social roles in multi-animal groups. This system uses a subject-aware approach: it evaluates the behavioral state of every individual in a group of two or more animals while factoring in its social and environmental surroundings. We demonstrate the performance of subject-aware deep-learning in different species and assays, from partner preference in freely-moving insects to primate social interactions in the field. Our subject-aware deep learning approach provides a controllable, interpretable, and efficient framework to enable new experimental paradigms and systematic evaluation of interactive behavior in individuals identified within a group.

SUBMITTER: Goss K

PROVIDER: S-EPMC11257443 | biostudies-literature | 2024 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Quantifying social roles in multi-animal videos using subject-aware deep-learning.

Goss Kelly K Bueno-Junior Lezio S LS Stangis Katherine K Ardoin Théo T Carmon Hanna H Zhou Jie J Satapathy Rohan R Baker Isabelle I Jones-Tinsley Carolyn E CE Lim Miranda M MM Watson Brendon O BO Sueur Cédric C Ferrario Carrie R CR Murphy Geoffrey G GG Ye Bing B Hu Yujia Y

bioRxiv : the preprint server for biology 20240710

Analyzing social behaviors is critical for many fields, including neuroscience, psychology, and ecology. While computational tools have been developed to analyze videos containing animals engaging in limited social interactions under specific experimental conditions, automated identification of the social roles of freely moving individuals in a multi-animal group remains unresolved. Here we describe a deep-learning-based system - named LabGym2 - for identifying and quantifying social roles in mu ...[more]

PMID: 39026890

Similar Datasets

Project description:BackgroundLaboratory testing is routinely used to assay blood biomarkers to provide information on physiologic state beyond what clinicians can evaluate from interpreting medical imaging. We hypothesized that deep learning interpretation of echocardiogram videos can provide additional value in understanding disease states and can evaluate common biomarkers results.MethodsWe developed EchoNet-Labs, a video-based deep learning algorithm to detect evidence of anemia, elevated B-type natriuretic peptide (BNP), troponin I, and blood urea nitrogen (BUN), as well as values of ten additional lab tests directly from echocardiograms. We included patients (n = 39,460) aged 18 years or older with one or more apical-4-chamber echocardiogram videos (n = 70,066) from Stanford Healthcare for training and internal testing of EchoNet-Lab's performance in estimating the most proximal biomarker result. Without fine-tuning, the performance of EchoNet-Labs was further evaluated on an additional external test dataset (n = 1,301) from Cedars-Sinai Medical Center. We calculated the area under the curve (AUC) of the receiver operating characteristic curve for the internal and external test datasets.FindingsOn the held-out test set of Stanford patients not previously seen during model training, EchoNet-Labs achieved an AUC of 0.80 (0.79-0.81) in detecting anemia (low hemoglobin), 0.86 (0.85-0.88) in detecting elevated BNP, 0.75 (0.73-0.78) in detecting elevated troponin I, and 0.74 (0.72-0.76) in detecting elevated BUN. On the external test dataset from Cedars-Sinai, EchoNet-Labs achieved an AUC of 0.80 (0.77-0.82) in detecting anemia, of 0.82 (0.79-0.84) in detecting elevated BNP, of 0.75 (0.72-0.78) in detecting elevated troponin I, and of 0.69 (0.66-0.71) in detecting elevated BUN. We further demonstrate the utility of the model in detecting abnormalities in 10 additional lab tests. We investigate the features necessary for EchoNet-Labs to make successful detection and identify potential mechanisms for each biomarker using well-known and novel explainability techniques.InterpretationThese results show that deep learning applied to diagnostic imaging can provide additional clinical value and identify phenotypic information beyond current imaging interpretation methods.FundingJ.W.H. and B.H. are supported by the NSF Graduate Research Fellowship. D.O. is supported by NIH K99 HL157421-01. J.Y.Z. is supported by NSF CAREER 1942926, NIH R21 MD012867-01, NIH P30AG059307 and by a Chan-Zuckerberg Biohub Fellowship.

Project description:Functional magnetic resonance imaging (fMRI) is increasingly used to characterize functional connectivity between brain regions. Given the vast number of between-voxel interactions in high-dimensional fMRI data, it is an ongoing challenge to detect stable and generalizable functional connectivity in the brain among groups of subjects. Component models can be used to define subspace representations of functional connectivity that are more interpretable. It is, however, unclear which component model provides the optimal representation of functional networks for multi-subject fMRI datasets. A flexible cross-validation approach that assesses the ability of the models to predict voxel-wise covariance in new data, using three different measures of generalization was proposed. This framework is used to compare a range of component models with varying degrees of flexibility in their representation of functional connectivity, evaluated on both simulated and experimental resting-state fMRI data. It was demonstrated that highly flexible subject-specific component subspaces, as well as very constrained average models, are poor predictors of whole-brain functional connectivity, whereas the best-generalizing models account for subject variability within a common spatial subspace. Within this set of models, spatial Independent Component Analysis (sICA) on concatenated data provides more interpretable brain patterns, whereas a consistent-covariance model that accounts for subject-specific network scaling (PARAFAC2) provides greater stability in functional connectivity relationships between components and their spatial representations. The proposed evaluation framework is a promising quantitative approach to evaluating component models, and reveals important differences between subspace models in terms of predictability, robustness, characterization of subject variability, and interpretability of the model parameters. Hum Brain Mapp 38:882-899, 2017. © 2016 Wiley Periodicals, Inc.

Project description:Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there is no consensus regarding optimal methods for quantifying uncertainty. Our purpose is to critically evaluate methods for quantifying uncertainty in deep learning for healthcare applications and propose a conceptual framework for specifying certainty of deep learning predictions. We searched Embase, MEDLINE, and PubMed databases for articles relevant to study objectives, complying with PRISMA guidelines, rated study quality using validated tools, and extracted data according to modified CHARMS criteria. Among 30 included studies, 24 described medical imaging applications. All imaging model architectures used convolutional neural networks or a variation thereof. The predominant method for quantifying uncertainty was Monte Carlo dropout, producing predictions from multiple networks for which different neurons have dropped out and measuring variance across the distribution of resulting predictions. Conformal prediction offered similar strong performance in estimating uncertainty, along with ease of interpretation and application not only to deep learning but also to other machine learning approaches. Among the six articles describing non-imaging applications, model architectures and uncertainty estimation methods were heterogeneous, but predictive performance was generally strong, and uncertainty estimation was effective in comparing modeling methods. Overall, the use of model learning curves to quantify epistemic uncertainty (attributable to model parameters) was sparse. Heterogeneity in reporting methods precluded the performance of a meta-analysis. Uncertainty estimation methods have the potential to identify rare but important misclassifications made by deep learning models and compare modeling methods, which could build patient and clinician trust in deep learning applications in healthcare. Efficient maturation of this field will require standardized guidelines for reporting performance and uncertainty metrics.

Dataset Information

Quantifying social roles in multi-animal videos using subject-aware deep-learning.

Publications

Quantifying social roles in multi-animal videos using subject-aware deep-learning.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets