Identifying musical pieces from fMRI data using encoding and decoding models.
Ontology highlight
ABSTRACT:
SUBMITTER: Hoefle S
PROVIDER: S-EPMC5797093 | biostudies-literature | 2018 Feb
REPOSITORIES: biostudies-literature
Publications
Identifying musical pieces from fMRI data using encoding and decoding models.
Hoefle Sebastian S Engel Annerose A Basilio Rodrigo R Alluri Vinoo V Toiviainen Petri P Cagy Maurício M Moll Jorge J
Scientific reports 20180202 1
Encoding models can reveal and decode neural representations in the visual and semantic domains. However, a thorough understanding of how distributed information in auditory cortices and temporal evolution of music contribute to model performance is still lacking in the musical domain. We measured fMRI responses during naturalistic music listening and constructed a two-stage approach that first mapped musical features in auditory cortices and then decoded novel musical pieces. We then probed the ...[more]
Similar Datasets
Project description:Spatial filtering strategies, combined with multivariate decoding analysis of BOLD images, have been used to investigate the nature of the neural signal underlying the discriminability of brain activity patterns evoked by sensory stimulation - primarily in the visual cortex. Previous research indicates that such signals are spatially broadband in nature, and are not primarily comprised of fine-grained activation patterns. However, it is unclear whether this is a general property of the BOLD signal, or whether it is specific to the details of employed analyses and stimuli. Here we applied an analysis strategy from a previous study on decoding visual orientation from V1 to publicly available, high-resolution 7T fMRI on the response BOLD response to musical genres in primary auditory cortex. The results show that the pattern of decoding accuracies with respect to different types and levels of spatial filtering is comparable to that obtained from V1, despite considerable differences in the respective cortical circuitry.
| S-EPMC5887073 | biostudies-literature
Project description:Previous functional magnetic resonance imaging (fMRI) studies have shown that activity in premotor and parietal brain-regions covaries with the intensity of upcoming grip-force. However, it remains unclear how information about the intended grip-force intensity is initially represented and subsequently transformed into a motor code before motor execution. In this fMRI study, we used multivoxel pattern analysis (MVPA) to decode where and when information about grip-force intensities is parametrically coded in the brain. Human participants performed a delayed grip-force task in which one of four cued levels of grip-force intensity had to be maintained in working memory (WM) during a 9-s delay-period preceding motor execution. Using time-resolved MVPA with a searchlight approach and support vector regression, we tested which brain regions exhibit multivariate WM codes of anticipated grip-force intensities. During the early delay period, we observed above-chance decoding in the ventromedial prefrontal cortex (vmPFC). During the late delay period, we found a network of action-specific brain regions, including the bilateral intraparietal sulcus (IPS), left dorsal premotor cortex (l-PMd), and supplementary motor areas. Additionally, cross-regression decoding was employed to test for temporal generalization of activation patterns between early and late delay periods with those during cue presentation and motor execution. Cross-regression decoding indicated temporal generalization to the cue period in the vmPFC and to motor-execution in the l-IPS and l-PMd. Together, these findings suggest that the WM representation of grip-force intensities undergoes a transformation where the vmPFC encodes information about the intended grip-force, which is subsequently converted into a motor code in the l-IPS and l-PMd before execution.
| S-EPMC11815324 | biostudies-literature
Project description:Representations from transformer-based unidirectional language models are known to be effective at predicting brain responses to natural language. However, most studies comparing language models to brains have used GPT-2 or similarly sized language models. Here we tested whether larger open-source models such as those from the OPT and LLaMA families are better at predicting brain responses recorded using fMRI. Mirroring scaling results from other contexts, we found that brain prediction performance scales logarithmically with model size from 125M to 30B parameter models, with ~15% increased encoding performance as measured by correlation with a held-out test set across 3 subjects. Similar logarithmic behavior was observed when scaling the size of the fMRI training set. We also characterized scaling for acoustic encoding models that use HuBERT, WavLM, and Whisper, and we found comparable improvements with model size. A noise ceiling analysis of these large, high-performance encoding models showed that performance is nearing the theoretical maximum for brain areas such as the precuneus and higher auditory cortex. These results suggest that increasing scale in both models and data will yield incredibly effective models of language processing in the brain, enabling better scientific understanding as well as applications such as decoding.
| S-EPMC11258918 | biostudies-literature
Project description:Invasive Brain-Computer Interfaces (BCIs) require surgeries with high health-risks. The risk-to-benefit ratio of the procedure could potentially be improved by pre-surgically identifying the ideal locations for mental strategy classification. We recorded high-spatiotemporal resolution blood-oxygenation-level-dependent (BOLD) signals using functional MRI at 7 Tesla in eleven healthy participants during two motor imagery tasks. BCI diagnostic task isolated the intent to imagine movements, while BCI simulation task simulated the neural states that may be yielded in a real-life BCI-operation scenario. Imagination of movements were classified from the BOLD signals in sub-regions of activation within a single or multiple dorsal motor network regions. Then, the participant's decoding performance during the BCI simulation task was predicted from the BCI diagnostic task. The results revealed that drawing information from multiple regions compared to a single region increased the classification accuracy of imagined movements. Importantly, systematic unimodal and multimodal classification revealed the ideal combination of regions that yielded the best classification accuracy at the individual-level. Lastly, a given participant's decoding performance achieved during the BCI simulation task could be predicted from the BCI diagnostic task. These results show the feasibility of 7T-fMRI with unimodal and multimodal classification being utilized for identifying ideal sites for mental strategy classification.
| S-EPMC6197258 | biostudies-literature
Project description:Goal-driven and feedforward-only convolutional neural networks (CNN) have been shown to be able to predict and decode cortical responses to natural images or videos. Here, we explored an alternative deep neural network, variational auto-encoder (VAE), as a computational model of the visual cortex. We trained a VAE with a five-layer encoder and a five-layer decoder to learn visual representations from a diverse set of unlabeled images. Using the trained VAE, we predicted and decoded cortical activity observed with functional magnetic resonance imaging (fMRI) from three human subjects passively watching natural videos. Compared to CNN, VAE could predict the video-evoked cortical responses with comparable accuracy in early visual areas, but relatively lower accuracy in higher-order visual areas. The distinction between CNN and VAE in terms of encoding performance was primarily attributed to their different learning objectives, rather than their different model architecture or number of parameters. Despite lower encoding accuracies, VAE offered a more convenient strategy for decoding the fMRI activity to reconstruct the video input, by first converting the fMRI activity to the VAE's latent variables, and then converting the latent variables to the reconstructed video frames through the VAE's decoder. This strategy was more advantageous than alternative decoding methods, e.g. partial least squares regression, for being able to reconstruct both the spatial structure and color of the visual input. Such findings highlight VAE as an unsupervised model for learning visual representation, as well as its potential and limitations for explaining cortical responses and reconstructing naturalistic and diverse visual experiences.
| S-EPMC6592726 | biostudies-literature
Project description:Speech comprehension is a complex process that draws on humans' abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 participants each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.
| S-EPMC10447563 | biostudies-literature
Project description:Functional magnetic resonance imaging (fMRI) is increasingly used to characterize functional connectivity between brain regions. Given the vast number of between-voxel interactions in high-dimensional fMRI data, it is an ongoing challenge to detect stable and generalizable functional connectivity in the brain among groups of subjects. Component models can be used to define subspace representations of functional connectivity that are more interpretable. It is, however, unclear which component model provides the optimal representation of functional networks for multi-subject fMRI datasets. A flexible cross-validation approach that assesses the ability of the models to predict voxel-wise covariance in new data, using three different measures of generalization was proposed. This framework is used to compare a range of component models with varying degrees of flexibility in their representation of functional connectivity, evaluated on both simulated and experimental resting-state fMRI data. It was demonstrated that highly flexible subject-specific component subspaces, as well as very constrained average models, are poor predictors of whole-brain functional connectivity, whereas the best-generalizing models account for subject variability within a common spatial subspace. Within this set of models, spatial Independent Component Analysis (sICA) on concatenated data provides more interpretable brain patterns, whereas a consistent-covariance model that accounts for subject-specific network scaling (PARAFAC2) provides greater stability in functional connectivity relationships between components and their spatial representations. The proposed evaluation framework is a promising quantitative approach to evaluating component models, and reveals important differences between subspace models in terms of predictability, robustness, characterization of subject variability, and interpretability of the model parameters. Hum Brain Mapp 38:882-899, 2017. © 2016 Wiley Periodicals, Inc.
| S-EPMC6866892 | biostudies-literature
Project description:Pattern recognition on neural activations from naturalistic music listening has been successful at predicting neural responses of listeners from musical features, and vice versa. Inter-subject differences in the decoding accuracies have arisen partly from musical training that has widely recognized structural and functional effects on the brain. We propose and evaluate a decoding approach aimed at predicting the musicianship class of an individual listener from dynamic neural processing of musical features. Whole brain functional magnetic resonance imaging (fMRI) data was acquired from musicians and nonmusicians during listening of three musical pieces from different genres. Six musical features, representing low-level (timbre) and high-level (rhythm and tonality) aspects of music perception, were computed from the acoustic signals, and classification into musicians and nonmusicians was performed on the musical feature and parcellated fMRI time series. Cross-validated classification accuracy reached 77% with nine regions, comprising frontal and temporal cortical regions, caudate nucleus, and cingulate gyrus. The processing of high-level musical features at right superior temporal gyrus was most influenced by listeners' musical training. The study demonstrates the feasibility to decode musicianship from how individual brains listen to music, attaining accuracy comparable to current results from automated clinical diagnosis of neurological and psychological disorders.
| S-EPMC5768727 | biostudies-literature
Project description:The way spatial memory retrieval is represented in the brain remains unclear to date. Previous studies have displayed a hippocampus-centered navigation network using functional magnetic resonance imaging (fMRI) analysis. There have been some studies on the representation of navigation behavior by signal distribution patterns, but only in the hippocampus and adjacent structures. In this study, we aimed to determine (1) the brain regions that represent information in both intensity and distribution patterns during spatial memory retrieval and (2) whether the patterns of neural responses represent spatial memory retrieval behavior performance. Both univariate analysis [general linear model (GLM)] and multivariate pattern analysis (MVPA) were employed to reveal the spatial distributions of brain responses elicited by spatial memory retrieval. Correlation analyses were performed to detect the correspondences between brain responses and behavior performance. We found that spatial memory retrieval occurred in widespread brain regions, including the bilateral hippocampi, bilateral superior frontal gyrus, bilateral superior parietal lobules, bilateral occipital lobes, and cerebellum. The amplitude of activation in the left hippocampus showed a significant negative correlation (r = -0.46, p = 0.039) with the number of task completions. Additionally, within-subject classification accuracies based on the blood oxygenation level-dependent (BOLD) signal patterns of the right middle temporal gyrus (rMTG) rostral areas in the Brainnetome Atlas showed a significant positive correlation (r = 0.78, p < 0.0001) with retrieval accuracy. In summary, our findings have implications for understanding the separation between navigational and non-navigational states and emphasizing the utility of MVPA in the whole brain.
| S-EPMC8168467 | biostudies-literature
Project description:This study examines the decoding times at which the brain processes structural information in music and compares them to timescales implicated in recent work on speech. Combining an experimental paradigm based on Ghitza and Greenberg (Phonetica, 66(1-2), 113-126, 2009) for speech with the approach of Farbood et al. (Journal of Experimental Psychology: Human Perception and Performance, 39(4), 911-918, 2013) for musical key-finding, listeners were asked to judge the key of short melodic sequences that were presented at a highly a compressed rate with varying durations of silence inserted in a periodic manner in the audio signal. The distorted audio signals comprised signal-silence alternations showing error rate curves that identify peak performance centered around an event rate of 5-7 Hz (143-200 ms interonset interval; 300-420 beats/min), where event rate is defined as the average rate of pitch change. The data support the hypothesis that the perceptual analysis of music entails the processes of parsing the signal into chunks of the appropriate temporal granularity and decoding the signal for recognition. The music-speech comparison points to similarities in how auditory processing builds on the specific temporal structure of the input, and how that structure interacts with the internal temporal dynamics of the neural mechanisms underpinning perception.
| S-EPMC4286482 | biostudies-literature