Dataset Information

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models.

ABSTRACT: Objective.Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field.Approach.In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task.Main results.We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech.Significance.These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

SUBMITTER: Berezutskaya J

PROVIDER: S-EPMC10510111 | biostudies-literature | 2023 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models.

Berezutskaya Julia J Freudenburg Zachary V ZV Vansteensel Mariska J MJ Aarnoutse Erik J EJ Ramsey Nick F NF van Gerven Marcel A J MAJ

Journal of neural engineering 20230920 5

<i>Objective.</i>Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable resu ...[more]

PMID: 37467739

Similar Datasets

Project description:BackgroundThe indirect method for generating parametric images in positron emission tomography (PET) involves the acquisition and reconstruction of dynamic images and temporal modelling of tissue activity given a measured arterial input function. This approach is not robust, as noise in each dynamic image leads to a degradation in parameter estimation. Direct methods incorporate into the image reconstruction step both the kinetic and noise models, leading to improved parametric images. These methods require extensive computational time and large computing resources. Machine learning methods have demonstrated significant potential in overcoming these challenges. But they are limited by the requirement of a paired training dataset. A further challenge within the existing framework is the use of state-of-the-art arterial input function estimation via temporal arterial blood sampling, which is an invasive procedure, or an additional magnetic resonance imaging (MRI) scan for selecting a region where arterial blood signal can be measured from the PET image. We propose a novel machine learning approach for reconstructing high-quality parametric brain images from histoimages produced from time-of-flight PET data without requiring invasive arterial sampling, an MRI scan, or paired training data from standard field-of-view scanners.ResultThe proposed is tested on a simulated phantom and five oncological subjects undergoing an 18F-FDG-PET scan of the brain using Siemens Biograph Vision Quadra. Kinetic parameters set in the brain phantom correlated strongly with the estimated parameters (K1, k2 and k3, Pearson correlation coefficient of 0.91, 0.92 and 0.93) and a mean squared error of less than 0.0004. In addition, our method significantly outperforms (p < 0.05, paired t-test) the conventional nonlinear least squares method in terms of contrast-to-noise ratio. At last, the proposed method was found to be 37% faster than the conventional method.ConclusionWe proposed a direct non-invasive DL-based reconstruction method and produced high-quality parametric maps of the brain. The use of histoimages holds promising potential for enhancing the estimation of parametric images, an area that has not been extensively explored thus far. The proposed method can be applied to subject-specific dynamic PET data alone.

Project description:BackgroundWe retrospectively evaluated the quality of deep learning (DL) reconstructions of on-scanner accelerated intraoperative MRI (iMRI) during respective brain tumor surgery.MethodsAccelerated iMRI was performed using dual surface coils positioned around the area of resection. A DL model was trained on the fastMRI neuro dataset to mimic the data from the iMRI protocol. The evaluation was performed on imaging material from 40 patients imaged from Nov 1, 2021, to June 1, 2023, who underwent iMRI during tumor resection surgery. A comparative analysis was conducted between the conventional compressed sense (CS) method and the trained DL reconstruction method. Blinded evaluation of multiple image quality metrics was performed by two neuroradiologists and one neurosurgeon using a 1-to-5 Likert scale (1, nondiagnostic; 2, poor; 3, acceptable; 4, good; and 5, excellent), and the favored reconstruction variant.ResultsThe DL reconstruction was strongly favored or favored over the CS reconstruction for 33/40, 39/40, and 8/40 of cases for readers 1, 2, and 3, respectively. For the evaluation metrics, the DL reconstructions had a higher score than their respective CS counterparts for 72%, 72%, and 14% of the cases for readers 1, 2, and 3, respectively. Still, the DL reconstructions exhibited shortcomings such as a striping artifact and reduced signal.ConclusionDL shows promise in allowing for high-quality reconstructions of iMRI. The neuroradiologists noted an improvement in the perceived spatial resolution, signal-to-noise ratio, diagnostic confidence, diagnostic conspicuity, and spatial resolution compared to CS, while the neurosurgeon preferred the CS reconstructions across all metrics.Relevance statementDL shows promise to allow for high-quality reconstructions of iMRI, however, due to the challenging setting of iMRI, further optimization is needed.Key pointsiMRI is a surgical tool with a challenging image setting. DL allowed for high-quality reconstructions of iMRI. Additional optimization is needed due to the challenging intraoperative setting.

Project description:Introduction: Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that use MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region of interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. Methods: To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. Results and discussion: We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

Dataset Information

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models.

Publications

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets