Wave physics as an analog recurrent neural network.
Ontology highlight
ABSTRACT:
SUBMITTER: Hughes TW
PROVIDER: S-EPMC6924985 | biostudies-literature | 2019 Dec
REPOSITORIES: biostudies-literature
Publications
Wave physics as an analog recurrent neural network.
Hughes Tyler W TW Williamson Ian A D IAD Minkov Momchil M Fan Shanhui S
Science advances 20191220 12
Analog machine learning hardware platforms promise to be faster and more energy efficient than their digital counterparts. Wave physics, as found in acoustics and optics, is a natural candidate for building analog processors for time-varying signals. Here, we identify a mapping between the dynamics of wave physics and the computation in recurrent neural networks. This mapping indicates that physical wave systems can be trained to learn complex features in temporal data, using standard training t ...[more]
Similar Datasets
Project description:The human brain can be considered as a complex dynamic and recurrent neural network. There are several models for neural networks of the human brain, that cover sensory to cortical information processing. Large majority models include feedback mechanisms that are hard to formalise to realistic applications. Recurrent neural networks and Long short-term memory (LSTM) inspire from the neuronal feedback networks. Long short-term memory (LSTM) prevent vanishing and exploding gradients problems faced by simple recurrent neural networks and has the ability to process order-dependent data. Such recurrent neural units can be replicated in hardware and interfaced with analog sensors for efficient and miniaturised implementation of intelligent processing. Implementation of analog memristive LSTM hardware is an open research problem and can offer the advantages of continuous domain analog computing with relatively low on-chip area compared with a digital-only implementation. Designed for solving time-series prediction problems, overall architectures and circuits were tested with TSMC 0.18 μm CMOS technology and hafnium-oxide (HfO 2) based memristor crossbars. Extensive circuit based SPICE simulations with over 3,500 (inference only) and 300 system-level simulations (training and inference) were performed for benchmarking the system performance of the proposed implementations. The analysis includes Monte Carlo simulations for the variability of memristors' conductance, and crossbar parasitic, where non-idealities of hybrid CMOS-memristor circuits are taken into the account.
| S-EPMC8506007 | biostudies-literature
Project description:Recurrent neural networks have seen widespread use in modeling dynamical systems in varied domains such as weather prediction, text prediction and several others. Often one wishes to supplement the experimentally observed dynamics with prior knowledge or intuition about the system. While the recurrent nature of these networks allows them to model arbitrarily long memories in the time series used in training, it makes it harder to impose prior knowledge or intuition through generic constraints. In this work, we present a path sampling approach based on principle of Maximum Caliber that allows us to include generic thermodynamic or kinetic constraints into recurrent neural networks. We show the method here for a widely used type of recurrent neural network known as long short-term memory network in the context of supplementing time series collected from different application domains. These include classical Molecular Dynamics of a protein and Monte Carlo simulations of an open quantum system continuously losing photons to the environment and displaying Rabi oscillations. Our method can be easily generalized to other generative artificial intelligence models and to generic time series in different areas of physical and social sciences, where one wishes to supplement limited data with intuition or theory based corrections. Adding prior experimentally or theoretically obtained knowledge to the training of recurrent neural networks may be challenging due to their feedback nature with arbitrarily long memories. The authors propose a path sampling approach that allows to include generic thermodynamic or kinetic constraints for learning of time series relevant to molecular dynamics and quantum systems.
| S-EPMC9700810 | biostudies-literature
Project description:Stress orientation information is invaluable to evaluate active tectonic forces within the Earth's crust. The global dataset provided by the World Stress Map offers a rich resource of stress indicators, facilitating the calibration of mechanical models to extract complete stress and displacement fields. However, traditional inversion processes are hampered by the manual tuning of geomechanical properties and boundary conditions to reconcile simulations with observations. In this study, we introduce ML-SEISMIC (machine learning for stress estimation integrating satellite image and computational modelling), a physics-informed deep neural network approach to autonomously align stress orientation data with an elastic model. It nearly completely bypasses the need for explicit boundary condition inputs and yields comprehensive distributions of material properties, displacements, and stress tensors. Application of this methodology to Australia, coupled with precise global navigation satellite systems observations, unveils a robust and scale-independent interpolation framework. Additionally, it pinpoints regions where stress orientation reinterpretation is warranted. Our results present a streamlined yet powerful process, offering a substantial leap forward in geodynamic investigations. This approach promises to unify velocity and stress orientation observations with physical models, ushering in a new era of insights into Earth's dynamic processes.
| S-EPMC10754839 | biostudies-literature
Project description:Compartmental models provide simple and efficient tools to analyze the relevant transmission processes during an outbreak, to produce short-term forecasts or transmission scenarios, and to assess the impact of vaccination campaigns. However, their calibration is not straightforward, since many factors contribute to the rapid change of the transmission dynamics. For example, there might be changes in the individual awareness, the imposition of non-pharmacological interventions and the emergence of new variants. As a consequence, model parameters such as the transmission rate are doomed to vary in time, making their assessment more challenging. Here, we propose to use Physics-Informed Neural Networks (PINNs) to track the temporal changes in the model parameters and the state variables. PINNs recently gained attention in many engineering applications thanks to their ability to consider both the information from data (typically uncertain) and the governing equations of the system. The ability of PINNs to identify unknown model parameters makes them particularly suitable to solve ill-posed inverse problems, such as those arising in the application of epidemiological models. Here, we develop a reduced-split approach for the implementation of PINNs to estimate the temporal changes in the state variables and transmission rate of an epidemic based on the SIR model equation and infectious data. The main idea is to split the training first on the epidemiological data, and then on the residual of the system equations. The proposed method is applied to five synthetic test cases and two real scenarios reproducing the first months of the Italian COVID-19 pandemic. Our results show that the split implementation of PINNs outperforms the joint approach in terms of accuracy (up to one order of magnitude) and computational times (speed up of 20%). Finally, we illustrate that the proposed PINN-method can also be adopted to produced short-term forecasts of the dynamics of an epidemic.
| S-EPMC11407682 | biostudies-literature
Project description:Molecular dynamics of charged particles in porous conductive media have received considerable attention in recent years due to their application in cutting-edge technologies such as batteries and supercapacitors. Due to the presence of long-range electrical interactions, induced charges present at the boundary, and the influence of boundary conditions, the simulation of these systems is more challenging than the simulation of typical molecular dynamic systems. Simulating these kinds of systems typically involves using a numerical solver to solve the Poisson equation, which is a very time-consuming procedure. Recently, Physics-Informed Neural Networks (PINNs) have been introduced as an alternative to numerical solutions of PDEs. In this paper, we present a new PINN-based model for predicting the potential of point-charged particles surrounded by conductive walls. As a result of the proposed PINN model, the mean square error is less than [Formula: see text] and [Formula: see text] score is more than [Formula: see text] for the corresponding example simulation. Results have been compared with typical neural networks and random forest as standard machine learning algorithms. The [Formula: see text] score of the random forest model was [Formula: see text], and a standard neural network could not be trained well.
| S-EPMC10462718 | biostudies-literature
Project description:The traditional approach of computational biology consists of calculating molecule properties by using approximate classical potentials. Interactions between atoms are described by an energy function derived from physical principles or fitted to experimental data. Their functional form is usually limited to pairwise interactions between atoms and does not consider complex multibody effects. More recently, neural networks have emerged as an alternative way of describing the interactions between biomolecules. In this approach, the energy function does not have an explicit functional form and is learned bottom-up from simulations at the atomistic or quantum level. In this study, we attempt a top-down approach and use deep learning methods to obtain an energy function by exploiting the large amount of experimental data acquired with years in the field of structural biology. The energy function is represented by a probability density model learned from a large repertoire of building blocks representing local clusters of amino acids paired with their sequence signature. We demonstrated the feasibility of this approach by generating a neural network energy function and testing its validity on several applications such as discriminating decoys, assessing qualities of structural models, sampling structural conformations, and designing new protein sequences. We foresee that, in the future, our methodology could exploit the continuously increasing availability of experimental data and simulations and provide a new method for the parametrization of protein energy functions.
| S-EPMC9476656 | biostudies-literature
Project description:Analog In-memory Computing (IMC) has demonstrated energy-efficient and low latency implementation of convolution and fully-connected layers in deep neural networks (DNN) by using physics for computing in parallel resistive memory arrays. However, recurrent neural networks (RNN) that are widely used for speech-recognition and natural language processing have tasted limited success with this approach. This can be attributed to the significant time and energy penalties incurred in implementing nonlinear activation functions that are abundant in such models. In this work, we experimentally demonstrate the implementation of a non-linear activation function integrated with a ramp analog-to-digital conversion (ADC) at the periphery of the memory to improve in-memory implementation of RNNs. Our approach uses an extra column of memristors to produce an appropriately pre-distorted ramp voltage such that the comparator output directly approximates the desired nonlinear function. We experimentally demonstrate programming different nonlinear functions using a memristive array and simulate its incorporation in RNNs to solve keyword spotting and language modelling tasks. Compared to other approaches, we demonstrate manifold increase in area-efficiency, energy-efficiency and throughput due to the in-memory, programmable ramp generator that removes digital processing overhead.
| S-EPMC11779922 | biostudies-literature
Project description:Analog in-memory computing is a promising future technology for efficiently accelerating deep learning networks. While using in-memory computing to accelerate the inference phase has been studied extensively, accelerating the training phase has received less attention, despite its arguably much larger compute demand to accelerate. While some analog in-memory training algorithms have been suggested, they either invoke significant amount of auxiliary digital compute-accumulating the gradient in digital floating point precision, limiting the potential speed-up-or suffer from the need for near perfectly programming reference conductance values to establish an algorithmic zero point. Here, we propose two improved algorithms for in-memory training, that retain the same fast runtime complexity while resolving the requirement of a precise zero point. We further investigate the limits of the algorithms in terms of conductance noise, symmetry, retention, and endurance which narrow down possible device material choices adequate for fast and robust in-memory deep neural network training.
| S-EPMC11335942 | biostudies-literature
Project description:Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical, medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network, which we term as Recurrent-MZ, 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over an extended depth-of-field. Using experiments on C. elegans and nanobead samples, Recurrent-MZ is demonstrated to significantly increase the depth-of-field of a 63×/1.4NA objective lens, also providing a 30-fold reduction in the number of axial scans required to image the same sample volume. We further illustrated the generalization of this recurrent network for 3D imaging by showing its resilience to varying imaging conditions, including e.g., different sequences of input images, covering various axial permutations and unknown axial positioning errors. We also demonstrated wide-field to confocal cross-modality image transformations using Recurrent-MZ framework and performed 3D image reconstruction of a sample using a few wide-field 2D fluorescence images as input, matching confocal microscopy images of the same sample volume. Recurrent-MZ demonstrates the first application of recurrent neural networks in microscopic image reconstruction and provides a flexible and rapid volumetric imaging framework, overcoming the limitations of current 3D scanning microscopy tools.
| S-EPMC7985192 | biostudies-literature
Project description:In this paper, a novel recurrent sigma‒sigma neural network (RSPSNN) that contains the same advantages as the higher-order and recurrent neural networks is proposed. The batch gradient algorithm is used to train the RSPSNN to search for the optimal weights based on the minimal mean squared error (MSE). To substantiate the unique equilibrium state of the RSPSNN, the characteristic of stability convergence is proven, which is one of the most significant indices for reflecting the effectiveness and overcoming the instability problem in the training of this network. Finally, to establish a more precise evaluation of its validity, five empirical experiments are used. The RSPSNN is successfully applied to the function approximation problem, prediction problem, parity problem, classification problem, and image simulation, which verifies its effectiveness and practicability.
| S-EPMC11696508 | biostudies-literature