Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning.

ABSTRACT: A large body of evidence has indicated that the phasic responses of midbrain dopamine neurons show a remarkable similarity to a type of teaching signal (temporal difference (TD) error) used in machine learning. However, previous studies failed to observe a key prediction of this algorithm: that when an agent associates a cue and a reward that are separated in time, the timing of dopamine signals should gradually move backward in time from the time of the reward to the time of the cue over multiple trials. Here we demonstrate that such a gradual shift occurs both at the level of dopaminergic cellular activity and dopamine release in the ventral striatum in mice. Our results establish a long-sought link between dopaminergic activity and the TD learning algorithm, providing fundamental insights into how the brain associates cues and rewards that are separated in time.

SUBMITTER: Amo R

PROVIDER: S-EPMC9624460 | biostudies-literature | 2022 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning.

Amo Ryunosuke R Matias Sara S Yamanaka Akihiro A Tanaka Kenji F KF Uchida Naoshige N Watabe-Uchida Mitsuko M

Nature neuroscience 20220707 8

A large body of evidence has indicated that the phasic responses of midbrain dopamine neurons show a remarkable similarity to a type of teaching signal (temporal difference (TD) error) used in machine learning. However, previous studies failed to observe a key prediction of this algorithm: that when an agent associates a cue and a reward that are separated in time, the timing of dopamine signals should gradually move backward in time from the time of the reward to the time of the cue over multip ...[more]

PMID: 35798979

Similar Datasets

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task.

Project description:Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.

| S-EPMC7771962 | biostudies-literature

Temporal difference error prediction signal dysregulation in cocaine dependence.

Project description:Cocaine dependence impacts drug-related, dopamine-dependent reward processing, yet its influence on non-drug reward processing is unclear. Here, we investigated cocaine-mediated effects on reward learning using a natural food reinforcer. Cocaine-dependent subjects (N = 14) and healthy controls (N = 14) learned to associate a visual cue with a juice reward. In subsequent functional imaging sessions they were exposed to trials where juice was received as learned, withheld (negative temporal difference error (NTDE)), or received unexpectedly (positive temporal difference error (PTDE)). Subjects were scanned twice in sessions that were identical, except that cocaine-dependent participants received cocaine or saline 10 min before task onset. In the insula, precentral and postcentral gyri NTDE signals were greater, and PTDE-related function was reduced in cocaine-dependent subjects. Compared with healthy controls, in the cocaine-dependent group PTDE signals were also reduced in medial frontal gyrus and reward-related function, irrespective of predictability, was reduced in the putamen. Group differences in error-related activity were predicted by the time as last self-administered cocaine use, but TDE function was not influenced by acute cocaine. Thus, cocaine dependence seems to engender increased responsiveness to unexpected negative outcomes and reduced sensitivity to positive events in dopaminergic reward regions. Although it remains to be established if these effects are a consequence of or antecedent to cocaine dependence, they likely have implications for the high-cocaine use recidivism rates by contributing to the drive to consume cocaine, perhaps via influence on dopamine-related reward computations. The fact that these effects do not acquiesce to acute cocaine administration might factor in binge-related escalated consumption.

| S-EPMC4023147 | biostudies-literature

An imperfect dopaminergic error signal can drive temporal-difference learning.

Project description:An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.

| S-EPMC3093351 | biostudies-literature

Dopamine reward prediction error responses reflect marginal utility.

Project description:BackgroundOptimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity.ResultsIn two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions' shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles.ConclusionsThese data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).

| S-EPMC4228052 | biostudies-literature

Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice.

Project description:Climbing fiber inputs to Purkinje cells are thought to be involved in generating the instructive signals that drive cerebellar learning. To investigate how these instructive signals are encoded, we recorded the activity of individual climbing fibers during cerebellum-dependent eyeblink conditioning in mice. We found that climbing fibers signaled both the unexpected delivery and the unexpected omission of the periocular airpuff that served as the instructive signal for eyeblink conditioning. In addition, we observed that climbing fibers activated by periocular airpuffs also responded to stimuli from other sensory modalities if those stimuli were novel or if they predicted that the periocular airpuff was about to be presented. This pattern of climbing fiber activity is markedly similar to the responses of dopamine neurons during reinforcement learning, which have been shown to encode a particular type of instructive signal known as a temporal difference prediction error.

| S-EPMC4754078 | biostudies-literature

Dopamine prediction error responses integrate subjective value from different reward dimensions.

Project description:Prediction error signals enable us to learn through experience. These experiences include economic choices between different rewards that vary along multiple dimensions. Therefore, an ideal way to reinforce economic choice is to encode a prediction error that reflects the subjective value integrated across these reward dimensions. Previous studies demonstrated that dopamine prediction error responses reflect the value of singular reward attributes that include magnitude, probability, and delay. Obviously, preferences between rewards that vary along one dimension are completely determined by the manipulated variable. However, it is unknown whether dopamine prediction error responses reflect the subjective value integrated from different reward dimensions. Here, we measured the preferences between rewards that varied along multiple dimensions, and as such could not be ranked according to objective metrics. Monkeys chose between rewards that differed in amount, risk, and type. Because their choices were complete and transitive, the monkeys chose "as if" they integrated different rewards and attributes into a common scale of value. The prediction error responses of single dopamine neurons reflected the integrated subjective value inferred from the choices, rather than the singular reward attributes. Specifically, amount, risk, and reward type modulated dopamine responses exactly to the extent that they influenced economic choices, even when rewards were vastly different, such as liquid and food. This prediction error response could provide a direct updating signal for economic values.

| S-EPMC3926061 | biostudies-literature

Dopamine reward prediction error signal codes the temporal evaluation of a perceptual decision report.

Project description:Learning to associate unambiguous sensory cues with rewarded choices is known to be mediated by dopamine (DA) neurons. However, little is known about how these neurons behave when choices rely on uncertain reward-predicting stimuli. To study this issue we reanalyzed DA recordings from monkeys engaged in the detection of weak tactile stimuli delivered at random times and formulated a reinforcement learning model based on belief states. Specifically, we investigated how the firing activity of DA neurons should behave if they were coding the error in the prediction of the total future reward when animals made decisions relying on uncertain sensory and temporal information. Our results show that the same signal that codes for reward prediction errors also codes the animal's certainty about the presence of the stimulus and the temporal expectation of sensory cues.

| S-EPMC5715768 | biostudies-literature

Causal evidence supporting the proposal that dopamine transients function as temporal difference prediction errors.

Project description:Reward-evoked dopamine transients are well established as prediction errors. However, the central tenet of temporal difference accounts-that similar transients evoked by reward-predictive cues also function as errors-remains untested. In the present communication we addressed this by showing that optogenetically shunting dopamine activity at the start of a reward-predicting cue prevents second-order conditioning without affecting blocking. These results indicate that cue-evoked transients function as temporal-difference prediction errors rather than reward predictions.

| S-EPMC7007380 | biostudies-literature

Effects of Gradual and Sudden Introduction of Perturbations on Adaptive Responses to Formant-Shift and Formant-Clamp Perturbations.

Project description:PurposeWhen the speech motor system encounters errors, it generates adaptive responses to compensate for the errors. Unlike errors induced by formant-shift perturbations, errors induced by formant-clamp perturbations do not correspond with the speaker's speech (i.e., degraded motor-to-auditory correspondence). We previously showed that adaptive responses to formant-clamp perturbations are smaller than responses to formant-shift perturbations when perturbations are introduced gradually. This study examined responses to formant-clamp and formant-shift perturbations when perturbations are introduced suddenly.MethodOne group of participants (n = 30) experienced gradually introduced formant-clamp and formant-shift perturbations, and another group (n = 30) experienced suddenly introduced formant-clamp and formant-shift perturbations. We designed the perturbations based on participant-specific vowel configurations such that a participant's first and second formants of /ɛ/ were perturbed toward their /æ/. To estimate adaptive responses, we measured formant changes (0-100 ms of the vowel) in response to the formant perturbations.ResultsWe found that (a) the difference between responses to formant-clamp and formant-shift perturbations was smaller when the perturbations were introduced suddenly and (b) responses to suddenly introduced (but not gradually introduced) formant-shift perturbations positively correlated with responses to formant-clamp perturbations.ConclusionsThese results showed that the speech motor system responds to errors induced by formant-shift and formant-clamp perturbations more differently when perturbations are introduced gradually than suddenly. Overall, the quality of errors (formant-shift vs. formant-clamp) and the manner of introducing errors (gradually vs. suddenly) modulate the speech motor system's evaluations of and responses to errors.Supplemental materialhttps://doi.org/10.23641/asha.22406422.

| S-EPMC10457088 | biostudies-literature

Spatio-temporal characterization of attosecond pulses from plasma mirrors.

Project description:Reaching light intensities above 1025 W/cm2 and up to the Schwinger limit of the order of 1029 W/cm2 would enable testing fundamental predictions of quantum electrodynamics. A promising - yet challenging - approach to achieve such extreme fields consists in reflecting a high-power femtosecond laser pulse off a curved relativistic mirror. This enhances the intensity of the reflected beam by simultaneously compressing it in time down to the attosecond range, and focusing it to sub-micrometre focal spots. Here we show that such curved relativistic mirrors can be produced when an ultra-intense laser pulse ionizes a solid target and creates a dense plasma that specularly reflects the incident light. This is evidenced by measuring the temporal and spatial effects induced on the reflected beam by this so-called 'plasma mirror'. The all-optical measurement technique demonstrated here will be instrumental for the use of relativistic plasma mirrors with the upcoming generation of Petawatt lasers that recently reached intensities of 5 × 1022 W/cm2, and therefore constitutes a viable experimental path to the Schwinger limit.

| S-EPMC7611525 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data