Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Active fixation as an efficient coding strategy for neuromorphic vision.

ABSTRACT: Contrary to a photographer, who puts a great effort in keeping the lens still, eyes insistently move even during fixation. This benefits signal decorrelation, which underlies an efficient encoding of visual information. Yet, camera motion is not sufficient alone; it must be coupled with a sensor specifically selective to temporal changes. Indeed, motion induced on standard imagers only results in burring effects. Neuromorphic sensors represent a valuable solution. Here we characterize the response of an event-based camera equipped with fixational eye movements (FEMs) on both synthetic and natural images. Our analyses prove that the system starts an early stage of redundancy suppression, as a precursor of subsequent whitening processes on the amplitude spectrum. This does not come at the price of corrupting structural information contained in local spatial phase across oriented axes. Isotropy of FEMs ensures proper representations of image features without introducing biases towards specific contrast orientations.

SUBMITTER: Testa S

PROVIDER: S-EPMC10167324 | biostudies-literature | 2023 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Active fixation as an efficient coding strategy for neuromorphic vision.

Testa Simone S Sabatini Silvio P SP Canessa Andrea A

Scientific reports 20230508 1

Contrary to a photographer, who puts a great effort in keeping the lens still, eyes insistently move even during fixation. This benefits signal decorrelation, which underlies an efficient encoding of visual information. Yet, camera motion is not sufficient alone; it must be coupled with a sensor specifically selective to temporal changes. Indeed, motion induced on standard imagers only results in burring effects. Neuromorphic sensors represent a valuable solution. Here we characterize the respon ...[more]

PMID: 37156822

Similar Datasets

Autonomous Development of Active Binocular and Motion Vision Through Active Efficient Coding.

Project description:We present a model for the autonomous and simultaneous learning of active binocular and motion vision. The model is based on the Active Efficient Coding (AEC) framework, a recent generalization of classic efficient coding theories to active perception. The model learns how to efficiently encode the incoming visual signals generated by an object moving in 3-D through sparse coding. Simultaneously, it learns how to produce eye movements that further improve the efficiency of the sensory coding. This learning is driven by an intrinsic motivation to maximize the system's coding efficiency. We test our approach on the humanoid robot iCub using simulations. The model demonstrates self-calibration of accurate object fixation and tracking of moving objects. Our results show that the model keeps improving until it hits physical constraints such as camera or motor resolution, or limits on its internal coding capacity. Furthermore, we show that the emerging sensory tuning properties are in line with results on disparity, motion, and motion-in-depth tuning in the visual cortex of mammals. The model suggests that vergence and tracking eye movements can be viewed as fundamentally having the same objective of maximizing the coding efficiency of the visual system and that they can be learned and calibrated jointly through AEC.

| S-EPMC6646586 | biostudies-literature

Active efficient coding explains the development of binocular vision and its failure in amblyopia.

Project description:The development of vision during the first months of life is an active process that comprises the learning of appropriate neural representations and the learning of accurate eye movements. While it has long been suspected that the two learning processes are coupled, there is still no widely accepted theoretical framework describing this joint development. Here, we propose a computational model of the development of active binocular vision to fill this gap. The model is based on a formulation of the active efficient coding theory, which proposes that eye movements as well as stimulus encoding are jointly adapted to maximize the overall coding efficiency. Under healthy conditions, the model self-calibrates to perform accurate vergence and accommodation eye movements. It exploits disparity cues to deduce the direction of defocus, which leads to coordinated vergence and accommodation responses. In a simulated anisometropic case, where the refraction power of the two eyes differs, an amblyopia-like state develops in which the foveal region of one eye is suppressed due to inputs from the other eye. After correcting for refractive errors, the model can only reach healthy performance levels if receptive fields are still plastic, in line with findings on a critical period for binocular vision development. Overall, our model offers a unifying conceptual framework for understanding the development of binocular vision.

| S-EPMC7084066 | biostudies-literature

Nearly Panoramic Neuromorphic Vision with Transparent Photosynapses.

Project description:Neuromorphic vision based on photonic synapses has the ability to mimic sensitivity, adaptivity, and sophistication of bio-visual systems. Significant advances in artificial photosynapses are achieved recently. However, conventional photosyanptic devices normally employ opaque metal conductors and vertical device configuration, performing a limited hemispherical field of view. Here, a transparent planar photonic synapse (TPPS) is presented that offers dual-side photosensitive capability for nearly panoramic neuromorphic vision. The TPPS consisting of all two dimensional (2D) carbon-based derivatives exhibits ultra-broadband photodetecting (365-970 nm) and ≈360° omnidirectional viewing angle. With its intrinsic persistent photoconductivity effect, the detector possesses bio-synaptic behaviors such as short/long-term memory, experience learning, light adaptation, and a 171% pair-pulse-facilitation index, enabling the synapse array to achieve image recognition enhancement (92%) and moving object detection.

| S-EPMC10602561 | biostudies-literature

Widefield Diamond Quantum Sensing with Neuromorphic Vision Sensors.

Project description:Despite increasing interest in developing ultrasensitive widefield diamond magnetometry for various applications, achieving high temporal resolution and sensitivity simultaneously remains a key challenge. This is largely due to the transfer and processing of massive amounts of data from the frame-based sensor to capture the widefield fluorescence intensity of spin defects in diamonds. In this study, a neuromorphic vision sensor to encode the changes of fluorescence intensity into spikes in the optically detected magnetic resonance (ODMR) measurements is adopted, closely resembling the operation of the human vision system, which leads to highly compressed data volume and reduced latency. It also results in a vast dynamic range, high temporal resolution, and exceptional signal-to-background ratio. After a thorough theoretical evaluation, the experiment with an off-the-shelf event camera demonstrated a 13× improvement in temporal resolution with comparable precision of detecting ODMR resonance frequencies compared with the state-of-the-art highly specialized frame-based approach. It is successfully deploy this technology in monitoring dynamically modulated laser heating of gold nanoparticles coated on a diamond surface, a recognizably difficult task using existing approaches. The current development provides new insights for high-precision and low-latency widefield quantum sensing, with possibilities for integration with emerging memory devices to realize more intelligent quantum sensors.

| S-EPMC10787069 | biostudies-literature

A Spike-Based Neuromorphic Architecture of Stereo Vision.

Project description:The problem of finding stereo correspondences in binocular vision is solved effortlessly in nature and yet it is still a critical bottleneck for artificial machine vision systems. As temporal information is a crucial feature in this process, the advent of event-based vision sensors and dedicated event-based processors promises to offer an effective approach to solving the stereo matching problem. Indeed, event-based neuromorphic hardware provides an optimal substrate for fast, asynchronous computation, that can make explicit use of precise temporal coincidences. However, although several biologically-inspired solutions have already been proposed, the performance benefits of combining event-based sensing with asynchronous and parallel computation are yet to be explored. Here we present a hardware spike-based stereo-vision system that leverages the advantages of brain-inspired neuromorphic computing by interfacing two event-based vision sensors to an event-based mixed-signal analog/digital neuromorphic processor. We describe a prototype interface designed to enable the emulation of a stereo-vision system on neuromorphic hardware and we quantify the stereo matching performance with two datasets. Our results provide a path toward the realization of low-latency, end-to-end event-based, neuromorphic architectures for stereo vision.

| S-EPMC7693562 | biostudies-literature

A spiking neural network for active efficient coding.

Project description:Biological vision systems simultaneously learn to efficiently encode their visual inputs and to control the movements of their eyes based on the visual input they sample. This autonomous joint learning of visual representations and actions has previously been modeled in the Active Efficient Coding (AEC) framework and implemented using traditional frame-based cameras. However, modern event-based cameras are inspired by the retina and offer advantages in terms of acquisition rate, dynamic range, and power consumption. Here, we propose a first AEC system that is fully implemented as a Spiking Neural Network (SNN) driven by inputs from an event-based camera. This input is efficiently encoded by a two-layer SNN, which in turn feeds into a spiking reinforcement learner that learns motor commands to maximize an intrinsic reward signal. This reward signal is computed directly from the activity levels of the first two layers. We test our approach on two different behaviors: visual tracking of a translating target and stabilizing the orientation of a rotating target. To the best of our knowledge, our work represents the first ever fully spiking AEC model.

| S-EPMC11775837 | biostudies-literature

Finding the gap: neuromorphic motion-vision in dense environments.

Project description:Animals have evolved mechanisms to travel safely and efficiently within different habitats. On a journey in dense terrains animals avoid collisions and cross narrow passages while controlling an overall course. Multiple hypotheses target how animals solve challenges faced during such travel. Here we show that a single mechanism enables safe and efficient travel. We developed a robot inspired by insects. It has remarkable capabilities to travel in dense terrain, avoiding collisions, crossing gaps and selecting safe passages. These capabilities are accomplished by a neuromorphic network steering the robot toward regions of low apparent motion. Our system leverages knowledge about vision processing and obstacle avoidance in insects. Our results demonstrate how insects might safely travel through diverse habitats. We anticipate our system to be a working hypothesis to study insects' travels in dense terrains. Furthermore, it illustrates that we can design novel hardware systems by understanding the underlying mechanisms driving behaviour.

| S-EPMC10821932 | biostudies-literature

Tetrachromatic vision-inspired neuromorphic sensors with ultraweak ultraviolet detection.

Project description:Sensing and recognizing invisible ultraviolet (UV) light is vital for exploiting advanced artificial visual perception system. However, due to the uncertainty of the natural environment, the UV signal is very hard to be detected and perceived. Here, inspired by the tetrachromatic visual system, we report a controllable UV-ultrasensitive neuromorphic vision sensor (NeuVS) that uses organic phototransistors (OPTs) as the working unit to integrate sensing, memory and processing functions. Benefiting from asymmetric molecular structure and unique UV absorption of the active layer, the as fabricated UV-ultrasensitive NeuVS can detect 370 nm UV-light with the illumination intensity as low as 31 nW cm-2, exhibiting one of the best optical figures of merit in UV-sensitive neuromorphic vision sensors. Furthermore, the NeuVS array exbibits good image sensing and memorization capability due to its ultrasensitive optical detection and large density of charge trapping states. In addition, the wavelength-selective response and multi-level optical memory properties are utilized to construct an artificial neural network for extract and identify the invisible UV information. The NeuVS array can perform static and dynamic image recognition from the original color image by filtering red, green and blue noise, and significantly improve the recognition accuracy from 46 to 90%.

| S-EPMC10121588 | biostudies-literature

RGB Color-Discriminable Photonic Synapse for Neuromorphic Vision System.

Project description:To emulate the functionality of the human retina and achieve a neuromorphic visual system, the development of a photonic synapse capable of multispectral color discrimination is of paramount importance. However, attaining robust color discrimination across a wide intensity range, even irrespective of medium limitations in the channel layer, poses a significant challenge. Here, we propose an approach that can bestow the color-discriminating synaptic functionality upon a three-terminal transistor flash memory even with enhanced discriminating capabilities. By incorporating the strong induced dipole moment effect at the excitation, modulated by the wavelength of the incident light, into the floating gate, we achieve outstanding RGB color-discriminating synaptic functionality within a remarkable intensity range spanning from 0.05 to 40 mW cm-2. This approach is not restricted to a specific medium in the channel layer, thereby enhancing its applicability. The effectiveness of this color-discriminating synaptic functionality is demonstrated through visual pre-processing of a photonic synapse array, involving the differentiation of RGB channels and the enhancement of image contrast with noise reduction. Consequently, a convolutional neural network can achieve an impressive inference accuracy of over 94% for Canadian-Institute-For-Advanced-Research-10 colorful image recognition task after the pre-processing. Our proposed approach offers a promising solution for achieving robust and versatile RGB color discrimination in photonic synapses, enabling significant advancements in artificial visual systems.

| S-EPMC11607236 | biostudies-literature

The Active Side of Stereopsis: Fixation Strategy and Adaptation to Natural Environments.

Project description:Depth perception in near viewing strongly relies on the interpretation of binocular retinal disparity to obtain stereopsis. Statistical regularities of retinal disparities have been claimed to greatly impact on the neural mechanisms that underlie binocular vision, both to facilitate perceptual decisions and to reduce computational load. In this paper, we designed a novel and unconventional approach in order to assess the role of fixation strategy in conditioning the statistics of retinal disparity. We integrated accurate realistic three-dimensional models of natural scenes with binocular eye movement recording, to obtain accurate ground-truth statistics of retinal disparity experienced by a subject in near viewing. Our results evidence how the organization of human binocular visual system is finely adapted to the disparity statistics characterizing actual fixations, thus revealing a novel role of the active fixation strategy over the binocular visual functionality. This suggests an ecological explanation for the intrinsic preference of stereopsis for a close central object surrounded by a far background, as an early binocular aspect of the figure-ground segregation process.

| S-EPMC5357847 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data