RetinaStudio: A bioinspired framework to encode visual information
Introduction
The retina is essentially a piece of brain tissue that gets direct stimulation from the outside world's lights and images. Visual input to the retina consists of a stream of photons, which can be unequivocally quantified in space and time. The retina performs spatial, temporal, and chromatic processing on visual information and converts it into spike trains. Thus our entire experience of the external visual world derives from the concerted activity of a restricted number of retinal ganglion cells, which have to send their information, via the optic nerve, to higher visual centers. This representation has to be unequivocal and fast, in order to ensure object recognition for any single stimulus presentation within a few milliseconds [1]. Therefore, the question of how the information about the external world is compressed in the retina, and how this compressed representation is encoded in spike trains is an important challenge for visual neuroscience and for applications as visual prosthesis [2].
Our group is working on the development of a cortical visual neuroprosthesis aimed to restore some functional vision to profoundly visual-impaired people. The goal of developing such a bioinspired retinal encoder is not simply recording a high-resolution image, but to transmit visual information in a meaningful way to the appropriate site(s) in the visual pathways. In order to achieve this goal we have to take into account the coding features of the biological visual system and design constraints related to the number and distribution of the electrodes where the visual scene is mapped to [2], [3].
A special problem to be addressed when dealing with different models and tools for the design of visual neuroprosthesis is that there are many issues involved. These issues usually overlap a number of heterogeneous disciplines such as neuroscience, neuroengineering, electronics engineering and computer science. This fact depicts a sort of questions that must be considered to design a reliable retina bio-model such as:
- •
Is the physical implementation target fast enough for a sustained in vivo stimulation?
- •
What is the maximum error (worst case) that could be generated using a given technology?
- •
What is the more convenient data-representation to be used, and how does it affects to the overall system speed?
- •
Can the model be embedded into a wearable device?
- •
How many variables from the visual processing system can operate as editable parameters for a rapid tuning of the model?
Our contribution to partially cover these topics is presented as a high-level abstraction framework. The system, named RetinaStudio, allows the specification, testing, simulation, validation and implementation of bioinspired retina models to be used within any visual neuroprosthesis. It offers a uniform platform that overcomes the drawback generated by the diverse disciplines that take place in the design of the final device. In this way, RetinaStudio is an interdisciplinary tool suitable for visual scientists, neuro/electronics/computer engineers, ophthalmologists and neurologists. The interaction of such heterogeneous disciplines and the environment has been carefully designed to be straightforward and efficient. The final objective is to progress towards a uniform functional tool able to:
- •
describe and simulate a custom bioinspired retina model at various levels of abstraction;
- •
perform an automatic optimization and acceleration of the processing by means of a sort of supported technologies: FPGA, COTS microprocessors or GPU architectures;
- •
assist the designer to decide the final target to be used in terms of: real-time capabilities, feasibility, portability (what is the hardware support that best fit the designer constraints?), plasticity (what variables can be defined as custom parameters?), etc.;
- •
easily compare synthetics and biological records using de facto standards for data sharing (i.e. the Neural Event Format [4]) to validate the correctness of the retina model;
- •
perform in vivo stimulation using optimized implementations generated by RetinaStudio.
At the moment there is not any tool able to offer all these requirements as a uniform framework. Currently, some research projects such as Retiner [5], Virtual Retina [6] or EPIRET3 [7] are developing tools in this direction. However these tools only support a reduced subset of the previous items and do not offer flexible and fast enough models to perform real-time stimulations.
The paper is organized as follows: next section presents the proposed framework to model and test retina models; Section 3 includes some experiments and use cases related with the tool. Finally Section 4 presents some concluding remarks.
Section snippets
A multidisciplinary framework to model artificial retinas
The bioinspired processing scheme used by RetinaStudio is based on the histology and physiology of vertebrate retinas (see Fig. 1). Briefly this scheme is composed of:
- •
A first stage, the image capturing stage, which is inspired in the Outer Plexiform Layer of the retina (OPL) where connections between rod and cones, vertically running bipolar cells and horizontally oriented horizontal cells occur. In this stage the image is captured (typically taken from one or more videocameras) and it is split
Case study
To assess the effectiveness offered by RetinaStudio to design, simulate and validate bioinspired retina models, we have designed and performed several experiments. Our results show the simplicity of describing several basic custom retina models using Flowlang language as well as its potential usefulness for neuroscientists working with retinal recordings due to its high performance. Post-simulation experimental output generated by RetinaStudio are conveniently discussed.
Conclusions
This paper presents a new functional framework to describe, simulate and validate custom retina models. The tool, named RetinaStudio, supports Flowlang, a new proposed high-level language for modeling the behavior of vertebrate retinas that can be also used to get a better understanding of visual processing of biological retinas. Flowlang models can be compiled using a retargetable compiler included in RetinaStudio, which can generate optimized code using SIMD-accelerated processing primitives
Acknowledgments
We would like to thank Markus Bongard for all his help with the electrophysiological recordings. This work has been supported in part by the ONCE (National Organization of the Spanish Blind), by the Research Chair on Retinitis Pigmentosa Bidons Egara and by the Grant SAF2008-03694 from the Spanish Government.
Antonio Martínez-Álvarez was born in Granada, Spain, in 1976. He received the M.S. and Ph.D. degrees in Electronics Engineering by the University of Granada, Spain, in 2002 and 2006, respectively. From 2002 to 2006 he joined the Department of Computer Architecture and Technology at the University of Granada, Spain. He is currently an Associate Professor with the Department of Computer Technology, University of Alicante, Spain. His main research interests deal with methods and tools for
References (14)
- et al.
The neural concert of vision
Neurocomputing
(2009) - et al.
Translating image sequences into spike patterns for cortical neuro-stimulation
Neurocomputing
(2004) - et al.
A neural interface for a cortical vision prosthesis
Vision Res.
(1999) - et al.
Development of a cortical visual neuroprosthesis for the blindthe relevance of neuroplasticity
J. Neural Eng.
(2005) - et al.
Toward the development of a cortically based visual neuroprosthesis
J. Neural Eng.
(2009) - Neural Event Format, 〈http://cyberkineticsinc.com/NEVspc20.pdf〉, [Online; accessed 12-May-2011],...
- et al.
Virtual Retinaa biological retina model and simulator, with contrast gain control
J. Comput. Neurosci.
(2009)
Cited by (14)
Biologically-inspired image processing in computational retina models
2019, Computers in Biology and MedicineCitation Excerpt :Although nonlinear RGC responses have been described early in the study of retina function [12,16], they initially received less attention, as linear RGCs [17] were initially assumed to have the central role in vision [13]. Many functional models of the retina are comprised of linear filters followed by a static non-linearity (linear-nonlinear or LN models) and a spike generation mechanism that can be either probabilistic or deterministic [18–20]. In models that mimic information processing in the retina, model parameters may be deduced from retina physical parameters [21].
Bio-inspired computer vision: Towards a synergistic approach of artificial and biological vision
2016, Computer Vision and Image UnderstandingModeling the role of fixational eye movements in real-world scenes
2015, NeurocomputingCitation Excerpt :Therefore a good retina model, as well as its physical implementation, should take into account this time constraint to be able to respond to stimuli in real time. In this paper several retina models which are sensitive to variations in luminance are described by mean of RetinaStudio [18], a framework to encode visual information that allows the specification, testing, simulation, validation and implementation of bioinspired retina models. Each retina model is defined as a matrix of different kinds of ganglion cells.
Fusion of ANNs as decoder of retinal spike trains for scene reconstruction
2022, Applied Intelligence
Antonio Martínez-Álvarez was born in Granada, Spain, in 1976. He received the M.S. and Ph.D. degrees in Electronics Engineering by the University of Granada, Spain, in 2002 and 2006, respectively. From 2002 to 2006 he joined the Department of Computer Architecture and Technology at the University of Granada, Spain. He is currently an Associate Professor with the Department of Computer Technology, University of Alicante, Spain. His main research interests deal with methods and tools for dependable design of digital integrated circuits and FPGAs, high-performance image-processing architectures and embedded systems based on reconfigurable devices. He is also interested in neuroengineering and neuroprosthesis devices. Currently he is working in the design and development of RetinaStudio.
Andrés Olmedo-Payá was born in Alcoy, Spain, in 1981. He received his Computer Engineering degree in 2010 from University of Alicante. Currently he is a Ph.D. student in the Computer Technology Department at University of Alicante and he works in the Artificial Vision Laboratory at the Bioengineering Institute of the University Miguel Hernández, Spain. His research interests include high-performance image-processing and neuroprosthesis devices. Currently he is working in the design and development of RetinaStudio.
Sergio A. Cuenca-Asensi is an associate professor in the Computer Architecture and Technology Department at University of Alicante, Spain. He received the B.S. degree in Electronic Physics in 1990 from University of Granada, Spain. He received a Ph.D. in Computer Engineering from the University Miguel Hernández of Elche, Spain, in 2002. His current research interests are reconfigurable computing, high-performance image-processing architectures, hardware/software co-design and soft error mitigation in embedded systems.
José Manuel Ferrández Vicente was born in Elche, Spain. He received the M.Sc. degree in Computer Science in 1995, and the Ph.D. degree in 1998, all of them from the Universidad Politécnica de Madrid, Spain. He is currently Associate Professor at the Department of Electronics, Computer Technology and Projects at the Universidad Politécnica de Cartagena and Head of the Electronic Design and Signal Processing Research Group at the same University. He is the Coordinator of the Spanish and the Iberoamerican Network on Natural and Artificial Computation. His research interests include bioinspired processing, neuromorphic engineering and low vision prosthesis.
Eduardo Fernández received the M.D. degree from the University of Alicante, Spain, in 1986 and the Ph.D. degree in Neurosciences in 1990. He is currently Full Professor at the University Miguel Hernéndez, Spain, and Director of the Artificial Vision Laboratory at the Bioengineering Institute of the University Miguel Hernández, Spain. In the last years he has been using histological as well as electrophysiological techniques to understand how mammalian retinal cells and the circuitry within the retina can manage and code visual information. He is actively working on the development of a visual neuroprosthesis for the profoundly blind.