Atif Hashmi
Abstract
The desire to create novel computing systems, paired with recent advances in neuroscientific understanding of the brain, has led researchers to develop neuromorphic architectures that emulate the brain. To date, such models are developed, trained, and deployed on the same substrate. However, excessive co-dependence between the substrate and the algorithm prevents portability, or at the very least requires reconstructing and retraining the model whenever the substrate changes. This paper proposes a well-defined abstraction layer -- the Neuromorphic instruction set architecture, or NISA -- that separates a neural application's algorithmic specification from the underlying execution substrate, and describes the Aivo framework, which demonstrates the concrete advantages of such an abstraction layer. Aivo consists of a NISA implementation for a rate-encoded neuromorphic system based on the cortical column abstraction, a state-of-the-art integrated development and runtime environment (IDE), and various profile-based optimization tools. Aivo's IDE generates code for emulating cortical networks on the host CPU, multiple GPGPUs, or as boolean functions. Its runtime system can deploy and adaptively optimize cortical networks in a manner similar to conventional just-in-time compilers in managed runtime systems (e.g. Java, C#).
We demonstrate the abilities of the NISA abstraction by constructing a cortical network model of the mammalian visual cortex, deploying on multiple execution substrates, and utilizing the various optimization tools we have created. For this hierarchical configuration, Aivo's profiling based network optimization tools reduce the memory footprint by 50% and improve the execution time by a factor of 3x on the host CPU. Deploying the same network on a single GPGPU results in a 30x speedup. We further demonstrate that a speedup of 480x can be achieved by deploying a massively scaled cortical network across three GPGPUs. Finally, converting a trained hierarchical network to C/C++ boolean constructs on the host CPU results in 44x speedup.
- Systems of neuromorphic adaptive plastic scalable electronics (synapse). 2008. URL http://www.darpa.mil/dso/solicitations/baa08-28.htm.Google Scholar
- Matlab neural network toolbox, July 2010. URL http://www.mathworks.com/ products/ neuralnet/.Google Scholar
- Java neuroph, July 2010. URL http://neuroph.sourceforge.net/index.html.Google Scholar
- G. M. Amdahl, G. A. Blaauw, and F. P. Brooks. Architecture of the ibm system/360. IBM Journal of Research and Development, 8(2):87--101, 1964. ISSN 0018-8646. doi: 10.1147/rd.82.0087. Google ScholarDigital Library
- J. Arthur and K. Boahen. Learning in silicon: Timing is everything. In Proceedings of Advances in Neural Information Processing Systems, volume 18, pages 75--82. Advances in Neural Information Processing Systems, 2006.Google Scholar
- R. K. B. Awerbuch. Competitive collaborative learning. Journal of Computer and System Sciences, 74(8):1271--1288, 2008. Google ScholarDigital Library
- T. Binzegger, R. Douglas, and K. Martin. A quantitative map of the circuit of cat primary visual cortex. J. Neurosci., 24(39):8441--8453, Sep 2004.Google ScholarCross Ref
- W. Calvin. Cortical columns, modules, and hebbian cell assemblies. In M. A. Arbib, editor, The Handbook of Brain Theory and Neural Networks, pages 269--272. MIT Press, Cambridge, MA, 1998. Google ScholarDigital Library
- L. Chua. Memristor-the missing circuit element. IEEE Transactions on Circuit Theory, 18(5):507--519, 1971.Google ScholarCross Ref
- D. DeSieno. Adding a conscience to competitive learning. In International Conference on Neural Networks, ICNN, volume 1, pages 117--124, 1988.Google ScholarCross Ref
- F. Folowosele, R. Vogelstein, and R. Etienne-Cummings. Real-time silicon implementation of v1 in hierarchical visual information processing. In Biomedical Circuits and Systems Conference, 2008. BioCAS 2008. IEEE, pages 181--184, 2008. doi: 10.1109/BIO-CAS.2008.4696904.Google ScholarCross Ref
- W. Freeman. Random activity at the microscopic neural level in cor-tex ("noise") sustains and is regulated by low-dimensional dynamics of macroscopic activity ("chaos"). International Journal of Neural Systems, 7(4):473--480, 1996.Google ScholarCross Ref
- K. Grill-Spector, T. Kushnir, T. Hendler, S. Edelman, Y. Itzchak, and R. Malach. A sequence of object-processing stages revealed by fmri in the human occipital lobe. Hum. Brain Map., 6:316--328, 1998.Google ScholarCross Ref
- A. Hashmi and M. Lipasti. Discovering cortical algorithms. In Proceedings of the International Conference on Neural Computation (ICNC 2010), 2010.Google Scholar
- A. Hashmi, H. Berry, O. Temam, and M. H. Lipasti. Leveraging progress in neurobiology for computing systems. In Proceedings of the Workshop on New Directions in Computer Architecture held in Conjunction with 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-42), 2009.Google Scholar
- J. Hawkins and D. George. Hierarchical temporal memory. 2006. URL www.numenta.com/Numenta HTM Concepts.pdf.Google Scholar
- M. Holler, S. Tam, H. Castro, and R. Benson. An electrically trainable artificial neural network (etann) with 10240 'floating gate' synapses. In Neural Networks, 1989. IJCNN., International Joint Conference on, pages 191--196 vol.2, June 1989. doi: 10.1109/IJCNN.1989.118698.Google ScholarCross Ref
- D. Hubel and T. Wiesel. Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology, 195:215--243, 1968.Google ScholarCross Ref
- K. Hynna and K. Boahen. Silicon neurons that burst when primed. Circuits and Systems, 2007. ISCAS 2007. IEEE International Symposium on, pages 3363--3366, May 2007. doi: 10.1109/IS-CAS.2007.378288.Google ScholarCross Ref
- E. Izhikevich. Which model to use for cortical spiking neurons? Neural Networks, IEEE Transactions on, 15(5):1063--1070, 2004. ISSN 1045--9227. doi: 10.1109/TNN.2004.832719. Google ScholarDigital Library
- H. Jang, A. Park, and K. Jung. Neural network implementation using cuda and openmp. In DICTA '08: Proceedings of the 2008 Digital Image Computing: Techniques and Applications, pages 155--161, Washington, DC, USA, 2008. IEEE Computer Society. ISBN 978-0-7695-3456-5. doi: http://dx.doi.org/10.1109/DICTA.2008.82. Google ScholarDigital Library
- J. Jones and L. Palmer. An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex. Journal of Neurophysiology, 58(6):1233--1258, December 1987.Google ScholarCross Ref
- S. Jung and S. su Kim. Hardware implementation of a real-time neural network controller with a dsp and an fpga for nonlinear systems. Industrial Electronics, IEEE Transactions on, 54(1):265--271, 2007. ISSN 0278-0046. doi: 10.1109/TIE.2006.888791.Google Scholar
- E. Kandel, J. Schwartz, and T. Jessell. Principles of Neural Science. McGraw-Hill, 4 edition, 2000.Google Scholar
- Y. LeCun, L. Bottou, Y. Bengio, and P. learning applied to document recognition. 86(11):2278--2324, 1998.Google Scholar
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. 86(11):2278--2324, November 1998. Haffner. Gradient-based Proceedings of the IEEE, Haffner. Gradient-based Proceedings of the IEEE,Google Scholar
- H. Markram. The blue brain project. In SC '06: Proceedings of the 2006 ACM/IEEE conference on Supercomputing, page 53, New York, NY, USA, 2006. ACM. ISBN 0-7695-2700-0. doi: http://doi.acm.org/10.1145/1188455.1188511. Google ScholarDigital Library
- T. Martinetz. Competitive hebbian learning rule forms perfectly topology preserving maps. In International Conference on Artificial Neural Networks, ICANN, pages 427--434, 1993.Google ScholarCross Ref
- M. Matthias and J. Born. Hippocampus whispering in deep sleep to prefrontal cortex for good memories? Neuron, 61:496--498, 2009.Google ScholarCross Ref
- V. Mountcastle. An organizing principle for cerebral function: The unit model and the distributed system. In G. Edelman and V. Mount-castle, editors, The Mindful Brain. MIT Press, Cambridge, Mass., 1978.Google Scholar
- V. Mountcastle. The columnar organization of the neocortex. Brain, 120:701--722, 1997.Google ScholarCross Ref
- A. Nere and M. Lipasti. Cortical architectures on a gpgpu. In GPGPU '10: Proceedings of the 3rd Workshop on General-Purpose Computation on Graphics Processing Units, pages 12--18, New York, NY, USA, 2010. ACM. ISBN 978-1-60558-935-0. doi: http://doi.acm.org/10.1145/1735688.1735693. Google ScholarDigital Library
- M. O'Neil. Neural network for recognition of handwritten digits, October 2010. URL http://www.codeproject.com/KB/library/NeuralNetRecognition.aspx.Google Scholar
- Y. V. Pershin, S. La Fontaine, and M. Di Ventra. Memristive model of amoeba learning. Phys. Rev. E, 80(2):021926, Aug 2009. doi: 10.1103/PhysRevE.80.021926.Google ScholarCross Ref
- R. Raina, A. Madhavan, and A. Ng. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 873--880. ACM, 2009. ISBN 978-1-60558-516-1. Google ScholarDigital Library
- K. L. Rice, T. M. Taha, and C. N. Vutsinas. Scaling analysis of a neocortex inspired cognitive model on the cray xd1. J. Supercomput., 47(1):21--43, 2009. ISSN 0920-8542. doi: http://dx.doi.org/10.1007/s11227-008-0195-z. Google ScholarDigital Library
- D. Ringach. Haphazard wiring of simple receptive fields and orientation columns in visual cortex. J. Neurophysiol., 92(1):468--476, Jul 2004.Google ScholarCross Ref
- U. Rokni, A. Richardson, E. Bizzi, and H. Seung. Motor learning with unstable neural representations. Neuron, 64:653--666, 2007.Google ScholarCross Ref
- J. Schemmel, J. Fieres, and K. Meier. Wafer-scale integration of analog neural networks. In Neural Networks, 2008. IJCNN 2008. (IEEE World Congress on Computational Intelligence). IEEE International Joint Conference on, pages 431--438, 2008. doi: 10.1109/IJCNN.2008.4633828.Google ScholarCross Ref
- T. Serre, A. Oliva, and T. Poggio. A feedforward architecture accounts for rapid categorization. Proc. Natl. Acad. Sci. USA, 104(15):6424--6429, Apr 2007. doi: 10.1073/pnas.0700622104.Google ScholarCross Ref
- T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio. Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell., 29(3):411--426, Mar 2007. doi: 10.1109/TPAMI.2007.56. Google ScholarDigital Library
- R. Vogels and G. Orban. How well do response changes of striate neurons signal difference in orientation: a study in the discriminating monkey. Journal of Neuroscience, 10(11):3543--3558, 1990.Google ScholarCross Ref
- H. Wersing and E. Korner. Learning optimized features for hierarchical models of invariant object recognition. Neural Computation, 15: 1559--1588, 2003. Google ScholarDigital Library
Index Terms
- A case for neuromorphic ISAs
Recommendations
A case for neuromorphic ISAs
ASPLOS XVI: Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systemsThe desire to create novel computing systems, paired with recent advances in neuroscientific understanding of the brain, has led researchers to develop neuromorphic architectures that emulate the brain. To date, such models are developed, trained, and ...
A case for neuromorphic ISAs
ASPLOS '11The desire to create novel computing systems, paired with recent advances in neuroscientific understanding of the brain, has led researchers to develop neuromorphic architectures that emulate the brain. To date, such models are developed, trained, and ...
An analog astrocyte---neuron interaction circuit for neuromorphic applications
Recent neurophysiologic findings have shown that astrocytes (the most abundant type of glial cells) are active partners in neural information processing and regulate the synaptic transmission dynamically. Motivated by these findings, in the present ...
Comments