Abstract
Event-based cameras have garnered growing interest in computer vision due to the advantages of sparsed spatio-temporal representation. Spiking neural networks (SNNs), as representative brain-inspired computing models, are inherently suitable for event-driven processing. However, event-based SNNs still have shortcomings in using multiple feature extraction methods, such as the loss of feature information. In this work, we propose an event-based hierarchical model using feature fusion and SNNs for object recognition. In the proposed model, input event stream is adaptively sliced into segment stream for the subsequent feature extraction and SNNs with Tempotron rule. And the model utilizes feature mapping to realize the fusion of the orientation features extracted by Gabor filter and spatio-temporal correlation features extracted by the clustering algorithm considering the surrounding past events within the time window. The experiments conducted on several event-based datasets (i.e., N-MNIST, MNIST-DVS, DVS128Gesture and DailyAction-DVS) show superior performance of the proposed model and the ablation study demonstrates the effectiveness of feature fusion for object recognition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Delbrück, T., Linares-Barranco, B., Culurciello, E., Posch, C.: Activity-driven, event-based vision sensors. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp. 2426-2429. IEEE (2010)
Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nat. Neurosci. 2(11), 1019–1025 (1999)
Zhao, B., Ding, R., Chen, S., Linares-Barranco, B., Tang, H.: Feedforward categorization on AER motion events using cortex-like features in a spiking neural network. IEEE Trans. Neural Netw. Learn. Syst. 26(9), 1963–1978 (2014)
Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N., Benosman, R.: Hfirst: a temporal approach to object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(10), 2028–2040 (2015)
Xiao, R., Tang, H., Ma, Y., Yan, R., Orchard, G.: An event-driven categorization model for AER image sensors using multispike encoding and learning. IEEE Trans. Neural Netw. Learn. Syst. 31(9), 3649–3657 (2019)
Tang, T., Jiang, R., Yan, R., Tang, H.: An event-driven object recognition model using activated connected domain detection. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 3049-3056. IEEE (2020)
Liu, Q., Ruan, H., Xing, D., Tang, H., Pan, G.: Effective AER object classification using segmented probability-maximization learning in spiking neural networks. Proc. AAAI Conf. Artif. Intell. 34, 1308–1315 (2020)
Lagorce, X., Orchard, G., Galluppi, F., Shi, B.E., Benosman, R.B.: Hots: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1346–1359 (2016)
Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., Benosman, R.: Hats: histograms of averaged time surfaces for robust event-based object classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1731-1740 (2018)
Nan, Y., Xiao, R., Gao, S., Yan, R.: An event-based hierarchy model for object recognition. In: 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2342-2347. IEEE (2019)
Liu, Q., Pan, G., Ruan, H., Xing, D., Xu, Q., Tang, H.: Unsupervised AER object recognition based on multiscale spatio-temporal features and spiking neurons. IEEE Trans. Neural Netw. Learn. Syst. 31(12), 5300–5311 (2020)
Liu, Q., Xing, D., Tang, H., Ma, D., Pan, G.: Event-based action recognition using motion information and spiking neural networks. In: IJCAI, pp. 1743–1749 (2021)
Gütig, R., Sompolinsky, H.: The tempotron: a neuron that learns spike timing-based decisions. Nat. Neurosci. 9(3), 420–428 (2006)
Orchard, G., Jayawant, A., Cohen, G.K., Thakor, N.: Converting static image datasets to spiking neuromorphic datasets using saccades. Front. Neurosci. 9, 437 (2015)
Serrano-Gotarredona, T., Linares-Barranco, B.: Poker-DVS and mnist-DVS their history, how they were made, and other details. Front. Neurosci. 9, 481 (2015)
Amir, A., et al.: A low power, fully event-based gesture recognition system. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7243–7252 (2017)
Xing, Y., Di Caterina, G., Soraghan, J.: A new spiking convolutional recurrent neural network (scrnn) with applications to event-based hand gesture recognition. Front. Neurosci. 14, 590164 (2020)
Shrestha, S.B., Orchard, G.: Slayer: spike layer error reassignment in time. Adv. Neural Inf. Process. Syst. 31 (2018)
He, W., et al.: Comparing SNNs and RNNs on neuromorphic vision datasets: similarities and differences. Neural Netw. 132, 108–120 (2020)
Acknowledgment
This work was supported by the National Natural Science Foundation of China sNSAF under Grant No. U2030204 and No. 62276235, and by the Leading Innovation Team of the Zhejiang Province under Grant 2021R01002.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Su, M., Yang, P., Jiang, R., Yan, R. (2024). Event-Based Object Recognition Using Feature Fusion and Spiking Neural Networks. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1961. Springer, Singapore. https://doi.org/10.1007/978-981-99-8126-7_37
Download citation
DOI: https://doi.org/10.1007/978-981-99-8126-7_37
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8125-0
Online ISBN: 978-981-99-8126-7
eBook Packages: Computer ScienceComputer Science (R0)