Abstract
This paper presents an approach for substantial reduction of the training and operating phases of Self-Organizing Maps in tasks of 2-D projection of multi-dimensional symbolic data for natural language processing such as language classification, topic extraction, and ontology development. The conventional approach for this type of problem is to use n-gram statistics as a fixed size representation for input of Self-Organizing Maps. The performance bottleneck with n-gram statistics is that the size of representation and as a result the computation time of Self-Organizing Maps grows exponentially with the size of n-grams. The presented approach is based on distributed representations of structured data using principles of hyperdimensional computing. The experiments performed on the European languages recognition task demonstrate that Self-Organizing Maps trained with distributed representations require less computations than the conventional n-gram statistics while well preserving the overall performance of Self-Organizing Maps.
Keywords
This work was supported by the Swedish Research Council (VR, grant 2015-04677) and the Swedish Foundation for International Cooperation in Research and Higher Education (grant IB2018-7482) for its Initiation Grant for Internationalisation, which allowed conducting the study.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Available online at http://www.statmt.org/europarl/.
References
Alahakoon, D., Halgamuge, S., Srinivasan, B.: Dynamic self-organizing maps with controlled growth for knowledge discovery. IEEE Trans. Neural Netw. 11(3), 601–614 (2000)
Appiah, K., Hunter, A., Dickinson, P., Meng, H.: Implementation and applications of tri-state self-organizing maps on FPGA. IEEE Trans. Circ. Syst. Video Technol. 22(8), 1150–1160 (2012)
Bandaragoda, T.R., De Silva, D., Alahakoon, D.: Automatic event detection in microblogs using incremental machine learning. J. Assoc. Inform. Sci. Technol. 68(10), 2394–2411 (2017)
Bishop, C.M., Svensén, M., Williams, C.K.: GTM: the generative topographic mapping. Neural Comput. 10(1), 215–234 (1998)
De Silva, D., Alahakoon, D.: Incremental knowledge acquisition and self learning from text. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2010)
De Silva, D., Alahakoon, D., Yu, X.: A data fusion technique for smart home energy management and analysis. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 4594–4600 (2016)
De Silva, D., et al.: Machine learning to support social media empowered patients in cancer care and cancer treatment decisions. PLoS One 13(10), 1–10 (2018)
De Silva, D., Yu, X., Alahakoon, D., Holmes, G.: A data mining framework for electricity consumption analysis from meter data. IEEE Trans. Industr. Inf. 7(3), 399–407 (2011)
Dittenbach, M., Merkl, D., Rauber, A.: The growing hierarchical self-organizing map. In: International Joint Conference on Neural Networks (IJCNN), vol. 6, pp. 15–19 (2000)
Eliasmith, C.: How to Build a Brain. Oxford University Press, Oxford (2013)
Frady, E.P., Kleyko, D., Sommer, F.T.: A theory of sequence indexing and working memory in recurrent neural networks. Neural Comput. 30, 1449–1513 (2018)
Hazan, H., Saunders, D.J., Sanghavi, D.T., Siegelmann, H.T., Kozma, R.: Unsupervised learning with self-organizing spiking neural networks. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6 (2018)
Hinton, G., McClelland, J., Rumelhart, D.: Distributed representations. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing. Explorations in the Microstructure of Cognition - Volume 1 Foundations, pp. 77–109. MIT Press, Cambridge (1986)
Jayaratne, M., Alahakoon, D., De Silva, D., Yu, X.: Bio-inspired multisensory fusion for autonomous robots. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3090–3095 (2018)
Joshi, A., Halseth, J.T., Kanerva, P.: Language geometry using random indexing. In: de Barros, J.A., Coecke, B., Pothos, E. (eds.) QI 2016. LNCS, vol. 10106, pp. 265–274. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52289-0_21
Kanerva, P.: Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1(2), 139–159 (2009)
Kaski, S., Honkela, T., Lagus, K., Kohonen, T.: WEBSOM-self-organizing maps of document collections1. Neurocomputing 21(1–3), 101–117 (1998)
Kleyko, D., Osipov, E.: Brain-like classifier of temporal patterns. In: International Conference on Computer and Information Sciences (ICCOINS), pp. 104–113 (2014)
Kleyko, D., Osipov, E., De Silva, D., Wiklund, U., Alahakoon, D.: Integer self-organizing maps for digital hardware. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)
Kleyko, D., Osipov, E., Gayler, R.: Recognizing permuted words with vector symbolic architectures: a cambridge test for machines. Proc. Comput. Sci. 88, 169–175 (2016)
Kleyko, D., Osipov, E., Papakonstantinou, N., Vyatkin, V.: Hyperdimensional computing in industrial systems: the use-case of distributed fault isolation in a power plant. IEEE Access 6, 30766–30777 (2018)
Kleyko, D., Osipov, E., Wiklund, U.: A hyperdimensional computing framework for analysis of cardiorespiratory synchronization during paced deep breathing. IEEE Access 7, 34403–34415 (2019)
Kleyko, D., Rahimi, A., Gayler, R., Osipov, E.: Autoscaling bloom filter: controlling trade-off between true and false positives. Neural Comput. Appl. 1–10 (2019)
Kleyko, D., Rahimi, A., Rachkovskij, D., Osipov, E., Rabaey, J.: Classification and recall with binary hyperdimensional computing: trade-offs in choice of density and mapping characteristic. IEEE Trans. Neural Netw. Learn. Syst. 29(12), 5880–5898 (2018)
Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences. Springer, Heidelberg (2001). https://doi.org/10.1007/978-3-642-56927-2
Kohonen, T., Somervuo, P.: Self-organizing maps of symbol strings. Neurocomputing 21(1–3), 19–30 (1998)
Kusiak, A.: Smart manufacturing must embrace big data. Nat. News 544(7648), 23 (2017)
Nallaperuma, D., De Silva, D., Alahakoon, D., Yu, X.: Intelligent detection of driver behavior changes for effective coordination between autonomous and human driven vehicles. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3120–3125 (2018)
Nawaratne, R., Bandaragoda, T., Adikari, A., Alahakoon, D., De Silva, D., Yu, X.: Incremental knowledge acquisition and self-learning for autonomous video surveillance. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 4790–4795 (2017)
Osipov, E., Kleyko, D., Legalov, A.: Associative synthesis of finite state automata model of a controlled object with hyperdimensional computing. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3276–3281 (2017)
Plate, T.A.: Holographic Reduced Representations: Distributed Representation for Cognitive Structures. Center for the Study of Language and Information (CSLI), Stanford (2003)
Quasto, U., Richter, M., Biemann, C.: Corpus portal for search in monolingual corpora. In: Fifth International Conference on Language Resources and Evaluation (LREC), pp. 1799–1802 (2006)
Rachkovskij, D.A.: Representation and processing of structures with binary sparse distributed codes. IEEE Trans. Knowl. Data Eng. 3(2), 261–276 (2001)
Rahimi, A., et al.: High-dimensional Computing as a Nanoscalable Paradigm. IEEE Trans. Circ. Syst. I: Regul. Pap. 64(9), 2508–2521 (2017)
Rahimi, A., Kanerva, P., Benini, L., Rabaey, J.M.: Efficient biosignal processing using hyperdimensional computing: network templates for combined learning and classification of ExG signals. Proc. IEEE 107(1), 123–143 (2019)
Rahimi, A., Kanerva, P., Rabaey, J.: A robust and energy efficient classifier using brain-inspired hyperdimensional computing. In: IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 64–69 (2016)
Rasanen, O., Kakouros, S.: Modeling dependencies in multiple parallel data streams with hyperdimensional computing. IEEE Signal Process. Lett. 21(7), 899–903 (2014)
Recchia, G., Sahlgren, M., Kanerva, P., Jones, M.: Encoding sequential information in semantic space models: comparing holographic reduced representation and random permutation. Computat. Intell. Neurosci. 1–18 (2015)
Santana, A., Morais, A., Quiles, M.: An alternative approach for binary and categorical self-organizing maps. In: International Joint Conference on Neural Networks (IJCNN), pp. 2604–2610 (2017)
Shah-Hosseini, H., Safabakhsh, R.: TASOM: a new time adaptive self-organizing map. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 33(2), 271–282 (2003)
Vesanto, J., Alhoniemi, E.: Clustering of the self-organizing map. IEEE Trans. Neural Netw. 11(3), 586–600 (2000)
Zolotukhin, M., Hamalainen, T., Juvonen, A.: Online anomaly detection by using n-gram model and growing hierarchical self-organizing maps. In: International Wireless Communications and Mobile Computing Conference (IWCMC), pp. 47–52 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kleyko, D., Osipov, E., De Silva, D., Wiklund, U., Vyatkin, V., Alahakoon, D. (2019). Distributed Representation of n-gram Statistics for Boosting Self-organizing Maps with Hyperdimensional Computing. In: Bjørner, N., Virbitskaite, I., Voronkov, A. (eds) Perspectives of System Informatics. PSI 2019. Lecture Notes in Computer Science(), vol 11964. Springer, Cham. https://doi.org/10.1007/978-3-030-37487-7_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-37487-7_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37486-0
Online ISBN: 978-3-030-37487-7
eBook Packages: Computer ScienceComputer Science (R0)