Abstract
Knowledge in rural communities is largely created, preserved, and is transferred verbally, and it is limited. This information is valuable to these communities, and managing and making it available digitally with state-of-the-art approaches enriches awareness and collective knowledge of people of these communities. The large amounts of data and information produced on the Internet are inaccessible to the population in these rural communities due to factors like lack of infrastructure, connectivity, and limited literacy. Knowledge internal to rural communities is also not conserved and made available in any global Big Data information systems. Artificial Intelligence (AI) technologies such as Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) provide substantial assistance when vast quantities of data, like Big Data, are available to build solutions. In the case of low-resource languages like Kannada and rural colloquial dialects, publicly available corpora are significantly less. Building state-of-the-art AI solutions is challenging in this context, and we address this problem in this work. Knowledge management in rural communities requires a low-cost and efficient approach that social workers can use. This paper proposes an architecture for oral knowledge management for rural communities speaking colloquial Kannada. The proposed architecture has an interface for oral knowledge retrieval using text processing on transcripts generated from the smallest state-of-the-art ASR model. We propose three interfaces to search for content: an n-gram based fuzzy search to search for texts in audios, the most frequent entities search based on the Kannada Named Entity Recognition (NER) model, and question-answering with Large Language Model (LLM) using a community knowledge vector store.
This work was supported by the Mphasis F1 Foundation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Demo app: http://103.156.19.244:33035/,
username: guest, password: guest123.
References
Aksënova, A., et al.: Accented speech recognition: benchmarking, pre-training, and diverse data (2022)
Baevski, A., Hsu, W.N., Conneau, A., Auli, M.: Unsupervised speech recognition. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 27826–27839. Curran Associates, Inc. (2021)
Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: Wav2vec 2.0: a framework for self-supervised learning of speech representations. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460. Curran Associates, Inc. (2020)
Cohn, D., Ghahramani, Z., Jordan, M.: Active learning with statistical models. In: Advances in Neural Information Processing Systems, vol. 7 (1994)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018). http://arxiv.org/abs/1810.04805
Duarte, F.: Amount of data created daily (2023). https://explodingtopics.com/blog/data-generated-per-day. Accessed 08 Oct 2023
Goodmann, E., Matienzo, M.A., VanCour, S., Dries, W.V.: Building the national radio recordings database: a big data approach to documenting audio heritage. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 3080–3086 (2019). https://doi.org/10.1109/BigData47090.2019.9006520
Kakwani, D., et al.: IndicNLPSuite: monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In: Findings of EMNLP (2020)
Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet Physics Doklady, vol. 10, pp. 707–710. Soviet Union (1966)
Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks (2021)
Mhaske, A., et al.: Naamapadam: a large-scale named entity annotated data for Indic languages. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10441–10456. Association for Computational Linguistics, Toronto (2023). https://doi.org/10.18653/v1/2023.acl-long.582, https://aclanthology.org/2023.acl-long.582
Najafabadi, M.M., Villanustre, F., Khoshgoftaar, T.M., Seliya, N., Wald, R., Muharemagic, E.: Deep learning applications and challenges in big data analytics. J. Big Data 2(1), 1–21 (2015)
Pan, X., Zhang, B., May, J., Nothman, J., Knight, K., Ji, H.: Cross-lingual name tagging and linking for 282 languages. In: Annual Meeting of the Association for Computational Linguistics (2017)
Pratap, V., et al.: Scaling speech technology to 1,000+ languages. arXiv (2023)
Radford, A., Kim, J.W., Xu, T., Brockman, G., Mcleavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 202, pp. 28492–28518. PMLR (2023). https://proceedings.mlr.press/v202/radford23a.html
Verma, J.P., Agrawal, S., Patel, B., Patel, A.: Big data analytics: challenges and applications for text, audio, video, and social media data. Int. J. Soft Comput. Artif. Intell. Appl. (IJSCAI) 5(1), 41–51 (2016)
Vryzas, N., Tsipas, N., Dimoulas, C.: Web radio automation for audio stream management in the era of big data. Information 11(4) (2020). https://doi.org/10.3390/info11040205, https://www.mdpi.com/2078-2489/11/4/205
Wang, P., Wang, X., Liu, X.: Selection of audio learning resources based on big data. Int. J. Emerg. Technol. Learn. (Online) 17(6), 23 (2022)
Zhang, J., et al.: Managing and analysing big audio data for environmental monitoring. In: 2013 IEEE 16th International Conference on Computational Science and Engineering, pp. 997–1004 (2013). https://doi.org/10.1109/CSE.2013.146
Zhang, Q., Yang, L.T., Chen, Z., Li, P.: A survey on deep learning for big data. Inf. Fusion 42, 146–157 (2018). https://doi.org/10.1016/j.inffus.2017.10.006, https://www.sciencedirect.com/science/article/pii/S1566253517305328
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Aparna, M., Srivatsa, S., Sai Madhavan, G., Dinesh, T.B., Srinivasa, S. (2024). AI-Based Assistance for Management of Oral Community Knowledge in Low-Resource and Colloquial Kannada Language. In: Sachdeva, S., Watanobe, Y. (eds) Big Data Analytics in Astronomy, Science, and Engineering. BDA 2023. Lecture Notes in Computer Science, vol 14516. Springer, Cham. https://doi.org/10.1007/978-3-031-58502-9_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-58502-9_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-58501-2
Online ISBN: 978-3-031-58502-9
eBook Packages: Computer ScienceComputer Science (R0)