Skip to main content
SpringerLink
Log in

Contextualized Embeddings from Transformers for Sentiment Analysis on Code-Mixed Hinglish Data: An Expanded Approach with Explainable Artificial Intelligence

  • Conference paper
  • First Online:
Speech and Language Technologies for Low-Resource Languages (SPELLL 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1802))

Abstract

Transformer-based models have gained traction for giving breakthrough performance on various Natural Language Processing (NLP) tasks in recent years. A number of studies have been conducted to understand the type of information learned by the model and its performance on different tasks. YouTube comments can serve as a rich source for multilingual data, which can be used to train state-of-the-art models. In this study, two transformer-based models, multilingual Bidirectional Encoder Representations from Transformers (mBERT) and RoBERTa, are fine-tuned and evaluated on code-mixed ‘Hinglish’ data. The representations learned by the intermediate layers of the models are also studied by using them as features for machine learning classifiers. The results show a significant improvement compared to the baseline for both datasets using the feature-based method, with the highest accuracy of 92.73% for Kabita Kitchen’s channel and 87.42% for Nisha Madhulika’s channel. Explanations of the model predictions using the Local Interpretable Model-Agnostic Explanations (LIME) technique show that the model is using significant features for classification and can be trusted.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://huggingface.co/bert-base-multilingual-uncased.

  2. 2.

    https://huggingface.co/roberta-base.

  3. 3.

    https://github.com/marcotcr/lime.

References

  1. YouTube India Statistics. https://www.statista.com/statistics/280685/number-of-monthly-unique-youtube-users/. Accessed 26 July 2022

  2. Arora, G.: iNLTK: natural language toolkit for Indic languages. arXiv preprint arXiv:2009.12534 (2020)

  3. Jain, K., Deshpande, A., Shridhar, K., Laumann, F., Dash, A.: Indic-transformers: an analysis of transformer language models for Indian languages. arXiv preprint arXiv:2011.02323 (2020)

  4. Kaur, G., Kaushik, A., Sharma, S.: Cooking is creating emotion: a study on Hinglish sentiments of YouTube cookery channels using semi-supervised approach. Big Data Cogn. Comput. 3(3), 37 (2019)

    Article  Google Scholar 

  5. Donthula, S.K., Kaushik, A.: Man is what he eats: a research on Hinglish sentiments of YouTube cookery channels using deep learning. Int. J. Recent Technol. Eng. (IJRTE) 8(2S11), 930–937 (2019)

    Google Scholar 

  6. Yadav, S., Kaushik, A., Sharma, S.: Cooking well, with love, is an art: transformers on YouTube Hinglish data. In: 2021 International Conference on Computational Performance Evaluation (ComPE), pp. 836–841. IEEE (2021)

    Google Scholar 

  7. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  8. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  9. Polignano, M., Basile, P., De Gemmis, M., Semeraro, G., Basile, V.: ALBERTO: Italian BERT language understanding model for NLP challenging tasks based on tweets. In: 6th Italian Conference on Computational Linguistics, CLiC-it 2019, vol. 2481, pp. 1–6. CEUR (2019)

    Google Scholar 

  10. de Vries, W., van Cranenburgh, A., Bisazza, A., Caselli, T., van Noord, G., Nissim, M.: BERTje: a Dutch BERT model. arXiv preprint arXiv:1912.09582 (2019)

  11. Lee, S., Jang, H., Baik, Y., Park, S., Shin, H.: KR-BERT: a small-scale Korean-specific language model. arXiv preprint arXiv:2008.03979 (2020)

  12. Martin, L., et al.: CamemBERT: a tasty French language model. arXiv preprint arXiv:1911.03894 (2019)

  13. Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018)

    Article  Google Scholar 

  14. Ribeiro, M.T., Singh, S., Guestrin, C.: “why should i trust you?” Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

    Google Scholar 

  15. Liu, Y., et al.: RoBERTA: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  16. Kabita’s Kitchen Cookery Channel. https://www.Youtube.com/ channel/CChqsCRFePrP2X897iQkyAA. Accessed 10 July 2022

  17. Nisha Madhulika’s Cookery Channel. https://www.Youtube.com/user/NishaMadhulika. Accessed 10 July 2022

  18. Kaushik, A., Kaur, G.: YouTube cookery channels viewers comments in Hinglish, May 2019. https://doi.org/10.5281/zenodo.2841848

  19. Mathew, B., Saha, P., Yimam, S.M., Biemann, C., Goyal, P., Mukherjee, A.: HateXplain: a benchmark dataset for explainable hate speech detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 17, pp. 14867–14875 (2021)

    Google Scholar 

  20. Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in BERTology: what we know about how BERT works. Trans. Assoc. Comput. Linguist. 8, 842–866 (2020)

    Article  Google Scholar 

  21. Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. arXiv preprint cs/0306050 (2003)

    Google Scholar 

  22. Yang, J., Zhao, H.: Deepening hidden representations from pre-trained language models. arXiv preprint arXiv:1911.01940 (2019)

  23. Su, T.C., Cheng, H.C.: SesameBERT: attention for anywhere. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 363–369. IEEE (2020)

    Google Scholar 

  24. McCoy, R.T., Pavlick, E., Linzen, T.: Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007 (2019)

  25. Kazhuparambil, S., Kaushik, A.: Classification of Malayalam-English mix-code comments using current state of art. In: 2020 IEEE International Conference for Innovation in Technology (INOCON), pp. 1–6. IEEE (2020)

    Google Scholar 

  26. Patwa, P., et al.: SemEval-2020 task 9: overview of sentiment analysis of code-mixed tweets. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 774–790 (2020)

    Google Scholar 

  27. Bhange, M., Kasliwal, N.: HinglishNLP at SemEval-2020 task 9: fine-tuned language models for Hinglish sentiment detection. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 934–939 (2020)

    Google Scholar 

  28. Szczepański, M., Pawlicki, M., Kozik, R., Choraś, M.: New explainability method for BERT-based model in fake news detection. Sci. Rep. 11(1), 1–13 (2021)

    Article  Google Scholar 

  29. BERT Word Embeddings Tutorial. https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/. Accessed 13 July 2022

  30. A Visual Guide to Using BERT for the First Time. https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/. Accessed 13 July 2022

  31. Wolf, T., et al.: HuggingFace’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)

  32. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019). https://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

  33. Scikit Learn: Machine Learning in Python. https://scikit-learn.org/stable/. Accessed 25 Oct 2022

  34. Shah, S.R., Kaushik, A.: Sentiment analysis on Indian indigenous languages: a review on multilingual opinion mining. arXiv preprint arXiv:1911.12848 (2019)

  35. Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning, vol. 4, no. 4. Springer, New York (2006)

    Google Scholar 

  36. Hand, D.J., Till, R.J.: A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach. Learn. 45(2), 171–186 (2001)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dublin Business School, Dublin, D02 WC04, Ireland

    Sargam Yadav

  2. Dundalk Institute of Technology, Dundalk, A91 K584, Ireland

    Abhishek Kaushik

Authors
  1. Sargam Yadav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abhishek Kaushik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sargam Yadav .

Editor information

Editors and Affiliations

  1. National Institute of Technology Karnataka, Mangalore, India

    Anand Kumar M

  2. National University of Ireland, Galway, Ireland

    Bharathi Raja Chakravarthi

  3. Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, India

    Bharathi B

  4. National University of Ireland, Galway, Ireland

    Colm O’Riordan

  5. Indian Institute of Technology Madras, Chennai, India

    Hema Murthy

  6. Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, India

    Thenmozhi Durairaj

  7. University of Hildesheim, Hildesheim, Germany

    Thomas Mandl

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yadav, S., Kaushik, A. (2023). Contextualized Embeddings from Transformers for Sentiment Analysis on Code-Mixed Hinglish Data: An Expanded Approach with Explainable Artificial Intelligence. In: M, A.K., et al. Speech and Language Technologies for Low-Resource Languages . SPELLL 2022. Communications in Computer and Information Science, vol 1802. Springer, Cham. https://doi.org/10.1007/978-3-031-33231-9_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-33231-9_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-33230-2

  • Online ISBN: 978-3-031-33231-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation