Skip to main content
SpringerLink
Log in

TensSent: a tensor based sentimental word embedding method

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The representation of words as vectors, conventionally known as word embeddings, has drawn considerable attention in recent years as feature learning techniques for natural language processing. The majority of these methods operate solely on the semantic and the syntactic aspects of a text, remaining oblivious to sentimental information. However, as numerous words with opposite polarities may appear in similar contexts, such as “interesting” and “boring”, the exclusive use of context-oriented information lacks the required particulars for generating word embeddings as features in sentiment analysis. Along this thread, the present study proposes two novel unsupervised models to integrating word polarity information and word co-occurrences as more tailored features for sentiment analysis. Word polarity and co-occurrence come together in the form of a tensor and tensor factorization is employed for generating the word embeddings. The experimental results on IMDB and SemEval-task2 datasets demonstrate the relatively higher performance of the proposed method compared to baseline approaches on the tasks of document-level sentiment analysis by 4.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Latent Semantic Indexing

  2. https://archive.org/details/viwiki-20170301

  3. https://www.cs.york.ac.uk/semeval-2013/task2/

References

  1. Deerwester S, Dumias ST, Furmas GW, Lander TK, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391–407. [Online]. Available: https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/(SICI)1097-4571(199009)41:6%3C391::AID-ASI1%3E3.0.CO;2-9

    Article  Google Scholar 

  2. Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  3. Mikolov T, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their copositionality. In: In Advances in neural information processing systems, pp 3111–3119

  4. Bojanowski P, Grave E, Joulin A, Mikolov T (2016) Enriching Word Vectors with Subword Information. Trans Assoc Comput Linguist 5:135–146. [Online]. Available: https://www.aclweb.org/anthology/Q17-1010.pdf

    Article  Google Scholar 

  5. Huang EH, Socher R, Manning CD, Ng AY (2012) Improve Word Representation via Global Context and Multiple Word prototypes. In: Proceedings of the 50th annual meeting of the association for computational linguistics no. July. Jeju Islan: Korea: Association for Computational Linguistics, pp 873–882

  6. Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language, Vol. 1 (Long Papers) Technologies. New Orleans, Louisiana: Association for Computational Linguistics, pp 2227–2237. [Online]. Available: https://www.aclweb.org/anthology/N18-1202

  7. Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies. Minneapolis, Minnesota: Association for Computational Linguistics, pp 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423

  8. Zhu L, Li W, Shi Y, Guo K (2020) SentiVec: Learning sentiment-context vector via kernel optimization function for sentiment analysis. IEEE Transactions on Neural Networks and Learning Systems. pp 1–12

  9. Hayashi T, Fujita H (2019) Word embeddings-based sentence-level sentiment analysis considering word importance. Acta Polytechnica Hungarica 16(7):7–24

    Google Scholar 

  10. Maas AL, Daly RE, Pham PT, Huang D, Ng AY, Potts C (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the association for computational linguistics (ACL-2011), no. February, pp 142–150. [Online]. Available: https://www.aclweb.org/anthology/P11-1015

  11. Hofmann T (1999) Probabilistic latent semantic analysis. In: Thomas H (ed) Probabilistic latent semantic analysis. Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publisher Inc. 289–296

  12. Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  13. Dhillon PS, Foster D, Ungar L (2011) Multi-View Learning of word embeddings via CCA. In: Proceedings of the 24th international conference on neural information processing (NIPS’11), no. December. [Online]. Available: https://papers.nips.cc/paper/4193-multi-view-learning-of-word-embeddings-via-cca

  14. Dhillon PS, Rodu J, Foster DP, Ungar LH (2012) Two step CCA: A new spectral method for estimating vector models of words. In: Proceedings of the 29th international conference on machine learning, ICML 2012, vol 2, Edinburgh, pp. 1551–1558. [Online]. Available: https://icml.cc/Conferences/2012/papers/763.pdf

  15. Dhillon PS, Foster DP, Ungar LH (2015) Eigenwords: Spectral word embeddings. J Mach Learn Res 16:3035–3078

    MathSciNet  MATH  Google Scholar 

  16. Faruqui M, Dyer C (2014) Improving vector space word representations using multilingual correlation. In: 14th Conference of the european chapter of the association for computational linguistics 2014, EACL 2014, pp 462–471

  17. Lebret R, Collobert R (2014) Word embeddings through Hellinger PCA. In: 14th Conference of the european chapter of the association for computational linguistics 2014, EACL 2014, pp 482–490

  18. Lebret R, Collobert R (2015) Rehabilitation of count-based models for word vector representations. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 9041, pp 417–429

  19. Levy O, Goldberg Y (2014) Neural word embedding as implicit matrix factorization. In: Proceedings of the 27th international conference on neural information processing systems, vol 2, no. December, pp 2177–2185. [Online]. Available: https://dl.acm.org/doi/10.5555/2969033.2969070

  20. Li J, Jurafsky D (2015) Do multi-sense embeddings improve natural language understanding?. In: Conference proceedings - EMNLP 2015: conference on empirical methods in natural language processing no. September pp 1722–1732

  21. Sen P, Ganguly D, Jones G (2019) Word-Node2Vec: improving word embedding with document-level non-local word co-occurrences. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics human language technologies, Volume 1 (Long and Short Papers), pp 1041–1051. [Online]. Available: https://www.aclweb.org/anthology/N19-1109

  22. Reisinger J, Mooney RJ (2010) Multi-prototype vector-space models of word meaning. In: NAACL HLT 2010 - human language technologies: the 2010 annual conference of the north american chapter of the association for computational linguistics, proceedings of the main conference, no June, pp 109–117

  23. Wu Z, Giles CL (2015) Sense-aware semantic analysis: A multi-prototype word representation model using wikipedia. In: Proceedings of the national conference on artificial intelligence, vol 3, pp 2188–2194

  24. Neelakantan A, Shankar J, Passos A, McCallum A (2014) Efficient non-parametric estimation of multiple embeddings perword in vector space. In: EMNLP 2014 - 2014 Conference on empirical methods in natural language processing, proceedings of the conference, pp 1059–1069

  25. Chen X, Liu Z, Sun M (2014) A unified model for word sense representation and disambiguation. In: EMNLP 2014 - 2014 Conference on empirical methods in natural language processing, proceedings of the conference, pp 1025–1035

  26. Tian F, Dai H, Bian J, Gao B, Zhang R, Chen E, Liu TY (2014) A probabilistic model for learning multi-prototypeword embeddings. In: COLING 2014 - 25th International conference on computational linguistics, proceedings of COLING 2014: technical papers, pp 151–160

  27. Camacho-Collados J, Pilehvar MT (2018) From word to sense embeddings: A survey on vector representations of meaning. J Artif Intell Res 63:743–788. [Online]. Available: https://dl.acm.org/doi/10.1613/jair.1.11259

    Article  MathSciNet  Google Scholar 

  28. Flekova L, Gurevych I (2016) Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In: 54th Annual meeting of the association for computational linguistics, ACL 2016 - Long Papers, vol 4, pp 2029–2041

  29. Iacobacci I, Pilehvar MT, Navigli R (2015) SENSEMBED: Learning sense embeddings forword and relational similarity. In: ACL-IJCNLP 2015 - 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing of the asian federation of natural language processing, proceedings of the conference, vol 1, no 1, pp 95–105

  30. Flati T, Navigli R (2014) Three birds (in the LLOD cloud) with one stone: BabelNet, Babelfy and the Wikipedia Bitaxonomy. In: Proceedings of SEMANTiCS. [Online]. Available: http://ceur-ws.org/Vol-1224/paper3.pdf

  31. Nguyen DQ, Nguyen DQ, Modi A, Thater S, Pinkal M (2017) A mixture model for learning multi-sense word embeddings. In: *SEM 2017 - 6th Joint conference on lexical and computational semantics, proceedings, vol 2, pp 121–127

  32. Ruas T, Grosky W, Aizawa A (2019) Multi-sense embeddings through a word sense disambiguation process. Expert Syst Appl 136:288–303. [Online]. Available: https://doi.org/10.1016/j.eswa.2019.06.026

    Article  Google Scholar 

  33. Jain S, Bodapati SB, Nallapati R, Anandkumar A (2019) Multi sense embeddings from topic models. [Online]. Available: arXiv:1909.07746

  34. Athiwaratkun B, Wilson G, Anandkumar A (2018) Probabilistic fasttext for multi-sense word embeddings. In: ACL 2018 - 56th Annual meeting of the association for computational linguistics, proceedings of the conference (long papers), pp 1–11

  35. Salama RA, Youssef A, Fahmy A (2018) Morphological word embedding for arabic. [Online]. Available: https://doi.org/10.1016/j.procs.2018.10.463, vol 142, pp 83–93

  36. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV (2019) XLNet: generalized autoregressive pretraining for language understanding. In: Advances in neural information processing systems (NIPS 2019), pre-proceedings. Curran Associates, pp 5754–5764. [Online]. Available: http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding.pdf

  37. Vaswani A, Brain G, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, no. Nips, pp 5998–6008. [Online]. Available: http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf

  38. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) RoBERTa: a robustly optimized bert pretraining approach. 1. [Online]. Available: arXiv:1907.11692

  39. Tang D, Wei F, Yang N, Zhou M, Liu T, Qin B (2014) Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (long papers), vol 1. Baltimore, Maryland: Association for Computational Linguistics, pp 1555–1565. [Online]. Available: https://www.aclweb.org/anthology/P14-1146

  40. Collobert R, Weston J (2008) A unified architecture for natural language processing. 160–167

  41. Zhang Z, Lan M (2016) Learning sentiment-inherent word embedding for word-level and sentence-level sentiment analysis. In: Proceedings of 2015 international conference on asian language processing, IALP 2015, vol 1, pp 94–97

  42. Yu L-C, Wang J, Lai KR, Zhang X (2017) Refining word embeddings for sentiment analysis. In: Proceedings of the 2017 conference on empirical methods in natural language processing Copenhagen, pp 534–539. [Online]. Available: https://www.aclweb.org/anthology/D17-1056

  43. Tang D, Wei F, Qin B, Yang N, Liu T, Zhou M (2016) Sentiment Embeddings with Applications to Sentiment Analysis. IEEE Trans Knowl Data Eng 28(2):496–509

    Article  Google Scholar 

  44. Fu P, Lin Z, Yuan F, Wang W, Meng D (2018) Learning sentiment-specific word embedding via global sentiment representation. In: 32nd AAAI Conference on artificial intelligence, AAAI 2018, pp 4808–4815

  45. Mao X, Chang S, Shi J, Li F, Shi R (2019) Sentiment-aware word embedding for emotion classification. Appl Sci (Switzerland) 9:7

    Google Scholar 

  46. Vo AD, Nguyen QP, Ock CY (2020) Semantic and syntactic analysis in learning representation based on a sentiment analysis model. Appl Intell 50(3):663–680

    Article  Google Scholar 

  47. Rezaeinia SM, Rahmani R, Ghodsi A, Veisi H (2019) Sentiment analysis based on improved pre-trained word embeddings. Expert Syst Appl 117:139–147. [Online]. Available: https://doi.org/10.1016/j.eswa.2018.08.044

    Article  Google Scholar 

  48. Zhang Y, Song D, Zhang P, Li X, Wang P (2019) A quantum-inspired sentiment representation model for twitter sentiment analysis. Appl Intell 49(8):3093–3108

    Article  Google Scholar 

  49. Toshitaka H, Fujita H (2018) Sentence-level sentiment analysis using feature vectors from word embeddings. In: SoMeT. [Online]. Available: http://ebooks.iospress.nl/volumearticle/49982, vol 303, pp 749–758

  50. Naseem U, Razzak I, Musial K, Imran M (2020) Transformer based deep intelligent contextual embedding for twitter sentiment analysis. Future Gener Comput Syst 113:58–69. [Online]. Available: https://doi.org/10.1016/j.future.2020.06.050

    Article  Google Scholar 

  51. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500

    Article  MathSciNet  Google Scholar 

  52. Baccianella S, Esuli A, Sebastiani F (2010) SENTIWORDNET 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010. vol 0, pp 2200–2204

  53. Nielsen FÅ (2011) A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In: Proceedings of the ESWC2011 workshop on ‘making sense of microposts’: big things come in small packages. vol 718, pp 93–98

  54. Hamilton WL, Clark K, Leskovec J, Jurafsky D (2016) Inducing domain-specific sentiment lexicons from unlabeled corpora. In: EMNLP 2016 - Conference on empirical methods in natural language processing, proceedings, pp 595–605

  55. Liu W, Zhou P, Wang Z, Zhao Z, Deng H, JU Q (2020) FastBERT: a Self-distilling BERT with adaptive inference time. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 6035–6044. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.537

Download references

Author information

Authors and Affiliations

  1. Amirkabir University of Technology, Tehran, Iran

    Zahra Rahimi & Mohammad Mehdi Homayounpour

Authors
  1. Zahra Rahimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Mehdi Homayounpour
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Mehdi Homayounpour.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors wish to express their thanks for the financial support of Iran National Science foundation (INSF), Project No 97009308.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahimi, Z., Homayounpour, M.M. TensSent: a tensor based sentimental word embedding method. Appl Intell 51, 6056–6071 (2021). https://doi.org/10.1007/s10489-020-02163-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-02163-8

Keywords

Search

Navigation