Zhongguo Wang
ABSTRACT
Measuring text similarity is a key research area in natural language processing technology. In this research, we proposed a multi-attention Siamese bi-directional long short-term memory (MAS-Bi-LSTM) to calculate the semantic similarity between two Chinese texts. The novel model used Bi-LSTM as the basic framework of the Siamese network, introduced a multi-head attention mechanism to capture the key features of the text, and used the Manhattan distance to calculate the similarity. Experiments were conducted on the large-scale Chinese question matching corpus dataset. Results showed that our model can achieve higher accuracy compared with other comparable models. The F1 value of our model reached 0.8070. The contribution of this research is to use the multi-head attention mechanism to re-weight the semantic features, and explore the influence of different pre-training corpus, distance formulas and heads of multi-attention on the model.
- A. Ben Abacha and P. Zweigenbaum, "MEANS: A medical question-answering system combining NLP techniques and semantic Web technologies," Inf. Process. Manag., vol. 51, no. 5, pp. 570-594, Sep 2015, doi: 10.1016/j.ipm.2015.04.006.Google Scholar
Digital Library
- T. Nakazawa, K. Yu, D. Kawahara, and S. Kurohashi, "Example-based machine translation based on deeper NLP," in International Workshop on Spoken Language Translation (IWSLT) 2006, 2006.Google Scholar
- A. F. Smeaton, "Using NLP or NLP Resources for Information Retrieval Tasks," in Natural Language Information Retrieval, T. Strzalkowski Ed. Dordrecht: Springer Netherlands, 1999, pp. 99-111.Google Scholar
- P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, "Learning deep structured semantic models for web search using clickthrough data," presented at the Proceedings of the 22nd ACM international conference on Information & Knowledge Management, San Francisco, California, USA, 2013. [Online]. Available: https://doi.org/10.1145/2505515.2505665.Google ScholarDigital Library
- Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil, "A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval," presented at the Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, Shanghai, China, 2014. [Online]. Available: https://doi.org/10.1145/2661829.2661935.Google ScholarDigital Library
- H. Palangi , "Semantic modelling with long-short-term memory for information retrieval," arXiv preprint arXiv:1412.6629, 2014.Google Scholar
- W. Bao, W. Bao, J. Du, Y. Yang, and X. Zhao, "Attentive Siamese LSTM Network for Semantic Textual Similarity Measure," in 2018 International Conference on Asian Language Processing (IALP), Nov 2018, pp. 312-317, doi: 10.1109/IALP.2018.8629212.Google Scholar
- Z. Lin , "A structured self-attentive sentence embedding," arXiv preprint arXiv:1703.03130, 2017.Google Scholar
- T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," arXiv preprint arXiv:1301.3781, 2013.Google Scholar
- J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532-1543.Google Scholar
- A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, "Bag of tricks for efficient text classification," arXiv preprint arXiv:1607.01759, 2016.Google Scholar
- D. Guthrie, B. Allison, W. Liu, L. Guthrie, and Y. Wilks, "A closer look at skip-gram modelling," in LREC, 2006, vol. 6: Citeseer, pp. 1222-1225.Google Scholar
- T. Kenter, A. Borisov, and M. De Rijke, "Siamese cbow: Optimizing word embeddings for sentence representations," arXiv preprint arXiv:1606.04640, 2016.Google Scholar
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems, vol. 25, pp. 1097-1105, 2012.Google Scholar
- F. Gers, "Long short-term memory in recurrent neural networks," Verlag nicht ermittelbar, 2001.Google Scholar
- P. Zhou , "Attention-based bidirectional long short-term memory networks for relation classification," in Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: Short papers), 2016, pp. 207-212.Google Scholar
- J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, "Attention-based models for speech recognition," arXiv preprint arXiv:1506.07503, 2015.Google Scholar
- J. Li, "Research on text semantic similarity combined with neural network," Master's thesis, Shandong University, 2019.Google Scholar
- C. Zhao, J. Guo, Z. Yu, Y. Huang, Q. Liu, and R. Song, "Correlation Analysis of News and Cases Based on Unbalanced Siamese Network," Journal of Chinese Information Processing, vol. 34, no. 3, pp. 99-106, 2020.Google Scholar
Recommendations
Comparison between Calculation Methods for Semantic Text Similarity based on Siamese Networks
DSIT 2021: 2021 4th International Conference on Data Science and Information TechnologyIn the era of information explosion, people are eager to obtain contents that meet their own needs and interests from massive amounts of information. Therefore, how to understand the needs of Internet users correctly and effectively is one of the urgent ...
A hybrid approach of Weighted Fine-Tuned BERT extraction with deep Siamese Bi – LSTM model for semantic text similarity identification
AbstractThe conventional semantic text-similarity methods requires high amount of trained labeled data and also human interventions. Generally, it neglects the contextual-information and word-orders information resulted in data sparseness problem and ...
Building siamese attention-augmented recurrent convolutional neural networks for document similarity scoring
Highlights- We proposed a new architecture - the Siamese attention-augmented recurrent convolutional neural network (S-ARCNN).
AbstractAutomatically measuring document similarity is imperative in natural language processing, with applications ranging from recommendation to duplicate document detection. State-of-the-art approach in document similarity commonly involves ...
Comments