Abstract
Graph Neural Networks (GNNs) have achieved remarkable results in several areas of pattern recognition, since GNNs handle complex structures well. Recently, GNNs have been used to learn text representation. However, the existing GNNs-based methods ignore the structure of graphs when generating graph representations from node representations, which limits their ability to learn hierarchical representations of graphs and capture hierarchical semantics. In this paper, we propose Multi-Level Semantic Graph Pooling Network (MLSGPool), a novel hierarchical graph pooling for text representation. MLSGPool consists of two parts: the local pooling layer and the semantic interaction layer. The local pooling layer utilizes GNNs to get a score for each node and then selects nodes with higher scores to form a smaller subgraph. We treat the representation of each subgraph as a semantic level. By stacking multiple local pooling layers, we can learn the hierarchical graph structure and extract multi-level semantics. In addition, we design a multi-head attention-based semantic interaction layer to capture the interaction between selected nodes and the nodes before the local pooling layer, which addresses the difficulty of interaction between a word and its distant neighbours. We apply our proposed model to text classification tasks, and experimental results on several benchmark datasets showed that the proposed method outperformed the baseline methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Deng, Z., Sun, C., Zhong, G., Mao, Y.: Text classification with attention gated graph neural network. Cogn. Comput. 14(4), 1464–1473 (2022)
Ding, K., Wang, J., Li, J., Li, D., Liu, H.: Be more with less: Hypergraph attention networks for inductive text classification. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4927–4936 (2020)
Gao, H., Ji, S.: Graph u-nets. In: international conference on machine learning, pp. 2083–2092. PMLR (2019)
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems 30 (2017)
Huang, L., Ma, D., Li, S., Zhang, X., Wang, H.: Text level graph neural network for text classification. arXiv preprint arXiv:1910.02356 (2019)
Huang, Z., Xu, W., Yu, K.: Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
Johnson, R., Zhang, T.: Deep pyramid convolutional neural networks for text categorization. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 562–570 (2017)
Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016)
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From word embeddings to document distances. In: International Conference on Machine Learning, pp. 957–966. PMLR (2015)
Lee, J., Lee, I., Kang, J.: Self-attention graph pooling. In: International Conference on Machine learning, pp. 3734–3743. PMLR (2019)
Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 (2015)
Liu, P., Qiu, X., Huang, X.: Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 (2016)
Ma, Y., Wang, S., Aggarwal, C.C., Tang, J.: Graph convolutional networks with eigenpooling. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 723–731 (2019)
Piao, Y., Lee, S., Lee, D., Kim, S.: Sparse structure learning via graph neural networks for inductive document classification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 11165–11173 (2022)
Su, X., et al.: A comprehensive survey on community detection with deep learning. IEEE Transactions on Neural Networks and Learning Systems (2022)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
Wang, G., et al.: Joint embedding of words and labels for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2321–2331 (2018)
Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., Weinberger, K.: Simplifying graph convolutional networks. In: International Conference on Machine Learning, pp. 6861–6871. PMLR (2019)
Yao, L., Mao, C., Luo, Y.: Graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 7370–7377 (2019)
Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., Leskovec, J.: Hierarchical graph representation learning with differentiable pooling. In: Advances in Neural Information Processing Systems 31 (2018)
Yun, X.L., Zhang, Y.M., Yin, F., Liu, C.L.: Instance gnn: a learning framework for joint symbol segmentation and recognition in online handwritten diagrams. IEEE Trans. Multimedia 24, 2580–2594 (2021)
Zhang, Q., Lei, Z., Zhang, Z., Li, S.Z.: Context-aware attention network for image-text retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3536–3545 (2020)
Zhang, Y., Yu, X., Cui, Z., Wu, S., Wen, Z., Wang, L.: Every document owns its structure: Inductive text classification via graph neural networks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 334–339 (2020)
Acknowledgment
This work was supported by National Key Research and Development Project (No.2020AAA0106200), the National Nature Science Foundation of China under Grants (No.61936005, 61872424, 61872199), and the Natural Science Foundation of Jiangsu Province (Grants No. BK20200037 and BK20210595).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Fu, T., Bao, BK., Shao, X. (2023). Multi-level Semantic Extraction Using Graph Pooling Network for Text Representation. In: Lu, H., et al. Image and Graphics . ICIG 2023. Lecture Notes in Computer Science, vol 14358. Springer, Cham. https://doi.org/10.1007/978-3-031-46314-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-46314-3_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46313-6
Online ISBN: 978-3-031-46314-3
eBook Packages: Computer ScienceComputer Science (R0)