Skip to main content
SpringerLink
Log in

Generating relation-specific weights for ConvKB using a HyperNetwork architecture

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Link prediction on knowledge graphs has become increasingly important in various fields, including recommender systems, question answering, and social networks. To address this challenge, ConvKB, a state-of-the-art model in deep learning approaches, has been proposed. However, ConvKB is limited in its ability to exploit general information about relations because it uses the same filters for all relations. Additionally, ConvKB's design, consisting of a convolution layer and a linear layer, may not provide enough parameters to store necessary information during feature learning. To address these limitations, this paper proposes the ConvHyper model, which combines the ConvKB model with a HyperNetwork architecture to create relation-specific weights for convolutional neural network layers. Specifically, the embedding of relations is linearly combined through a HyperNetwork to generate weights for the base neural network. This HyperNetwork architecture helps reduce the complexity of the search space to identify optimal weights and create a relation-specific weight structure for neural network layers. Experimental results show that ConvHyper significantly improves on all metrics in four well-known datasets. ConvHyper improves on H@1 up to 5.5% and achieves better results on other metrics, ranging from 0.8% to 1%. We also compared ConvHyper with other CNN-based models and found that many metrics have better values, particularly from 1.1% to 4.7% on the H@10 metric. Furthermore, we analyzed the influence of hyperparameters and found that the learning rate and negative samples significantly affect the model's results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The datasets generated and/or analysed during the current study are available in the GitHub repository, https://github.com/lnthanhhcmus/ConvHyper.

Code Availability

The code implemented during the current study are available in the GitHub repository, https://github.com/lnthanhhcmus/ConvHyper.

Notes

  1. https://github.com/daiquocnguyen/ConvKB

  2. https://github.com/lnthanhhcmus/ConvHyper

References

  1. Schneider EW (1973) Course modularization applied: the interface system and its implications for sequence control and data analysis. ERIC, Chicago, Illinois, p 21. https://eric.ed.gov/?id=ED088424

  2. Kejriwal M (2022) Knowledge Graphs. Applied Data Science in Tourism: Interdisciplinary Approaches, Methodologies, and Applications 423–449. https://doi.org/10.1007/978-3-030-88389-8_20

  3. Paulheim H (2017) Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic Web 8:489–508. https://doi.org/10.3233/SW-160218

    Article  Google Scholar 

  4. Noy N, Gao Y, Jain A et al (2019) Industry-scale knowledge graphs: lessons and challenges. Commun ACM 62:36–43. https://doi.org/10.1145/3331166

    Article  Google Scholar 

  5. Curtiss M, Becker I, Bosman T et al (2013) Unicorn: a system for searching the social graph. Proc VLDB Endow 6:1150–1161. https://doi.org/10.14778/2536222.2536239

    Article  Google Scholar 

  6. Bollacker K, Evans C, Paritosh P et al (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: ACM SIGMOD international conference on Management of data. pp 1247–1250

  7. Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57:78–85

    Article  Google Scholar 

  8. Auer S, Bizer C, Kobilarov G, et al (2007) Dbpedia: a nucleus for a web of open data. In: The Semantic Web. ISWC ASWC 2007 2007. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp 722–735. https://doi.org/10.1007/978-3-540-76298-0_52

  9. Miller GA (1995) WordNet: a lexical database for English. Commun ACM 38:39–41. https://doi.org/10.1145/219717.219748

    Article  Google Scholar 

  10. Mahdisoltani F, Biega J, Suchanek F (2015) YAGO3: A knowledge base from multilingual wikipedias. In: Proceedings of the 7th Biennial Conference on Innovative Data Systems Research

  11. Ji S, Pan S, Cambria E et al (2021) A survey on knowledge graphs: Representation, acquisition and applications. IEEE Trans Neural Netw Learning Syst 33:494–514. https://doi.org/10.1109/TNNLS.2021.3070843

    Article  MathSciNet  Google Scholar 

  12. Dong X, Gabrilovich E, Heitz G et al (2014) Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. Association for Computing Machinery, New York, NY, USA, pp 601–610

  13. Ben Seghier MEA, Corriea JAFO, Jafari-Asl J et al (2021) On the modeling of the annual corrosion rate in main cables of suspension bridges using combined soft computing model and a novel nature-inspired algorithm. Neural Comput Appl 33:15969–15985. https://doi.org/10.1007/s00521-021-06199-w

    Article  Google Scholar 

  14. Mai SH, Ben Seghier MEA, Nguyen PL et al (2022) A hybrid model for predicting the axial compression capacity of square concrete-filled steel tubular columns. Eng Comput 38:1205–1222. https://doi.org/10.1007/s00366-020-01104-w

    Article  Google Scholar 

  15. Ohadi S, HashemiMonfared SA, AzhdaryMoghaddam M, Givehchi M (2023) Feasibility of a novel predictive model based on multilayer perceptron optimized with Harris hawk optimization for estimating of the longitudinal dispersion coefficient in rivers. Neural Comput Appl 35:7081–7105. https://doi.org/10.1007/s00521-022-08074-8

    Article  Google Scholar 

  16. Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. pp 1811–1818

  17. Jiang X, Wang Q, Wang B (2019) Adaptive convolution for multi-relational learning. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp 978–987

  18. Nguyen DQ, Nguyen TD, Nguyen DQ, Phung D (2018) A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp 327–333

  19. Le T, Nguyen D, Le B (2021) Learning Embedding for Knowledge Graph Completion with Hypernetwork. In: Proceedings of the 13th International Conference on Computational Collective Intelligence. Springer International Publishing, Cham, pp 16–28

  20. Bordes A, Usunier N, Garcia-Duran A et al (2013) Translating embeddings for modeling multi-relational data. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. Curran Associates Inc. pp 2787–2795

  21. Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence. p 8

  22. Sun Z, Deng Z-H, Nie J-Y, Tang J (2019) RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In: Proceedings of 7th International Conference on Learning Representations. OpenReview.net

  23. Peng Y, Zhang J (2020) LineaRE: Simple but Powerful Knowledge Graph Embedding for Link Prediction. In: Proceedings of 2020 IEEE International Conference on Data Mining. IEEE, pp 422–431

  24. Chao L, He J, Wang T, Chu W (2021) PairRE: Knowledge Graph Embeddings via Paired Relation Vectors. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, pp 4360–4369

  25. Balažević I, Allen C, Hospedales TM (2019) Hypernetwork knowledge graph embeddings. In: Proceedings of the 2019 International Conference on Artificial Neural Networks. Springer, pp 553–565

  26. Guo L, Sun Z, Hu W (2019) Learning to exploit long-term relational dependencies in knowledge graphs. In: Proceedings of the 36th International Conference on Machine Learning. pp 2505–2514

  27. Nguyen DQ, Vu T, Nguyen TD, et al (2019) A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp 2180–2189

  28. Schlichtkrull M, Kipf TN, Bloem P et al (2018) Modeling Relational Data with Graph Convolutional Networks. In: Gangemi A, Navigli R, Vidal M-E et al (eds) The Semantic Web. Springer International Publishing, Cham, pp 593–607

    Chapter  Google Scholar 

  29. Ye R, Li X, Fang Y et al (2019) A Vectorized Relational Graph Convolutional Network for Multi-Relational Network Alignment. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. pp 4135–4141

  30. Ha D, Dai A, Le QV (2017) Hypernetworks. In: Proceedings of the 5th International Conference on Learning Representations

  31. Lin Y, Liu Z, Sun M et al (2015) Learning Entity and Relation Embeddings for Knowledge Graph Completion. In: Proceedings of the AAAI Conference on Artificial Intelligence

  32. Toutanova K, Chen D (2015) Observed versus latent features for knowledge base and text inference. In: Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. Association for Computational Linguistics, Beijing, China, pp 57–66

  33. Rossi A, Firmani D, Matinata A et al (2021) Knowledge Graph Embedding for Link Prediction: A Comparative Analysis. ACM Trans Knowl Discov Data 15:1–49. https://doi.org/10.1145/3424672

    Article  Google Scholar 

  34. Shen T, Zhang F, Cheng J (2022) A comprehensive overview of knowledge graph completion. Knowl-Based Syst 255:109597. https://doi.org/10.1016/j.knosys.2022.109597

    Article  Google Scholar 

  35. Duchi J, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 12:2121–2159

    MathSciNet  MATH  Google Scholar 

  36. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations

  37. Kazemi SM, Poole D (2018) SimplE embedding for link prediction in knowledge graphs. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, pp 4289–4300

  38. Nathani D, Chauhan J, Sharma C, Kaul M (2019) Learning attention-based embeddings for relation prediction in knowledge graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 4710–4723

  39. Xie X, Zhang N, Li Z, et al (2022) From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer. In: Companion Proceedings of the Web Conference 2022. Association for Computing Machinery, New York, NY, USA, pp 162–165

  40. Kamigaito H, Hayashi K (2021) Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, pp 5517–5531

  41. Huang J, Zhang T, Zhu J et al (2021) A deep embedding model for knowledge graph completion based on attention mechanism. Neural Comput & Applic 33:9751–9760. https://doi.org/10.1007/s00521-021-05742-z

    Article  Google Scholar 

  42. Junhua D, Yucheng H, Yi-an Z, Dong Z (2022) Attention-Based Relational Graph Convolutional Network for Knowledge Graph Reasoning. In: 2022 21st International Symposium on Communications and Information Technologies (ISCIT). pp 216–221

  43. Das R, Godbole A, Monath N et al (2020) Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion. In: Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, pp 4752–4765

  44. Le T, Le N, Le B (2023) Knowledge graph embedding by relational rotation and complex convolution for link prediction. Expert Syst Appl 214:119122. https://doi.org/10.1016/j.eswa.2022.119122

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam

    Thanh Le, Duy Nguyen & Bac Le

  2. Vietnam National University, Ho Chi Minh City, Vietnam

    Thanh Le, Duy Nguyen & Bac Le

Authors
  1. Thanh Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Duy Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bac Le
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Thanh Le: Conceptualization, Methodology, Validation, Supervision, Writing – original draft. Duy Nguyen: Methodology, Software, Formal analysis, Visualization, Writing – original draft, Writing – review & editing. Bac Le: Conceptualization, Methodology, Supervision, Review, ValidationDeclarations.

Corresponding author

Correspondence to Bac Le.

Ethics declarations

Informed consent

Informed consent was obtained from all of the subjects involved in this study.

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 12

Table 13

Table 14

Table 15

Table 12 H@10 values on the validation set using different negative sample sizes on the WN18RR and FB15k-237 datasets
Full size table
Table 13 H@10 values in validation set using different filter sizes on WN18RR and FB15k-237 datasets
Full size table
Table 14 H@10 values in validation set using different dimensions of the hidden vector on WN18RR and FB15k-237
Full size table
Table 15 Effect of negative sample sizes on metrics using WN18RR dataset and FB15k-237 dataset
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Le, T., Nguyen, D. & Le, B. Generating relation-specific weights for ConvKB using a HyperNetwork architecture. Appl Intell 53, 21092–21115 (2023). https://doi.org/10.1007/s10489-023-04670-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04670-w

Keywords

Search

Navigation