ABSTRACT
In link prediction evaluation, an embedding model assigns plausibility scores to unseen triples in a knowledge graph using an input partial triple. Performance metrics like mean rank are useful to compare models side by side, but do not shed light on their behavior. Interpreting link prediction evaluation and comparing models based on such interpretation are appealing. Current interpretation methods have mainly focused on single predictions or other tasks different from link prediction. Since knowledge graph embedding methods are diverse, interpretation methods that are applicable only to certain machine learning approaches cannot be used. In this paper, we propose a model-agnostic method for interpreting link prediction evaluation as a whole. The interpretation consists of Horn rules mined from the knowledge graph containing the triples a model deems plausible. We combine precision and recall measurements of mined rules using Fβ score to quantify interpretation accuracy. To maximize interpretation accuracy when comparing models, we study two approximations to the hard problem of merging rules. Our quantitative study shows that interpretation accuracy serves to compare diverse models side by side, and that these comparisons are different from those using ranks. Our qualitative study shows that several models globally capture expected semantics, and that models make a common set of predictions despite of redundancy reduction.
- Ralph Abboud, .Ismail. Ilkan Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020. BoxE: A Box Embedding Model for Knowledge Base Completion. In NeurIPS. 9649--9661.Google Scholar
- Naser Ahmadi, Viet-Phi Huynh, Venkata Vamsikrishna Meduri, Stefano Ortona, and Paolo Papotti. 2020. Mining Expressive Rules in Knowledge Graphs. ACM J. Data Inf. Qual., Vol. 12, 2 (2020), 8:1--8:27.Google Scholar
- Farahnaz Akrami, Mohammed Samiul Saeef, Qingheng Zhang, Wei Hu, and Chengkai Li. 2020. Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study. In SIGMOD. 1995--2010.Google Scholar
- Daniel Ayala, Agust'i n Borrego, Inma Herná ndez, Carlos R. Rivero, and David Ruiz. 2019. AYNEC: All You Need for Evaluating Completion Techniques in Knowledge Graphs. In ESWC. 397--411.Google Scholar
- Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. TuckER: Tensor Factorization for Knowledge Graph Completion. In EMNLP. 5184--5193.Google Scholar
- Iti Bansal, Sudhanshu Tiwari, and Carlos R. Rivero. 2020. The Impact of Negative Triple Generation Strategies and Anomalies on Knowledge Graph Completion. In CIKM. 45--54.Google Scholar
- Max Berrendorf, Evgeniy Faerman, Laurent Vermue, and Volker Tresp. 2020. Interpretable and Fair Comparison of Link Prediction or Entity Alignment Methods. In WI/IAT. 371--374.Google Scholar
- Patrick Betz, Christian Meilicke, and Heiner Stuckenschmidt. 2022. Adversarial Explanations for Knowledge Graph Embeddings. In IJCAI. 2820--2826.Google Scholar
- Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In SIGMOD. 1247--1250.Google ScholarDigital Library
- Stephen Bonner, Ian P. Barrett, Cheng Ye, Rowan Swiers, Ola Engkvist, and William L. Hamilton. 2021. Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery. CoRR, Vol. abs/2105.10488 (2021).Google Scholar
- Antoine Bordes, Nicolas Usunier, Alberto Garc'i a-Durá n, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In NIPS. 2787--2795.Google ScholarDigital Library
- Agust'i n Borrego, Daniel Ayala, Inma Herná ndez, Carlos R. Rivero, and David Ruiz. 2019. Generating Rules to Filter Candidate Triples for their Correctness Checking by Knowledge Graph Completion Techniques. In K-CAP. 115--122.Google Scholar
- Vasek Chvá tal. 1979. A Greedy Heuristic for the Set-Covering Problem. Math. Oper. Res., Vol. 4, 3 (1979), 233--235.Google ScholarDigital Library
- Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D Knowledge Graph Embeddings. In AAAI. 1811--1818.Google Scholar
- Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In KDD. 601--610.Google Scholar
- Takuma Ebisu and Ryutaro Ichise. 2018. TorusE: Knowledge Graph Embedding on a Lie Group. In AAAI. 1819--1826.Google Scholar
- Luis Antonio Galá rraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. 2013. AMIE: Association Rule Mining under Incomplete Evidence in Ontological Knowledge Bases. In WWW. 413--422.Google Scholar
- Xinbo Gao, Bing Xiao, Dacheng Tao, and Xuelong Li. 2010. A Survey of Graph Edit Distance. Pattern Anal. Appl., Vol. 13, 1 (2010), 113--129.Google ScholarCross Ref
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2019. A Survey of Methods for Explaining Black Box Models. ACM Comput. Surv., Vol. 51, 5 (2019), 93:1--93:42.Google ScholarDigital Library
- Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. 2018. Knowledge Graph Embedding with Iterative Guidance from Soft Rules. In AAAI. 4816--4823.Google Scholar
- Arthur Colombini Gusm a o, Alvaro Henrique Chaim Correia, Glauber De Bona, and Fá bio Gagliardi Cozman. 2018. Interpreting Embedding Models of Knowledge Bases: A Pedagogical Approach. CoRR, Vol. abs/1806.09504 (2018).Google Scholar
- Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. OpenKE: An Open Toolkit for Knowledge Embedding. In EMNLP. 139--144.Google Scholar
- Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. 2017. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, Vol. 6 (2017), e26726.Google ScholarCross Ref
- Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard de Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan F. Sequeda, Steffen Staab, and Antoine Zimmermann. 2022. Knowledge Graphs. ACM Comput. Surv., Vol. 54, 4 (2022), 71:1--71:37.Google Scholar
- Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2022. A Survey on Knowledge Graphs: Representation, Acquisition, and Applications. IEEE Trans. Neural Networks Learn. Syst., Vol. 33, 2 (2022), 494--514.Google ScholarCross Ref
- Been Kim, Oluwasanmi Koyejo, and Rajiv Khanna. 2016. Examples are not enough, learn to criticize! Criticism for Interpretability. In NIPS. 2280--2288.Google Scholar
- Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. 2019. Anytime Bottom-Up Rule Learning for Knowledge Graph Completion. In IJCAI. 3137--3143.Google Scholar
- Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. 2018. Fine-Grained Evaluation of Rule- and Embedding-Based Systems for Knowledge Graph Completion. In ISWC. 3--20.Google Scholar
- George A. Miller. 1995. WordNet: A Lexical Database for English. Commun. ACM, Vol. 38, 11 (1995), 39--41.Google ScholarDigital Library
- W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Interpretable Machine Learning: Definitions, Methods, and Applications. CoRR, Vol. abs/1901.04592 (2019).Google Scholar
- Maximilian Nickel, Lorenzo Rosasco, and Tomaso A. Poggio. 2016. Holographic Embeddings of Knowledge Graphs. In AAAI. 1955--1961.Google ScholarDigital Library
- Pouya Pezeshkpour, Yifan Tian, and Sameer Singh. 2019. Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications. In AKBC.Google Scholar
- Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, and Paolo Merialdo. 2021. Knowledge Graph Embedding for Link Prediction: A Comparative Analysis. ACM Trans. Knowl. Discov. Data, Vol. 15, 2 (2021), 14:1--14:49.Google ScholarDigital Library
- Andrea Rossi, Donatella Firmani, Paolo Merialdo, and Tommaso Teofili. 2022. Explaining Link Prediction Systems based on Knowledge Graph Embeddings. In SIGMOD. 2062--2075.Google Scholar
- Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020. You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings. In ICLR.Google Scholar
- Andrey Ruschel, Arthur Colombini Gusm ao, Gustavo Padilha Polleti, and Fabio Gagliardi Cozman. 2019. Explaining Completions Produced by Embeddings of Knowledge Graphs. In ECSQARU. 324--335.Google Scholar
- Afshin Sadeghi, Hirra Malik, Diego Collarana, and Jens Lehmann. 2021. Relational Pattern Benchmarking on the Knowledge Graph Link Prediction Task. In NeurIPS Datasets and Benchmarks.Google Scholar
- Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In NIPS. 926--934.Google ScholarDigital Library
- Fernando Sola, Daniel Ayala, Rafael Ayala, Inma Hernández, Carlos R. Rivero, and David Ruiz. 2023. AYNEXT - tools for streamlining the evaluation of link prediction techniques. SoftwareX, Vol. 23 (2023), 101474.Google ScholarCross Ref
- Tengwei Song, Jie Luo, and Lei Huang. 2021. Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding. In NeurIPS. 24695--24706.Google Scholar
- Josua Stadelmaier and Sebastian Padó. 2019. Modeling Paths for Explainable Knowledge Base Completion. In ACL Workshops. 147--157.Google Scholar
- Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In WWW. 697--706.Google Scholar
- Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In ICLR Posters.Google Scholar
- Pedro Tabacof and Luca Costabello. 2020. Probability Calibration for Knowledge Graph Embedding Models. In ICLR.Google Scholar
- Sudhanshu Tiwari, Iti Bansal, and Carlos R. Rivero. 2021. Revisiting the Evaluation Protocol of Knowledge Graph Completion Methods for Link Predictionn. In TheWebConf. 809--820.Google Scholar
- Kristina Toutanova and Danqi Chen. 2015. Observed Versus Latent Features for Knowledge Base and Text Inference. In ACL Workshops. 57--66.Google Scholar
- Trung-Kien Tran, Mohamed H. Gad-Elrab, Daria Stepanova, Evgeny Kharlamov, and Jannik Strötgen. 2020. Fast Computation of Explanations for Inconsistency in Large-Scale Knowledge Graphs. In TheWebConf. 2613--2619.Google Scholar
- Thé o Trouillon, Johannes Welbl, Sebastian Riedel, É ric Gaussier, and Guillaume Bouchard. 2016. Complex Embeddings for Simple Link Prediction. In ICML, Vol. 48. 2071--2080.Google Scholar
- Brian Walsh, Sameh K. Mohamed, and V'i t Nová cek. 2020. BioKG: A Knowledge Graph for Relational Learning On Biological Data. In CIKM. 3173--3180.Google Scholar
- Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge Graph Embedding: A Survey of Approaches and Applications. IEEE Trans. Knowl. Data Eng., Vol. 29, 12 (2017), 2724--2743.Google ScholarCross Ref
- Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. In AAAI. 1112--1119.Google Scholar
- Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In ICLR Posters.Google Scholar
- Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019d. Quaternion Knowledge Graph Embeddings. In NeurIPS. 2731--2741.Google Scholar
- Wen Zhang, Shumin Deng, Han Wang, Qiang Chen, Wei Zhang, and Huajun Chen. 2019a. XTransE: Explainable Knowledge Graph Embedding for Link Prediction with Lifestyles in e-Commerce. In JIST, Vol. 1157. 78--87.Google Scholar
- Wen Zhang, Bibek Paudel, Liang Wang, Jiaoyan Chen, Hai Zhu, Wei Zhang, Abraham Bernstein, and Huajun Chen. 2019b. Iteratively Learning Embeddings and Rules for Knowledge Graph Reasoning. In TheWebConf. 2366--2377.Google Scholar
- Wen Zhang, Bibek Paudel, Wei Zhang, Abraham Bernstein, and Huajun Chen. 2019c. Interaction Embeddings for Prediction and Explanation in Knowledge Graphs. In WSDM. 96--104.Google Scholar
- Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. 2020. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction. In AAAI. 3065--3072.Google Scholar
Index Terms
- A Model-Agnostic Method to Interpret Link Prediction Evaluation of Knowledge Graph Embeddings
Recommendations
Hierarchy-Based Link Prediction in Knowledge Graphs
WWW '16 Companion: Proceedings of the 25th International Conference Companion on World Wide WebLink prediction over a knowledge graph aims to predict the missing entity h or t for a triple (h,r,t). Existing knowledge graph embedding based predictive methods represent entities and relations in knowledge graphs as elements of a vector space, and ...
Multimodal Link Prediction Method for Commodity Knowledge Graph
HP3C '23: Proceedings of the 2023 7th International Conference on High Performance Compilation, Computing and CommunicationsKnowledge graph link prediction is to predict whether there is a relation between two entities according to the existing knowledge graph data, which is of great significance for improving the existing knowledge graph. Based on the application of ...
Hierarchical-aware relation rotational knowledge graph embedding for link prediction
AbstractKnowledge graph embedding, as the upstream task of link prediction which aims to predict new links between entities under the premise of known relations, its reliability greatly affects the performance of link prediction. However, ...
Comments