Abstract
Instability of trained models, i.e., the dependence of individual node predictions on random factors, can affect reproducibility, reliability, and trust in machine learning systems. In this paper, we systematically assess the prediction instability of node classification with state-of-the-art Graph Neural Networks (GNNs). With our experiments, we establish that multiple instantiations of popular GNN models trained on the same data with the same model hyperparameters result in almost identical aggregated performance, but display substantial disagreement in the predictions for individual nodes. We find that up to 30% of the incorrectly classified nodes differ across algorithm runs. We identify correlations between hyperparameters, node properties, and the size of the training set with the stability of predictions. In general, maximizing model performance implicitly also reduces model instability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Code and supplementary material are available at https://github.com/mklabunde/gnn-prediction-instability.
References
Bahri, D., Jiang, H.: Locally adaptive label smoothing improves predictive churn. In: ICML (2021)
Bhojanapalli, S., et al.: On the reproducibility of neural network predictions. arXiv preprint arXiv:2102.03349 (2021)
Dwivedi, V.P., Joshi, C.K., Laurent, T., Bengio, Y., Bresson, X.: Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982 (2020)
Gao, Z., Isufi, E., Ribeiro, A.: Stability of graph convolutional neural networks to stochastic perturbations. Signal Process. 188, 108216 (2021)
Hu, W., et al.: Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 (2020)
Jiang, H., Narasimhan, H., Bahri, D., Cotter, A., Rostamizadeh, A.: Churn reduction via distillation. In: ICLR (2022)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)
Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: ICML (2019)
Liu, H., S., A.P.V., Patwardhan, S., Grasch, P., Agarwal, S.: Model stability with continuous data updates. arXiv preprint arXiv:2201.05692 (2022)
Madani, O., Pennock, D., Flake, G.: Co-validation: using model disagreement on unlabeled data to validate classification algorithms. In: NeurIPS (2004)
Mernyei, P., Cangea, C.: Wiki-CS: a Wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901 (2020)
Milani Fard, M., Cormier, Q., Canini, K., Gupta, M.: Launch and iterate: reducing prediction churn. In: NeurIPS (2016)
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2009)
Schumacher, T., Wolf, H., Ritzert, M., Lemmerich, F., Grohe, M., Strohmaier, M.: The effects of randomness on the stability of node embeddings. In: Workshop on Graph Embedding and Mining, co-located with ECML PKDD (2021)
Shamir, G.I., Coviello, L.: Anti-distillation: improving reproducibility of deep networks. arXiv preprint arXiv:2010.09923 (2020)
Shchur, O., Mumme, M., Bojchevski, A., Günnemann, S.: Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018)
Summers, C., Dinneen, M.J.: Nondeterminism and instability in neural network optimization. In: ICML (2021)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: ICLR (2018)
Wang, C., Rao, W., Guo, W., Wang, P., Liu, J., Guan, X.: Towards understanding the instability of network embedding. IEEE Trans. Knowl. Data Eng. 34(2), 927–941 (2022)
Yang, Z., Cohen, W., Salakhudinov, R.: Revisiting semi-supervised learning with graph embeddings. In: ICML (2016)
Zhuang, D., Zhang, X., Song, S.L., Hooker, S.: Randomness in neural network training: characterizing the impact of tooling. arXiv preprint arXiv:2106.11872 (2021)
Zügner, D., Akbarnejad, A., Günnemann, S.: Adversarial attacks on neural networks for graph data. In: KDD (2018)
Zügner, D., Günnemann, S.: Certifiable robustness and robust training for graph convolutional networks. In: KDD (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Klabunde, M., Lemmerich, F. (2023). On the Prediction Instability of Graph Neural Networks. In: Amini, MR., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2022. Lecture Notes in Computer Science(), vol 13715. Springer, Cham. https://doi.org/10.1007/978-3-031-26409-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-26409-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-26408-5
Online ISBN: 978-3-031-26409-2
eBook Packages: Computer ScienceComputer Science (R0)
