Keke Huang
Pietro Liò
ABSTRACT
Graph Neural Networks (GNNs), known as spectral graph filters, find a wide range of applications in web networks. To bypass eigendecomposition, polynomial graph filters are proposed to approximate graph filters by leveraging various polynomial bases for filter training. However, no existing studies have explored the diverse polynomial graph filters from a unified perspective for optimization.
In this paper, we first unify polynomial graph filters, as well as the optimal filters of identical degrees into the Krylov subspace of the same order, thus providing equivalent expressive power theoretically. Next, we investigate the asymptotic convergence property of polynomials from the unified Krylov subspace perspective, revealing their limited adaptability in graphs with varying heterophily degrees. Inspired by those facts, we design a novel adaptive Krylov subspace approach to optimize polynomial bases with provable controllability over the graph spectrum so as to adapt various heterophily graphs. Subsequently, we propose AdaptKry, an optimized polynomial graph filter utilizing bases from the adaptive Krylov subspaces. Meanwhile, in light of the diverse spectral properties of complex graphs, we extend AdaptKry by leveraging multiple adaptive Krylov bases without incurring extra training costs. As a consequence, extended AdaptKry is able to capture the intricate characteristics of graphs and provide insights into their inherent complexity. We conduct extensive experiments across a series of real-world datasets. The experimental results demonstrate the superior filtering capability of AdaptKry, as well as the optimized efficacy of the adaptive Krylov basis.
Supplemental Material
- Richard Askey. 1974. Positive Jacobi Polynomial Sums, III. In Linear Operators and Approximation II. 305--312.Google Scholar
- Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gauzere, Sebastien Adam, and Paul Honeine. 2020. Bridging the gap between spectral and spatial domains in graph neural networks. arXiv preprint arXiv:2003.11702 (2020).Google Scholar
- Filippo Maria Bianchi, Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. 2021. Graph neural networks with convolutional arma filters. IEEE transactions on pattern analysis and machine intelligence (2021).Google ScholarCross Ref
- Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. 2004. Convex optimization. Cambridge university press.Google ScholarDigital Library
- Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Process. Mag. , Vol. 34, 4 (2017), 18--42.Google ScholarCross Ref
- Sudhanshu Chanpuriya and Cameron Musco. 2022. Simplified Graph Convolution with Heterophily. In NeurIPS.Google Scholar
- Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi Feng. 2019. Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution. In ICCV. 3434--3443.Google Scholar
- Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. 2021. Adaptive Universal Generalized PageRank Graph Neural Network. In ICLR.Google Scholar
- Fan RK Chung and Fan Chung Graham. 1997. Spectral graph theory. Number 92. American Mathematical Soc.Google Scholar
- Michaë l Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NIPS. 3837--3845.Google Scholar
- Wenqi Fan, Yao Ma, Qing Li, Yuan He, Yihong Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph Neural Networks for Social Recommendation. In WWW. 417--426.Google Scholar
- Rida T. Farouki. 2012. The Bernstein polynomial basis: A centennial retrospective. Comput. Aided Geom. Des. , Vol. 29, 6 (2012), 379--419.Google ScholarDigital Library
- Joel N Franklin. 2012. Matrix theory. Courier Corporation.Google Scholar
- Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti. 2020. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198 (2020).Google Scholar
- Walter Gautschi. 2004. Orthogonal polynomials: computation and approximation. OUP Oxford.Google Scholar
- Tao Guo and Baojiang Cui. 2021. Web Page Classification Based on Graph Neural Network. In IMIS, Vol. 279. Springer, 188--198.Google Scholar
- Yuhe Guo and Zhewei Wei. 2023. Graph Neural Networks with Learnable and Optimal Polynomial Bases. In ICML. 12077--12097.Google Scholar
- Martin H Gutknecht. 2007. A brief introduction to Krylov space methods for solving linear systems. In ISFCS. 53--62.Google Scholar
- David K Hammond, Pierre Vandergheynst, and Rémi Gribonval. 2011. Wavelets on graphs via spectral graph theory. Applied and Computational Harmonic Analysis , Vol. 30, 2 (2011), 129--150.Google ScholarCross Ref
- Mingguo He, Zhewei Wei, and Ji-Rong Wen. 2022. Convolutional Neural Networks on Graphs with Chebyshev Approximation, Revisited. In NeurIPS.Google Scholar
- Mingguo He, Zhewei Wei, Hongteng Xu, et al. 2021. BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation. In NeurIPS. 14239--14251.Google Scholar
- Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 (2020).Google Scholar
- Keke Huang, Jing Tang, Juncheng Liu, Renchi Yang, and Xiaokui Xiao. 2023. Node-wise Diffusion for Scalable Graph Learning. In WWW. 1723--1733.Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.Google Scholar
- Johannes Klicpera, Aleksandar Bojchevski, and Stephan Gü nnemann. 2019a. Predict then Propagate: Graph Neural Networks meet Personalized PageRank. In ICLR.Google Scholar
- Johannes Klicpera, Stefan Weißenberger, and Stephan Gü nnemann. 2019b. Diffusion Improves Graph Learning. In NeurIPS. 13333--13345.Google Scholar
- Runlin Lei, Zhen Wang, Yaliang Li, Bolin Ding, and Zhewei Wei. 2022. EvenNet: Ignoring Odd-Hop Neighbors Improves Robustness of Graph Neural Networks. In NeurIPS.Google Scholar
- David A Levin and Yuval Peres. 2017. Markov chains and mixing times. Vol. 107. American Mathematical Soc.Google Scholar
- Chang Li and Dan Goldwasser. 2019. Encoding Social Information with Graph Convolutional Networks for political Perspective Detection in News Media. In ACL. 2594--2604.Google Scholar
- Mingjie Li, Xiaojun Guo, Yifei Wang, Yisen Wang, and Zhouchen Lin. 2022. G(^mbox2 )CN: Graph Gaussian Convolution Networks with Concentrated Graph Filters. In ICML, Vol. 162. 12782--12796.Google Scholar
- Jörg Liesen and Zdenek Strakos. 2013. Krylov subspace methods: principles and analysis. Oxford University Press.Google Scholar
- Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser-Nam Lim. 2021. Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods. In NeurIPS. 20887--20902.Google Scholar
- John C Mason and David C Handscomb. 2002. Chebyshev polynomials. Chapman and Hall/CRC.Google Scholar
- Sunil K. Narang, Akshay Gadde, and Antonio Ortega. 2013. Signal processing techniques for interpolation in graph structured data. In ICASSP. 5445--5449.Google Scholar
- Dianne P O'Leary. 1998. Conjugate gradients and related KMP algorithms: the beginnings. Technical Report.Google Scholar
- Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the web. Technical Report. Stanford InfoLab.Google Scholar
- Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. 2020. Geom-GCN: Geometric Graph Convolutional Networks. In ICLR.Google Scholar
- Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, and Jie Tang. 2018. DeepInf: Social Influence Prediction with Deep Learning. In SIGKDD. 2110--2119.Google Scholar
- Yousef Saad. 2022. The Origin and Development of Krylov Subspace Methods. Computing in Science & Engineering , Vol. 24, 4 (2022), 28--39.Google ScholarCross Ref
- Aravind Sankar, Yozen Liu, Jun Yu, and Neil Shah. 2021. Graph Neural Networks for Friend Ranking in Large-scale Social Platforms. In WWW. 2535--2546.Google Scholar
- Franco Scarselli, Ah Chung Tsoi, and Markus Hagenbuchner. 2004. Computing personalized pageranks. In WWW. 382--383.Google Scholar
- JJ Seidel. 1989. Graphs and their spectra. Banach Center Publications , Vol. 25, 1 (1989), 147--162.Google ScholarCross Ref
- Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine, Vol. 29, 3 (2008), 93--93.Google Scholar
- David I. Shuman. 2020. Localized Spectral Graph Filter Frames: A Unifying Framework, Survey of Design Considerations, and Numerical Comparison. IEEE Signal Process. Mag. , Vol. 37, 6 (2020), 43--63.Google ScholarCross Ref
- David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. 2013. The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains. IEEE Signal Process. Mag. , Vol. 30, 3 (2013), 83--98.Google ScholarCross Ref
- Tomohiro Sogabe. 2023. Krylov Subspace Methods for Linear Systems: Principles of Algorithms. Vol. 60. Springer Nature.Google Scholar
- Xiyuan Wang and Muhan Zhang. 2022. How Powerful are Spectral Graph Neural Networks. In ICML, Vol. 162. 23341--23362.Google Scholar
- Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. 2021. Dissecting the Diffusion Process in Linear Graph Convolutional Networks. In NeurIPS. 5758--5769.Google Scholar
- Stephen Wright, Jorge Nocedal, et al. 1999. Numerical optimization. Springer Science, Vol. 35, 67--68 (1999), 7.Google Scholar
- Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. 2019a. Simplifying Graph Convolutional Networks. In ICML, Vol. 97. 6861--6871.Google Scholar
- Qitian Wu, Hengrui Zhang, Xiaofeng Gao, Peng He, Paul Weng, Han Gao, and Guihai Chen. 2019b. Dual Graph Attention Networks for Deep Latent Representation of Multifaceted Social Effects in Recommender Systems. In WWW. 2091--2102.Google Scholar
- Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. 2022. Nodeformer: A scalable graph structure learning transformer for node classification. NeurIPS , Vol. 35, 27387--27401.Google Scholar
- Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In SIGKDD. 974--983.Google Scholar
- Lulu Zhang, Buqing Cao, Mi Peng, Yueying Qing, Guosheng Kang, Jianxun Liu, and Kenneth K Fletcher. 2021. Bilinear Graph Neural Network-Enhanced Web Services Classification. In HPCC. 189--196.Google Scholar
- Xin Zheng, Miao Zhang, Chunyang Chen, Qin Zhang, Chuan Zhou, and Shirui Pan. 2023. Auto-HeG: Automated Graph Neural Network on Heterophilic Graphs. In WWW. 611--620.Google Scholar
- Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. 2021. Interpreting and Unifying Graph Neural Networks with An Optimization Framework. In WWW. 1215--1226.Google Scholar
Index Terms
- Optimizing Polynomial Graph Filters: A Novel Adaptive Krylov Subspace Approach
Recommendations
MINRES-QLP: A Krylov Subspace Method for Indefinite or Singular Symmetric Systems
CG, SYMMLQ, and MINRES are Krylov subspace methods for solving symmetric systems of linear equations. When these methods are applied to an incompatible system (that is, a singular symmetric least-squares problem), CG could break down and SYMMLQ's ...
Preconditioned Krylov Subspace Methods Solving Dense Nonsymmetric Linear Systems Arising from BEM
ICCS '07: Proceedings of the 7th international conference on Computational Science, Part III: ICCS 2007Discretization of boundary integral equations leads, in general, to fully populated nonsymmetric linear systems of equations. We research the comparative performances of iterative techniques based on Krylov subspace solvers as GMRES(m), QMR and Bi-...
Preconditioned Krylov subspace and GMRHSS iteration methods for solving the nonsymmetric saddle point problems
AbstractIn the present paper, we propose a separate approach as a new strategy to solve the saddle point problem arising from the stochastic Galerkin finite element discretization of Stokes problems. The preconditioner is obtained by replacing the (1,1) ...
Comments