Extreme Learning Machine to Graph Convolutional Networks

Abstract

Graph Convolutional Network (GCN) is a powerful model to deal with data arranged as a graph, a structured non-euclidian domain. It is known that GCN reaches high accuracy even when operating with just 2 layers. Another well-known result shows that Extreme Learning Machine (ELM) is an efficient analytic learning technique to train 2 layers Multi-Layer Perceptron (MLP). In this work, we extend ELM theory to operate in the context of GCN, giving rise to ELM-GCN, a novel learning mechanism to train GCN that turns out to be faster than baseline techniques while maintaining prediction capability. We also show a theoretical upper bound in the number of hidden units required to guarantee the GCN performance. To the best of our knowledge, our approach is the first to provide such theoretical guarantees while proposing a non-iterative learning algorithm to train graph convolutional networks.

Appendices

Appendix 1. Proofs

In this appendix we present the proofs of Theorems 1 and 2, which support the theory of Extreme Learning Machine to Graph Convolutional Networks. We also show that RELM-GCN matches ELM-GCN when $\gamma \rightarrow \infty$.

Theorem 1

Proof

We interpret the convolved graph signal matrix $\hat{X}$ rows as the input samples of classic ELM Theorem (Theorem 2.1 from [10]). Since the hypotheses that rows of $\hat{X}$ are distinct, $\sigma$ is an infinitely differentiable activation function, and $\varTheta ^{(1)}$ is randomly sampled from some continuous probability distribution, we conclude the hypothesis of the classic ELM Theorem is satisfied, thus $\sigma (\hat{X} \varTheta ^{(1)}) = \sigma (\hat{A} X \varTheta ^{(1)}) =: Y_h$ is invertible with probability one.

Since the edge weights of G are non-negative, all the elements in matrix $\tilde{A}$ (Eq. 2) are also non-negative. Moreover, diagonal elements of $\tilde{A}$ are positive. Thus, diagonal elements of $\tilde{D}$ (Eq. 2) are also positive and hence $\tilde{D}$ and $\tilde{D}^{-1/2}$ are invertible matrices. Furthermore, from the hypothesis that $\tilde{A}$ is invertible, Eq. 2 shows that $\hat{A}$ must also be an invertible matrix. Therefore, with probability one, $(\hat{A}\ Y_h)$ is invertible and by defining $\varTheta ^{(2)} := (\hat{A} Y_h)^{-1}T$ we have $\Vert \hat{A} Y_h \varTheta ^{(2)} - T \Vert = 0$, concluding the proof of the Theorem.

Theorem 2

Proof

Following exactly the same proof of classic ELM Theorem (Theorem 2.2 from [10]), the validity of the Theorem comes from the fact that, otherwise, one could choose $H = N$ which makes $\Vert \hat{A} Y_h \varTheta ^{(2)} - T \Vert = 0 < \epsilon$ according to Theorem 1.

Now we show that ELM-GCN is a special case of RELM-GCN when $\gamma \rightarrow \infty$.

Proof

When $\gamma \rightarrow \infty$ we have $\frac{1}{\gamma }I \rightarrow 0$. Thus the analytical assignment to $\varTheta ^{(2)}$ by RELM-GCN (last instruction of Algorithm 3) becomes:

$$\begin{aligned} \varTheta ^{(2)}&= \bigg ( \frac{1}{\gamma }I + (\hat{A}Y_h)^T (\hat{A}Y_h) \bigg )^\dagger (\hat{A}Y_h)^T\,T = \bigg ( (\hat{A}Y_h)^T (\hat{A}Y_h) \bigg )^\dagger (\hat{A}Y_h)^T\,T\\&= (\hat{A}Y_h)^\dagger \Big ( (\hat{A}Y_h)^T \Big )^\dagger (\hat{A}Y_h)^T\,T = (\hat{A}Y_h)^\dagger \Big ( (\hat{A}Y_h)^\dagger \Big )^T (\hat{A}Y_h)^T\,T\\&= (\hat{A}Y_h)^\dagger \Big ( (\hat{A}Y_h) (\hat{A}Y_h)^\dagger \Big )^T \,T = (\hat{A}Y_h)^\dagger (\hat{A}Y_h) (\hat{A}Y_h)^\dagger \,T = (\hat{A}Y_h)^\dagger \,T\\ \end{aligned}$$

which is the assignment to $\varTheta ^{(2)}$ given by ELM-GCN (last instruction of Algorithm 2).

Appendix 2. Additional Experiment Details

In the following, we further validate the results obtained in the experiments involving real data. Specifically, we analyse the different runs that produced Fig. 3 using Wilcoxon Signed-Rank Test [5]. This hypothesis test is a non-parametric statistical test that compares two samples that are paired. Precisely, the test compares accuracy and training time produced by either ELM-GCN or RELM-GCN against the ones resulting from the other two algorithms. First, we consider the null hypothesis that one of our approaches and competing algorithms generate results according to the same distribution. If the null hypothesis is rejected, we proceed to the next step which considers another null hypothesis that ELM-GCN or its regularized version performs worst (i.e. lower accuracy or higher training time) than the other learning techniques. In all tests we use significance of 99.9%.

Regarding ELM-GCN, the Wilcoxon test rejected the hypothesis that this technique produces output at least as accurate as BP or fastGCN. However in terms of training time, the null hypothesis that ELM-GCN comes from same distribution as its competitors is rejected. Moreover, the Wilcoxon test also rejects the hypothesis that ELM-GCN has higher training time than BP or fastGCN.

Comparing RELM-GCN with BP, the Wilcoxon test could not reject the null hypothesis that both techniques produce the same accuracy. However, when RELM-GCN is compared against fastGCN, we get an interesting outcome. The null hypothesis can not be rejected when both learning algorithms are compared in the inductive paradigm, but the hypothesis is rejected in the transductive scenario. Moreover, the hypothesis that RELM-GCN is less accurate than fastGCN in the same paradigm is rejected. Indeed, a careful analysis of Fig. 3a shows that RELM-GCN consistently outperforms fastGCN on the first 3 datasets while performing comparably on Reddit.

Furthermore, the Wilcoxon test rejected the null hypothesis that RELM-GCN is as fast as competing techniques, regardless of the algorithm and paradigm chosen to compare. Moreover, the second step of the test showed that we should also reject the hypothesis that RELM-GCN is slower than the other algorithms in any learning paradigm. Conclusions provided by the Wilcoxon test are consistent with training times shown in Fig. 3, since RELM-GCN outperforms the competing algorithms in most datasets, being only comparable with fastGCN in Pubmed.