TAMNR: a network embedding learning algorithm using text attention mechanism

Problem description

Networks are also called graphs in data science and computer science, and are generally represented by G = (V, E), with V representing the set of all nodes in the network, E representing the set of all edges in the network, and |V| representing the number of nodes in the network. This article uses R_v ∈ R^k to represent the network embedding vector obtained from training, which is a k-dimensional matrix, and each row in the matrix represents a node’s k-dimensional network embedding vector, where k is expected to be much smaller than |V|, and att is the attention parameter.

In real-world network structure data, there is usually a certain degree of similarity between two connected nodes. Given a graph G = (V, E), for any two nodes V_i, V_j, if there is an edge between the two nodes, that is e_i,j ∈ E, then V_iandV_j are said to be similar to each other. However, many nodes are far apart in terms of topology, but have similarities and simply considering the network structure does not account for these nodes. Therefore, a text-based feature is designed and attention is added for modeling, allowing this network embedding model to incorporate the text features of the nodes. During the network learning process, the network topology and text features of the nodes are preserved, so that the nodes that are close to each other or have similar text features in the network topology. In the subsequent model design procedures, an attention mechanism for text features is added, so that text features with high contributions have a positive impact on network structure modeling, avoiding the negative impact of unimportant features on network structure modeling.

CBOW (Continuous Bag-of-Words) model using negative sampling

Word2Vec has designed two models: CBOW and Skip-Gram, and provides two optimization methods: negative sampling and hierarchical SoftMax. DeepWalk is a network embedding algorithm proposed for large-scale networks based on the Word2Vec model, and inherits the training model and optimization method provided by Word2Vec. Although the training accuracy of the CBOW adding negative sampling optimization model is slightly lower than that of the Skip-Gram adding hierarchical SoftMax optimization model, the training speed of the former is much faster than that of the latter. Therefore, the negative sampling optimized CBOW algorithm is used in the model proposed in this article to train the embeddings in the model.

For the current node v, the context node of node v is v_nb, node v is a positive sample, other nodes are negative samples, and the negative sample set of node v is NEG(v), NEG(v) ≠ null. The sampling label of node v is $L\left(v\right)$. For ∀u ∈ D, D is the node set. The sampling result of node v by $L\left(v\right)$ is defined as follows: (1)${\displaystyle L^v\left(u\right)=\left\{\begin{array}{c}1,u=v,\hfill \\ 0,u\ne v.\hfill \end{array}\right.}$

with the positive sample of $L^v\left(u\right)$ being 1, and the negative sample of $L^v\left(u\right)$ being 0.

For a node v and its context node v_nb, its positive sample is $\left(v,v_{nb}\right)$, and the goal is to maximize the following probability: (2)${\displaystyle g_c\left(v\right)=\prod _{v_i\in \left\{v\right\}\cup NEG\left(v\right)}p\left(v_i|v_{nb}\right).}$

And (3)${\displaystyle p\left(v_i|v_{nb}\right)={\left[\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]}^{L^v\left(v_i\right)}\cdot {\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]}^{1-L^v\left(v_i\right)},}$

where $\sigma \left(x\right)$ is the sigmoid function. e_v is the cumulative sum of vectors of each node in v_nb, and θ^v_i is the vector to be trained for node v_i,

Therefore, $g_c\left(v\right)$ can be changed based on Eqs. (2) and (3) as follows: (4)${\displaystyle g_c\left(v\right)=\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\prod _{v_i\in \left\{v\right\}\cup NEG\left(v\right)}\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right].}$

For the entire node set C, the overall optimization objective function is as follows: (5)${\displaystyle G=\prod _{v\in C}{g}_c\left(v\right).}$

The objective function of the CBOW model based on negative sampling is to take the following logarithmic operation:

(6)${\displaystyle L\left(v\right)=logG=log\prod _{v\in C}{g}_c\left(v\right)=\sum _{v\in C}log{g}_c\left(v\right)=\sum _{v\in C}\sum _{v_i\in \left\{v\right\}\cup NEG\left(v\right)}\left\{L^v\left(v_i\right)\cdot log\left[\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]+\left(1-L^v\left(v_i\right)\right)\cdot log\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]\right\}.}$ (7)${\displaystyle L\left(v,v_i\right)=\left(1-L^v\left(v_i\right)\right)\cdot log\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]+L^v\left(v_i\right)\cdot log\left[\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right].}$

Given the function $\mathrm{L}\left(v,v_i\right)$ about θ^v_i, the gradient calculation is: (8)${\displaystyle \frac{\partial L\left(v,v_i\right)}{\partial {\theta }^{v_i}}=\frac{\partial }{\partial {\theta }^{v_i}}\left\{\left(1-L^v\left(v_i\right)\right)\cdot log\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]+L^v\left(v_i\right)\cdot log\left[\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]\right\}.}$

Using the derivative function optimization Eq. (8) of $log\sigma \left(x\right)$ and $log\left(1-\sigma \left(x\right)\right)$ results in: (9)${\displaystyle \frac{\partial L\left(v,v_i\right)}{\partial {\theta }^{v_i}}=L^v\left(v_i\right)\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]e_v-\left[1-L^v\left(v_i\right)\right]\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)e_v=\left\{L^v\left(v_i\right)\left[1-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]-\left[1-L^v\left(v_i\right)\right]\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right\}{e}_v=\left[L^v\left(v_i\right)-\sigma \left(e_v^{\mathrm{T}}{\theta }^{v_i}\right)\right]e_v.}$

In Eq. (6), C is the corpus after random walk of nodes, which is optimized by the stochastic gradient ascent method. This then leads to the update formula of the parameters. The update formula of θ^v_i is: (10)${\displaystyle {\theta }^{v_i}:={\theta }^{v_i}+\mu \left[L^v\left(v_i\right)-\sigma \left(e_v^{\mathrm{T}}\cdot {\theta }^{v_i}\right)\right]e_v.}$

Considering the gradient of e in $L\left(v,v_i\right)$, and using e_v and θ^v_i symmetry, the update formula of the context total node embedding vector $\mathrm{v}\left(u\right)$ is: (11)${\displaystyle v\left(u\right):=v\left(u\right)+\mu \sum _{v_i\in \left\{v\right\}\cup NEG\left(v\right)}\left[L^v\left(v_i\right)-\sigma \left(e_v^{\mathrm{T}}\cdot {\theta }^{v_i}\right)\right]\cdot {\theta }^{v_i}}$

Finally, Eq. (6) can be simplified as follows: (12)${\displaystyle L\left(v\right)=\sum _{v\in C}\sum _{v_i\in \left\{v\right\}\cup NEG\left(v\right)}logp\left(v_i|v_{nb}\right).}$

TAMNR modeling

A simple and efficient joint learning model is needed to meet the requirements of large-scale network embedding learning tasks. This article proposes a joint network representation learning framework based on the textual attention mechanism. The framework consists of two parts: network node relationship modeling and node text relationship modeling. This model uses text as the input of the relationship model, ensuring that the relationship has the same words in the node text when constructing the node relationship. Through these improvements, the TAMNR model proposed in this article is expected to solve the problem of joint modeling of network structure features and text features, and obtain better quality vector representations. Specific information about the model is shown in Fig. 1.

As shown in Fig. 1, for the current central node v_i,  the network modeling uses its former two nodes v_i−2 and v_i−1, and its next two nodes v_i+1 and v_i+2 to predict the probability of the current central node v_i appearing. The network modeling part of the model continuously adjusts the values in the network vector so that the node pairs with connected edges have a closer vector distance to each other, and the node pairs with multi-hop edges or no edges have a farther vector distance. The node text relationship modeling models relationships between node pairs with common text features, and introduces an attention mechanism to give more learning opportunities for node pairs with same words, so that node pairs with the same words have a closer vector distance. Network node relationship modeling and node text feature modeling share a node representation vector in the TAMNR model, allowing these two parts of the model to obtain and exchange feature information through the shared vector, so that the TAMNR algorithm can be used in the modeling learning process. TAMNR can have stronger generalization ability in various tasks by obtaining valuable feature information from neighbor nodes and node text features.

The CBOW model only considers the local contextual information of words, thus failing to effectively capture the quantity and importance of co-occurring words in the text of neighboring nodes. To address this problem, an attention mechanism is added to the TAMNR model to incorporate co-occurring word information, aiming to maximize the appearance probability of the co-occurring words. The overall objective function proposed is, as follows: (13)${\displaystyle L\left(v\right)=\sum _{v\in C}\left(log{g}_c\left(v\right)+\rho log{g}_w\left(v\right)\right).}$

The left part of the function is the objective function of the relational modeling, the right part is the objective function of the text attention modeling, and ρ is the harmonic coefficient that balances the network structure modeling and the text feature modeling. The goal of this article is to maximize Eq. (13) by using the stochastic gradient ascend method to update each parameter, where $g_c\left(v\right)$ and $g_w\left(v\right)$ have the same form, and the left term in the above equation is the objective function of CBOW, and its parameters are updated in Eqs. (10) and (11). The network structure model and the text feature model share the same node representation vectors, so they can obtain information from each other through shared representations, enabling the node representation vectors to comprehensively utilize multiple features during training, thereby obtaining higher-quality node representation vectors. The right term of the above equation is the objective function of the text attention model, which is essentially a CBOW model with an attention mechanism added, so the optimization solution is the same as the right term. The right term is defined, as follows: (14)${\displaystyle F_1=\alpha \cdot \sum _{v\in C}\sum _{u\in w_v}log\left(g_w\left(u\right)\right)=\alpha \cdot \sum _{v\in C}\sum _{u\in w}\sum _{v_i\in \left\{v\right\}\cup NEG\left(u\right)}\left\{L^{v_i}\left(u\right)\cdot log\left[\sigma \left(e_u\cdot {\theta }^{v_i}\right)\right]+\left(1-L^{v_i}\left(u\right)\right)log\left[1-\sigma \left(e_u\cdot {\theta }^{v_i}\right)\right]\right\}.}$

The left term is defined, as follows: (15)${\displaystyle f_1=L^{v_i}\left(u\right)\cdot log\left[\sigma \left(e_u\cdot {\theta }^{v_i}\right)\right]+\left(1-L^{v_i}\left(u\right)\right)log\left[1-\sigma \left(e_u\cdot {\theta }^{v_i}\right)\right].}$

In Eq. (14), the target node is a positive sample, and the remaining nodes are negative samples, and then f₁ is used to obtain partial derivatives of θ^v_i and e_u respectively, and the parameter update formula is obtained, as follows: (16)${\displaystyle {\theta }^{v_i}:={\theta }^{v_i}+\mu \left[L^{v_i}\left(u\right)-\sigma \left(e_u\cdot {\theta }^{v_i}\right)\right]\cdot e_u.}$

The update formula of the embedding vector $\mathrm{w}\left(u\right)$ of each word is as follows: (17)${\displaystyle w\left(u\right):=w\left(u\right)+\mu \sum _{v_i\in \left\{v\right\}\cup NEG\left(v\right)}\left[L^{v_i}\left(u\right)-\sigma \left(e_u\cdot {\theta }^{v_i}\right)\right]\cdot {\theta }^{v_i}.}$

with µ representing the learning rate. After obtaining the updated values of θ^v_i and e_u, the objective function can be iteratively optimized. Weight α needs to be multiplied into Eqs. (13) and (14) µ before balancing the network structure model and text feature model. e_u is the sum of the expression vectors of the text words, and $\mathrm{w}\left(u\right)$ is the representation vector of the word u in the target node text. The attention function att is used to weigh the contribution rate of different text words to the model.

When context words act as context nodes, the sum of context vectors is e_u, and its calculation is as follows: (18)${\displaystyle e_u=\sum _{j=1}^{\left|S\right|}att\left(w_j\right)\cdot d_j.}$ In Eq. (15), att(w_j) is the attention weight of the word w_j to the target node, d_j is the expression vector of the word w_j, and $\left|S\right|$ is the number of words in the node text.

The attention function att is calculated as follows: (19)${\displaystyle att\left(w_j\right)=\frac{exp\left(d_j\cdot \mathrm{w}\left(u\right)\right)}{\sum _{k=1}^{\left|S\right|}exp\left(d_j\cdot \mathrm{w}\left(u\right)\right)}.}$

The attention weight in Eq. (16) is calculated by the vector of words in the text and the vector of context nodes.

Dataset

To evaluate the effectiveness of the TAMNR model, three citation network datasets are used: the academic network dataset CiteSeer (M10), Data Base systems and Logic Programming (DBLP; V4), and Simplified Data Base systems and Logic Programming (SDBLP). SDBLP is used to remove the nodes with less than three references in the DBLP. The average degree of the CiteSeer (M10) dataset is only 2.57, making it a typical sparse dataset; DBLP (V4) has an average degree of 11.297, making it a relatively dense dataset; and SDBLP has an average degree of 25.337, making it to be a denser dataset. These three datasets are selected to test the model on a variety of dataset sizes and to simulate different types of network types in real life, verifying that the model has good machine learning performance in various networks.

Dataset descriptions can be found in Table 1.

Introduction to comparison algorithms

DeepWalk: DeepWalk originates from the Word2Vec algorithm. The DeepWalk algorithm is the most classic network embedding learning algorithm based on neural networks. Most subsequent network embedding learning algorithms are based on the DeepWalk algorithm. DeepWalk can use the CBOW model with fast training speed and the Skip-Gram model with high training accuracy to train the representation learning model based on the neural network, and can also use negative sampling and hierarchical SoftMax to accelerate the network training process. In this article, DeepWalk is trained using CBOW and negative sampling.

LINE: LINE is a network representation algorithm that encodes the network structure of a very large-scale network into a low-dimensional network by sacrificing accuracy. Therefore, the training speed of LINE is very fast, but the accuracy is low, especially in sparse networks. LINE’s speed improvement comes from only considering the first-order similarity or second-order similarity of the network, and the concatenated representation of the two.

GraRep: The GraRep algorithm is based on the idea that the high-order similarity between nodes is important in generating the global representation of nodes. The algorithm uses k-step similarity and calculates its state transfer matrix for different k values. The algorithm transforms the optimization problem of the loss function into a matrix decomposition problem, and directly obtains the global representation matrix of the graph through SVD decomposition, with each row representing the global representation vector of a node.

MFDW: Because DeepWalk decomposes the matrix M = (A + A²)/2, MFDW uses the SVD algorithm to decompose the matrix M, and uses W = U⋅S^0.5 as the embedding vector of the network.

Text Feature (TF): TF converts the text content of the network node into a co-occurrence matrix, and then uses SVD to decompose this co-occurrence matrix to obtain a text feature vector with a column dimension of 100. The TF method is a content-based contrast algorithm.

TADW: TADW is a matrix factorization algorithm that decomposes a matrix M and text matrix T. TADW does not consider context information, and its T matrix cannot preserve the order of words.

CAHNE: CAHNE learns context embeddings for nodes by introducing the context node sequence, and the attention mechanism is also integrated into the model to better reflect the impact of context nodes on the current node.

Experimental setup

The network node classification task is used to evaluate the algorithm proposed in this article against the comparison algorithms introduced in this article, with Liblinear as the baseline classifier. To verify the generalization ability of the algorithm, the training set is set to 0.1∼0.9, and the remaining network nodes are used as the test set. The network embedding vector obtained by the network embedding learning algorithm is uniformly set to 100 dimensions, the random walk length is set to 40, the number of random walks to 10, the window size to 5, the negative sampling to 5, the minimum node frequency to 5, and the learning rate of the neural network is set to 0.05. All experiments in this article are repeated 10 times and then averaged for the final result.

Analysis of experimental results

Three real network datasets, CiteSeer, DBLP, and SDBLP, are used as evaluation datasets, with 10% to 90% of the dataset used as the training set, and the remaining data used as the test set. Table 2 lists the network node classification accuracy.

As shown in Table 2, the network node classification performance of the LINE algorithm is the worst. MFDW is the matrix factorization form of DeepWalk, and its network node classification performance is better than the DeepWalk algorithm in the training sets of various proportions. On the CiteSeer dataset, the MFDW text features of network nodes outperformed the the DeepWalk algorithm in the node classification task. The TAMNR model proposed in this article adds the node text feature relationship, so its performance is also better than the DeepWalk algorithm.

In the DBLP and SDBLP networks, the network node classification performance of DeepWalk is slightly inferior to that of LINE and GraRep. The MFDW algorithm based on DeepWalk performed better than DeepWalk. As the training set ratio increased, the network node classification performance of Text Feature surpassed DeepWalk. Since DBLP and SDBLP are denser networks, the TAMNR algorithm performed better than the comparison algorithms on these networks.

On the CiteSeer dataset, the average classification accuracy of the TAMNR algorithm is 69.59%, and the average classification accuracy of the DeepWalk algorithm is 61.88%. On the DBLP dataset, the average classification accuracy of the TAMNR algorithm is 73.92%, and the average classification accuracy of the DeepWalk algorithm is 65.64%. On the SDBLP dataset, the average classification accuracy of the TAMNR algorithm is 85.18%, and the average classification accuracy of the DeepWalk algorithm is 81.94%.

The comparison graph of these results is shown, as follows:

As shown in Fig. 2, the network node classification accuracy value span of these algorithms on the CiteSeer and DBLP datasets are larger than on the SDBLP dataset. On the SDBLP dataset, the network node classification accuracy of the six comparison algorithms showed a significant upward trend. However, on the CiteSeer and DBLP datasets, the accuracy curve showed a relatively slower upward trend. The main reason for this observed difference is that the features obtained by different algorithms are quite different on a sparse network. If an algorithm can obtain features that fully reflect the network structure, its network classification performance is better. On sparse networks, the network representation learning algorithm based on joint learning can make up for the insufficient training caused by the sparse edges. On dense networks, different algorithms can also obtain effective network structure features from sufficient edge connections, so the difference between the classification performance is smaller.

Four main results are obtained from this comparison study: (1) On the CiteSeer sparse network, the classification performance of GraRep based on high-order representation is not as good as that of DeepWalk, but on dense network datasets, such as DBLP and SDBLP, the classification performance of GraRep is better than that of DeepWalk. (2) On dense network datasets, such as DBLP and SDBLP, the network representation learning algorithm based on matrix decomposition is more effective than the network representation learning algorithm based on a shallow neural model, and the former performed slightly better than the latter in the network node classification task. (3) There are many ways to integrate the text features of network nodes. The simple vector concatenate method cannot bring about significant performance improvement in network representation tasks. Induced matrix completion is a very effective text feature integration framework, and it achieved excellent network representation performance on the three real datasets. The text feature integration framework proposed in this article overcomes the computational limitation of matrix decomposition, and uses node text features to constrain the DeepWalk training procedure, so that the network embedding vectors contain more semantic information. (4) On sparse networks, such as CiteSeer, the classification performance of TAMNR model proposed in this article is better than the comparison algorithms, but as the average degree of network nodes increased, this difference in performance became smaller. On the DBLP and SDBLP datasets, the classification performance difference between TAMNR and DeepWalk is small.

Network embedding visualization

The main purpose of network representation visualization is to check whether the representation vectors obtained by training show a significant clustering phenomenon. The clustering phenomenon shows whether the network representation has learned the community information of the network. If the community division based on the network representation obtained is more accurate, it has better reliability in the network node classification task. In this experiment, four classes of nodes are randomly selected from the CiteSeer dataset, and 150 nodes are randomly selected for each class. The t-SNE algorithm is used to visualize the learned network representation. The results are shown in Fig. 3.

DeepWalk performed poorly in network representation classification tasks, so DeepWalk also displayed the worst results in visualization tasks. The network representation learning performance of the TAMNR model on the three data sets demonstrated its excellent node classification performance, so the embedding vectors obtained by TAMNR showed obvious clustering phenomenon and clustering boundaries. This visualization experiment shows that the proposed strategy of combining node text features can improve the performance of network embedding.

Case analysis

In order to verify the feasibility of the TAMNR model proposed in this article, the target node “Quantum Field Theory as Dynamical System” is randomly selected on the CiteSeer dataset. Three nodes with higher similarity with the target node are obtained by cosine calculation. The result is shown in Table 3.

As shown in Table 3, the DeepWalk model only considers the similarity of network structure features, and does not consider the similarity of text features, so similar nodes did not reflect the text feature similarity. The TAMNR model takes both the network structure features and the text features into account, so the returned similar nodes showed word co-occurrence. Both the TADW algorithm and the TAMNR algorithm proposed in this article consider the text feature information, but the returned nodes are different. The main reason for this difference is that the two algorithms have different mechanisms for modeling text features, so the TADW or TAMNR algorithms are likely best suited to different tasks.