ST-AFN: a spatial-temporal attention based fusion network for lane-level traffic flow prediction

Problem statement

Before ST-AFN is introduced in detail, this section will describe the specific ground road lane selection strategy and traffic flow prediction problem.

Definition 1: Lane Selection. In the previous lane-level traffic flow prediction research, the experimental data came from high-ways or elevated freeways. Under this road condition, following is the corresponding solution: when on high-ways or elevated freeways, the long and straight roads are divided into multiple sub-sections based on the sensors in the main line and the ramp as shown in Fig. 2. There are often some shortcomings in the corresponding strategies under this research background: firstly, the width of the ramp is narrow, which easily becomes a bottleneck in the morning and evening peaks with dense traffic. Existing strategy does not consider a large number of inflow and outflow vehicles in the ramp (as shown by the red arrows in Fig. 2) which may lead to inaccurate results. Secondly, due to the limitation of the research background, the experiment can only use a single straight flow lane as the research object. It has poor scalability and limited practical application range. In comparison, urban ground roads have the characteristics of a higher proportion of total road mileage, wider coverage area, and more complex traffic patterns (Kamal, Hayakawa & Imura, 2020). However, there is still no mature and efficient lane selection strategy for ground roads.

Considering the above research strategies and the characteristics of the ground road network, this paper proposes the selection strategy shown in Fig. 3. On the ground roads in urban areas, the intersections are highly correlated. The basis for selecting the lanes with intersections alone is sufficient to support the subsequent prediction. This method not only takes the adjacent lanes below the same intersection into consideration, but also selects lanes in the upstream and downstream intersections, including straight, and often overlooked left-turn and right-turn lanes. This strategy is based on the physical connection structure of the road, and perfectly captures the information in lanes. In this way, it not only can improve the prediction accuracy, but also broaden the application range.

Definition 2: Flow Prediction. After the lane traffic data is collected, we divide the training data set into n parts at fixed time intervals. For convenience, this article records the total number of lanes under k intersections as m. Each lane has h- dimensional attributes. The traffic characteristic parameters can be denoted by an n × h × m matrix T^±. In detail $T_q^{+}$ denotes the forward volume, and $T_s^{+}$ denotes forward speed. The predicted goal can be described as: (1)${\displaystyle \left[X_{n+1}^{q+},X_{n+2}^{q+},\dots ,X_{n+\tau }^{q+}\right]=\mathrm{f}\left(\mathrm{g}\left(T_q^{+}\right)\oplus \mathrm{p}\left(T_s^{+}\right)\right)}$

where g is the encoding processing function, p is the speed processing function, f is the final decoding and output function, X^q+ is the lane volume value, τ is the forecast time length.

ST-AFN framework

ST-AFN is mainly composed of three parts: deep speed processing network, the spatial encoder, and the temporal decoder. Deep bidirectional LSTM are connected in series to form the speed processing network. The encoder with the spatial attention blocks are utilized to analyze the spatial characteristics of the traffic parameters. Then the output of the above two networks are merged to build the information matrix which is the input of decoder. After the decoder extracting temporal feature, we use full connected layer (FC) to complete the final prediction, as shown in Fig. 4. The following will introduce each network in order.

Speed process network

The current traffic situation of the road section is closely related to its upstream and downstream sections. The upstream traffic flows toward the downstream section, and the state of the downstream section (such as congestion) will gradually accumulate, which in turn will impact the upstream traffic flow. In the direction of the time dimension, bidirectional LSTM is superimposed and fused by forward LSTM and backward LSTM. While overcoming the problems of gradient disappearance and gradient explosion, it also considers the forward and backward propagation of sequential data.

As shown in the Fig. 5, ${\overleftarrow{h}}_{ve}^{u_l}$ is derived from the forward LSTM, and ${\overrightarrow{h}}_{ve}^{u_l}$ is derived from backward LSTM. They represent the forward hidden state and backward hidden state in u-th unit. Correspondingly, ${\overrightarrow{c}}_{ve}^{u_l}$ and ${\overleftarrow{c}}_{ve}^{u_l}$ represent the forward cell state and backward cell state respectively. After each subunit completes forward and backward propagation, ${\overrightarrow{h}}_{ve}^{u_l}$ and ${\overleftarrow{h}}_{ve}^{u_l}$ are concatenated to form $h_{ve}^{u_l}$. (2)${\displaystyle H_{ve}^u=W_{ve}^u\cdot {\left[h_{ve}^{u_1},h_{ve}^{u_2},\dots ,h_{ve}^{u_m}\right]}^{\mathrm{T}}+b_{ve}^u}$

where $h_{ve}^u$ is the result of splicing $h_{ve}^{u_l}$ (l = 1, 2, …, m), and then it is changed linearly to get the final hidden state $H_{ve}^u$.$W_{ve}^u$ is the weight matrix and $b_{ve}^u$ is the bias term. $C_{ve}^u$ is calculated at the same time, representing the final cell state. Then $C_{ve}^u$ and $H_{ve}^u$ are used for initialization in the next unit, and are collected to build the context vector.

The spatial encoder

When predicting traffic flow, the processing methods of time dependence and space dependence directly affect the accuracy of the experimental results. The application of attention mechanism can help networks to accurately analyze the dependencies between lanes at each moment in real time. After selecting the target lane, it pays more attention to lanes with high correlation, reduces the weight value of irrelevant lanes, and efficiently allocates weights dynamically and optimally in a parallel strategy. The proposed encoder network is shown in the Fig. 6.

The encoder with embedded spatial attention mechanism blocks is composed of another bidirectional LSTM network. The spatial blocks in the unit can be summarized as: firstly we transform query $Q_e^i$, key $K_e^i$, and value $V_e^i$to get ${\hat{Q}}_e^i$, ${\hat{K}}_e^i$, and ${\hat{V}}_e^i$. Secondly, we calculate the dot product and use Softmax function to normalize it. Finally, we get the weights. The attention mechanism formula in i-th spatial unit is: (3)${\displaystyle e_i^j=tanh\left(W_{ae}^i\cdot \left[H_{qe}^{i-1},C_{qe}^{i-1},X_i^{q_j^{+}}\right]+b_{ae}^i\right)}$

where $H_{qe}^{i-1}$ is the final hidden state of i − 1- th spatial unit, $C_{qe}^{i-1}$ is the final cell state. $X_i^{q_j}$ represents the volume of j- th lane in current unit. $W_{ae}^i$ and $b_{ae}^i$ respectively represent the weight item and bias item, tanh is one of the activation functions.

Now, we have obtained $e_i^j$ for each lane. Then a softmax function is used for the normalization, and get attention weight ${\alpha }_i^j$ for each candidate lane. Finally we weight the original data $X_i^{q_j^{+}}$ with ${\alpha }_i^j$ to get ${\hat{X}}_i^{q^{+}}$, which will be the input data of i- th Spatial Unit.

The temporal decoder

The temporal attention mechanism blocks are embedded in the decoder network, they are employed to distinguish the importance of each period and assign the temporal weights. Context vector θ is the input of the decoder network. As shown in Fig. 7, speed encoded consists of hidden layers of each subunit in the speed processing neural network. Similarly, volume encoded consists of hidden layers of each subunit in the encoder neural network. The batch_size in the figure means the number of batches per training, num_layer is the number of network loop iterations, T is the time step required for prediction, and input_size represents the number of lanes.

where $H_{qe}^n$ and $H_{ve}^n$ is the final hidden state in each unit of speed process network and the encoder network respectively. H_ve and H_qe are the matrices formed by the final hidden state. W_qv is the weight item and b_qv is the bias item in the linear transform.

As shown in the Fig. 8, the input of the i′- th temporal unit is θ_i′. We mark the final hidden state and final cell state as $H_{qd}^{i^{\prime }-1}$ and $C_{qd}^{i^{\prime }-1}$ respectively. Temporal attention mechanism formula can be written as: (5)${\displaystyle d_{i^{\prime }}^k=tanh\left(W_{ad}^{i^{\prime }}\cdot \left[H_{qd}^{i^{\prime }-1},C_{qd}^{i^{\prime }-1},{\theta }_{i^{\prime }}^k\right]+b_{ad}^{i^{\prime }}\right)}$

where $W_{ad}^{i^{\prime }}$ is the weight matrix and $b_{ad}^{i^{\prime }}$ is the bias item, which are all the learnable parameters.

After obtaining $d_{i^{\prime }}^k$ for each time, we apply Softmax function to normalize it, and get attention weight ${\beta }_{i^{\prime }}^k$. Finally we weight the original data θ_i′ with ${\beta }_{i^{\prime }}^k$ to get ${\hat{\theta }}_{i^{\prime }}$. Then ${\hat{\theta }}_{i^{\prime }}$ is used together with historical real traffic flow of target lane $y_{his}^{i^{\prime }}$ to get ${\hat{y}}_{his}^{i^{\prime }}$: (6)${\displaystyle {\hat{y}}_{his}^{i^{\prime }}=W_{f,d}^{i^{\prime }}\left[\begin{array}{c}{\hat{\theta }}_{i^{\prime }}\hfill \\ y_{his}^t\hfill \end{array}\right]+b_{f,d}^{i^{\prime }}}$ According to the above method, each unit in decoder network iteratively allocates temporal weights until the N- th unit. N is the size of the dimension of θ(excluding the temporal dimension). Finally, we concatenate $H_{qd}^N$ and ${\hat{\theta }}_N$, and feed the matrix into fully connected (FC) layer to get the result of predicted traffic volume $q_{tag}^{i^{\prime }+\tau }$: (7)${\displaystyle q_{tag}^{i^{\prime }+\tau }=W_{fd}^N\left[\begin{array}{c}{\hat{\theta }}_N\hfill \\ H_{qd}^N\hfill \end{array}\right]+{\hat{b}}_{fd}^N}$

The training process

In the training process of ST-AFN, training data are normalized and scrubbed to make up each batch, and the learning rate is adjusted using ReduceLROnPlateau: the maximum tolerance threshold is set, and the learning rate is dynamically adjusted downward using the loss value in each epoch as an index. The specific process is as follows:

Code 1. The training method of ST-AFN

Evaluation

In this chapter, we use the real-world traffic data to evaluate the proposed ST-AFN and the benchmark algorithms. The experiments run on 64G Ubuntu 18.04 system, which is equipped with Intel Xeon Silver and NVDIA Quadro M4000. The hyperparameters in each experimental group were set uniformly: the learning rate is set to 0.001, batch size is set to 128, the epoch size is set to 100.

Data set

The real traffic data required for the experiment are collected in Xiaoshan District, Hangzhou, China from July 1, 2017 to October 1, 2017, for a total of 123 days. These data are collected by surveillance cameras at various intersections. The main format of the original data is shown in Table 1 (At a 4-lane intersection, lane 1 usually refers to the left-turn or U-turn lane, and lane 4 is the right-turn lane. Similarly, at a 5-lane intersection, lane 5 is the right turn lane).

Because the original data are relatively scattered, we filter out the required vehicles based on the location attributes of the vehicle’s departure time and the camera location, and eliminate the error data (the departure time is earlier than the entry time, the license plate number is empty, etc.). The distance is a fixed value, and the vehicle travel time can be obtained from the entry and departure time. What’s more, the average speed is calculated by dividing the distance between intersections by the driving time. Then we filter out the data of each lane in each intersection.

As shown in Fig. 9A, each lane under the intersection of Tonghui Road-Changyuan Road and the intersection of Shixin Road-Boxue Road is used as the target lane in the two experiments, and the remaining lanes under the road participated in the experiment as candidate lanes. Data is counted at five-minute intervals (sum the flow in five minutes and average the speed), so each detector can generate 288 records per day. Then we draw a curve according to the obtained characteristics parameters to judge the noise points and make it as smooth as possible. A large amount of data is sufficient to support the training and testing of the model. In order to reduce the interference caused by noisy data, abnormal points are repaired by adjacent normal records. Then the StandardScaler method is applied to the repaired data.

The data of Xiaoshan District, Hangzhou is used as the main experimental data and to prove ST-AFN’s portability, we collect real traffic data from Qingdao for verification. The data is provided by the traffic big data competition held in Qingdao in October 2019. The original data is mainly composed of laneID, speed, cameraPos, timestamp, etc., from which we selected an arterial with different directions and adjacencies as the data set. The processing strategy is similar to the above, data is counted at five-minute intervals.

As shown in Fig. 9B, each lane under the intersection of Jinshui Road-Dongchuan Road is used as the target lane.

The lanes selected for each of the three experimental datasets are presented in Table 2, arranged from top to bottom in the order in which they connect each road in a north-south direction. And in the Jinshui Road experiment set, they are arranged form left to right. The intersections where the target lanes are located in each experiment are marked by ‘*’ (e.g., *Changyuan Road-N), and the N, E, and W after the intersections are directional characters (e.g., -N for the North intersection, -N&E&W for the North, East, and West intersections).

Baseline methods

To compare with our model, serval parametric, non-parametric, and neural network models were selected as baseline methods (including a state of the art lane-level forecasting method).

• SVR: Support vector regression is a common method in traffic flow prediction. As an important branch of support vector machines, their difference is mainly in the evaluation index. The optimal hyperplane in SVM allows multiple categories to be the farthest apart, while the optimal hyperplane for SVR is the one that minimizes the sum of deviations of all sample points

• ARIMA: Autoregressive integrated moving average is one of the time series forecasting methods. It contains an auto regressive block and a moving average block.

• LSTM: LSTM composed of input gates, output gates, and forget gates is an improved version of Recurrent Neural Network (RNN).

• TM-CNN: TM-CNN (Ke et al., 2020) refers a two-stream multi-channel convolutional neural network. Firstly the authors convert traffic speed data and volume data into matrices. Then they expand each lane data into each channel matrix, as the input of CNN. And obtain the final result after fusion.

• FDL: FDL (Gu et al., 2019) refers a fusion deep learning model. Firstly, entropy-based gray relational analysis method is utilized to judge the dependency between lanes. Then they utilize the LSTM-GRU structure network to complete the lane-level speed forecast. It is one of state of the art models in lane level prediction.

Performance metric

In this study, we evaluate the prediction results with Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE). (8)${\displaystyle MAE=\frac{1}{N}\sum _{i=1}^N\left|{\hat{y}}_i-y_i\right|}$ (9)${\displaystyle RMSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left({\hat{y}}_i-y_i\right)}^2}}$ (10)${\displaystyle MAPE=\frac{100\text{\%}}{N}\sum _{i=1}^N\left|\frac{{\hat{y}}_i-y_i}{y_i}\right|}$

where N is the number of forecast targets, y_i is the history value and ${\hat{y}}_i$ is the forecast result.