Accelerating Deformable Convolution Networks with Dynamic and Irregular Memory Accesses

2.1 Deformable Convolutional Networks

Deformable convolution may sample arbitrary locations of the input features for convolution such that it can capture the objects with scale or transformation. The unique feature makes it attractive in visual recognition tasks with geometric variations such as lighting and rotation. There have been many deformable convolution architectures proposed recently [1, 14]. They typically include two standard convolution operations. The first convolution calculates the indices for the input feature sampling while the second convolution convolves on the sampled features. Usually, a BLI operation is used to bridge the two convolution operations. It approximates the input features based on the non-integer indices generated by the first convolution and provides the features as inputs to the second convolution. The structures of DCNs mainly differ on the index reuse and can be roughly divided into two categories. The first category of DCNs has a unique index for each data in the feature plane and the indices are reused across the different channels [14]. The second category of DCNs also has the indices shared across the feature channels, but it has a unique index for each data in each convolution window [1]. Basically, the same data in the feature plane has different indices when it is located in different convolution windows. The second category of DCNs requires a larger convolution to calculate more sampling indices and produces more deformed features than the first category of DCNs. The two categories of DCNs are abbreviated as DCN-I and DCN-II, respectively.

2.2 Unified Deformable Convolution Model

The different deformable convolution can be represented in a unified model as formulated in Equations (1)–(3). Basically, it has a convolution operation to determine the deformed locations or indices of the input features in the first step. This step is formulated in Equation (1) where \(\boldsymbol {X}\), \(\boldsymbol {W_1}\), \(\boldsymbol {b_1}\), and \(\boldsymbol {y^{^{\prime }}}\) refer to the input feature, weight, bias, and output of the first-stage convolution, respectively. As each convolution window across the input feature has an independent index and input features at different channels share the same index, a 3D (\(L \times H \times D\)) data is utilized for \(\boldsymbol {y^{^{\prime }}}\). H and D stand for the number of convolution windows on X dimension and Y dimension, respectively. \(L=2 \times K \times K\) stands for indices at different window locations and axis dimensions. Suppose \(\alpha _{m}\) and \(\beta _{n}\) are the corresponding coordinates in X axis and Y axis, respectively, and they are located at the same position and the same convolution window. \(\alpha _{m}\) and \(\beta _{n}\) are not integers, so they cannot be used to retrieve the input features directly for the deformable convolution. To that end, a bilinear interpolation approach is utilized to calculate the deformed features using the neighboring features around the location (\(\alpha _{m}\), \(\beta _{n}\)) in the second step. The calculated feature data \(x_{c,\alpha _{m},\beta _{n}}^{^{\prime }}\) can be obtained using Equation (2) where \(BLI(.)\) refers to the bilinear interpolation function, c is the channel index of the feature data, \(\Delta \alpha _{m}=\alpha _{m}- \lfloor \alpha _{m} \rfloor\) and \(\Delta \beta _{m}=\beta _{m}- \lfloor \beta _{m} \rfloor\). \(x_{c,\alpha _{m},\beta _{n}}^{^{\prime }}\) is an element of the deformed feature \(\boldsymbol {X^{^{\prime }}}\). A vivid description of the BLI function can be found in Figure 2. When the input features are retrieved and organized according to the deformed indices, the deformable convolution can be obtained using a standard convolution over the reorganized features as shown in Equation (3) where \(\boldsymbol {W_2}\), \(\boldsymbol {b_2}\), and \(\boldsymbol {y}\) refer to weights, bias, and output feature of the second convolution operation in a deformed convolution: (1) \(\begin{equation} \begin{aligned}\boldsymbol {y^{^{\prime }}}=\boldsymbol {X} \ast \boldsymbol {W_1} + \boldsymbol {b_1}, \end{aligned} \end{equation}\) (2) \(\begin{equation} \begin{aligned}x_{c,\alpha _{m},\beta _{n}}^{^{\prime }}&= F_{BLI}(x_{\left\lfloor \alpha _{m} \right\rfloor ,\left\lfloor \beta _{n} \right\rfloor },x_{\left\lfloor \alpha _{m} \right\rfloor ,\left\lceil \beta _{n} \right\rceil },x_{\left\lceil \alpha _{m} \right\rceil ,\left\lfloor \beta _{n} \right\rfloor },x_{\left\lceil \alpha _{m} \right\rceil ,\left\lceil \beta _{n} \right\rceil },\Delta \alpha _{m},\Delta \beta _{n})\\ &=\left(1-\Delta \alpha _{m} \right)\left(1-\Delta \beta _{n} \right)x_{\left\lfloor \alpha _{m} \right\rfloor ,\left\lfloor \beta _{n} \right\rfloor }+\left(1-\Delta \alpha _{m} \right)\Delta \beta _{n}x_{\left\lfloor \alpha _{m} \right\rfloor ,\left\lceil \beta _{n} \right\rceil }+\Delta \alpha _{m}\left(1-\Delta \beta _{n} \right)x_{\left\lceil \alpha _{m} \right\rceil ,\left\lfloor \beta _{n} \right\rfloor }\\ &\;\;\;\;+\Delta \alpha _{m}\Delta \beta _{n}x_{\left\lceil \alpha _{m} \right\rceil ,\left\lceil \beta _{n} \right\rceil }, \end{aligned} \end{equation}\) (3) \(\begin{equation} \boldsymbol {y}=\boldsymbol {X^{^{\prime }}} \ast \boldsymbol {W_2} + \boldsymbol {b_2}. \end{equation}\)

Fig. 2.

View Figure

2.3 Neural Network Accelerator Redesigning

The great success of neural networks in massive domains of applications inspires considerable efforts devoted to developing neural network accelerators [20, 21, 22, 23, 24, 25]. In spite of the notable efforts, newer network operations proposed for higher performance may go beyond the capability of the existing neural network accelerators. Although it is usually possible to offload the unsupported operations to the attached GPPs while leaving the rest of the conventional neural network operations on the accelerator, the performance of the co-designed implementation may drop dramatically due to the massive data communication between GPP and the accelerator. Also complex operations offloaded to GPPs may still become the performance bottleneck due to the insufficient computing capability of GPP and degrade the overall neural network execution. Thereby, the neural network accelerators are usually redesigned to meet the requirements of the new neural network operations on top of the existing neural network accelerators. For instance, unified neural network accelerators are also proposed to perform deconvolution used in generative adversarial neural networks other than the conventional convolution [26, 27, 28, 29]. A novel accelerator is developed to support 3D neural networks in References [30, 31, 32]. The authors in Reference [33] proposed to add a bilinear interpolation calculation module to an existing ReRAM neural network accelerator to enable in-situ DCN calculation.

Inspired by prior works, deformable convolution networks that cannot be fitted to existing neural network accelerators also attract efforts for specialized accelerator designs [17, 18] as illustrated in Section 1. Nevertheless, they mainly proposed to limit the random sampling in deformable convolution to ensure better data locality and smaller memory access footprints. Although the limitation is beneficial to the hardware design, it compromises the model accuracy, which is the major design goal of DCNs. Table 1 reveals the model accuracy comparison between native DCNs [1] and the DCNs with additional sampling constraints proposed in References [17, 18]. We have ShuffleNet V2, Faster R-CNN, and VGG16 used for the comparison. Twelve epochs, 100 epochs, and 20 epochs are used for the training of Faster R-CNN, ShuffleNet V2, and VGG16, respectively. It can be observed that there are generally 1% to 2% accuracy loss, which is non-trivial. In addition, the sampling constraints are coupled with the hardware designs, so the model training must take the hardware design parameters into consideration, which also blurs the hardware and software boundary and complicates the model development. Unlike prior works, we aim to develop a DCN accelerator to enable native deformable convolution efficiently without any accuracy penalty or model training constraints.

In addition, since existing neural network accelerators are mostly fixed point and DCNs need to be quantized to take advantage of the hardware acceleration, we apply 8bit fixed point quantization to the native DCNs with the approach proposed in Reference [34] and evaluate the influence of quantization on the model accuracy. The quantized model accuracy is presented in the last column of Table 1, it can be observed that 8 bit fixed point quantization has negligible influence on the prediction accuracy of DCNs.

4.1 Overall Accelerator Architecture

Deformable convolution is the major barrier that hinders the deployment of DCNs on existing neural network accelerators. Thereby, taming the deformable convolution to existing neural network accelerators is key to accelerate DCNs. As formulated in Section 2.2, deformable convolution consists of three processing stages including convolution, BLI, and convolution. Since the convolution operations in DCNs can be deployed on existing neural network accelerators directly, the major challenge of DCN acceleration is to optimize the BLI operation that samples the input features according to the irregular indices calculated with the first convolution in DCNs and conducts the BLI calculation based on the sampled input features. Since the indices depend on the input features and thus change at runtime, the BLI sampling leads to complicate memory access patterns. To address this dynamic and irregular memory access problem, we have BLI divided into tiles and track the tile dependency at runtime with a TDT. On top of the TDT, we have a tile scheduler to decide the order of the output tile execution and the input tile loading at the same time such that the tiles loaded to the on-chip buffers can be fully utilized and the memory accesses to the external memory can be reduced. As for the BLI calculation, we reorganize it to multiple small vector-based dot product and have the processing performed in parallel on top of the 2D computing array in the neural network accelerator for the sake of higher performance.

Fig. 5.

View Figure

The proposed DCN accelerator architecture is shown in Figure 5. The components without filling any color belong to a classic neural network accelerator while the rest of the components filled with grey are designed specially for the deformable convolution. The entire accelerator is generally controlled with a sequence of instructions compiled from the target neural networks. The instructions are stored in the instruction buffer and decoded at runtime to generate control signals for the entire accelerator. With the controlling, neural network operations are mapped and executed on the regular 2D computing array. For the convolution operations, weights from different filters are streamed to the different columns of the computing array from top to bottom in parallel while input features are streamed from left to right in different rows according to the output stationary data flow proposed in Reference [19].

For the deformable convolution, it is divided into three dependent operations. The first convolution operation starts when inputs and weights are ready in the on-chip buffers. The outputs of the first convolution are indices and will be utilized to sample from the input features. Since they are not integers and cannot be utilized to retrieve the input features directly, we have them stored in an index buffer and have an address converter to obtain the four neighboring integer indices. Meanwhile, BLI coefficients as proposed in Section 4.2 are generated with a coefficient calculation block at the same time. To this end, we can start the BLI calculation with the retrieved input features and BLI coefficients by taking advantage of the 2D computing array of the accelerator. The output of the BLI are essentially deformed features and will be utilized as inputs of the second convolution. While the address conversion and the BLI calculation are conducted in pipeline manner, the output buffer and the index buffer are separated in case of conflicts. After the BLI calculation, the deformed features will be stored and swapped as inputs of the computing array for the second convolution. When the features or weights exceed the on-chip buffers, BLI and the second convolution need to be tiled and fused to avoid intermediate data exchange through the external memory. In addition, we have a TDT to keep a record of the tile dependency based on the generated indices and have a runtime tile scheduler to optimize the output tile execution and input tile loading ordering based on the TDT to enhance the on-chip data reuse and memory access efficiency.

4.2 BLI Implementation

To implement BLI, we have an address converter module to convert the original non-integer indices to neighboring integer addresses of the input feature buffer and a coefficient calculation block to produce the BLI coefficients at the same time. Each feature requires four coefficients \(\eta , \mu , \theta , \gamma\), and they can be calculated with Equation (2). The BLI coefficients are stored in the weight buffer while the converted buffer addresses are used to retrieve features from the the input buffer directly. The retrieved features and coefficients read from the weight buffer will then be fed to the 2D computing array following a standard weight stationary data flow [19] for the BLI computing. Details of the processing will be illustrated in the rest of this section.

BLI Mapping: Each deformed feature depends on four neighboring input features and it can be viewed as a vector-based dot product. One of the vector includes four BLI coefficients and the other vector includes four neighboring input features as shown in Figure 6. Each deformed feature processing can be mapped to four neighboring PEs organized as a cluster with a weight-stationary data flow. Since weights of the BLI are shared among the different input feature channels, they can be distributed to the PEs in the same cluster and broadcast to different clusters. The corresponding four input features in the same channel will be streamed in parallel for the multiplication among PEs in the same cluster. Features from different channels will be distributed to the different clusters of the computing array for higher throughput, but additional wires from wide input feature buffers to the PEs across the computing array are required accordingly. The clustered computing array on top of the original 2D regular computing array is shown in Figure 7. Unlike the output of the conventional computing array that are aligned in column, the clustered computing array output are extracted and aligned in cluster. Thereby, a DEMUX is added to the output port of each PE cluster to extract the output from the computing array when BLI is mapped.

Fig. 6.

View Figure

Fig. 7.

View Figure

Input Feature Layout: To make best use of the entire computing array, the neighboring input features must be fed to the computing array continuously. However, when the input features are sequentially stored in a single port buffer, it can not load the four features in different rows and columns of the input features from the on-chip buffer in a single cycle simply with wider read port. A four-port on-chip buffer can meet the computing requirement, but it is extremely resource-consuming in terms of both power and chip area. To address the problem, we modify both the input feature layout and the structure of the input buffer as shown in Figure 8. Since the four features for each BLI processing of a deformed output feature are located in adjacent rows and columns, we separate the input features into four partitions based on the feature coordinate parity in the feature map and the four features will always be located in different partitions. Accordingly, we have the buffer divided into four banks and each bank accommodate an input feature partition. Four features required by the BLI processing of any feature can be loaded in a single cycle. In addition, we have the input features stored in channel-major order and widen the port of the buffer bank such that features of multiple channels are read at the same time for all the different PE clusters.

Fig. 8.

View Figure

Address Converter: To fetch neighboring features for the BLI, we need to calculate the buffer addresses of the four input features based on the non-integer indices. The basic idea is to obtain the four neighboring integers of the non-integer feature indices first and then deduct the base index of the four features to calculate the on-chip buffer addresses of the required features. Essentially, it is a conversion from 3D feature map indices to the 1D on-chip buffer indices and the higher dimension indices including the channel index and the height index need to be scaled accordingly as formulated in Equation (4). \(\alpha _{m}\) and \(\beta _{n}\) denote the original non-integer feature indices, i.e., coordinates in the 2D feature plane. \(index_{lb}\), \(index_{rb}\), \(index_{lt}\), and \(index_{rt}\) denote the four buffer addresses of the features located at the left bottom, right bottom, left top, and right top, respectively. \(\boldsymbol {A}\) denotes the number of PEs in the computing array in the neural network accelerator, \(\boldsymbol {H}\) denotes the height of the input feature map, \(\boldsymbol {c}\) denotes the channel number of the input feature, \(T_{0}\) denotes the base index of the four neighboring features. As the address conversion in different channels are the same, the formulation only illustrates the conversion in the 2D feature plane. To enable runtime BLI, we have a specialized address converter module implemented. It can be easily pipelined as shown in Figure 9. The generated indices will be aligned and sent to the different input buffer banks to retrieve the corresponding four features for the BLI calculation: (4) \(\begin{equation} \left\lbrace \begin{matrix}index_{lb}=\left(\left\lfloor \left\lfloor \beta _{n} \right\rfloor /2 \right\rfloor \times j+\left\lfloor \left\lfloor \alpha _{m} \right\rfloor /2 \right\rfloor \right) \times i-T_{0}\\ index_{rb}=\left(\left\lfloor \left\lfloor \beta _{n} \right\rfloor /2 \right\rfloor \times j+\left\lfloor \left\lceil \alpha _{m} \right\rceil /2 \right\rfloor \right) \times i-T_{0}\\ index_{lt}=\left(\left\lfloor \left\lceil \beta _{n} \right\rceil /2 \right\rfloor \times j+\left\lfloor \left\lfloor \alpha _{m} \right\rfloor /2 \right\rfloor \right) \times i-T_{0}\\ index_{rt}=\left(\left\lfloor \left\lceil \beta _{n} \right\rceil /2 \right\rfloor \times j+\left\lfloor \left\lceil \alpha _{m} \right\rceil /2 \right\rfloor \right) \times i-T_{0} \end{matrix}\right. \; \; \; \; \; \begin{pmatrix}i=\left\lceil c/(A/4) \right\rceil \\ j = H/2 \end{pmatrix}. \end{equation}\)

Fig. 9.

View Figure

Coefficient Calculation: To enable runtime BLI, coefficients also need to be calculated at runtime and they are formulated in Equation (5) according to Equation (2). The four coefficients can be reused across the different channels, so the formulation only illustrates the calculation in the 2D feature plane. We notice that the multiplication result \(\Delta \alpha _{m}\Delta \beta _{n}\) is required by all the four coefficient calculation, so it is calculated first. Then the rest of the coefficient calculation can be conducted with only addition and subtraction. The pipelined architecture is shown on the left of Figure 9. The coefficients will be stored in the weight buffer for the BLI calculation according to the BLI mapping: (5) \(\begin{equation} \begin{aligned}\left\lbrace \begin{matrix} \eta =\left(1-\Delta \alpha _{m} \right)\left(1-\Delta \beta _{n} \right)\\ \mu =\left(1-\Delta \alpha _{m} \right)\Delta _{n}\; \; \; \, \; \; \; \; \; \; \; \\ \theta =\Delta \alpha _{m}\left(1-\Delta \beta _{n} \right)\; \; \; \; \; \; \; \; \\ \gamma =\Delta \alpha _{m}\Delta \beta _{n}\: \: \: \: \: \: \: \: \: \: \: \: \: \: \: \; \; \; \; \; \, \end{matrix}.\right. \end{aligned} \end{equation}\)

4.3 Runtime Tile Scheduling

To address the dynamic and irregular memory access problem in deformable convolution, we propose to track the data dependency at runtime and optimize the execution with runtime scheduling based on the tracked data dependency. The dependency tracking and scheduling optimization will be illustrated in the rest of this sub section.

Tile Dependency Tracking: To track the dynamic memory accesses, we need a dependency table to record all the required input features for each deformable convolution output feature. Due to the limited on-chip buffer in the accelerator, neural network processing is usually tiled and the dependency table is constructed in the granularity of a tile accordingly. The tile dependency table is abbreviated as TDT. The dependency of the deformable convolution output tile to the input feature tile can be inspected with the deformed feature indices as described in Figure 10. Assume both the input and output features are divided into fixed \(5 \times 5\) tiles. The feature indices, i.e., \(\alpha _{m}\) and \(\beta _{n}\) are compared to the different boundary indices and the comparison result vectors can be used to determine the row index and the column index of the tile.

Fig. 10.

View Figure

As shown in Figure 10, the comparison result vector for \(\alpha _{m}\) is (1,1,0,0) and it means \(\alpha _{m}\) is between 0.4H and 0.6H. With a decoder, we can obtain the row index of the dependent input tile. In this example, the row index is 2. Similarly, we can also obtain the column index of the dependent input tile and it is 1 in this example. Given the row index and the column index, we can decide the dependent input tile index and it is 11 in this example as highlighted with red color. With this index, we can further determine the dependent bit vector of which each bit refers to the dependency of the indexed input tile. If the corresponding input tile is dependent, then it will be set to 1, while the remaining bits are set to 0. By continuously inspecting all the deformed features required by an output tile and conducting OR operation with the bit vectors, we can obtain the entire tile dependency vector of an output tile. The tile dependency table is constructed right after the first convolution of the deformable convolution. Although all the coefficients required for the tile tracking can be generated based on the total tile size and configured at runtime, the maximum tile size remains limited by the hardware component setups such as TDT and dependency vector length.

Tile Scheduling: Input tiles loaded to the on-chip buffers can be reused among the different output tile calculation, but both the ordering of the input tile loading and output tile execution can affect the tile reuse especially when the input buffer is limited and some of the input tiles have to be replaced. The input tile utilization varies greatly during the DCN processing as observed in the motivation section, which further aggravates the influence of the ordering of the input tile loading and the output tile execution. To address the above problem, we propose a unified tile scheduling algorithm that handles both the output tile scheduling and input tile scheduling based on the TDT that is updated at runtime.

The proposed scheduling algorithm is presented in Algorithm 1. It includes an output tile scheduling procedure and an input tile scheduling procedure based on the output tile scheduling result. For the output tile scheduling, it essentially selects an output tile that can reuse the input tiles that are already loaded and stored in the on-chip buffer. When the on-chip buffer is empty, it simply selects the output tile that requires the most input tiles, which are more likely to be reused. When the output tile is selected, we need to determine the loading order of the dependent input tiles of the selected output tile. Although it is possible to sort all the dependent input tiles based on the its potential reuse, but the input tile reuse is expensive to estimate at runtime. In this work, we have the dependent input tiles divided into three parts as illustrated in the \(input\_tile\_scheduling(.)\) function. The first part is the input tiles that are already stored in on-chip buffers. They will be scheduled first to ensure the on-chip data reuse. It can be determined by comparing the input tile on-chip status bit vector OC and the dependency bit vector \(B[nextID]\). The second part is the tiles that will be reused by the next output tile calculation. They will be loaded at last such that they can reside in the on-chip buffer for reuse. The next output tile calculation is obtained with the procedure \(output\_tile\_scheudling(.)\). By comparing the current output tile dependency bit vector and the next output tile dependency bit vector, we can determine the overlapped input tiles that can be reused, but the tiles that are already stored in the on-chip buffer will be removed. The rest of the input tiles will be scheduled between the first part and the second part. As the input tile reuse are already considered in the scheduling, a first in first out (FIFO) strategy is used for the input tile replacement for efficient hardware implementation.

The proposed tile scheduling is implemented with customized hardware rather than a software scheduling algorithm on CPUs to ensure efficient tile-based execution. The tile scheduling module is shown in Figure 11. It mainly depends on the TDT to select the next output tile for execution on the computing array in the accelerator. The basic idea is to choose the output tile that has the most dependent input tiles overlapped with that required by current output tile. Thus, we have the dependency bit vector of current output tile AND with the bit vectors of all the un-executed output tiles. The AND result will be sent to an non-zero (NZ) bit counter module that mainly consists of an adder tree to count the number of the non-zero bits. The number will pass through a pipelined comparator to determine the maximum value. The corresponding output tile has the most input tiles overlapped with that of the current output tile, so it will be scheduled for the execution next. Instead of having the output tile scheduling and the execution conducted sequentially, we adopt a pre-scheduling strategy that performs the next output tile scheduling in parallel with the current output tile execution. Since the execution does not have to wait for the immediate scheduling result, more complex scheduling algorithm can be implemented. When the next output tile is selected, we will schedule the dependent input tiles. The input tile scheduling mainly depends on three hardware-friendly bit-wise operations, which divide the input tiles into three parts as already discussed in Algorithm 1. By inspecting the non-zero bit number of the three resulting bit vectors with corresponding NZ bit counters, we can determine the IDs of the input tiles in each partition and push them into three independent queues. As each queue belongs to different scheduling priorities, they can be scheduled sequentially and the input tile scheduling is completed when all the queues are empty.

Fig. 11.

View Figure

4.4 BLI and Convolution Fusion

We notice that massive data movement is required when the different processing stages of the deformable convolution is performed sequentially due to the limited on-chip buffer. Inspired by the neural network fusion techniques [36], we try to fuse the different processing stages such that the intermediate data can be reused via on-chip buffers without additional external memory accesses. Basically, we have the input data of the upstream processing stage tiled. When a tile of the output data is obtained in the upstream processing stage, they will be used by the downstream processing stage immediately by simply swapping the input buffer and output buffer. Compared to the stage by stage processing, the fused processing on top of the tiling avoids transferring the intermediate data to and back from the external memory, which is beneficial to both the performance and energy efficiency. While the amount of the deformed indices is usually small compared to the feature data and can be buffered on-chip directly, we mainly have the second processing stage and the third processing stage tiled and fused in practice.

5.1 Experiment Setup

Hardware Platforms: The proposed DCN accelerator is implemented with Verilog and synthesized with Synopsys Design Compiler under TSMC 40 nm library. It works at 800 MHz. The configurations of the DCN accelerator are shown in Table 2. Processing elements in the accelerator adopt 8 bit fixed point. We also have DCN implemented on a set of different architectures including an ARM processor (ARM), ARM+TPU, GPU, DCN Accelerator (DCNA), MEDCN [37], and DSEDCN [38]. MEDCN and DSEDCN are also customized DCN accelerators, but they target at FPGAs, which are difficult to compare directly. In this work, we utilize the GOPS presented in their work and scale them to the same clock used in DCNA for a fair performance comparison. The ARM processor is ARM-A7@900 MHz equipped with 1 GB DRAM (DDR3), which is the core of Raspberry Pi 3. The GPU is 256-core NVIDIA Pascal GPU with 8 GB GDDR5 memory and it is the core of Nvidia TX2. Experiments on the ARM processor and the GPU were implemented with PyTorch 1.3 on real platforms, i.e., Raspberry Pi and TX2, respectively. Experiments for ARM+TPU were conducted in a mixed manner. The second stage of the deformable convolution is not supported by TPU and it was performed on the ARM processor instead, while the rest of the neural networks was performed on TPU and evaluated with Scale-Sim [39]. The configurations of the TPU architecture are the same with that used in the DCN accelerator. In addition, both the ARM processor and TPU have 1 GB DRAM equipped. The average power of the ARM processor is 1.3 W and its idle power is 0.3 W. To evaluate the power consumption of DRAM, we accumulate the power consumption of the different memory operations such as Activation (ACT), Read (RD), Write (WR), and Background (BG) power based on Table 3 according to Micron’s power calculators [40].

Neural Network Benchmark: To evaluate the proposed DCNA, we have two typical neural network models including VGG19 [41] and SegNet [42] used as our benchmark. Deformable convolution can be used to replace any convolution in neural networks, but the replacement configurations can lead to different trade-offs between computation and model accuracy. In this case, we have three typical deformable convolution configurations set for each model and they are denoted as VGG19/SegNet-3, VGG19/SegNet-8, and VGG19/SegNet-F. As the convolution layers close to the output layer are usually smaller, we have deformable convolution placed from the output layer to input layer of the neural networks to minimize the deformable convolution induced computation. VGG19/SegNet-3 and VGG19/SegNet-8 represents that the last three convolution layers and the last eight convolution layers of VGG19/SegNet are replaced with the deformable convolution layers, respectively. VGG19/SegNet-F represents that all the convolution layers are replaced with deformable convolution layers. Details of the benchmarks are summarized in Table 4.

Fig. 12.

View Figure

5.2 Performance Evaluation

The performance of the DCN execution on the different computing architectures is normalized to that on an ARM processor and shown in Figure 12 while the wall time of the execution om ARM processor is detailed in Table 5. In general, DCNA achieves 515\(\times\) and 621\(\times\) higher performance on DCN-I and DCN-II, respectively, on average compared to a general ARM processor. DCN-II requires more sampling locations in deformable convolution operations of DCNs, so it has more computation and random accesses involved, which leads to larger execution time. Accordingly, higher performance speedup is achieved for DCN-II on DCNA. On the other hand, we notice that DCNA achieves much higher performance speedup on VGG19/SegNet-F compared to the other two configurations (i.e., VGG19/SegNet-3 and VGG19/SegNet-8) with less deformable convolution operations in the neural networks. The main reason is that the deformable convolution is rather challenging for the ARM processor due to the irregular memory accesses. The execution of deformable convolution operations on an ARM processor dominates the total DCN execution time and becomes the performance bottleneck of DCNs. In contrast, convolution with regular memory accesses is usually intensively optimized on ARM processors in PyTorch and the performance speedup achieved on customized accelerators is relatively lower. When DCNs are deployed on the ARM+TPU architecture, we can only have the convolution operations accelerated with TPU while the deformable convolution is still executed on the ARM processor. According to Amdahl’s law, the deformable convolution remains the performance bottleneck. Thereby, the performance speedup of the ARM+TPU architecture to the ARM processor is rather low especially for VGG19/SegNet-F. Unlike the ARM+TPU architecture, GPU in TX2 can have the entire DCNs implemented and the deformable convolution operations can also benefit from the GPU parallel processing due to its powerful support on general tensor operations. Thus, significant performance speedup is achieved compared to the ARM+TPU architecture. DCNA that is built on top of a conventional neural network accelerator has customized circuit design for both the standard convolution and the new deformable convolution outperforms GPU and exhibits 2.21\(\times\) performance speedup on average. Similar to DCNA proposed in this work, MEDCN also adopts the layer fusion technique to merge the second convolution and the deformed feature generation and reduce the memory accesses. The major difference between DCNA and MEDCN is the way to address the irregular memory accesses in DCNs. MEDCN mainly utilizes a specialized register array to address the irregular memory access induced on-chip bank conflicts. In contrast, DCNA proposed an odd/even IO/buffer suit to the parallel BLI data flow to address the on-chip data conflicts, and it has additional tile scheduling based on pre-computed indices to investigate the input feature locality. Hence, DCNA outperforms MEDCN in general. DSEDCN is built on top of MEDCN. In addition, the DCN is divided into different layers and allocate to different groups. The overall throughput of the DSEDCN is improved through load balancing and global pipeline. While the locality problem remains unresolved.

Fig. 13.

View Figure

5.3 Energy Consumption Evaluation

The energy consumption of DCNs on the four computing architectures normalized to the baseline ARM processor is presented in Figure 13 while the actual energy consumption of the ARM processor is shown in Table 6. Particularly, Figure 13 shows both the total energy consumption and the energy consumption distribution. Generally, DCNA with customized hardware acceleration shows the lowest energy and it is 612\(\times\) less than the ARM processor. The energy consumption benefit is attributed to both the much smaller execution time brought by the performance acceleration as illustrated in Figure 12 and the lower power consumption. For the ARM+TPU architecture, although TPU can accelerate the convolution operations and consumes little energy, it still consumes considerable time and energy for the deformable convolution operations on the ARM processor. GPU on Nvidia Jetson TX2 can have both the convolution and deformable convolution accelerated, so it greatly reduces the execution time but its energy consumption remains \(9 \times\) higher on average than DCNA due to the much higher power consumption. Moreover, we notice that the percentage of the DRAM energy consumption on VGG19/SegNet-F is larger than that on VGG19/SegNet-3 on DCNA. The main reason is that VGG19/SegNet-F with more deformable convolution involves many irregular memory accesses and lowers the memory access efficiency. Particularly, DCNA handles the irregular memory accesses with the granularity of a tile. Usually only a portion of the data in a tile is required and many of the data remain unused though the dependency is considered by the on-chip tile scheduler. In addition, some of the tiles may have to be repeatedly loaded due to the limited on-chip buffer and irregular data reuse. Thereby, the memory access efficiency is much lower than that of a standard convolution with regular memory access patterns. The lower memory accesses eventually lead to higher DRAM energy consumption.

5.4 Chip Area Evaluation

The baseline neural network accelerator can be roughly divided into on-chip Data Buffer (input feature/output feature/weight/bias buffer), PE Array, and the Original Control Logic while DCNA requires additional components including Index Buffer, Tile Dependency Table, and the Added Control Logic. The chip area of the different components is presented in Figure 14. It can be seen that DCNA induces only 6.6% chip area compared to the baseline design. Among the added hardware blocks, index buffer that usually needs to store the generated feature indices in a deformable convolution operation takes up the most chip area. In contrast, the additional control logic and the tile dependency table consume negligible chip area. Since the feature indices generated in deformable convolution is usually reused among the different channels, it is much less than the input/output features and weights. Hence, the index buffer size is much smaller compared to the data buffer in the baseline neural network accelerator.

Fig. 14.

View Figure

5.5 Optimization Evaluation

Tile Scheduling: To evaluate the influence of the proposed tile scheduling on the DCN performance, we compare it with a naive implementation without bit vector-based dependency tracking (W/O bit vector) and an implementation with bit vector-based tracking but without tile scheduling (W/ bit vector + W/O scheduling). The naive implementation executes the output features sequentially and loads all the dependent input tiles as needed due to the lack of the overall dependency information. The implementation with bit vector-based tracking but without tile scheduling sequentially executes all the output tiles, but it loads the dependent input tiles of the entire output tile rather than a single output feature based on the corresponding TDT. The loaded input tiles will also be used for the calculation of all the features in the output tile such that the loaded tiles can be reused as much as possible. The proposed implementation has both the bit vector-based tracking and tile scheduling considered (W/ bit vector + W/ scheduling). It optimizes the ordering of both the output tile execution and the input tile loading for more efficient data reuse. The comparison is shown in Figure 15. It can be seen that VGG19/SegNet-F with all the convolution layers replaced with deformable convolution benefits most from the tile-based dependency tracking and scheduling. In contrast, VGG19/SegNet-3 with only three small convolution layers replaced with the deformable convolution exhibits marginal performance speedup. It is mainly caused by two reasons. First, the computation of the deformable convolution operations takes up only a small portion of the entire neural network computation, so there is little space left for the performance improvement. Second, the sizes of the deformable convolution operations are small and the data including input features, weights, and output features can be mostly fully buffered. Hence, the tiles can be reused without any scheduling and the proposed tile scheduling shows minor performance improvement in this case. The performance improvement on DCN-I and DCN-II also differs. This is mainly caused by the fact that DCN-II involves more random sampling and thus benefits more from the DCNA acceleration in general.

Fig. 15.

View Figure

Fig. 16.

View Figure

On top of the performance improvement, we also evaluate the influence of the tile scheduling on the energy consumption of the DCN execution. The experiment result is shown in Figure 16. Again, it can be seen that VGG19/SegNet-F with most deformable convolution operations benefits most and shows the least energy consumption while VGG19/SegNet-3 with small deformable convolution operations has little optimization space for the tile scheduling. Generally, the energy reduction is mainly attributed to both the reduced execution time according to Figure 15 and the lower power consumption brought by the reduced memory accesses. As the performance improvement is already discussed in prior subsection, we mainly investigate the memory access reduction in this subsection. The total memory accesses issued by DCNA during the DCN processing is shown in Figure 17. By comparing the implementation without bit vector-based dependency tracking and the implementation with bit vector but no scheduling, we observe that the bit vector-based tile dependency tracking removes substantial memory accesses. It is mainly achieved by avoiding repeatedly loading the same input tiles required by the calculation of different output features in the same output tile. The scheduling further reduces the memory accesses by inspecting the input tile reuse among the different output tiles according to the tile dependency table. The experiment shows that the proposed scheduling reduces the memory accesses by 40.7% on VGG19/SegNet-F on average compared to the implementation with only bit vector-based dependency tracking.

Fig. 17.

View Figure

Tile Sizing: Tile size is an important design parameter and it determines the granularity of the tile dependency tracking and scheduling. Since most of the memory access latency in DCNA can be overlapped by the computing latency, tile size mainly affects the memory access efficiency and the DRAM energy consumption eventually, and has little influence on the performance. We take VGG19/SegNet-F with most deformable convolution operations as an example and analyzed the DRAM energy consumption under different tile sizes. The experiment result is shown in Figure 18. It can be seen that the DCN processing with smaller tile size benefits most from DCNA and consumes the least DRAM energy. The main reason is that smaller tile size allows more efficient tile dependency tracking and scheduling. Thus, the on-chip buffer utilization and DRAM memory access efficiency can be improved. In contrast, when the tile size is large, the dependent input features of each output tile can spread across all the input tiles. In this case, all the input tiles have to be repeatedly loaded for each output tile calculation, leaving little optimization space for the proposed tile dependency tracking and scheduling.

Fig. 18.

View Figure

Layer Fusion: To reduce the intermediate data transmission to and from DRAM, we propose to fuse the processing stages in each deformable convolution operation. The influence of the layer fusion on the energy consumption is shown in Figure 19. It can be seen that the fusion reduces the energy consumption by more than 20% on VGG19/SegNet-F with DCN-II structure. The main reason is that the memory access time in the two DCNs cannot be fully overlapped by the computation time due to the involved large deformable convolution operations and the fusion that avoids large intermediate data transmission improves the DCN performance other than the memory access efficiency. Unlike VGG19/SegNet-F, the deformable convolution operations in the rest scenarios are relatively small and the fusion only reduces the memory accesses while the performance has little improvement. As the DRAM energy consumption takes up only around 20% of the entire DCN energy consumption according to Figure 13, the energy reduction brought by the memory reduction is limited. Particularly, for VGG19/SegNet-3, the entire intermediate data can be mostly fully buffered and there is even little memory access optimization space left for the fusion. Thereby, the fusion shows little energy reduction.

Fig. 19.

View Figure