BlockNet: A Deep Neural Network for Block-Based Motion Estimation Using Representative Matching

Abstract

Owing to the limitations of practical realizations, block-based motion is widely used as an alternative for pixel-based motion in video applications such as global motion estimation and frame rate up-conversion. We hereby present BlockNet, a compact but effective deep neural architecture for block-based motion estimation. First, BlockNet extracts rich features for a pair of input images. Then, it estimates coarse-to-fine block motion using a pyramidal structure. In each level, block-based motion is estimated using the proposed representative matching with a simple average operator. The experimental results show that BlockNet achieved a similar average end-point error with and without representative matching, whereas the proposed matching incurred 18% lower computational cost than full matching.

1. Introduction

Motion estimation is a process that searches for movement between two sequential images. It is widely used in video applications such as video compression [1], global motion estimation [2,3], and frame rate up-conversion [4,5]. Recently, deep neural networks, which exhibit superiority in the field of computer vision [6,7,8,9,10,11,12], were used to estimate motion [13,14,15,16]. Dosovitskiy et al. [13] proposed FlowNet by directly applying a deep network to estimate motion from input image pairs. Ilg et al. [14] proposed FlowNet2, which was designed as a cascade of FlowNet. The first network in FlowNet2 estimates the motion between the current and reference images. This motion is exploited to warp the reference image, which constitutes the input to the following network. Although FlowNet2 achieved outstanding performance, the network had over 160M parameters. Recently, Sun et al. [15] and Hui et al. [16] proposed lightweight and effective networks that estimate the coarse-to-fine motion by exploiting a pyramidal structure for features named PWC-Net and LiteFlowNet, respectively. However, the aforementioned methods suffer from extremely high computational costs and large numbers of model parameters because they estimate motion in the pixel domain, which constitutes a limitation in practical realizations.

Block-based motion is widely used as an alternative to pixel-based motion [17]. To estimate block-based motion, block-matching algorithms, where blocks of two sequential images are compared, are usually used owing to their simplicity. A naïve block-matching method consists of exploiting all pixels of each candidate block for matching. Although the naïve block-matching method can be used to find the optimal motion between input image pairs, the computational complexity is high. To reduce this complexity in a matching manner, several alternative methods using representative values for each block were studied, such as sub-block mean [18], certain patterns [19], and some noticeable pixels [20]. However, because these methods matched each block of input image pairs in the intensity domain, it is difficult to find the effective values for representing the block.

In this paper, we propose a new deep neural architecture for block-based motion estimation. Our contribution is as follows. First, unlike conventional representative matching in the intensity domain [18,19,20], we conduct representative matching in the feature domain where the features are obtained from convolutional neural networks (CNNs). Owing to the powerful ability of CNNs to represent features, the proposed representative matching achieves similar performance to naïve full matching with lower computation complexity. Secondly, it can be easily implemented by the typical pooling operator used in deep learning because the process to find the representative value in each feature is shared. Finally, to maximize the efficiency of the proposed representative matching, we optimize deep neural network using dilated convolution, which can expand the receptive field preserving feature shape without the additional computation and pyramidal structure that causes BlockNet to use a small search range.

2. BlockNet

Figure 1 shows the architecture of BlockNet, the proposed block-based motion estimation network using representative matching. Next, we explain each component of BlockNet in detail.

2.1. Feature Extractor

When a current image $I_c$ and reference image $I_r$ are given as input, BlockNet extracts features using three convolutional layers that construct compact architecture. In each layer, a convolution with a filter size 3 $\times$ 3 and stride of 2 is first used to obtain the appropriate spatial shape in each level; a dilated convolution with filter size of 3 $\times$ 3 and rate of 2 is then used to enlarge the receptive field while retaining the spatial shape. The numbers of filters in each convolutional layer are 16, 32, and 64, respectively, similar to [15]. The filter weights in each convolutional layer are shared across two input images, namely $I_c$,$I_r$, to extract features with common patterns.

2.2. Representative Matching for Motion Estimation

2.2.1. Proposed Algorithm

In a compact deep neural network, a hand-designed architecture such as the matching approach may perform better to estimate motion [13]. Based on this, it is reasonable for our network to utilize a standard method of block-matching motion estimation. Block matching consists of finding the most similar block by comparing the block in the current feature and candidate blocks of the reference feature in the search range (Figure 2, left side).

For pixel matching, the matching cost is defined in [13] as the correlation of vectors corresponding to each pixel of the current and reference features. This correlation is accumulated for the search range of each pixel. As a result, a 3D cost volume, which has dimensions of $R^2\times H\times W$, where $R\times R$ is the search range and $H$ and $W$ respectively denote the height and width of the features, is constructed. When this pixel matching is applied on block matching, the matching cost is calculated at block level instead of pixel level. Thus, the 3D cost volume has dimensions $R^2\times \frac{H}{d}\times \frac{W}{d}$, where $d\times d$ is block size, that is:

$$\mathit{Cost} \mathit{Volume}\left(i,j\right)=\frac{1}{N{d}^2}{\left({\mathit{b}}_i^c\right)}^{\mathrm{T}}{\mathit{b}}_j^r$$

(1)

where ${\mathit{b}}_i^c$ is the column vector vectorizing the $i$-th block of the current feature, ${\mathit{b}}_j^r$ is the column vector vectorizing the $j$-th candidate block of the reference feature in the search range, and $N{d}^2$ is the dimension of the column vector.

When the block size is $d\times d$ and the search range is $R\times R$, the number of multipliers for a block matching is $N{d}^2\times R^2$. If the block size is larger, the total number of the computation is much higher and thus it may not be suitable for practical realizations. To reduce the computation in the block matching, we could find the representative value in the block. In this study, we propose a representative matching method using the simple average operator defined next (see Figure 2):

$$\mathit{Cost} \mathit{Volume}\left(i,j\right)=\frac{1}{N}{({\overline{\mathit{b}}}_i^c)}^{\mathrm{T}}{\overline{\mathit{b}}}_j^r$$

(2)

where ${\overline{\mathit{b}}}_i^c$ and ${\overline{\mathit{b}}}_j^r$ are the average of the column vectors ${\mathit{b}}_i^c$ and ${\mathit{b}}_j^r$, respectively. Owing to the proposed representative matching, the number of multipliers for block matching is reduced as much as $d\times d$.

In a conventional block-matching algorithm using representative values of the block [18], the average value of the block is also used to reduce the computational cost. However, because the representative matching is applied in the intensity domain, it is insufficient to represent the block. By contrast, because the image can be analyzed as various features through a CNN, our representative matching works well in the feature domain.

With the 3D cost volume and current feature as input, block-based motion is obtained using a CNN with filter size 3 $\times$ 3 and stride 2 (Figure 1, motion estimation network). The numbers of filters at each convolutional layer are 32, 24, 16, and 8, respectively.

2.2.2. Implementation Details

Instead of extracting the representative values whenever performing the block-level matching, it can be considered that the representative value of each feature is first extracted, and then matching is performed to implement efficiently. To this end, the average-pooling operator, which is widely exploited in the deep-learning framework, can be used.

The implementation procedure of the proposed representative matching is described in Algorithm 1. Because the representative values of the block in the current and reference features should be extracted at intervals of block size and pixel, respectively, the stride of the average-pooling operator is set to block size and 1, respectively. Moreover, the average-pooling size is the same as the block size (steps 1 and 2). Then, the 3D cost volume is obtained by a matching process that extracts the patches from each average-pooled feature in the search range (steps 3–5) and multiplies them (step 6).

Algorithm 1. Proposed representative matching

Definition

$avg\_pool\left(·\right)$: The average-pooling operator

$repeat\left(·\right)$: Duplication of each element of the matrix

$extract\_patch\left(·\right)$: Extraction of the patches repeatedly

Input: current feature ${\mathit{f}}_c$, reference feature ${\mathit{f}}_r$, block size $d\times d$, and search range $R\times R$

${\overline{\mathit{f}}}_c=avg\_pool\left({\mathit{f}}_c, size=\left[d,d\right], stride=\left[d,d\right]\right)$ ${\overline{\mathit{f}}}_r=avg\_pool\left({\mathit{f}}_r, size=\left[d,d\right], stride=\left[1,1\right]\right)$ ${\overline{\mathit{f}}}_c^{repeat}=repeat\left({\overline{\mathit{f}}}_c, size=\left[R,R\right]\right)$ ${\mathit{p}}_c=extract\_patch({\overline{\mathit{f}}}_c^{repeat}, size=\left[R,R\right], stride=\left[R,R\right])$ ${\mathit{p}}_r=extract\_patch\left({\overline{\mathit{f}}}_r, size=\left[R,R\right], stride=\left[d,d\right]\right)$ $\mathit{Cost} \mathit{Volume}$$={\mathit{p}}_c\ast {\mathit{p}}_r$

Output: $\mathit{Cost} \mathit{Volume}$

2.3. Pyramidal Structure with Feature Warping

To maximize the efficiency of our representative matching, we adopted the pyramidal structure in PWC-Net [15]. At level $l$, the reference feature is warped toward the current feature using a $\times$2 up-sampled motion estimated from the previous level (Figure 1, feature warping). We first estimate the motion utilizing a 3-level pyramid structure with $\left\{{\mathit{f}}_c^3,{\mathit{f}}_r^3\right\},$ and $\left\{{\mathit{f}}_c^2,{\mathit{f}}_r^2\right\}$ among the 4 possible levels. We then simply up-sample the estimated motion as much as the remaining levels to obtain the final motion. This architecture can reduce the computation complexity while obtaining the motion with a similar accuracy to that reported in [13].

3. Experiments

3.1. Experimental Setup

To train BlockNet, we used the FlyingChairs dataset [13], which is composed of 22,872 image pairs with ground-truth motion. We cropped 384 $\times$ 512 images to 384 $\times$ 448 patches and used 90% and 10% of the dataset for training and to test, respectively. We used the multi-scale training loss $\mathcal{ℒ}\left(\theta \right)$ described in [15] as follows:

$$\mathcal{ℒ}\left(\theta \right)={\displaystyle \sum }_l{\alpha }_l{\displaystyle \sum }_{\mathit{x}}\parallel \mathit{M}{\mathit{V}}_{\theta }^l\left(\mathit{x}\right)-\mathit{M}{\mathit{V}}_{GT}^l\left(\mathit{x}\right){\parallel }_2+\gamma \parallel \theta {\parallel }_2$$

(3)

where $\theta$ is the network parameter, ${\alpha }_l$ is the loss weight for layer $l$, $\mathit{x}$ is the block index, $\mathit{M}{\mathit{V}}_{\theta }^l$ is the estimated block-based motion vector in layer $l$,$\mathit{M}{\mathit{V}}_{GT}^l$ is the ground-truth block-based motion vector in layer $l$, $\parallel ·{\parallel }_2$ is the $L_2$ norm operator, and $\gamma$ is the regularization parameter. To obtain the ground truth of block-based motion, we down-sampled the pixel-level ground-truth motion by a factor given by the block size. As in [15], the ground truth was down-sampled by a factor of 2 at each level. Moreover, it was identically scaled by $1/20$ at all levels. This made the estimated motion have identical scale at all levels. Thus, the up-sampled motion had to be scaled from the previous level before passing through the warping operator. We set the scale values for the up-sampled motion as $20/2^3$ and $20/2^2$ at levels 3 and 2, respectively. We used a block size of 4 $\times$ 4 and a search range of 15 $\times$ 15, which are determined by experiments on hyperparameters in Section 3.2.

We first trained BlockNet using the MPI Sintel dataset [21] with 600 epochs. We fine-tuned the network using the FlyingChairs dataset. The initial learning rate was 0.0001. It was halved at iterations 0.2 M, 0.25 M, 0.3 M, and 0.35 M. We used a mini-batch size of 4 and the Adam optimizer [22]. The weights were set to ${\alpha }_4=0.32$, ${\alpha }_3=0.08$, and ${\alpha }_2=0.02$, and the regularization parameter $\gamma$ was set to 0.0004 as in [15]. BlockNet was implemented using TensorFlow 1.7.0.

3.2. Results

To verify the effectiveness of the proposed deep neural architecture, BlockNet was compared to a conventional block motion estimation (BME) that exploits all pixels of each candidate block in the search range for matching. We also compared each algorithm with or without the proposed representative matching (RM). All results are evaluated in terms of end-point error (EPE), with the $L_2$ norm between the estimated motion and ground truth [15].

The average and standard deviation of EPE are summarized in

Table 1. Quantitative results on the FlyingChairs dataset.

	BME with Full Matching	BME with Proposed RM	BlockNet with Full Matching	BlockNet with Proposed RM
Average EPE	10.33	16.86	3.74	4.09
Std. of EPE	7.64	7.09	3.53	3.57

. Experimental results show that BlockNet with full matching had lower average EPE than BME with full matching. This is because the CNNs in BlockNet can extract rich features, and the matching errors of BlockNet were lower than those of BME. Moreover, average EPEs of BlockNet with full matching and proposed RM were similar, while average EPEs of BME with full matching and proposed RM significantly differed. This result implies that the proposed representative value, which reduced the computational complexity as much as $1/16$ for each matching, was more effective in the feature domain than in the intensity domain. Figure 3 shows qualitative results of BlockNet with full matching and proposed RM. The results of BlockNet with full matching and proposed RM are quite similar (Figure 3; top, chair leg). However, proposed RM occasionally fails to estimate the detailed motion of an object compared to full matching (Figure 3; bottom, chair leg).

Detailed experiments were conducted to verify the effect of some hyper-parameters (block size, search range) in BlockNet with RM (Figure 4). Although a large block size reduced the computational complexity, the average EPE was increased because of the reduction in the resolution of the estimated motion. For the search range, the average EPE with a large value was slightly decreased at the expense of high computational complexity. The proposed RM was reduced by 18% compared to full matching, using the best hyper-parameter (Figure 4, red diamond), with respect to computational complexity while archiving similar average EPE.

4. Conclusions

In this paper, we proposed BlockNet using an efficient representative matching. The proposed network can extract rich features for block-based motion estimation. A representative matching was performed with these features by using the average operator and implemented simply by using the average-pooling operator, widely employed in the deep-learning framework. To maximize the efficiency of the proposed representative matching, a pyramidal structure with feature warping was adopted in BlockNet. Experimental results show that BlockNet with and without our representative matching achieved similar average EPE, while our matching exhibited lower computational cost than full matching. In future work, we will apply BlockNet to various real-time applications based on motion estimation, such as frame rate up-conversion because it has less computational cost and is easy to implement.