Advanced weight graph transformation matching algorithm

: An efficient and accurate point matching algorithm named advanced weight graph transformation matching (AWGTM) is proposed in this study. Instead of relying only on the elimination of dubious matches, the method iteratively reserve correspondences which have a small angular distance between two nearest-neighbour graphs. The proposed algorithm is compared against weight graph transformation matching (WGTM) and graph transformation matching (GTM). Experimental results demonstrate the superior performance in eliminating outliers and reserving inliers of AWGTM algorithm under various conditions for images, such as duplication of patterns and non-rigid deformation of objects. An execution time comparison is also presented, where AWGTM shows the best results for high outlier rates.


Introduction
Image registration is the process of aligning corresponding positions between images acquired under different conditions, such as time, viewpoint and sensor modality [1]. The most crucial and difficult step of registration process is the matching of corresponding positions between the images to be registered, which is the key to determine whether the success of registration or not. There is an urgent need to formulate a highly efficient matching algorithm.
There are two major categories according to the matching methods: area-based methods and feature-based methods. Area-based methods usually use image region content to match points between two images. These methods compare the similarity of image regions utilising measures such as Correlation-like [2], Fourier [3], and Mutual Information [4]. Area-based methods are preferably applied when the images have few prominent details and distinctive information provided by local shapes and structure. However, not only computationally expensive, area-based methods are also sensitive to the intensity variation, noise level, imaging conditions, and sensor modalities. On the other hand, feature-based methods can utilise distinctive features to establish matching correspondences or estimate the registration parameters between two images. This local structural information allows registering images of completely different nature and handling complex distortions between images.
Feature-based methods are more commonly adopted in accurate image registration process due to the higher efficiency. Most recent works focus on the concept of graph which utilises the spatial relationships between feature points and their adjacent feature points to establish the matching. The main idea of these approaches is that the spatial relationship formed by feature points is maintained regardless of the transformation between the two images. Aguilar et al. [5] introduced a robust point-matching method named graph transformation matching (GTM) based on matching K-nearest-neighbour (KNN) graphs with limitations of distances to eliminate vertices that introduced structural dissimilarity between the two graphs. Liu et al. [6] proposed a restricted spatial orders constraints algorithm, which compared the similarity of the cyclic strings representing the local spatial order of the neighbouring points. Zhao et al. [7] proposed a Bilateral K nearest neighbours spatial order around geometric centres (Bi-SOGC) algorithm. Considering both the bilateral adjacent relations and the spatial angular orders, the vertices with maximum bilateral K-nearest-neighbours (Bi-KNN) difference were deemed as candidate outliers, and the invariant spatial angular orders were used to deal with outliers in pseudo isomorphic structures. To find the outliers, Zhang et al. [8] introduced a affine invariant descriptor calculated by the triangle-area representation of KNN. Sanromà et al. [9] present a smooth simultaneous structure graph matching method, which poses a mixture model to evaluate the geometrical arrangement of nodes and their structural relations and use expectation maximisation algorithm to solve the graph matching problem. Considering the 3D reconstruction of scenes, Izadi and Saeedi [10] proposed a robust weighted graph transformation matching (WGTM) method. This method utilised the angular distances between matching graphs of point correspondences. The results showed a better performance comparing to both GTM and RANSAC. A common problem that all the above mentioned methods have difficulties with is that the inliers whose KNN are almost outliers will be removed.
The proposed algorithm in this paper is named advanced weight graph transformation matching (AWGTM). Instead of removing all the identified outliers, the AWGTM algorithm inspired by [10] (WGTM) utilise the strategy of inliers reserving, which can save the inliers when their KNNs contain outliers. Besides, due to the simplified process about iterations, the computation complexity decreased significantly in this algorithm. This paper is organised into four main sections including the present section. First, this paper reviews the GTM and WGTM algorithm in Section 2. Section 3 describes the principle of AWGTM algorithm. Section 4 tests the performance of AWGTM under the presence of outliers and compares it with that of WGTM and GTM approaches respectively. Section 5 summarises and concludes this paper.

Related work
Assuming that there are two images in which significant points have been extracted, the aim of points matching is to find the correct correspondences between these two sets of points. After the matching process, two sets of corresponding points are found: P = {p i } and P′ ={p′ i } of size N (where p i matches p′ i ). Since the algorithm of identifying the initial matching is not very effective, there are several incorrect corresponding pairs. The WGTM algorithm is inspired by GTM algorithm to remove these erroneous correspondences in [10]. It takes the angular distance as a criterion to judge the outliers, which is advantageous due to the fact that the angular distance is invariant with respect to scale and rotation and sensitive to the noise originating from lens distortion.
To eliminate correspondences that disrupt the neighbourhood relationships, WGTM creates a median KNN graphs G p =(V p , E p ) using the following definition: define a vertex v i for each match point p i , such that V p ={v 1 , …, v N }. An edge (i, j) exists when p j is one of the closest neighbours of p i and also p i − p j ≤ h. Here η is the median of all distances between pairs of vertices, as defined below: Thus, two KNN graphs for the two sets, G p and G′ p are generated.
if (i, j) ∈ E p and 0 otherwise. Similarly, the graph G′ p =(V′ p , E′ p ) for points p′ i also has an N × N adjacency matrix A′ p . If there are some vertices which have less than two edges with , it is hard to classify them as the inliers or the outliers without enough adjacency vertices that support the structure of them. Remove all the vertices like above and their correspondences from G p and G′ p , and recompute both G p and G′ p until all vertices of G p and G′ p have two edges at least. WGTM relies on the points that maintain the invariance with respect to the image transformation. Therefore, a weight matrix W is generated for each point p i using G p to identify those points. The weight value of edge that connects vertex v i to v m is computed by: Here p i and p m represent the coordinates for vertices v i and v m in image. k min represents the optimal rotation angle which minimises the sum of angular distances between vectors v i and v′ i : and where In the above equation, v x and v y is the x and y coordinates of v in the image. Assign 0 to all the elements which have not been set to any value in W. At this stage, the sum of ith row of weight matrix W represents the minimal total of angular distances between edges in the graphs generated by p i and its KNN and p' i and p i 's KNN.
When the percentage of edges connected to v i with their correspondences connected to v' i is smaller than 50%, the weight value of all different edges in the weighted matrix W should be replaced by π. In other words: For the invariant points under transformation, the weight values are very small. Considering the points with less than K neighbours, the mean of all weights related to all edges connected each vertex v i of G p is computed to avoid the impact of the number of its neighbours The point corresponding to the maximum value of ω will be regarded as outlier and eliminated with its corresponding point from P and P′.
If ω max < π and | μ new − μ old |<ε, stop iterating; otherwise, let μ old = μ new and continue to iterate. Where Set The values of ε and initial μ old are set to 0.001 and 2π in [10]. WGTM is designed to remove correspondences that disrupt the neighbourhood relationships one by one. WGTM's performance quality is superior to that of GTM, RANSAC [11] and Softassign [12] according to Aguilar et al. [ 5,Izadi and Saeedi [10]. However, there are still some problems with WGTM.

Proposed method
In WGTM method, one pair of points which are regarded as outliers will be eliminated from the list of corresponding matches at every iteration. According to the removal criterion, some inliers surrounded by real outliers will be taken as false correspondences and removed. When one inlier has been removed, all the graphs connected it will regenerate, even including the graphs composed of K inliers. There is high probability to add an outlier to the new graphs, which can increase the value of the angular distances. It is adverse to reserve inliers from corresponding sets. Here, a new method is proposed to discover inliers and save them.
Instead of complementing the outliers of original corresponding sets after iterations, the proposed method identifies the inliers at every iteration. Identifying the inliers at every iteration comparing after iterations is advantageous because the inliers will be reserved even if they are eliminated at following iterations. When ω(i)i s less than j, the algorithm in this paper defines that all the vertices in G P i > G P i ′ are candidate inliers. After finding all the points whose ω < j at each iteration, a candidate inliers set I candidate will be generated.
In WGTM, only the point corresponding to the maximum value of ω will be removed, as well as its pair point, at each iteration. However, this method is ineffective when the weight value ω max match with more than one point. Apparently, the other un-removed points corresponding to maximum value still work at the next iteration but the weight value of the rest points need to be recomputed, which results in expensive computing time in this case. To solve this problem, the proposed algorithm will eliminate all the points with maximum value of ω and their correspondences at each iteration. Compared with WGTM, this strategy can reduce the computing complexity significantly.
Therefore, at every iteration, one candidate inliers set I candidate and one outliers set O elimination will be generated from the list of corresponding matches. For identifying the inliers further, AWGTM algorithm remove the points in the intersection of I candidate and O elimination from I candidate , which products a inliers set I iteration I iteration = I candidate − I candidate > O elimination (11) Uniting I iteration generated at every iteration, the proposed algorithm acquires the final inliers set. The algorithm is shown in Fig. 1.

Experimental result
In [10], there is a comparison between WGTM and RANSAC [12], which indicates the superiority of WGTM, so it is meaningless to compare RANSAC with the proposed algorithm. This section presents the results of removing outliers using the AWGTM which is compared with the WGTM [10] and GTM [5]. The algorithm will be tested in three different conditions: camera movement, correspondence ambiguity and deformation. In these images, corner features are extracted [13] and all the inliers are selected manually. In the tests, 60 correct matches are chosen randomly. Then, outliers are added by matching arbitrary points between two images. The percentages of the additive outliers are changed from 5 to 95% with increment of 10%. The number of outliers is computed by [10]: No.of outliers = outlier percent × No.of inliers 1 − outlier percent (12) Since all the algorithms in our paper use KNN graphs, the performance is evaluated for two values of K = 5 and K = 10. The value of j is set to 0.07 (found empirically) to all the results presented in this work. The algorithms execute each case for 100 times. According to Izadi and Saeedi [10], there are four rules used for the measuring results. The accuracy value represents the degree of identifying a match to a true match, which is denoted in (13). Precision means the proportion of the true matches in inlier set. Recall stands for the percentage of true matches which are correctly identified, and (16) Figs. 1a and c, it can be seen that the AWGTM algorithm in accuracy and recall performs better than the WGTM algorithm and the GTM algorithm when the value of K is fixed. Meanwhile, there are little differences in the values of precision and specificity among the three algorithms in most of the outlier percentages. It is noteworthy that the WGTM algorithm has a poor performance when outlier percentage is 95% and K is set as 5. It is because the WGTM algorithm always identifies all the points as outliers. In this case, this paper deems the algorithm failed to remove the outliers. If the algorithm fails, the values of the four measures are set as 0, which is different from [10]. According the performance of the WGTM algorithm, it can be concluded that the WGTM algorithm is not suitable for the situation that the number of outliers is very big when K = 5. Due to enforcing a tighter bound between the networks of neighbouring points, the AWGM algorithm and the WGTM algorithm perform better for K = 10 than K = 5. However, the GTM algorithm is superior when K =5. A visual example of the quality difference in the matching set is shown in Fig. 3 that gives the resulting matching set for all algorithms when the initial matching are composed by 60 inliers and 65% outliers. Here yellow lines show correct matching correspondences. Blue lines depict falsely identified inliers. Fig. 3 shows that only AWGTM has eliminated all the outliers comparing to GTM and WGTM. Meanwhile, the number of inliers reserved using AWGTM is the most.

Correspondence ambiguity
The performance of the algorithm is also tested under ambiguous conditions including partial occlusion and multiple match correspondences due to duplicity of some of matching features. The second testing set contains 10 pairs of images. The feature points are found using Harris corner detector [13] and are matched against each other manually. Initial matching contains 60 inliers selected in original matching sets randomly and from 5 to 95% of outliers without repetition. This images set is fed to the GTM, WGTM and AWGTM algorithms, using the same parameters as above. Fig. 4 shows the values of accuracy, precision, recall and specificity respectively. It can be seen that AWGTM and WGTM algorithm provide higher accuracy and recall values for K = 10 than K = 5. When the percentage of outliers is less than 45%, it is obvious that AWGTM can obtain the best results whatever the value of K.
AWGTM outperforms the other two algorithms when the K value is fixed, which proves its capacity to keep the true positives. Because of the same reason, the WGTM algorithm obtain a poor    AWTGM algorithm has reserved the most inliers than GTM and WGTM. At the meantime, all the outliers have been removed. This experiment demonstrates again the strong ability of AWGTM algorithm on keeping correct matches and eliminating outliers.

Deformation
The third testing set contains 20 images of non-rigid objects such as plastic wrappers and papers. Considering the deformable nature of the plastic wrapper, the transition from one image of a pair to the other is just modelled by non-rigid transformation. The Harris corner points are extracted and matched using manual selection. Fig. 6 shows the accuracy, precision, recall, and specificity values obtained by the different algorithms. From Fig. 6a, it can be noted that AWGTM gives the highest accuracy value among all the considered methods when outlier percentage is less than 45%, no matter what the K value is. For Fig. 6c, the recall value indicates the same consequence. Meanwhile, according to the Figs. 6b and d, the values of precision and specificity for all the algorithms are very close except for the GTM algorithm when K = 10. Fig. 7 gives two typical examples of matching obtained the three algorithms when the initial matching contains 65% of outliers. From this figure, it is further evidence that AWGTM is an efficient and accurate point matching algorithm.

Running time
In addition to comparing the efficiency of the algorithms in removing outliers, it is also useful to compare their respective running time. The running time for all the three algorithms is presented in Table 1. In each case 60 inliers were selected and outliers was randomly chosen and added to them. Each time value in this table represents the average execution time for 50 runs of the algorithm on above three sets of images. All the experiments of this section were executed on a CORE™ i5, 2.5 GHz computer with 4GB of RAM. As can be seen in the Table 1, the time required by the three algorithms is almost same when the percentage of outliers is below 25%. After that, the superiority in the execution time of AWGTM begins to be demonstrated, especially when 95% outliers.

Conclusion
This paper has presented a point matching algorithm which establishes the matches of points between two images under conditions including translation, rotation, scale variation, partial occlusion and deformation. The algorithm validates every match through the spatial configuration and mapping relations of the points by using a graph approach. The designed reservation strategy can hold inliers at every iteration, even though they will be removed at following iterations. Meanwhile, eliminating all the points which have the maximum weight value decreases the computing complexity significantly when a large number of outliers exist. Compared with the state of the art methods, the proposed algorithm increases the values of accuracy and recall by about 5%, when all the algorithms are tested on above images conditions. Most importantly, the test results on running time have proved that the proposed algorithm is a more efficient method in matching feature points.