THE DIRECT REGISTRATION OF LIDAR POINT CLOUDS AND HIGH RESOLUTION IMAGE BASED ON LINEAR FEATURE BY INTRODUCING AN UNKNOWN PARAMETER

The registration between optical images and point clouds is the first task when the combination of these two datasets is concerned. Due to the discrete nature of the point clouds, and the 2D-3D transformation in particular, a tie points based registration strategy which is commonly adopted in image-to-image registration is hard to be used directly in this scenario. A derived collinear equation describing the map relationship between an image point and a ground point is used as the mathematical model for registration, with the point in the LiDAR space expressed by its parametric form. such a map relation can be viewed as the mathematical model which registers the image pixels to point clouds. This model is not only suitable for a single image registration but also applicable to multiple consecutive images. We also studied scale problem in image and point clouds registration, with scale problem is defined by the optimal corresponding between the image resolution and the density of point clouds. Test dataset includes the DMC images and point clouds acquired by the Leica ALS50 II over an area in Henan Prov., China. Main contributions of the paper includes: [1] an derived collinear equation is introduced by which a ground point is expressed by its parametric form, which makes it possible to replace point feature by linear feature, hence avoiding the problem that it is almost impossible to find a point in the point clouds which is accurately corresponds to a point in the image space; [2] least square method is used to calculate the registration transformation parameters and the unknown parameterλ in the same time;[3] scale problem is analyzed semi-quantitatively and to the authors’ best knowledge, it is the first time in literature that clearly defines the scale problem and carries out semi-quantitative analysis in the context of LiDAR data processing. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B4, 2012 XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia


INSTRUCTIONS 1.1 Instructions
Airborne Light Detection and Ranging technology has enjoyed rapid development in the photogrammetry and remote sensing community in recent years.Though many applications of LiDAR data have been carried out in fields such as topographic mapping, forestry parameter retrieval (Hyypp, 2000) (Andersen, 2007;Melzer, 2004), power-line detection (Feng, 2009;Melzer, 2004;Xu, 2008), 3D urban modeling (Gamba, 2000;Jie, 2006;Rottensteiner, 2002;Sampath, 2007;Habib, 2009), and true orthophoto production (Kim, 2006;Riao, 2007), to mention only a few.However, it is a pity that LiDAR data lack of spectral information due to the monochromatic property a laser transmitter adopted (Baltsavias, 1999), no matter what kinds of laser sources used.On the other hand, imageries acquired by conventional air-or-space borne sensors are imaged within visible or near-infrared band of spectrum, therefore, full of semantic information.To combine their respective advantages of LiDAR data and imagery can not only provide extra information for thematic mapping, it is also the requirement of ortho-photo production, in which accurate Digital Surface Model is a must.Co-registration of airborne laser scanning data and imagery is the first step when combining the usages of the two datasets are considered.Since 3D property in nature, laser scanning datasets are difficult to register with imagery by conventional image-toimage registration methods.Though there is lot of literature describes image-to-image registration (Brown, 1992), both automatically or semi-automatically, however, little literature concerns the problem of registration between LiDAR data and imagery up to date.In some research straight line features (Habib, 2005) and planer patches (Kwak, 2006;Bang, Habib 2008) were used in two separate methodologies as the primitive of choice for the co-registration of the photogrammetric datasets to the LiDAR coordinate system.The approach using straight line features and planer patches starts with generating a photogrammetric model through a photogrammetric triangulation using an arbitrary datum without knowledge of any control information (Habib, 2005).To incorporate photogrammetric straight line in the registration model, the end points of "tie line" have to be identified in one or more images, providing for collinearity equations.Intermediate points are measured on this line in all images where it appears.For each intermediate points, a coplanarity constraints is used.This constrain states that the vector from the perspective center to any intermediate point along the line is contained within the plane defined by the perspective center of that image and the two points defining the straight line in the object space.Similar to the case of the line features, on the characteristics of planar patches in both datasets.The core principle behind this methodology is that in the absence of systematic error, LiDAR points belonging to a certain planarsurface should coincide with the photogrammetric path representing the same object space surface.In other words, the volume of the pyramid with its vertex at the LiDAR point and its base at the corresponding photogrammetric patch should equal to zero.Though good results achieved, Harbib's method has shortages in term of the following aspects: a) only multiple frames with adequate overlapped regions can be registered, b) in their model, traditional photogrammetric workflow is used to obtain exterior elements, which is a two-step procedure consists of relative and absolute orientation.Errors can be accumulated in the procedure, therefore, affect the final registration accuracy, c) their procedure is complex and is not a computational costeffective one.
Another factor must be considered when registering LiDAR data and imagery is the correspondence between image resolution and LiDAR data density.When the application of urban planning and management is concerned, resolution of photogrammetric images usually ranges from 5cm to several tens of centimeters according to the configuration of modern photogrammetric cameras, while point spacing of LiDAR point clouds ranges from 0.2 to 2 meters.What is the optimal resolution an image should be when it is registered to LiDAR data with given density or vice versa?We view this problem as scale analysis.To the authors best knowledge, there is no literature concerning this problem.In summary, the existing problems of registration of LiDAR point clouds data and remote sensing images are as following: registration process is complex, requiring to complete in two steps.First, three-dimensional relative orientation is used to generate image corresponding point clouds; second, corresponding point clouds is taken to register with LiDAR point clouds; (2) mathematic formulation of registration primitives is not concise enough; (3) lack of registration method which is suitable for single frame image and multiple frame images with LiDAR point clouds simutanuously; (4) there is lack of scale analysis.

The Selection of Registration Primitives
Though much different in nature concerning the LiDAR data and imagery, the registration of them can also be decomposed into four essential problems: the extraction of registration primitives, the establishment of similarity measurement, the selection of transformation function and the strategy for matching (Brown, 1992).Registration primitives should satisfy the remote sensing data's characteristics that are obvious, even distribution and easy to extract, and it cannot be affected by different sensor data's geometrical or radiation deformation, noise, scene changes and other factors.The primitives which can be used for registration in current literature mainly are areas, such as forest, lake and champaign (Zitova, 2003,Flusser, 1994, Goshtasby, 1986) and points, such as angular point of areas, point of lines intersection (Stockman, 1982, ) and inflection point with large radian (Ali, 1998), as well as linear features, including ground objects' edge (Dai, 1997)  the extraction of them is not an easy task neither from imagery nor from point clouds.So straight lines are chosen as the registration primitives in our algorithm especially in urban area

The Extraction of Linear Primitives
In our procedure, we use accurate lines as the registration primitives in our transformation function.Bearing this in mind and considering the line extraction strategies mentioned above, we propose a new algorithm which is based on the fact that many sidewalls of buildings could reflect laser pulses, since most of time the data acquisition scanning angle is not zero degree.So many laser echoes are reflected from the sidewalls of the buildings (Fig. 1) .From above, we proposed a so-called Differential Volume Statistics Method (DVSM) method to extract precise linear features, which makes use of the property that the density of point clouds increases dramatically in the vicinity of edge lines (Fig. 2) (Ma, 2010) .
,where threshold h is the elevation difference threshold, o D is the distance threshold in the x-y plane,in order to make sure that the point P and Q are not very close.(c)Z values of the point P and Q can be interpolated using the corresponding original linear feature, and turn to (3) unless all the original linear features have been analysised.；(7) Output all the linear features which can be used as registration primitives.

THE REGISTRATION MODEL BASED ON COLLINEARITY EQUATION
LiDAR points are taken as images' object space, by collinearity equation (Wang Zhizhuo,1979), principal point of photograph, image point and object space point are in the same line, being expressed to be mathematical equation shown as formula (2): Where, x，y：corresponding to image coordinates of ground surface (3) Where, P λ : is the proportion parameter of P shown by line AB.
Assume that elements of interior orientation are known and camera does not have any system error, formula (2) is expanded according to linear Taylor formula and error equation is set up, then the formula (3) is: Thus, the mathematical model between image coordinates and point clouds space straight line.This model can be considered as traditional collinearity equation formed by straight line instead of point feature. in object space.It is suitable both for the registration of single and multiple images and LiDAR points data.As for single frame image, its normal equation is simplified as formula ( 4)： ] [ Where, ： t means elements of exterior orientation of the image; ： λ means the matrix formed by unknown parameters of "line" feature replacing "point" feature; : n means the combinations of registration primitives of point and line; ： L constant variables;

： V residual vector
As for multiple images, its normal equation formula ( 5): Where, x y means the p image point of the i image.

REGISTRATION SCALE ANALYSIS
There is a group of best combinations between point clouds density and image resolution which makes registration accuracy optimal.If it is a rather low or high image resolution in relative to point clouds density, registration result will bring loss or damage of the two data information.The method difined in this paper to resolve this problem is called scale analysis.Firstly, the influence of image resolution should be removed, then registration point position pixel error p C is introduced.The coordinate of P in LiDAR poins space,corresspoding to ( , ) p x y in image.With registration transformation model, P is inversed to image coordinate system, and its image coordinate x y ′ ′ , so | ( , ) x y ′ ′ | is p and p ′ 's pixel coordinates deviation existed in image coordinate system.

Data Explanation
DMC aerial image of Henan Province in china is adopted, and the respective flying heights are 600m，1000m and 2500m, corresponding to the aerial image resolutions of 0.056m ， 0.11m and 0.25m respectively; point clouds data is acquired by ALS-50II, and its density is 1.306 point/ 2 m .In order to It can be seen from the tables above that in the registration based on linear feature instead of point feature when flying height is no more than 2500m and point clouds density is more than 1 point/m 2 , coordinate deviation accuracy is in a pixel with accuracy of 0.15m, the result is shown in figure7 a-c.By comparing table 2 and 3, the accuracy using linear primitives is much better than using point registration primitive.And, the registration accuracy of multi-images is better than single images.What can be seen from table 4 and table 2 is that the higher resolution of DMC aerial image, the higher absolute accuracy of registration.When points' density is fixed, the higher image's resolution, the larger registration scale and the bigger registration deviation.When image resolution is fixed, the smaller density of points, the larger value of registration scale factor and the bigger deviation of registration point position.
From figure 8, it can be seen that under original point clouds density, the deviation of checkpoint is rather steady in the range of [0,1.5].Resolution of aerial image satisfies centimeter accuracy level.Assume image resolution is 0.05m, points density then should be 1.7 point / 2 m , so it can get the registration accuracy in a pixel.

The analysis of the patameter λ
In this paper, λ is introduced to make the line features in the LiDAR points data as registration primitives corresponding to the tie point selected in the image data.In order to verify the impaction of the initial λ value, we compare three groups of the λ values: "true values", "the results with good original values" and "the results with 0 original values " .
true The table above shows that: the original λ value does not affect the registration result.

CONCLUSION
This paper utilizes a kind of registration model using linear feature instead of point feature, achieving high-accuracy registration of LiDAR points and image, and conducts a semiquantitative registration scale analysis.This registration model is fit not only for the registration of multi-image and LiDAR point clouds data, but also for the registration of single image and LiDAR point cloud.In the experiment, the registration of DMC images with different resolutions and LiDAR points data with different density are used.In single image condition, when flying height is less than 2500m, the absolute accuracy of registration is less than 0.5m; In multi-images condition, the registration accuracy is better than that of single image.When flying height is less than 2500m, absolute accuracy is less than 0.15m.The experimental results show that using linear feature is better than that of point feature.
As for the images of different resolutions which are registrated with the points data of the same density, the higher resolution, the higher accuracy of registration result.In order to enhance the universality of algorithm, the registration method using various feature on this basis is needed to be further researched and breakthrough, especially when the linear features is absent.

Figure 1 :
Figure 1: extraction of linear features.The red ones are the ideal linear features The details of DVSM are as following:(1) Firstly, use the progressive TIN filtering method to get the ground points and grid DEM； (2) Secondly, using Gradient operator to detecte edges as the initial Linear features from depth-image which is generated from the LiDAR points by height ； (3) Scanning the initial linear features, project the current line onto the DEM, then the feature plane is formed whose normal vector is n 

Figure
Figure 4．The line AB on the LiDAR point data and the corresponding points in the image space Firstly, take line AB(Figure.5)extracted from LiDAR points data as a line in object space.As points data is threedimensional data, AB is three-dimensional line.P, a point in AB, has a correspondent corresponding image point P ′ in image.P λ , an unknown parameter, is introduced.So P's true value coordinate can be expressed by coordinates of A and B, as well as P λ to be [ , , ] T p p p X Y Z , as shown by formula (1): is the three-dimensional coordinate of A point in line in LiDAR space' Since this P point should be a corresponding point P ′ in image space (assumed not in the occluded area), ( , , ) corresponding point of P in LiDAR points space.Different parameters P λ correspond to a series of such corresponding points.It can be seen that P can be expressed by line AB and parameter P λ ， without seeking corresponding points in LiDAR data, therefore, it overcomes the difficulty of selecting tie points in LiDAR points data.
of pixel coordinates deviations above of all checkpoints.The ratio of LiDAR point clouds average point distance and image resolution is defined as scale factor S. Registration's scale problem can be semi-quantitatively analyzed by checking the relation of scale factor S and p C .

Figure 8 :
Figure 8: registration results' position deviation of check points Unlike image data, the selection of tie points from LiDAR dataset for registration is almost impossible, since LiDAR data are so called point clouds, what are the discrete return echoes representing the 3D geographic coordinate values.Therefore, it is of great difficulty to precisely register LiDAR data and image if we use tie points as the registration primitives.Though patches are usually good candidates as registration primitives, and split surfaces' intersection lines.This paper regards linear feature as registration primitives, mainly because LiDAR is characteristic of discreteness, which cannot accurately choose control points.Moreover, linear feature has the following advantages:  Linear feature has a certain degree of scalability.Any two points of linear feature can state the whole linear feature;

The Registration Result of Single Image and LiDAR Points Data The
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B4, 2012 XXII ISPRS Congress, 25 August -01 September 2012, Melbourne, Australia facilitate scale analysis, point cloud is appropriately vacuated.So the following data combinations are acquired: (0.056m， registration of one-chip frame aerial image and LiDAR point clouds data.Registration model is shown as (2).16 checkpoints, absolute accuracy of registration result is shown in table 1, and the unit is m.

Table 1 :
registration result of single image and LiDAR points data (experiment 1) 5

.3 The registration result of multi-images and LiDAR points data
The registration model of multi-mages and LiDAR points data still is as shown in (2). 10 checkpoints are used to check the registration result of airborne LiDAR point clouds data with point feature and remote sensing image, as well as the registration result of airborne LiDAR point clouds data with linear feature instead of point feature and remote sensing image.The registration results are shown as table2 and table 3.