FAÇADE RECONSTRUCTION USING GEOMETRIC AND RADIOMETRIC POINT CLOUD INFORMATION

This paper aims at façade reconstruction for subsequent enrichment of LOD2 building models. We use point clouds from dense image matching with imagery both from Mobile Mapping systems and oblique airborne cameras. The interpretation of façade structures is based on a geometric reconstruction. For this purpose a pre-segmentation of the point cloud into façade points and non-façade points is necessary. We present an approach for point clouds with limited geometric accuracy where a geometric segmentation might fail. Our contribution is a radiometric segmentation approach. Via local point features, based on a clustering in hue space, the point cloud is segmented into façade-points and non-façade points. This way, the initial geometric reconstruction step can be bypassed and point clouds with limited accuracy can still serve as input for the façade reconstruction and modelling approach.


INTRODUCTION
Building reconstruction and modelling has been an on-going research topic for the last years.Nowadays commercial tools are available to reconstruct complete city models from airborne imagery or LiDAR.Mostly digital surface/terrain models and 2D building floor plans are used as data source.So far, such approaches deliver only coarse models.Building blocks consist of walls extruded from the floor plan and have simple roof geometries.However, meanwhile for many applications this representation is not sufficient.Detailed virtual city models have become important for a variety of applications like 3D pedestrian or car navigation, evacuation or flood planning, augmented reality apps and so forth.In this context, not only the actual geometry of buildings is important, but also its semantics.For evacuation planning it is e.g.crucial that windows or doors are not only geometrically represented, but explicitly tagged as such.With the evolving market of data acquisition applications, two techniques particularly suitable for façade reconstruction have emerged: Mobile Mapping systems and aerial imagery from oblique camera systems.Mobile Mapping systemsplatforms mounted onto vehicles -capture the environment of urban areas efficiently.These platforms can carry camera systems, LiDAR sensors or combinations of several data acquisition devices.Within the scope of our approach, camera-based systems are of interest.Continuously acquired stereo images are processed by means of image matching approaches and lead to a point cloud for the whole urban area covered by the vehicle.In case of aerial imagery, oblique camera systems gather increasing popularity.Most systems combine nadir camera heads with oblique ones, hence giving the opportunity to almost completely capture buildings, including their façades.However, both terrestrial and airborne data collection also have their drawbacks and problems.Compared to dense image matching for the standard nadir case (Haala, 2013), oblique imagery has to cope with additional challenges like the big difference in scale for foreground and background objects, disadvantageous viewing angles, as well as many obscured areas (Cavegn et al., 2014).Mobile Mapping platforms can only move along streets.Thus, it is not always possible to capture all relevant building sides.Furthermore, other vehicles, pedestrians or vegetation lead to obstructed views on buildings.In addition, high buildings can vertically not be captured at sufficient resolution.This results in building point clouds with partially limited accuracy and completeness.Point clouds from dense image matching provide both geometry and colour information.In the following section it will be discussed, how this can be exploited by our radiometric segmentation approach for geometric reconstruction support.Yet, this does not overcome the problem of missing or obstructed building areas.Here, the advantage of a grammar based reconstruction and modelling approach becomes apparent, which is also discussed in the next section.For façade parts with sufficient point coverage, the reconstruction process is performed and subsequently rules defining a grammar are derived.By means of this grammar it is possible to synthetically model building parts that were not or insufficiently covered.In most cases data representations of point cloud, building model and images are not treated as a whole or handled in different applications.The presented approach aims at fusing all these input sources in suitable data formats for the needs of semiautomatic building reconstruction.New contributions are an interactive component that shall offer the possibility to select façades of interest for the reconstruction and modelling process in complete city areas on one hand.On the other hand a radiometric segmentation approach gives the possibility to perform a reconstruction also for point clouds with limited geometric accuracy.The remainder of this paper is structured as follows.The next section gives an overview of the related work.In section 3, our semi-automatic façade reconstruction approach is presented.3.1 gives a short overview, in section 3.2 the geometric reconstruction process is discussed.Thereafter, section 3.3 deals with the radiometric segmentation support approach, followed by the modelling step in 3.4.Some experimental results are shown in section 4. Finally, section 5 concludes the paper.

RELATED WORK
Façade reconstruction can be distinguished into 2D and 3D based approaches.The first category uses image data to reconstruct building primitives like windows and doors by segmentation.Among the first generation of grammar-based approaches (Alegre et al., 2004) proposed their work.A Bayesian generative model is constructed from stochastic context-free attribute grammar, subsequently a MCMC solution is used to optimize parameters.(Müller et al., 2007) use mutual information for symmetry detection in rectified façade images.Similarly, (Wenzel et al., 2008) detect multiple repeated groups and determine their translations in the image.Thereby a model for compact description of repetitive structures in façades is derived using a heuristic search method.A semi-automatic image-based façade modeling system is presented by (Musialski et al., 2012).Façade elements that exhibit partial symmetries across the image can be grouped and edited in synchronized manner.The user is in control of the modeling workflow, but is supported by automatic modeling tools, where they utilize unsupervised clustering in order to robustly detect significant elements in orthographic images.
Approaches in 3D exploit point cloud information gathered by dense image matching or LiDAR.(Boulaassal et al., 2007) utilize RANSAC to detect layered planes for segmentation.In (Zhan et al., 2009) geometric information is fused with colour information using a 3D region growing.Prior knowledge derived from a ground truth parameter database and encoded in probability density functions (PDF) is used in (Schmittwilken and Plümer, 2010).These PDF are then applied in a classification process to detect building parts.(Pu and Vosselman, 2009) perform geometric reconstruction as a combination of polygon fitting to extract walls, windows, doors etc. from terrestrial laser data and applying knowledge-based assumptions for occluded parts.
In terms of incomplete or erroneous data, model-based approaches are suitable since the reconstruction process is supported by integration of object knowledge.In (Brenner andRipperda 2006, Ripperda andBrenner, 2009) primitives are detected from façade orthophotos and laser scans.A context-free grammar is derived from a database of façade images and fitted to new ones using rjMCMC.(Van Gool et al., 2013) use an Inverse Procedural Modelling approach.A top-down grammar is combined with an image-based Structure-from-Motion bottomup approach.Completion of ruined temples for example are presented.However, style-specific grammars have to be predefined.For a complete overview of façade and building reconstruction approaches one might refer to (Musialski et al., 2013).
If realistic results shall be achieved, also grammar based approaches depend on data source of good quality to learn their grammars from.The presented paper addresses this issue.Point clouds, as well as coarse building models can nowadays be generated quite fast and efficiently.Therefore exploitation and fusion of both input sources seems reasonable.Our approach uses point cloud information to gradually modify the coarse building model.Primitive edges detected by point coverage analysis enable 3D cell decomposition for the coarse building model.Subsequently rules can be derived to model the remaining façade.

Overview
The presented approach extends the work of (Becker, 2011), so that it is capable of generating building models compliant with CityGML.Furthermore the data input is not restricted to a single façade point cloud and building anymore.To deal with large point clouds, processing of data stored in the LAS format is possible.A GUI has been implemented to simultaneously handle point cloud and LOD2 model of complete city areas.This gives the possibility for semi-automatic reconstruction by selecting façades of interest.For the modelling of buildings the OGC standard CityGML is particularly suitable, since it gives the possibility to store buildings in multiple levels of detail on one hand and on the other hand apart from geometry also semantic information can be provided.Therefore the data model after the reconstruction and modelling process is represented in CityGML as well.Regarding the point cloud representation, for each point alongside its coordinate also the intensity or color information is stored.For this purpose the LAS format has emerged as a standard.Figure 1 shows data which is used in our investigations a coarse city model in CityGML LOD2 overlaid with a point cloud generated from a stereo-based Mobile Mapping system, covering several streets and blocks of buildings.Users have the possibility to select façades of interest for the reconstruction process.Prior to the reconstruction, points belonging to the selected façade have to be extracted from the point cloud.For that purpose all points have to be transformed into a local coordinate system defined by the selected façade.Only points within a clip box around the façade are stored for further processing.Figure 3 shows exemplarily a selected façade (depicted in green) with the corresponding local coordinate system.The underlying building model consists of LOD2.
Typically city administrations provide these data models for urban areas nowadays (Haala and Kada, 2010), which is an additional reason to use it as input source.based approach models the remaining building and leads to CityGML model in LOD3.

Geometric Reconstruction
The generated point cloud and its corresponding LOD2 building model are used to reconstruct façade primitives to subsequently enrich the model to LOD3 representation.Point clouds from LiDAR or dense image matching are possible input sources.Within the scope of this work a tool was implemented, which extracts points of a point cloud based on façade planes from a CityGML model.

3D Cell decomposition:
Our approach initially fits a plane to the façade point cloud.Based on this plane, a presegmentation into façade points and non-façade points is realised.Subsequently, horizontal and vertical edge points located at the borders of windows and doors are searched.Four categories can be distinguishedupper, lower, left and right edge point.An upper edge point for example is considered to be found if inside a given buffer no further façade point is located below the query point.This way all façade points are tested if they belong to one of the four edge point categories and as a result window edges in both horizontal and vertical direction are found.By means of the extracted edges and the estimated window depth the coarse building model is into three dimensional cells.Initially, these cells have only geometric information.The top left of Figure 4 shows the detected window edges and resulting cell decomposition.On the top right, the reconstructed model is depicted.The bottom row of Figure 4 shows the enriched LOD3 model generated by the grammar based approach.

Classification:
Subsequently, the previously generated cells are classified into façade and window cells with help of a point availability map.This results in the reconstructed façade, which serves as input for a grammar based approach.There, rules are derived from the façade structure and applied to model the remaining building parts.The implemented GUI visualises intermediate steps and gives the possibility to adapt parameters like the clip box of point selection for testing purposes.However, the described reconstruction of doors and windows presumes, that they differ significantly from the main façade plane in depth.Which means that main façade and primitives should not be coplanar.Our geometric reconstruction is additionally aggravated by the limited accuracy of point cloud coordinates.Figure 5 gives an example of particularly suitable input data.The upper part of the depicted building was acquired by a terrestrial LiDAR sensor.Conditioned by the acquisition technique, penetration of glass leads to no proper reflexion of the laser beams and therefore windows appear as holes in the point cloud.Lateral and upper edges of window frames, as well as curtains in front of windows lead to reflexions.However, as shown in Figure 5 in the top middle, these points are clearly shifted in depth and therefore a segmentation into façade points and non-façade points is accomplished quite well (Figure 5 top right).The bottom of Figure 5 depicts the result of the 3D cell decomposition, where all windows could be correctly reconstructed.Difficult viewing angles and acquisition platforms in rapid motion lead to façade clouds with much higher clutter in depth.Additionally, reflective surfaces cause difficulties in the matching process and result in noisy data.Whereas LiDAR has its advantages in reflective surfaces for our reconstruction process, 3D data generated from dense image matching leads to problems in these areas.Due to uncertainties of the points in depth façade and primitives might become coplanar and not be distinguishable anymore.In this case, geometric reconstruction will presumably fail.As will be discussed in the following section though, radiometric information contained in point clouds generated from imagery can help to overcome these difficulties.

Radiometric Segmentation
Radiometric segmentation is investigated under the assumption that homogenous façade parts have the same or very similar colour schemes.This fact is not only based on architectural and symmetric aspects but is also often conditioned by the used construction material.Thus, topologically coherent as well as scattered, yet radiometric similar regions are searched.A complete overview of the radiometric segmentation approach can be comprehended in Figure 6.
To enable reconstruction even for point clouds with limited geometric accuracy, the radiometric data for each point is additionally used during segmentation.The outcome of the consideration of radiometry is a labelling into two classesfaçade points and non-façade points, analogous to the geometric segmentation.For the segmentation process, first RGB colour information of each point is transformed into HSV space.Only the hue channel is regarded as it can be considered independent from lighting conditions.For colour-based clustering of the cloud all points are mapped to bins according to their hue value.Thereby a hue histogram is constructed.Additionally, local features are computed for every point.Hence, for each individual point a K-nearest-neighbour search is performed.The first point feature, a local hue difference is determined by determining the median of all differences.Median is preferred over mean to give robustness against outliers in hue space.Another feature represents the local planarity for a point.A plane is fitted to all nearest neighbours and the distance to these points is stored.Again, the median of the distances is picked and represents the planarity for the current point.Subsequently, for each cluster the mean hue and mean planarity is determined.If both, mean hue and planarity are below a defined threshold, homogenous façade plane points are supposed to be found.Different segmentation approaches are currently still investigated with respect to their capability and run time.However, the final embedding of the radiometric support should work smoothly since it can be regarded as a separate module delivering a labelled point cloud.By segmenting the points into two disjunct classes a differentiation into façade points and non-façade points is possible.This way the pre-segmentation in the actual geometric reconstruction process can be bypassed.Subsequently, the following steps are performed as before within the geometric reconstruction.Figure 7 shows some results of the radiometric segmentation.Building primitives, roofs, as wells as indents can be distinguished from the main façade plane quite well.Figure 8 shows an example with very low radiometric information.Note, that this is the same building as shown in the example of Figure 5.However, this time the point cloud was generated from oblique airborne imagery.In this case, the segmentation quality decreases due to the low radiometric information.Nevertheless, with an adapted cluster size in contrast to Figure 7 most of the primitive outlines are still detected.Table 1 depicts   A tile either consists of a homogenous wall part or contains a geometry object.Thus, a sequence of façade and geometry tiles emerge which can be considered as topologically correct, if it begins and ends with a wall segment and both tile types alternate in between.In other words, the same tile type must not appear consecutively.Using a sequential cluster approach the tiles are classified by means of their size and type.This results in different tile types to which correspondingly a symbol is assigned.The relation or rather sequence of symbols is then further investigated.The objective is to compress the symbols by replacing redundant sequences by a new symbol.Thereby hierarchical relations between the tiles are detectable and can be expressed by replacement rules, as well as production rules can be derived.Afore-gained knowledge can be used for the modelling.The entirety of production-and replacement rules depicts the façade grammar.

Model-based completion
The façade grammar gathered in the previous section can be used to model the remaining façades of a building synthetically.Based on the previously determined number of floors, for each of them an application of the production rules is performed.Starting point is respectively the centre of the façade since it contains the main appearance features most of the time.This way, first the middle and subsequently the remaining façade parts are modelled with new tiles from left to right.If there are multiple production rules possible for a new tile, a corresponding rule is selected probability-based.

EXPERIMENTAL RESULTS
Figure 9 shows the outcome of the algorithm.The upper part contains some bureau complexes as well as single-family homes.Below a complete block of buildings along a street is depicted.
Additionally, Figure 10 shows the advantage of working with georeferenced data.After modelling, the generated building or block of buildings can be directly viewed in the global coordinate system and furthermore converted into the kmz format e.g., hence giving the possibility to display the results in Google Earth.

CONCLUSION
Within this work an approach for exploitation of geometric and radiometric point cloud information has been presented.Erroneous or incomplete point clouds due to occlusions or lack of image coverage can still serve as data basis for the modelbased approach that uses automatically derived grammars.In terms of very noisy data where geometric reconstruction comes to its limits, a radiometric segmentation can support the reconstruction process.The implemented GUI serves as tool for interactive reconstruction and adaption of parameters.
Yet, the existing approach should be extended in a way that for coarse building models with point cloud coverage on more than one façade reconstruction is performed individually.So far detailed analysis of results is not feasible in a practical manner since there is a lack of ground truth.Hence an evaluation of the correctness of the reconstructed building becomes cumbersome.The reason for the application of grammar based completion is at the same time the reason why even visual evaluation is sometimes prevented -due to the nature of the input data large parts of the building might not have been acquired at all.Therefore the generation of a benchmark dataset for terrestrial building or rather façade reconstruction with geometric as well as semantic annotation would be an interesting task for the future.This way, prospective as well as existing approaches could be evaluated in terms of their geometric, as well as semantic correctness.In the future, point cloud data of higher quality, could lead to even better results for building modelling.Data acquired by Mobile Mapping can exploit its ground proximity and therefore cover lower building parts with high density.
Whereas point clouds generated from oblique airborne imagery might provide better results for upper building parts.For that reason fusion of both sources covering the same area could provide an interesting data basis for façade reconstruction and modelling approaches.Apart from that, processed results of point clouds in terms of meshes with sharp edges on building borders might serve as additional data source or even make coarse building models redundant.However, this requires intelligent meshing as well as noise-robust point clouds.

Figure 1 .
Figure 1.GUI showing CityGML LOD2 overlaid with Mobile Mapping point cloud

Figure 2 .
Figure 2. Overview of the presented approach

Figure 4 .
Figure 4. (a): Cell decomposition of façade shown in middle row of figure 7, (b): Reconstructed model, (c) and (d): Enriched CityGML LOD3 model c as façade else label points in c as non façade end

Figure 7 .
Figure 7. Left column: Point cloud from Mobile Mapping data.Right column: Segmentation into façade plane (green points) and primitives (red)

Figure 9 .
Figure 9. LOD3 results for grammar based modelling