DATA OPTIMIZATION FOR 3D MODELING AND ANALYSIS OF A FORTRESS ARCHITECTURE

: Thanks to the recent worldwide spread of drones and to the development of structure from motion photogrammetric software, UAV photogrammetry is becoming a convenient and reliable way for the 3D documentation of built heritage. Hence, nowadays, UAV photogrammetric surveying is a common and quite standard tool for producing 3D models of relatively large areas. However, when such areas are large, then a signiﬁcant part of the generated point cloud is often of minor interest. Given the necessity of efﬁciently dealing with storing, processing and analyzing the produced point cloud, some optimization step should be considered in order to reduce the amount of redundancy, in particular in the parts of the model that are of minor interest. Despite this can be done by means of a manual selection of such parts, an automatic selection is clearly much more viable way to speed up the ﬁnal model generation. Motivated by the recent development of many semantic classiﬁcation techniques, the aim of this work is investigating the use of point cloud optimization based on semantic recognition of different components in the photogrammetric 3D model. The Girifalco Fortress (Cortona, Italy) is used as case study for such investigation. The rationale of the proposed methodology is clearly that of preserving high point density in the model in the areas that describe the fortress, whereas point cloud density is dramatically reduced in vegetated and soil areas. Thanks to the implemented automatic procedure, in the considered case study, the size of the point cloud has been reduced by a factor ﬁve, approximately. It is worth to notice that such result has been obtained preserving the original point density on the fortress surfaces, hence ensuring the same capabilities of geometric analysis of the original photogrammetric model


INTRODUCTION
Given the complexity and specificity of restoration issues in medieval fortress architectures, detailed, accurate and reliable analysis of the current status of such cultural heritage architectures should be done in order to provide appropriate information to restorers.In particular, the specific structure characteristics and the consequent applicable conservation and valorization strategies are fundamental factors to be taken into account in order to properly design the 3D survey and successfully extract and fully exploit the information from the obtained 3D model of the structure.
This work considers the survey and 3D modeling process of the Girifalco Fortress (Cortona, Tuscany, Italy).A fortress on the Cortona hilltop was originally built in the 5th century BC.However, after such building has been damaged during battles, hence it has been rebuilt during the 17th century: the original walls were substituted in order to cope with the newly developed cannons.The current look of the fortress (visible in Fig. 1) is actually the same obtained after the realization of such new walls.The cultural heritage and historical importance of the fortress is also due to the famous families that owned it, and, in particular to the De Medici family, which is worldwide known for its important role during Italian Renaissance.
Since the final interest in this work is that of providing suitable information to the restorers, this work aims at investigating a proper integration between restoration goals and geomatics tools.The rationale is that of designing a survey and data processing driven by the needs of extracting the information suitable for properly supporting the conservation and restoration process.
A preliminary step was the definition of the survey goals, in order to determine the building characteristics to be determined and detectable in the produced 3D model.Then, both UAV photogrammetry (Fig. 2 shows an orthophoto produced by properly processing UAV imagery) and Terrestrial Laser Scanning have been used in order to acquire the 3D information of interest for properly describing the fortress.Nevertheless, this work focuses on the automatic processing of the 3D model obtained by means of UAV photogrammetry.UAV photogrammetric survey of the Girifalco Fortress has been carried out on September 8, 2017, with the drone DJI Phantom 4. The UAV flew at an average altitude of 40 m over the ground (ground sample distance was 1.4 cm, approximately), acquiring 960 images of the fortress area, with (at least) 60% of overlapping between images (both along the same strip and between the closest images from different strips).Given the importance of properly describing the fortress walls, both nadir and oblique images have been collected during the flight.Actually this ensured also a better camera network geometry (Fraser andStamatopoulos, 2014, Chiabrando et al., 2017), which is of fundamental importance in order to ensure reliable reconstruction results, in particular in the camera self-calibration case.
Then Agisoft Photoscan 1.3.2 has been used in order to obtain a photogrammetric 3D model of the fortress.To this aim, the photogrammetric has been aided by the use of 27 targets, whose positions have been measured in a GNSS RTK survey, with average accuracy (root mean square (RMS) error) of 1.6 cm. 10 targets have been used as control points, and 7 as check points, leading to an average planimetric and altimetric error of 2.0 cm and 2.4 cm, respectively.
Agisoft Photoscan generated a photogrammetric point cloud of 40 Million points.The average point spacing on the central part of the model (the fortress area) is 3.0 cm.
Actually, 3D modelling of large sites can lead to huge point clouds, which can lead to dramatically long computational times for processing and information extraction from such models.Consequently, several works recently considered the problem of reducing the size of the generated point cloud while preserving its geometric information in order to ease the processing phase and make it faster (Błaszczak-Bak, 2016).However, this work aims at exploiting a semantic interpretation of the points in the model in order to properly determine which areas should be sub-sampled more.To be more specific, since the fortress walls in the 3D model shall be further analyzed for restoration purposes, the goal is that of automatically extract such parts from the model and preserve their model resolution, whereas, the other parts of the generated point cloud (e.g.vegetation) will be significantly subsampled (actually they can also be discarded, if needed).
Aiming at formulating a processing strategy that can be used for easily managing, interpreting, understanding and extracting information from the generated model, in this work the following steps of 3D data processing has been considered: • automatic recognition of vegetated areas, by exploiting both spatial characteristics of tree point clouds and their intensity measurements (on either RGB imagery or LiDAR), • optimization of the point density in order to make the 3D model easier to be managed.This step can take advantage also from recently developed data optimization methods (Błaszczak-Bak, 2016), if needed, • in particular, the produced dataset is optimized in order to ease the automatic extraction of information about the building walls.This kind of procedure can be useful for instance for the automatic generation of semantic models (after a proper segmentation and classification (Vosselman et al., 2004, Rabbani et al., 2006, Makuti et al., 2018)), e.g.BIMs and CityGML models.
The the paper is organized as follows: first, Section 2. presents the proposed point cloud segmentation and classification procedure.Then, Section 3. deals with the data reduction and optimization problem.Finally, some conclusions will be drawn on the proposed procedure and on the possibility of automatically extracting information about damages on the building walls in Section 4..

SEGMENTATION AND CLASSIFICATION
This section aims at the automatic extraction of certain geometric information from the photogrammetric point cloud.In particular, it is worth to notice that vegetated areas are of minor interest in the 3D model of the reconstructed fortress.Furthermore, the generated point cloud is very noisy on the vegetated areas.Since most of the fortress elements can be well approximated with simple geometric shapes (e.g.planar surfaces), an automatic procedure based on the Random Sample Consensus (RANSAC) (Fischler and Bolles, 1981) is used in this work in order to extract the fortress parts from the point cloud.However, the presence of very noisy vegetated areas can negatively affect the outcomes of the RANSAC algorithm.Consequently, a vegetation classification and segmentation procedure is applied before detecting the surfaces associated to the fortress.Despite the different implementation, the rationale of the proposed approach is relatively similar to the semantic segmentation principle (Long et al., 2015, Girshick et al., 2014, Noh et al., 2015).

Predictors
Similarly to certain land cover classification methods (Huang et al., 2002, Foody, 2002, Gislason et al., 2006), intensities of the radiation coming from the area of interest and measured at several wavelengths can be used to estimate the vegetated areas.
However, differently from multi-spectral classification techniques, which can exploit the Normalized Difference Vegetation Index (NDVI) in order to make such classification, in this case the use of a standard camera does not allow to compute such index.Nevertheless, such intensities can clearly be indicative of the vegetation presence.In particular, the ratio between the green component and each of the other two is used in this work.
Actually, the availability of 3D information about the point positions and about the local point densities can also be exploited in order to determine if a point correspond to a vegetated area.
To this aim, an approach based on considerations similar to those presented in (Habib and Lin, 2016) has been implemented.A local surface fitting is used to check whether points approximately lie on planar surfaces.For each considered point, all its neighborhood contained in the sphere centered in such point and with radius 0.5 m are used in order to determine the local variability of the point cloud in such area.In particular, the covariance matrix of such point positions is computed: points on a fortress wall shall form a locally smooth surface, hence the third eigenvalue of the computed covariance matrix should be small.Differently, points on vegetated areas should be associated to comparable values of the covariance matrix eigenvalues.
It is worth to notice that: • if the last eigenvalue of the covariance matrix is small with respect to the other ones, the neighborhood of the considered point can be quite well approximated by a planar surface • when the last eigenvalue is not small with respect to the others it is not possible to automatically conclude about such point being or not part of the vegetation.
In accordance with the description provided above, the smallest eigenvalue of the local sample covariance matrix, along with the local point density and with the point intensities, is used as a predictor of the point class, as described in the following subsection.

SVM classifier
A number of methods have been proposed in the literature in order to properly classify data appertaining to certain categories.
In particular machine learning and deep learning techniques have been widely used in order to improve the performance of classifiers.In particular, certain features are usually either automatically determined (as usual in deep learning methods) or manually selected (machine learning) is order to ease the solution of the classification problem (LeCun et al., 2015, Bishop, 2006, Facco et al., 2013).
Among the possible methods, a Support Vector Machine (SVM) classifier has been used in this work (Suykens and Vandewalle, 1999).Two rectangular regions, containing approximately 1000 points each, have been manually selected on a fortress wall and on a vegetated area on the north-east area of the reconstructed model and their predictor values fed into the SVM in order to properly learn the classifier parameters.Then, the trained SVM is used to classify the vegetated areas on all the model.).Such result is also confirmed by the top view of the fortress model after filtering out the vegetation points (Fig. 5).

Detection of planar surfaces
Once the vegetation has been filtered out, the remaining point cloud (Fig. 5) is used in order to detect the fortress surfaces.
In particular, first, RANSAC is used to detect vertical surfaces.Then, horizontal regular surfaces are determined: surfaces at heights comparable with the altitudes of the walls are classified as ceilings, whereas the other points (and planar surfaces) are generically classified as "ground".Actually, it is worth to notice that "ground" points can be segmented further in order to determine the presence of other objects (e.g.cars).Despite this can be of great interest in several applications, for instance for digital terrain model generation and for more advanced information extraction from the point cloud (Costantino and Angelini, 2011), this is out of the scope of this work.

SEMANTIC BASED MODEL OPTIMIZATION
The previously presented segmentation procedure provides a semantic classification of the 3D point cloud that can be conveniently exploited in order to change the size reduction of each point class depending on the specific user interests.
In practice, since in this work the main interest is that of preserv- ing as much geometric information about the fortress building as possible, all its points are kept.However, if a higher compression is needed, model compression can be obtained by exploiting for instance the approach presented in (Błaszczak-Bak, 2016) for point cloud optimization while maintaining most of the geometric information of the original model.
Differently, here vegetation and ground areas are highly compressed, by a factor (greater than) 100 and 20, approximately.
Given the minor interest in these areas, random sub-sampling is applied to such points in order to obtain the desired compression at a low computational cost.Nevertheless, point cloud optimization can be applied to them as well, if needed.
The resulting model is shown in Fig. 7, where is quite clear the different point density for the considered classes.The obtained model has a significantly smaller point cardinality with respect to the original one (it is reduced by more than a factor five).

DISCUSSION
This paper presented a point cloud segmentation and compression method, inspired by semantic segmentation, in order to reduce the of point cloud size while preserving most of the geometric information of interest.Indeed, since the currently common methods for 3D modeling of built heritage usually provide huge point clouds including also parts of minor interest for the user, it is important to reduce the size of such models in order to ease the computational burden for analyzing it and extracting any information of interest, and also to reduce the storage requirements for such models.
Actually, when dealing with applications of cultural heritage restoration, it is of fundamental importance to keep as much information as possible about the object of interest.This motivated the development of an automatic segmentation procedure in order to identify the most relevant parts of the model point cloud.
Thanks to such implemented segmentation and classification procedure, then it is possible to apply segmentation based point cloud optimization: the optimization function can be tailored to the user's needs and specified differently for the identified point classes.
In the considered case study this allowed the application of very different compression factors to the determined point classes.Despite the points associated to fortress surfaces (e.g.walls, ceilings) have not been sub-sampled, the overall point cloud compression in our case study was more than a factor five.Furthermore, the application of recently developed point cloud optimization methods can reduce the cloud size further while minimizing the loss of geometric information (Błaszczak-Bak, 2016).
Thanks to the preservation in the final model of the original point density, then, similarly to (Masiero et al., 2019), the obtained model is suitable for the application of surface and structural analysis (Korumaz et al., 2017, Tucci et al., 2016, Masiero et al., 2015).
A secondary result of the applied method is the production of a model affected by less noise, and, in particular, once properly improved, the ground classification can also be considered for the generation of digital terrain models of the surveyed areas.

Figure 3 ,
Figure 3, 4 and 5 show the results of the classification.In particular, Fig. 3 shows the vegetated areas, whereas Fig. 4 compares the side view of the model before (a) and after (b) filtering out the vegetation points.It is worth to notice that, differently from the original point cloud, the filtered point cloud contains much less noisy points (Fig. 4(b)).Such result is also confirmed by the top view of the fortress model after filtering out the vegetation points (Fig. 5).

Fig. 6
Fig. 6 shows the resulting classification: "ground" points are shown (a), whereas the extracted fortress walls and ceiling are shown in (b) and (c).

Figure 5 .
Figure 5. Top view of the point cloud generated by Agisoft Photoscan after vegetation removal.

Figure 6 .
Figure 6.Points labeled as ground (a) and as part of the fortress (side view (b), and top view (c).