MOBILE PHONE IMAGING FOR CH FAÇADE MODELLING

: The use of digital images as a source data for analysis and knowledge of the built environment is a common and well-known topic, which has seen a significant increase in the last fifteen years thanks both to algorithmic and technological development. One of the sensor families which has shown significant development in the last period are those mounted on smartphones, which offer ever increasingly advanced solutions from the technological and technical point of view. Considering the portability, handling and continuous availability of these instruments, it is legitimate to wonder if their cameras can be considered nowadays an effective working tool for architects and restorers. So, is the metric data, that comes from smartphones images acquisition and processing, reliable and usable for the two main phases of analysis and representation of the built environment? Starting from the case study of a façade of a church in the historic centre of Rome, the article aims to determine the reliability of the data acquired from a smartphone for architectural analysis and deterioration mapping.


INTRODUCTION
Architectural survey, meant as a process of investigation about building geometric, spatial and material consistency, represents a basic research tool to frame the state of art of the artefact respect to its phases of life and its historic stratification.An accurate survey allows to obtain structural data, geometric space proportions, materials analysis, promoting a typological, structural and functional knowledge of the building (De Angelis D'Ossat, 1971).Besides, a precise graphic reproduction of the data acquired force to study and understand in depth geometrical and materials characteristics, state of deterioration, alignments, solutions of continuity, all signs of the historicalarchitectural peculiarities.In the last decade the increasing use of close-range photogrammetric applications to acquire and represent architecture has showed the possibility to survey complex buildings with a multi-scale approach, analysing both the global artefact and the single details, obtaining reliable models from which extract orthogonal projections, sections, axonometries.In addition, the presence of an accurate radiometric data greatly helps in the identification of possible phenomena of surface degradation.Although this consolidated use in architecture and restoration, the possibility of applying smartphones for photogrammetric acquisition of architecture presents several issues, that must be considered in advance in order to understand and justify their application.First, it is a fact that in recent years the introduction of increasingly high-performance cameras equipped with multiple sensors and lenses has led to acquire images closer to the real data, opening a company competition on a global scale.Moreover, the smartphone camera simplicity and availability by end-users has certainly contributed to exponentially increase the sharing of digital images.Finally, a ubiquitous connection to the web network boosted data transmission, favouring online platforms for the generation of 3D models, providing both remote or local data processing which are increasingly accessible to non-experts.The coexistence of these conditions leads to wonder if today a smartphone can be used also as a working instrument by an architect or restorer.These professional figures may not have high skills in the photogrammetry field, but they need metric and radiometric reliable data in relation to the scale of representation, carrying out the different steps of representation and analysis of the building.The article, starting from a state of the art on smartphones imaging, aims to verify this possibility in the field of Cultural Heritage.For this, a survey of a church façade in Rome is performed by smartphone application.The process foresees a planning step, an acquisition campaign, a modelling passage and a metric comparison between the photogrammetric model and a gold standard given by a 3D laser scanner.At the end some conclusive considerations about the potential and limits in the use of smartphones in the representation and restoration of Cultural Heritage are presented.

STATE OF ART
The 3D survey methodologies based on close-range photogrammetry (Kraus, 1997) and SfM (Szeliski, 2010;Luhmann et al. 2013) have already presented many applications in recent history, from the first experiments on the reconstruction of 3D models from stereo-images (Pollefeys et al. 2003) to the comparison between active and passive systems (Remondino, El-Hakim, 2006) up to the definition of multiscale and multi-resolution methodologies (Remondino et al. 2014).In the Cultural Heritage field, many experiments concerned the comparison between different sensors (Remondino, 2011) framed in archaeological and architectural applications (Roncella et al. 2011).Their application has seen the definition of architecture photo-modelling activity (De Luca, 2011), the introduction of protocols aimed at 3D modelling definition and optimization (García-Gago et al. 2014) and a general comparison between photogrammetry and computer vision, analysing pros and cons of each single approach (Aicardi et al. 2018).In CH domain a specific declination is represented by the acquisition and 3D modelling of mainly flat built elements or territorial area and the generation of ortho-images.Some important applications in such sense can be the survey of archaeological sites (Chiabrando et al. 2015), the acquisition of building façades (Russo et al. 2018;Carnevali et al. 2018) or the sampling activity of painted portions (Chiabrando et al. 2014).All these processes have been strongly supported and boosted by the introduction of drones in different fields (Nex, Remondino, 2014), improving the use of aerial images and the integration with terrestrial ones.The application of smartphones for survey purposes has certainly took advantage from the advances in imaging field.
Besides, also the ranging scale has been verified, starting from the architectural analysis (Boboc et al. 2018) to the sculptural and decorative studies (Di Paola, Inzerillo, 2018).The image acquisition activity has been tested in static conditions with inertial sensors (Pintore et al. 2016) and working in real time set-up (Ondruska et al. 2015;Al Hamad, El Sheimy, 2014).
The great potential of these systems is mainly due to the integration of functionalities for the end-user, which benefits from the considerable hardware and software development of the systems (Valentini et al. 2018), from a more close to real radiometric acquisition (Leão, Westland, 2019), from a simplification in the data transmission (Daffara et al. 2019) and from the presence of web platforms devoted to the construction of 3D models from images (Tefera et al. 2018), which can be acquired in a stand-alone or collaborative working mode (Snavely et al. 2006;Golodetz et al. 2018).

CASE STUDY
The case study defined for the experimental phase refers to the church of San Rocco (Figure 1), framed in Campo Marzio district in Rome, an area which once housed the Porto di Ripetta.The harbour was designed and built in 1704 by the Italian architect Alessandro Specchi and demolished at the beginning of XX century during the construction of the new Muraglioni del Tevere (Lombardo, 2009).In 1499 Pope Alexander VI authorized the Confraternity of San Rocco for church construction adjacent to the Mausoleo di Augusto.
Figure 1.In the high image sequence, the historical sources relating to the San Rocco façade evolution, at the bottom the actual façade and the inclusion in the Roman building context.
The new church faced the river and not eastward and originally had a brick façade, a central portal surmounted by a tympanum, two side doors surmounted by two arched windows and a clock in the tympanum, as reproduced in the Rome guide by Girolamo Francini of 1558.The building was consecrated in 1502, year in which began the construction of the contiguous hospital, governed by the Confraternity until 1866.
Over the centuries the church has undergone many transformations and extension works that completely changed its original aspect, such as the 1833 project by Giuseppe Valadier for the new façade.The project of Valadier, of Palladian inspiration, is characterized by a base with steps interposed between the street level and the church floor, running along the entire facade and supporting the pedestal of the giant order of the central part while on both sides there is a minor order with half tympanum corresponding to the secondary aisles (Salerno, Spagnesi, 1962).After the demolition of the Porto di Ripetta and the construction of the Muraglioni, between 1893 and 1896, the street level was raised, covering the access stairs, drastically changing the facade.

Instruments
Some preliminary checks were carried out on the smartphone image acquisition process.This test phase has been necessary to understand the geometric basic principles in a multiple cameras and sensors system (Figure 2), evaluating working data extremely difficult to obtain from the manufacturers, for obvious secrecy reasons.In  The experiment has foreseen the acquisition of a geometric grid and a colour checker, to verify and compare the two cameras used alone and together (Figure 2).Not being able to disable cameras at software level, a black scotch on the corresponding lens has been placed alternatively, cancelling its contribution in the image acquisition process.The test showed that only the 16 Mpix lower camera is used for image acquisition, while the upper one integrated with AI system plays an important role only in portrait mode, improved by Bokeh effect.That behaviour has been demonstrated also by the comparison between the single camera and the integrated system, which highlighted no difference in image quality, colour or pixel position.For this, the centre of the projective geometric schema has been placed in the bottom camera.
Besides, a 3D phase shift laser scanner (Focus 3D X120, Faro) has been employed to acquire a reference range cloud.

Survey campaign
Initially, a survey campaign was planned with a 3D laser scanner to obtain gold standard data, as well as using a common instrument in the architecture and restoration field.The acquisition process has been carried out with a the 3D laser scanner, planning three main stations: one located at the centre of the balustrade (close to point 3, see Figure 4), to obtain the best frontal view reducing the shadowing effects, and two at the sides (close to 6 and 10 points, see Figure 4).For the central acquisition, a sampling step of 3mm@10metres has been set, in order to collect the whole surface with a high sampling step.The lateral scans were set with a resolution of 6mm@10meters.The average acquisition uncertainty has been 0.5 mm (1 sigma).The survey photogrammetric project has been planned carefully, considering the possibility to introduce both horizontal and vertical baseline, thanks to the presence of Ara Pacis building in front of the church (Figure 3).A balustrade at a height of 3 meters from the road plane and an access corridor located about 1 meter below the road were used as acquisition points, introducing a vertical baseline of 4 meters and a horizontal one of 5-6 meters, respect to a working distance of 10 to 15 meters (Figure 4).Smartphone has been set-up in manual mode, 1/800 shutter speed, focal length of 4 mm, ISO 400.In order to cover the entire façade with an optimal image distribution, 10 acquisition points have been planned, acquiring for each one 6 images, one with an axis perpendicular to the façade and 5 with a sloped axis (see diagram).The photogrammetric block has been planned to preserve an overlap of more than 60-80% in the two main directions.
The average ground sampling distance has been 3,6 mm, as shown in Table 2, respect to the main three working distances.
The GSD and working distance analysis refer to the nadiral images, but the massive presence of oblique images introduces a string scale variation, considering a maximum working distance of 21 meters and a GSD of 6 cm.

Data process
The range maps alignment has been carried out with JRC Reconstructor software (Gexcel), obtaining a unique point cloud of 8,3 x 10 6 points.It was than cleaned, eliminating the outliers present in the scene.That allowed obtaining two different output for comparison step: the raw data coming from the first central scan and a complete aligned point cloud coming from the whole range survey.On this latter a coordinates extraction of visible points has been carried out, in order to define a GCPs system to apply in the photogrammetric bundle adjustment process (Figure 5).In this extraction there is a clear awareness that the precision in the determination of these points has been conditioned by the sampling step, the accuracy and the uncertainty of the instrument, very different from a topographic result.But the intention of verifying the reliability of the photogrammetric product is comparative with 3D laser scanner output, so the point cloud has been considered as a gold standard both in the punctual and global verification.Besides, the images were treated within Metashape (Agisoft), through the well-known photogrammetric and SfM data process.After orienting the images, the GCPs coordinates have been imported in the program, using only a part of all GCPs for the bundle adjustment process of the photogrammetric block.The deviations obtained from the check points and control points (Table 3) showed an error contained within 14 cm, with an increased error in the farer points.The lower error refers not only to the closer points respect to the camera acquisition but also to that points laying on planar surfaces with small corners or low façade deep variations (i.e. points 6 and 11).The sparse point cloud was then translated into a dense point cloud of 12,3 x 10 6 points and a numerical model of 25,3 x 10 6 polygons.from which the relative orthophoto was extracted, with a resolution of 1,09 cm/pix (Figure 6).The deviations obtained by the GPCs are clearly consistent, but coherent both with the camera limitations and the GCPs reliability.Therefore, they are not enough to understand whether the orthophoto can be used in architecture or restoration analysis, for this reason a geometrical and visual comparison step has been planned between image and range data.

DATA ANALYSIS
Data analysis and comparison has been planned in three different steps: a metrical verification of the photogrammetric point cloud, a geometric comparison between the orthophoto and the geometry of the façade, a visual verification of the orthoimage quality for architectonic investigation, material recognition and degradation analysis.The first step focused on the metrical analysis of the point cloud, obtained from one side through sections extraction and comparison, from the other from range to image point cloud superimposition.The sections analysis has shown the presence of a not very noisy data, with some lacks in architectural edges definition, such as the rocks of the columns, pilasters or cornices (Figure 7).This depth error, almost due to the optical limitations of the camera, prevents the recognition and classification of some portions of the architectural system, showing on the opposite a good reliability on regular wide surfaces.The comparison between the two-point clouds verified this interpretation, highlighting a deviation map with a consistent error distribution on the entire façade.In this superimposition low deviations in correspondence of the flat surfaces has been defined, while high noise values were concentrated in correspondence of facade break lines (Figure 7).The cloud comparison has been carried on using a single range map acquired from the central point, limiting accuracy and uncertainly only to instrumental errors.The second step has been focused on the comparison between the drawing extracted from range data and the orthophoto obtained from the photogrammetric point cloud.Both these two outputs are metrical data, but they represent a results of a postprocessing passage and/or personal interpretation step.The drawing representation is defined as the result of a strong interpretative process respect to a discretized reality defined by the range point cloud, while the orthophoto is the final output of a re-projection process, determined by the definition of an arbitrary projection plane and an almost unpredictable texture re-projection.The comparison scope of these two indirect metric outputs is to verify the reliability of the orthophoto on the XY plane, that is, the plane parallel to the facade, for drawing purposes, which is one of the most important passage in the architectonic representation and analysis.
The comparison has shown a quite good global correspondence of the whole façade (Figure 8) Going into detail, few centimetres of variation are highlighted in the lower part of the church, while a shift close to 15 centimetres is present in the upper part (Figure 8).A careful analysis of the comparison has shown a roto-translation of the latter, with a behaviour that may find an answer in several factors, as the optical characteristics of the camera, the working distance, the acquisition of many oblique images, with relative GSD scale variation, and not perfectly reliable GPC.
The last analysis has been based on visual approach, through a careful study of the architectonic and material elements present in the façade and showed in the orthoimage.This passage was aimed at reading the palimpsest from an architectural point of view, carrying out a degradation mapping activity.The general view allowed to frame clearly the architectural style, with the two orders, the relationships between the vertical supports, the architraves, the tympanums, the cornices, the large openings and the panels in the background.The colour of material and finishing treatment, such as plaster, travertine and marble, were clearly recognizable, identifying certain deterioration effects as incrustations, biological attack, rainwater runoff and thin surface deposits.Besides, a strong limitation in the correct interpretation of some architectural elements is evident, due to the presence of wavy or broken silhouettes.These may not allow to recognize these areas (Figure 9), making it difficult to analyse the historical and stylistic function in relation with the measurement unit and proportioning systems used for monument construction.In addition, this "wavy" restitution of horizontal architectonical edges or the vertical grooves of the columns limits the correct identification of some important deterioration aspects such as the disintegration, detachment or surface coating lack.At the end, shadowing effects due to the working position lead to an important data loss, which represent an unresolved question respect to the simple use of camera without RPAS or telescopic poles.Close to all these areas, some façade portion resulted distorted (Figure 9), probably due to a compensation passage, which supply additional unreliable data for analysis purposes.

CONCLUSIONS
The technological smartphones evolution of the last years leads to the question of whether their low-cost sensors can be used as a working tool by architects and restorers.Undoubtedly the photogrammetric process with smartphones cameras applied in CH field for the reconstruction of small and medium sized artefacts is a well-known aspect, while their application at architectural scale still presents several bottlenecks.The article tries to suggest an answer in this sense, showing a methodological path of acquisition, data processing and comparison of a church façade in Rome.The results obtained from the experimentation lead to some initial observations.The photogrammetric point cloud presents several lacks in the tangential surfaces with respect to the camera axis and more generally in the façade edges, while a good data quality in the flat surfaces, parallel to the acquisition plane, has been highlighted, with an average distance from the reference range cloud of less than 10 cm.The geometric comparison between the elevation drawing extracted from the range cloud and the orthophoto confirmed a good reliability of the data up to the first string-course of the façade, while an increasing error occurs in the upper part, with a maximum of roto-translation around 15 cm.Finally, the visual analysis has emphasised a good photographic quality, allowing to start a degradation mapping and a material analysis, but showing on the other side some important limitations in the missing part definition due to the presence of many irregular edges, not clearly dependent to the façade morphology or error in the acquisition phase.In general, the result has been comforting, considering the size of the sensor and the lens used respect to the façade dimension.
The application of a smartphone has showed a very easy and quick acquisition, certainly useful for a first dimensional and material analysis, but the metric and radiometric output is not completely exhaustive compared to the needs of analysis of the monument.Some shortcomings can be easily overcome by integrating the survey with detailed photos, but a slight dimensional approximation of some construction lines of the architectural system remain.
In conclusion, the case study discussed has demonstrated that is possible to carry on an architectonic and restoration analysis of a big façade with smartphone, obtaining a reliable point cloud in the most regular areas for a global volumes' evaluation.
The restitution and restoration analysis from the orthophoto can be fixed at a scale of 1:100 on the condition that the results and measurement errors obtained at the lower part of the facade are gained for the entire building.That should happen foreseeing a topographic survey, in order to minimize errors in GCP determination, introducing in the meanwhile a third vertical baseline, which would improve the images distribution of the photogrammetric block, decreasing the scale variation.
For the future, the progressive improvement of cameras performances connected to smartphones, the increasingly possibility to manage photogrammetric blocks with strong scale variations and the application of image recognition algorithms lead to hope in an increasingly effective use of smartphones in the field of architecture and restoration analysis.

Figure 2 .
Figure 2. Above the external and internal back view of the A6000 (OnePlus) smartphone, below two testing phases of the smartphone camera with b/w and coloured targets.

Figure 3 .
Figure 3. Road section with walking plane (in red) and two photogrammetric lines of view from the balustrade and the underground corridor beside Ara Pacis entrance.

Figure 4 .
Figure 4. Photogrammetric survey schema.The plan shows the cameras acquisition network and the horizontal image rotation from point of view #3; in section a pair of acquisition points whit vertical baseline and rotation with 2 images.

Figure 5 .
Figure 5. Point cloud from laser scanner with GCPs.

Figure 6 .
Figure 6.Final orthoimage of the whole façade with metric scale and sculptural detail above the main portal.

Figure 7 .
Figure 7. Deviation map between the photogrammetric cloud and range data, with two relative sections.

Figure 8 .
Figure 8. Overlapping of the façade drawing with the orthophoto, with close-up on two specific areas.

Table 1 .
Table 1 some technical details on smartphone used are summarized: General smartphone parameters.

Table 2 .
Acquisition parameters of photogrammetric campaign.