DOCUMENTATION OF 40 KILOMETERS OF HISTORICAL PORTICOES – THE CHALLENGE

In the last years the image-based pipeline for 3D reconstruction purposes has received large interest leading to fully automated methodologies able to process large image datasets and deliver 3D products with a level of detail and precision variable according to the applications. Different open issues still exist, in particular when dealing with the 3D surveying and modeling of large and complex scenarios, like historical porticoes. The paper presents an evaluation of various surveying methods for the geometric documentation of ca 40km of historical porticoes in Bologna (Italy). Finally, terrestrial photogrammetry was chosen as the most flexible and productive technique in order to deliver 3D results in form of colored point clouds or textured 3D meshes accessible on the web. The presented digital products are a complementary material for the final candidature of the porticoes as UNESCO WHS.


INTRODUCTION
In the last years, 3D reconstruction methods became very popular with a large proliferation of 3D geometric models, often accessible on the web via dedicated portals and viewers.Such models are often created with fully automated methodologies or low-cost sensors and feature a level of detail and precision variable according to the applications and needs.Automation is undoubtedly very important, however different open issues still exist, in particular when dealing with the 3D surveying and modeling of large and complex scenarios, like historical porticoes.The paper presents an investigation on various 3D surveying techniques for the digital reconstruction of ca 43 km of historical porticoes in Bologna, Italy (Fig. 1), that contribute to classify the historical center of Bologna as one of the most unique and best preserved in Europe, but with fragmentary knowledge.The considered techniques (hardware and software) are reported with their advantages and disadvantages, specifically for the documentation of complex scenarios like the porticoes.Terrestrial photogrammetry was identified as the most suitable, flexible, reliable and productive technique for the 3D reconstruction aim.An ad-hoc image pre-processing methodology was developed to increase the processing performances of the image orientation and dense image matching phases.The achieved 3D results are shared online with a point-based viewer for valorization and dissemination purposes.At the same time, the 3D surveying results are the base for new studies and new management and planning techniques.For these latter aims, such complex scenario needs new approaches and IT solutions, e.g. for an accurate management of facades conservation (planning and single unit design), surfaces and pavements cleaning, unified lighting systems, etc.

The historical porticoes in Bologna
The porticoes were built through 11 th and 20 th century and can be regarded as unique from an architectural viewpoint in terms of their authenticity and integrity.Thanks to their great extension, permanence, use and history, the porticoes of Bologna are considered as outstanding universal value.Some of them are still composed of the same materials they had been originally built with.Their appearance and public use have always been left in place, even in case of renovations works.They span approximately 40km, mainly in the historic city centre of Bologna, they represent a high-quality architectural work and they feature a social, anthropological and community meaning, as meeting place between public and private sector.The "Bolognese portico" is not only an "open book" on the city and its history, declined in a large variety of forms and materials, but also a superset of the family of cultural objects that it belongs to.These arcades are probably the largest unified private space existing with a public use in an ancient city and are the most important feature but also the most complex to address the problems of the entire urban system: its knowledge, its management, its use, but also its daily maintenance, its preservation and its transformation.Since 2006 the porticoes are in the tentative list of UNESCO World Heritage Site (whc.unesco.org/en/tentativelists/5010/)and now they are undergoing a definitive recognition of the nomination as part of the program of the current Municipal Council.The nomination is aimed at highlighting the portico, not only as a high-quality architectural work, which in the past centuries has become a distinctive feature of the town, but also in its social, community and anthropological meanings, as a meeting place, a protected space that serves as the backdrop of the events of daily life.The nomination project refers to different subjects and is divided into many levels of action, with the aim to develop a platform conceived for on-line accessing the wealth of data and resources related to the Bologna porticoes system, such as historical, artistic, architectural resources, besides all those data regarding its actual management.This information system platform, bearers of specific skills, is aimed to give back a comprehensive, structured and coherent semantic interpretation of Bologna landscape, by means shapes realistically reconstructed from historical sources and surveys, overcoming the traditional display mode.For these reason, photo-realistic 3D data are needed as complementary material for conservation, management and communication needs (Apollonio et al., 2013a, Apollonio et al., 2014a).In this direction, new 3D reconstruction requirements were fixed allowing a more general approach able to integrate urbanarchitectural scales of representation of typical architectural heritage.It can be used to better understand their essence and their characters trying to fix the actual state, the past and predict the future.

The complexity of the 3D surveying project
As most of the heritage projects, this work faces many issues and challenges, namely:  Environmental matters: we need to consider the length of the porticoes, repeated patterns, uniform colors, lack of texture, illumination changes, occlusions, narrow spaces, policies and regulations, detailed decorations, pedestrians, etc.  Hardware/Software matters: we need to identify the most productive and reliable 3D technique for the delivery of the necessary 3D products with the given specifications. Accessibility/sharing/utilization of the products: the large 3D data must be easily accessible to the public, usable by policy makers and respectful of colors and geometries.

THE PROJECT SPECIFICATIONS
The project aims to produce a collection 3D reality-based models designed as an evolutionary system, capable not only to fix the actual status, but also of being useful for the design of the future.Therefore the task was -besides delivering several kilometers of 3D data of arcades -the realization of a reliable working methodology able to capture, replicate and display the shape and color of existing artefacts within reality-based models, possibly semantically enriched, with the level of detail at the scale of architecture, and using web-based teal-time rendering visualization solutions.The geo-referenced 3D digitization and rendering of the portico's geometries, faithful also in color reproduction, should be accessible and useful to all actors of the porch lifecycle, from researchers to managers or citizens and it should be incorporated as Architectural Knowledge System and within the information system of the Municipality of Bologna.Within the targets set-out for the construction of a knowledge/information system, standards and procedures are essential to ensure the consistent meaning of shared contents and to trace and record the 'history' of the processed data (Koller et al., 2009).Therefore, standardized characteristics of 3D models need to be defined a priori, in order to enable the integration of different heterogeneous information, coming from different sources, and to collect 3D models produced by different operators, at different places and times, keeping a substantial homogeneity of representation and data quality.
In order to address these problems, authors defined, a-priori, and verified, a-posteriori, widely shared standardized characteristics/specs of 3D models, able to evaluate the fidelity of the results with accurate pre-defined quality metric standards.
For the reality-based 3D modeling process, a set of standards was adopted in order to be used by the different surveying/modeling groups (at beginning of the the project four different groups tried different approaches, while today just one operator manage the whole process) in order to ensure accuracy quality during the experimental development of the strategy but also over time.The core of our standards definition is based on the concept of a 'Master/A-level model' (Apollonio et al., 2012) -meant as a replica of the original object based on its intrinsic characteristics and obtained at the end of a dense point cloud and/or mesh reconstruction process -able to supply the highest quality in terms of spatial and surface/color information content.Following this approach, metric standards were referred to the real building features.Starting from the 'A-level', we defined two further different qualitative Levels of Details (LoDs) according to the different uses of the 3D data (Fig. 2): -'B-level derived models': on-site visualization for management and professional purposes (geometry simplification: at least 20% of the original model; at least 50% reduction of original texture resolution) based on interpolation algorithms to achieve semantic model simplification without loss of quality; -'C-level derived models': on-line visualization, that can be used as 3D interfaces or included in large VR scenarios (geometry simplification: at least 7,5% of the original model; at least 50% reduction of original texture resolution), to be used as interface in visual 3D databases or for dissemination purposes.Therefore, another critical point in this procedure is related to the 3D models remeshing/decimation for multi-resolution models production.Summarizing, the outcomes of the 3D digitization project should be a metric 3D product of the 40km of porticoes at various geometric resolutions in order to highlight geometric details where necessary.The 3D data should be re-usable for various purposes, they should have a single point accuracy of 3-5 mm while 2D products should obtainable with a texture resolution of 2-3mm.The original mesh, considering the Alevel model, has an average edges length of 2 cm, while the reduced mesh model has an average edges length of 15 cm.Following the Shannon-Nyquist theorem, texture sampling resolution to accurately represent the structural characteristics was fixed at least twice the inverse amplitude of finer detail, i.e. in our case the double of geometric resolution.
A last requirement concerns the fidelity of the acquired colors, measured on the 'Master/A-level model' and visualized at OpenGL runtime.This is a key feature to allow the use of our 3D products for conservation and restauration purposes (Apollonio et al., 2013a).

TECHNOLOGIES AND SENSORS
Due to the complex and challenging task of the project, various techniques and sensors were investigated and tested (Fig. 3).Pros and contra are summarized in Table 1, bearing in mind the difficult accessibility conditions of historical porticoes, repeated patterns, uniform/lack of texture, illumination changes, narrow passages, arcades, etc.
Considering the performed investigations and tests, we finally decided to employ terrestrial photogrammetry, coupled with topographic surveying for scaling and geo-referencing.Aerial photogrammetry by means of aerial oblique imagery (Fig. 3b) will be then used to complete the 3D reconstructions of the upper parts of the porticoes.

Related works in image-based 3D reconstruction
In the last years the image-based pipeline for 3D reconstruction applications is increasingly becoming a key approach, ensuring ease of use, sufficient automation and efficient results, even for non-professional users.Significant progresses have been recently achieved in all core components of the image-based pipeline, starting from pre-processing (Maini and Aggarwal, 2010; Verhoeven et al., 2015) to efficient algorithms for scalable tie point extraction (Apollonio et al., 2014b;Hartmann et al., 2015), large-scale bundle adjustment (Agarwal et al., 2010;Wu et al., 2011) and dense and precise points clouds generation (Remondino et al., 2014).Fully automated methodologies are nowadays able to process large image datasets and deliver 3D products with a level of detail and precision variable according to the applications (Snavely et al., 2008;Crandall et al., 2013).Undoubtedly the great work of the past years devoted to the integration of automated computer vision algorithms with reliable and precise photogrammetric methods led to successful (commercial and open) solutions -often called Structure from Motion (SfM) -for automated tie point extraction and 3D reconstructions from large image datasets (Frahm et al., 2010;Remondino et al., 2012).Nowadays, particularly for terrestrial applications, the level of automation is reaching very high standards and it is increasing the impression that few randomly acquired images (or even found on the Internet (Heinly et al., 2015)), a blackbox tool or a tablet/mobile device (Kolev et al., 2014;Schoeps et al., 2015) are sufficient to produce a metrically precise 3D point cloud or textured 3D model.Such approaches are able to process large quantities of data (often exploiting GPU) andalmost always -deliver an apparently successful solution and nice-looking sparse point cloud.But when it comes to complex heritage scenarios within projects where a detailed and accurate geometric documentation for conservation and valorization purposes is requested, automated image-based method can easily fail and not deliver results which fulfil project constraints.

Techniques comparison
During our investigation, 3D surveying techniques were applied to the same building and, considering the project constraints, analyses and considerations were stated.Hereafter we report the assessment results for a building with 5 arcades surveyed with photogrammetry and laser scanning (Fig. 4).Photogrammetry required ca 250 images -acquired with a Nikon D3100, sensor size 23.1 × 15.4 mm, 18 mm nominal focal length -whereas 10 stations/scans were performed with a ToF scanner (Leica C10).The assessment is summarized in Figure 5 where the different digitization phases are considered: data acquisition, data processing (image orientation and dense point cloud generation or scan registration), 3D data editing and modeling, texture generation and 3D visualization.

DEVELOPED PROCESSING METHODOLOGY
Considering the complexity of the porticoes and the potentialities/limitations of photogrammetry for such scenarios, the processing methodology was split in 2 stages (Fig. 6): 1) image pre-processing and enhancement: the procedure includes color balancing, image denoising, color-to-gray conversion and image content enrichment by means of Wallis filtering (Gaiani et al., 2016); this stage was developed to increase the processing performances in successive image orientation and dense image matching phases.The main idea is to minimize typical failures caused by SIFT-like algorithms due to changes in the illumination conditions or low contrast areas and to improve the performances of feature extraction algorithms.2) image processing: for the 3D reconstruction we combined a calibrated version of the SIFT detector/descriptor (Apollonio et al., 2014), a photogrammetric bundle adjustment and the Nframes SURE matching method.The employed camera are a Nikon D3100 (23.1 x 15.4 mm CMOS sensor, 4608 x 3072 px) and a Nikon 5300 (23.5 x 15.6 mm APS-C CMOS sensor, 6000 x 4000 px) coupled with 2 lenses: 18mm (for the narrow and lower parts of the porticoes) and 35 mm (for the higher parts) nominal focal length.The image networks are composed up to 7000 images (Fig. 7).The performances of the methodology were evaluated using different approaches and indicators, namely: number of oriented images, reprojection error at the end of the bundle adjustment, the number of 3D points in the dense point cloud, the pairwise matching efficiency, local density computation and cloud-tocloud point distribution errors (Gaiani et al., 2016).
The developed methodology allowed to: (i) increase the number of correct image correspondences, particularly in textureless areas; (ii) track image features along a largest number of images to increase the reliability of the computed 3D coordinates; (iii) correctly orient a largest number of images; (iv) deliver 3D coordinated with sub-pixel accuracy at the end of the bundle adjustment procedure; (v) provide dense, complete and noisefree 3D point clouds.The developed photogrammetric method allows to obtain 3D models at 'Master/A-level' (Section 2) consisting of approximately five million points (dense point model) or one million polygons (mesh model) per arcade.The models thus obtained, therefore, can be optimized for the online mesh-based visualization platform, reducing the number of polygons that constitute the model tessellation, and defining an optimal resolution of the texture.We tested that, for the 'C-level derived model' a tessellation of about 100,000 polygons per arcade gives back a good compromise in architectural quality, resolution, visual appearance and handling.The difference between the original mesh and the reduced one is a negligible amount considered the scale model definition.Table 2: Some areas of the porticoes with related information about the digitization procedure.For segmentation and semantic reasons, during the processing the images were grouped according to architectural reasons.This has sometime affected the processing time, leading to longer time even if the number of images was not extreme.The processing time include orientation and dense matching steps.

SOME RESULTS OF THE PROJECT
The digitization work delivered 3D results as dense and coloured point cloud as well as textured polygonal models (Fig. 8), in april 2016, of almost 10 km and 353 buildings (approximately 1/4 of the whole porticos extension).The former (coloured point cloud -Fig.8b) is the pure outcome of the photogrammetric procedure and there is no interpolation procedure.The latter (textured polygonal models -Fig.8a) involves a meshing/interpolation process with possible smoothing effects.Table 2 reports some numbers of the 3D digitization procedure for four interesting portico areas.The length and number of arcades depicts the complexity of the porticoes.The processing time is not reflecting the dimension of the image block (n images).Indeed each block was subdivided in smaller groups following architectural reasons and this has affected the processing time, in particular the stereo-pairs matching which is proportional to (,2)=!/(−2)!2!
To share the achieved results, valorize the historical areas and disseminate their unique characteristics, a web-based point cloud viewer was developed (Fig. 8b), deploying the PoTree tool (http://potree.org;Schütz and Wimmer, 2015).PoTree is point-based rendering solution specifically developed for visualizing large point clouds using standard web-based technologies that work within a web browser.The point clouds are encoded in an octree data structure using a dedicated converter (Schütz, 2014) and PoI (Point of Interest) can be added linking the geometry to external information like text, videos, images, hyperlinks, etc.The visualization is available at http://3dom.fbk.eu/repository/3Dpointclouds/Bologna/index.html.The viewer ensures high geometrical and visual accuracy while being accessible to non-specialized users, unskilled operators and policy makers.

CONCLUSIONS
The paper presented the challenge of digitizing more than 40km of historical porticoes for documentation, conservation, management and communication needs.After extensive evaluations and tests, terrestrial photogrammetry was selected as primarily 3D surveying technique as more flexible, reliable, economic and performing with respect to other available solutions.The image processing is based on a meticulous preprocessing procedure to minimize typical failures in the feature points extraction or dense matching procedure.So far ca 25% of the porticoes were digitized but now that the methodology is established the last year of the project should achieve its complete goal.The 3D results are available to the public as point clouds and polygonal models, both coupled with annotations and technical information.The achieved photorealistic 3D data are part of the city Architectural Knowledge System and are part of the undergoing candidature program for the porticoes as UNESCO WHS site.

Figure 1 :
Figure 1: Examples of historical porticoes in Bologna digitally reconstructed for conservation, management & communication issues (UNESCO WHS candidature).Note repeated patterns, uniform colors, lack of texture, illumination changes, occlusions, narrow spaces, etc.

Figure 2 :
Figure 2: The three qualitative LoD defined according to different use of the 3D textured polygonal modelsfrom urban and merely touristic web-based visualization to professional stand-alone 3D model editing and analysis with conservation, management and study purposes.

Figure 3 :
Figure 3: Examples of 3D reconstructions achieved with the evaluated sensors and techniques: portable/handheld 3D scanning (a) and oblique aerial images (b).

Figure 4 :
Figure 4: The building with porticoes used for the reported assessment (a).Result of the range-based (b-left) and image (b-right) 3D surveying.


Ideal for building roofs and, partly, façades  Availability of existing images (open access, geoportal, etc.)  Large coverage  Complete upper occlusions of terrestrial acquisitions  Small scale (resolution hardly better than 10 cm on the ground)  Integration with higher-resolution terrestrial data  Inadequate in narrow passages/streets  Impossible to document interiors / under arcades  Low-resolution on building façades Airborne laser scanning (LiDAR)  Availability of existing point clouds (open access, geoportal, etc.)  Direct geo-referenced dense point clouds  Generally low resolution  Sparse point clouds on building façades  High costs for dense point clouds  No / low-quality texture  Integration with higher-resolution terrestrial data RPAS/UAV photogrammetry  High-resolution aerial images  Possibility of oblique views to survey building roofs and façades  Flying regulation issues  Difficult flight planning  Difficulties in piloting inside narrow streets with cables and arcades  Scaling / Geo-referencing RPAS/UAV laser scanning  High-resolution (density) point clouds  Possible surveying of building roofs and façades  Flying regulation issues  Difficult flight planning  Difficulties in piloting inside narrow streets with many cables and arcades  Generally no texture  High costs  Low accuracy Terrestrial photogrammetry with frame cameras/images  Cheap & flexible technique  Quick acquisitions  Geometry & texture  Productive  Illumination changes  Low texture areas  Scaling / Geo-referencing Terrestrial photogrammetry with panoramic cameras/images  360 deg FOV  Quick acquisitions  Geometry and texture  Low radiometric and geometric resolutions  Image processing issues  Scaling / Geo-referencing Tablet / mobile image-based devices  Fast acquisitions  Cheap hardware  3D on-the-fly  Noisy results  Short range  Too new technology  Mainly for indoor Terrestrial laser scanning  360 deg FOV  Metric data  High-resolution point clouds  No / low-quality texture  Many stations / acquisitions needed  Occlusions  Long acquisition time  Long editing time Van-based mobile mapping  360 deg FOV  Geometry & texture  Fast acquisitions  Geo-referencing  Impossible to document interiors / under arcades  Expensive  GPS outages in historic centers  Narrow passages  Occlusions and lack of data on high buildings Portable / handheld mobile mapping  Quite fast acquisitions  360 deg FOV  Metric data  Ideal for small urban scenarios  No / low quality texture  Generally expensive hardware  Short range  Too new technology

Figure 5 :
Figure 5: Results of the 3D surveying comparison for the building depicted in Figure 3. Efforts are differently spread inside the acquisition / processing pipeline but the overall numbers suggested photogrammetry as the most suitable and productive technique for the porticoes

Figure 6 :
Figure 6: The photogrammetric pipeline for the 3D digitization and visualization of historical porticoes.

Figure 7 :
Figure 7: Example of the complex photogrammetric networks.

Figure 8 :
Figure 8: Examples of porticoes digitized using the presented methodology, shown as mesh models (a) and inside the point-based web viewer (c).

Table 1 :
Considered techniques for the 3D surveying and documentation of the historical porticoes.Pros and cons are reported considering techniques' specifications and scenes' characteristics.