ENABLING SCAN-TO-BIM WORKFLOW FOR HERITAGE CONSERVATION AND MANAGEMENT PROCESS

: The paper stresses the potential of digital technology in the field of Cultural Heritage (CH), focusing on Architectural Heritage (AH) conservation and management process. It presents a workflow based on non-parametric automatic 3D modeling and Industry Foundation Classes (IFC) authoring to combine geometry and semantic information. The goal of such a methodology is to overcome time-consuming manual tasks that still limit the Historical Building Information Modelling (HBIM), which nevertheless proves to be one of the most effective digital tools for the conservation and management of AH in the medium to long term. The proposed workflow consists of five steps: the 3D survey campaign and raw data processing, the semantic annotation of images according to different information layers, the annotation transfer from images to the point cloud representing the studied architecture, the obtained Point Cloud Information Model (PCIM) automatic segmentation and 3D mesh wrapping, and finally the IFC generation for the HBIM representation. This unconventional digital chain was tested to support the knowledge representation of Villa Leonardi, an historical house in the Marche Region (Italy). The final output is a 3D model of the building, automatically processed from digital survey data, and semantically enriched thanks to the material consistency and decay analysis carried out and annotated on spherical panoramas. The effectiveness of this approach was proven in the conservation project for the façades of the house: enriched data on materials and decay supported an action plan of restoration works, which was then recorded in the HBIM model, thus permitting time and cost control.


INTRODUCTION
Any discussion on Cultural Heritage (CH) involves the concept of 'complexity'. Focussing on the built heritage, several factors contribute to this complexity: the first one is undoubtedly the dynamic condition of architecture, which is the result of interrupted transformations mainly related to changing uses but also to present-day pressures such as climate change and the pandemic crisis -to name but the most challenging. Other factors are the close interdependence between buildings and their context, multi-disciplinary contents and competences, and the procedural dimension of actions for passing on the built heritage to future generations, which includes knowledge, geometric representation, critical interpretation, conservation project, building site and, finally, management over time. To manage this complexity, a new approach to preservation has emerged, increasingly based on preventive and planned conservation strategies shifting the focus of the culture of restoration from projects to the processes (Della Torre, 2021). At the same time, restoration has become very contaminated with digital technologies to improve the effectiveness and sustainability of conservation practices, at least since the 1990s, and has marked a change of pace in recent years with the spread of applications that integrated traditional and new methods and tools (Greco et al. 2020). At the European level, these issues have been addressed in several policy documents and guidelines that highlighted the importance of creating synergies between heritage conservation and digitisation (European Commission, 2021;European Heritage Alliance, 2020).
In this framework, an ICOMOS expert group drew up a document to provide guidance on quality principles for all stakeholders directly or indirectly engaged in EU-funded heritage conservation and management. Among these key principles were 'knowledge-based actions' and the 'good governance' (Dimitrova et al., 2020), which can find great support in digital technologies in terms of both data collection systems and process organisation. The European Union's commitment in building competences and tools to embrace the digital transition and to encourage qualified innovation in the field of cultural heritage has been confirmed in some of the most relevant Italian national programmes, including the National Strategy for Intelligent Specialisation (SNSI), the National Research Plan (PNR), the National Recovery and Resilience Plan (PNRR) and, above all, the National Plan for the Digitisation of Cultural Heritage (PND), drafted by the Central Institute for the Digitisation of Cultural Heritage -Digital Library of the Ministry of Culture. On the one hand, the cultural paths outlined by these core programmes have consistently understood and enhanced the concept of 'valorisation' as described by the Code of Cultural Heritage and Landscape -the national legislation on heritage preservation in Italy -as a sequence of actions focused on 'knowledge', 'conservation' and 'fruition' (Legislative Decree 42/2004, Art. 6). On the other hand, these policy documents have boosted the development of innovative digital solutions for historical architectures and sites -e.g., 3D surveying, Historical or Heritage Building Information Modelling (HBIM), Geographic Information System (GIS), Laser Imaging Detection And Ranging (LIDAR), Virtual or Augmented and Mixed Reality (VR, AR, MR), and these are just a few.
Among these, HBIM has proved to be one of the most effective digital tools for the conservation and management of Architectural Heritage (AH), in the medium to long term (Di Stefano et al., 2020;Jouan and Hallot, 2019). The implementation of digital information modeling on AH started over ten years ago (Murphy et al., 2009) and produced a growing scientific literature as well as challenging and promising outcomes . Nevertheless, advantages in using HBIM are not without criticality due to the construction of digital models that faithfully represent real geometries of diachronic architectures, and to the management of heterogeneous data according to a semantics coherent with all phases and competencies involved in the conservation process (Murphy et al., 2013;Lopez et al., 2018;Simeone et al., 2019). So, HBIM involves architecture 3D modelling, which is typically built on laser scanning and photogrammetric 3D data (Diara and Rinaudo, 2019;Zaker et al., 2021). Up to now, several workflows have been studied and developed (Allegra et al., 2020;Quattrini et al., 2015), which mainly exploit software designed for the parametric representation of new buildings, resulting in non-automatic and time-consuming scan-to-BIM processes. Regarding the issue of data enrichment, several approaches have been proposed to manage annotations on geometric models (Ponchio et al., 2020). Basically, 'annotation' refers to a process of selecting a region on the model, then associating descriptive attributes to the selected region, which are related, for example, to the name of a structural/functional subcomponent, to the results of historical analyses, cognitive studies, diagnostic and chemical investigations, material sampling and so on (Bacci et al., 2019;Apollonio et al., 2018). As stated in (Croce et al., 2021), orthophotos, i.e., orthorectified images extracted from realitybased 3D models, are among the most frequently used media for the insertion of annotations, which implies the previous processing of a 3D model resulting from a 3D digital survey that is not always feasible in preventive conservation studies. Another issue in HBIM concerns to the use of IFC format, which has a fundamental role in data exchange and the interoperability of informative BIM platforms (Borrmann et al., 2018). Different approaches to IFC schema application to CH have been reported in literature. (Diara and Rinaudo, 2020) presented an experimental IFC classification implemented into an HBIM open-source software, while (Oostwegel et al., 2022) worked within the boundaries of the standardised IFC schema and used existing methods to include heritage information, i.e., property sets. In this framework, the present study defines a digital chain that can lead to HBIM representation while trying to break down some of the criticality briefly highlighted here. [CM]

AIM OF THE RESEARCH
The aim of this research is to enable a smarter but no less performing and quality Scan-to-BIM workflow for the conservation and management of AH. As the state of the art on this topic has demonstrated, most workflows are still limited by non-automatic and timeconsuming processes; this is a relevant crux that has so far been widely accepted in the face of HBIM potential to govern the complexity of CH. As a matter of fact, it has been normally balanced by the benefit in using and updating this digital tool over time. Such a critical issue represents the precondition for this study, which presents a non-parametric automatic modelling process, based on an opensource IFC authoring tool to integrate semantic information. This process takes full advantage of integrated digital survey data, which it uses not only as a geometric reference, but also for knowledge-based data enrichment. In particular, knowledge representation includes information on historical events from indirect sources (such as archive or published documents), materials and construction techniques, and on the state of conservation of the studied architecture. Exploiting semantically annotated spherical panoramas, a Point Cloud Information Modelling (PCIM) (Park et al., 2020) is processed to be subsequently segmented and wrapped into geometrically accurate 3D meshes referred to different information layers. Thanks to this process, it is possible to overcome manual modeling of the investigated AH geometry. IFC objects are automatically created from these meshes, processed and segmented according to the information layers previously annotated on images. As for previous Scan-to-BIM workflows, a shortcoming is the need for different softwares to manage the steps required; anyway, file formats used along this whole workflow enable a full interoperability, which is a fundamental requirement in heritage digital processes. [CM]

The proposed Scan-to-BIM workflow
This work presents an approach aimed at generating a HBIM model in a semi-automatic way by exploiting the potential of integrated digital survey data. The methodology mainly relies on two procedures, related to the semantic annotation and Scanto-BIM processes, respectively. The semantic annotation connects the geometric representation (quantitative and metric information) and the complementary information produced within specialist studies (qualitative information), which are crucial to support AH conservation and management processes. Different semantic annotation pipelines have already been described in literature and classified in relation to the type of annotated media, the degree of automation, and the part/whole relationship (Croce et al., 2020). The solution proposed in this study relies on a hybrid 2D/3D approach, that assumes that images are easier for semantic annotation and often more suitable for conservation-oriented analyses. Consistently, panoramic images calibrated and spatially oriented on the AH point cloud are exploited as supports for thematic mapping. Once annotated, images are projected on the AH point cloud to automatically transfer the qualitative information that is stored as points Scalar Fields (SF). Therefore, the output of semantic annotation is a PCIM that provides the basis for the second procedure, i.e., the construction of the HBIM model. The proposed Scan-to-BIM process takes advantage of an opensource IFC authoring tool to assign an IFC class to a 3D model regardless of its shape. This is a solution adopted to overcome time-consuming manual 3D modeling, taking advantage of 3D meshes automatically processed from point clouds. Consistently, the PCIM is segmented according to architectural element information layers, and each obtained part is wrapped into a 3D mesh. All the PCIM SFs are then transferred to the 3D meshes vertices. Finally, 3D meshes are used to generate the different IFC objects. On this basis, the proposed workflow can be summarized in 5 steps ( Fig. 1 Raw data processing aimed at providing 2D and 3D digital graphic support for the conservation and management process; 3. Critical understanding and semantic annotation on spherical panoramas (e.g., classification of architectural elements, identification of material consistency and state of conservation); 4.
Semantic information transferring between images and point cloud to generate a PCIM; 5.
Automatic PCIM wrapping to obtain IFC objects directly from 3D meshes, thus creating an HBIM representation. [RA]

The case study: Villa Leonardi, Treia (MC), Italy
The described workflow was developed and tested for the knowledge representation, as well as for the design and management of conservation works of the Villa Leonardi, an historical house located near the town centre of Treia in the province of Macerata, in the Marche Region (Italy). The house was built at the end of the 19th century on the remains of a pre-existing farmhouse, at the beginning of the 20th century was owned for about one hundred years by a local family called Leonardi, from whom it got its named. At present, it is part of a 7,000 square metre garden including two outbuildings and a row of holm oaks leading to the Potenza River, which were declared of historical-artistic interest in 2006, by the Italian Legislative Decree 42/2004. The architecture of this house consists of a compact three-storey volume with two smaller buildings leaning against the side of the entrance façade, once used as a family chapel and tool shed. In terms of construction, the building is of load-bearing masonry: the main volume is entirely made of brickwork, while the side volumes are of mixed masonry of sandstone blocks and bricks. The entrance façade is in exposed brick and is embellished with a three-arched loggia on the ground floor and half pilasters framing the windows on the main floor. On the other hand, the rear façade, as well as part of the sides of the house, are covered with white concrete plaster. On an overall view, the architectural contrast featuring the entrance façade and the rear one does not go unnoticed, being the result of recent transformation works that partially changed the house, its geometry, and surfaces. To date, the house has been uninhabited for decades, and the lack of use and consequently ordinary maintenance are apparent. The need to design a conservation plan to limit and solve the decay phenomena, to rehabilitate the house for new uses and thus enhance its values made it a suitable case study for the proposed experimentation. [CM; LC]

Digital survey: tools and data processing
A preliminary direct inspection of the AH is mandatory to evaluate the type of survey to be carried out and tools to be used. In the case of Villa Leonardi, TLS, high-res 360° imaging, and photogrammetry were integrated to ensure geometric and colorimetric accuracy. Moreover, the whole 3D survey was carefully planned to obtain a uniform lighting and to avoid sharp shadows. The data acquisition lasted one day, and it involved two operators. Firstly, in order to define the reference system, black and white targets were placed on the ground to be surveyed by a GPS HiPER HR with RTK (Real Time Kinematic) method. A Terrestrial Laser Scanner (TLS) Leica Geosystem P40 ScanStation was then used to scan from the ground Villa Leonardi exterior and interior. Set to a scan density of 6.3 mm at 10 m, a total amount of 50 scans were required to document the whole building. In addition, from each scan station, 18 images were performed via a Sony α9 digital camera, with a CMOS sensor of 35.6 x 23.8 mm, and approximately 24.2 megapixels. All the images were captured by rotating the camera mounted on a Nodal Ninja 3 panoramic head to keep the nodal point fixed and aligned to the TLS sensor. In addition, images including an X-Rite ColorChecker Classic were shot to correct the white balance and adjust the exposure of each set of images. Lastly, an Unmanned Aerial Vehicle (UAV) photogrammetric survey was performed using a DJI Mavic Mini. This UAV system is equipped with a CMOS sensor of 6,17 x 4,55 mm, and 12 megapixels. A sequence of 76 images representing the upper parts of the building was acquired flying at an altitude of 30 m above ground level. Once the survey campaign was over, using the software Leica Cyclone Core, each TLS point cloud was coloured with the RGB values of the 360° panoramic image obtained by stitching the 18 images captured from the corresponding scan station. Then, all the TLS point clouds were aligned into one representing interior and external spaces visible from the ground. In parallel, images captured by UAV were processed in Agisoft Metashape according to the SfM-DMVR method. So, another RGB coloured point cloud was obtained, representing the roof and the two terraces on the first floor. Finally, the coordinates of black and white targets were used to refer both TLS and SfM-DMVR point clouds to the WGS84 coordinate system and, therefore, to merge them. The result is a point cloud of the whole building, a digital replica which represents the reference for an accurate as-built 3D modeling within the scanto-BIM process. (Fig. 2). [RA; LC]

Knowledge-based contents: critical understanding and semantic annotation
Data from the 3D survey provided the media support for further analysis and investigations with the aim of enriching the digital representation and fulfilling the crucial need for knowledge of the Villa Leonardi. Since CH is a common good that is neither renewable nor replaceable, it must be preserved in full knowledge of its material and immaterial values. With this in mind, investigations were geared towards building both an active knowledge for heritage conservation projects and an upto-date quality database for informed, long-term heritage management. The knowledge of this historical house was structured on thematic levels of information concerning its history, physical and material consistency and state of conservation (alteration, decay and damage). The critical understanding of the Villa Leonardi began with a bibliographic and archival researchwork. The study of textual, graphic and photographic documents from indirect sources clarified the right period in which this house was built, dated to 1871 and still unknown today. In addition, all documents were digitised and classified, creating a virtual archive to be linked to the digital representation of this architecture. Then, the researchwork moved on to the material and decay analysis of the building. Even though the Villa Leonardi has been abandoned for several years, the interior spaces are still in a fairly good state of conservation, while its exterior façades present the greatest criticality, especially the south-facing one. For this purpose, the present study focused on the exterior façades of the building. The material analysis highlighted the different types of masonry, which has a thickness of four brick heads on the The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-M-2-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy ground floor, and which is partly made of exposed handmade bricks, partly a mixture of bricks and sandstone blocks, and finally partly covered with concrete plaster but where the decay now reveals the use of perforated industrial bricks as a result of interventions carried out later in time. Other materials on the façades are fir wood for door and window frames and shutters, and iron for parapets. Visual inspections also guided the detection and description of major decay and damage phenomena, by pointing out their causes and level of intensity from which types and priorities of conservation works must be derived. Regarding the south façade only, the analyses returned a synoptic picture with localised cracks and deformations, phenomena of detachment and erosion of mortars, especially at the top, partial washing out, risingdamp, hygroscopic behaviour of concrete plasters due to the presence of salts, as well as the presence of biological colonization mostly concentrated at the lower parts of the building. In general, causes of decay are related to infiltration and rising water, but also, and above all, to the lack of use and ordinary maintenance; in addition, there were forms of anthropic degradation due to modernisation works, which were not always congruent with the features and value of this historical house. These critical analyses were conducted through direct on-site inspections and reported on the 360° images. Thus, 360° images were semantically annotated according to three information layers: architectural elements, material consistency, and decay. Particularly, the spherical panoramas referring to the scan station at the centre of each façade were imported in Adobe Photoshop and mapped with regions coloured according to the different information layer legends (Fig. 3). As a matter of fact, graphic mapping was the first step for building of an information system, and it was developed according to a well-established way of representation for the discipline of restoration, and to the existing lexicons provided by the main national and international Institution for Cultural Heritage Protection. With regard to the state of conservation, the Italian prescriptions of Normal 1/88 Macroscopic alterations of stone materials: lexicon and its updating UNI 11182:2006, as well as the ICOMOS-ISCS Illustrated glossary on stone deterioration patterns were used (Vergès-Belmin, 2008). Moreover, this phase of critical interpretation set an essential cognitive step to define an action plan of restoration works and to enable active and predictive conservation strategies for this heritage. [CM]

From point cloud to PCIM
Once the different information was annotated on the 360° images, the thematic layers were transferred to the point cloud obtained by 3D survey. Spherical panoramas were already aligned to the TLS point cloud to transfer RGB values to its points. By replacing them with semantically annotated 360° images, it was possible to store also their RGB values, obtaining different point clouds colored according to each information layer. Then, thanks to the open-source software CloudCompare, it was possible to store in a single point cloud each thematic layer as a Scalar Field (SF). At the end of this step, the result was a PCIM, an information system representing the as-is condition of Villa Leonardi, which was classified according to the architectural element layer annotated on the spherical panoramas and reporting also material consistency and decay as constant SFs (Fig. 4). [RA]

3D modelling and data enrichment
The PCIM was then automatically wrapped into a 3D mesh using the Poisson Surface Reconstruction tool provided by CloudCompare. Moreover, it made it possible to generate a Density SF associated to the vertices of the output mesh, which is useful to reduce its extents to fit as much as possible the input point cloud. Subsequently, in order to transfer the semantic information from the PCIM to the 3D mesh, the SFs associated to the points of the former were interpolated to the vertices of the latter. To further enrich this representation, the potential of HBIM was exploited by generating IFC objects from 3D mesh in Blender, a free and opensource 3D modelling software. By using its BlenderBIM add-on, an IFCProject, and other IFC objects from different types of elements, e.g., meshes, were generated. According to the architectural element information layer, the 3D mesh of each façade was then automatically segmented. The obtained meshes were imported in Blender, and an IFC Class was assigned to each one as IFCElement, which is defined as a generalization of all components that make up an AEC product. The next step was the transfer of the material consistency and decay information. IFC Schema provides the possibility to create an IFCMaterial which is defined as a homogenous or inhomogeneous substance that can be used to form elements. Moreover, an IFCMaterial can be assigned to an IFC object both as a single material or as a material part of an IFCMaterialConstituentSet, which is a collection of individual material constituents, each assigning a material to a part of an element. The shortcoming of this approach was that parts are only identified by keywords, it is not possible to assign each material constituent to a part of the mesh surface. Concerning the decay information, no IFCDecay has been already standardized. A valuable option was the assignment of decay information by defining a Property Set. Indeed, within the IFCProject a Property Set Template applicable to every IFC object can be defined. So, a Property Set was designed listing all the decay and damage phenomena reported on the façades of Villa Leonardi, provided with their description. The Property Set was then associated to each architectural element, stating whether each phenomenon had been observed or not (Fig.5).
The result was a non-parametric HBIM representation, automatically modeled from the 3D survey data, and semantically enriched according to analysis carried out by annotating the 360° images. However, this representation did not make it possible to identify the part of the IFC object surface associated to a specific material or affected by a decay phenomenon, as it was instead possible by switching PCIM SFs.
To overcome this issue, the 3D meshes of each architectural element were segmented in CloudCompare, according to the material consistency and decay layer information respectively. The output meshes were then imported in Blender, to generate new IFC objects assigning them the same IFC Class of the architectural element to which they belong. Moreover, each object obtained by material information segmentation was assigned the corresponding IFCMaterial, while each one obtained by decay information segmentation was assigned the Decay Property Set. Furthermore, to facilitate the visualization of the semantic segmented meshes, two IFCGroup were added, one containing those relating to materials and another one those relating to decay. Lastly, bibliographical, and archival data were manually added Figure 5. The Villa Leonardi IFC generation. Within the IFCProject created in Blender, a 3D mesh of a pilaster is classified as IFCCovering. Moreover, the IFCMaterial "Bricks" and the "Decay Property Set" are assigned to it, providing material an decay information.
as IFC documents to the IFCProject and assigned to the IFCBuilding, gaining a knowledge modelling process for components that cannot be modelled directly. This digital representation of Villa Leonardi proved its effectiveness supporting the conservation project of the external façades. Data related to materials and decay were crucial to define required tasks, which were then recorded within the HBIM model, thus permitting time and cost control. In fact, in the IFCProject, an IFCWorkPlan was defined, containing two IFCWorkSchedule, related to conservation works on structures and finishings respectively. Tasks named and defined according to the Regional Price List (Prezziario della Regione Marche) were added to each work schedule, also assigning expected starting and ending dates to each task. A work calendar was also defined, which made it possible to generate a Gantt chart, also updating task duration according to workdays. Moreover, each task was assigned to the IFC object. Particularly, IFCObjectQuantitySets of meshes segmented according to decay information layers made it possible to automatically estimate costs of tasks needed to restore them. [RA] 6. CONCLUSIONS

Main achievements
This paper presented a methodological approach for the propagation of semantic information from spherical panoramas to a point cloud. Moreover, it showed how the obtained PCIM can be wrapped into a 3D mesh and segmented to generate an HBIM model. The critical investigations on the studied architecture were essential to support its conservation and management processes, providing an adequate knowledge of its current maintenance condition. Since these studies are based on an interdisciplinary approach, the use of images was mandatory to facilitate the semantic annotation process, enabling it also to non-experts in 3D modelling. However, while images provided a qualitative information recording, a digital replica based on digital survey data from laser scanning and photogrammetry was crucial to provide quantitative information. Thus, the strong point of the proposed workflow relied on the information transferring from 2D representations to a 3D one. Once mapped on the 360° images, semantic annotations were transferred to a point cloud by means of appropriate 2D/3D projection relationships. In particular, the presented study took advantage of the same panoramic images acquired from TLS scan stations for point clouds RGB mapping. So, the projection relationship between images and point cloud were already known. Nevertheless, the same approach could be extended to images from different acquisitions, even performed at different times, providing information on the changing state of conservation of the building. So, the first achievement of this study was the definition of a PCIM, an information system which recorded the as-is condition of the AH, gathering point cloud geometrical information and specialistic studies semantic information. Moreover, a subsequent step was presented to enable an HBIM representation overcoming manual time-consuming modelling of parametric Scan-to-BIM processes. The PCIM was automatically wrapped into a 3D mesh and automatically segmented according to the information layers transferred from previous images annotation. IFC objects were then generated from 3D meshes thanks to the opensource IFC authoring tool BlenderBIM.
Since the IFC schema is designed for AEC industry, its implementation on built architectural heritage presents some shortcomings, e.g., the absence of a standardization for decay information. Moreover, to exactly identify the specific areas affected by a degradation phenomenon, it was necessary to reimport the 3D mesh of each architectural element segmented according to decay information layers. However, the IFC schema was crucial to support the further enrichment of the AH representation with information regarding restoration works and conservation plans, exploiting IFC robust standardization for time and cost control. Moreover, the use of the IFC schema ensured the interoperability of the proposed HBIM model, thanks to Industry Foundation Classes open standards for the exchange of BIM information across different software. [RA; CM; LP]

Future works
Future works could be addressed to overcome shortcomings related to the different steps structuring the proposed workflow.
Concerning semantic annotations, this paper described a manual procedure, but it is possible to rely on a semi-automatic one, by exploiting Artificial Intelligence for images and point cloud segmentation.
Regarding PCIM, it was generated in CloudCompare and saved in binary format. A future work could be focused on defining an object-oriented data structure for PCIM, designed to record different information useful to support AH conservation and management process. Finally, involving all the actors of the heritage conservation the same digital tools could be used to define a more structured digitization workflow, capable of supporting AH preventive conservation plans. Moreover, a digitization workflow structured over time and on different levels of investigation could be defined, automatically providing an integration of different information, optimizing the usage of available resources. [RA; CM; LP]