A Digital Reconstruction of a Historical Building and Virtual Reintegration of Mural Paintings to Create an Interactive and Immersive Experience in Virtual Reality

Featured Application

This work describes the process used to obtain a final product of interest to cultural heritage. It shows how the details of a completely deteriorated historic building have been recovered. The digital documentation obtained has been displayed as an interactive virtual reality product. Of special interest is the use of the DStretch^® application in the process to recover the mural paintings.

Abstract

Nowadays, virtual reality technologies and immersive virtual reality (VR) apps allow people to view, explore, engage with and learn about historic monuments and buildings, historic sites, and even historic scenes. To preserve our cultural heritage for future generations. it is essential that damaged and dilapidated historic artefacts are accurately documented, and that steps are taken to improve user experiences in the areas of virtual visits, science and education. This paper describes an approach to reconstruct and restore historic buildings and mural paintings. The work process uses digital models that are then inserted into an interactive and immersive VR environment. Windows-Mixed Reality is used to visualize the said VR environment. The work method was applied at a United Nations Educational, Scientific and Cultural Organisation (UNESCO) World Heritage Site in Tenerife (Canary Islands, Spain), thereby creating a virtual three dimensional (3D) rendering of the architectural structures of the St Augustine Church in La Laguna and its murals. A combination of topography and terrestrial photogrammetry was used to reconstruct its architectural features, and the digital imaging tool DStretch^® to recover its murals. The resulting 3D model was then inserted into an immersive and interactive VR environment created using the cross-platform game engine Unity. One of the greatest challenges of this project revolved around recovering the dilapidated and virtually nonexistent mural paintings using DStretch^®. However, the final result is an immersive and interactive VR environment containing architectural and artistic information created within the video game engine Unity, which thereby allows the user to explore, observe and interact with a cultural heritage site in real time.

1. Introduction

Ever since virtual reality (VR) and augmented reality (AR) technologies became feasible options, there has been great interest in using these technologies to represent, restore and reconstruct cultural heritage [1]. These technologies provide us with an interesting and attractive channel through which it is possible to educate a wide audience about historic artefacts or sites [2]; or equally, to assist audiences in visiting sites that are difficult, if not impossible, to access in real life. Inserting information into a virtual environment has created entertaining and interactive experiences for users [3,4], and consequently, these technologies are becoming increasingly common, given their appeal to users and the endless possibilities on offer.

Logically, the field of Architectural Heritage Conservation and Restoration could benefit immensely from immersive VR apps offering realistic interactive images; in fact, such apps are already being used by many museums and tourist attractions. For example, Gaitatzes, Christopoulos and Roussou [5] have used CAVE^® to create virtual tours of historic sites, and more recently, museums have begun to offer immersive virtual tours to help familiarise visitors with sites and scenes from the past, using headsets (HMD) such as HTC Vive [6].

The recent advances in VR and AR technologies now enable these technologies to be used on smartphones or tablets, a fact that has provided cultural heritage organisations with a wealth of new opportunities for attracting and engaging with users [7].

The creation of interactive experiences has provided a more efficient and effective way to enrich creative processes, learning, collaboration and also training in the education sector [8]. In fact, the personalised interactive experiences offered by VR help different stakeholders (conservator-restorers, tourists, teachers, museums, etc.) create and enjoy visual products that embody scientific excellence.

In the professional world, VR technologies have proven apt for cultural heritage preservation and have become a powerful tool for professional conservator-restorers, archaeologists and the suchlike. Such professionals, having completed the necessary groundwork and research, can use VR to restore elements found to be in a state of ruin or disrepair [9]. As such, VR can be used to assist in conserving, reconstructing, documenting, researching and promoting cultural heritage. Different methods can be used to obtain 3D models [10], and graphic engines for videogames can be used to add factors to virtual environments, such as interactivity, visualisation and realism [11].

If one of the important challenges faced in the architectural conservation of historic buildings and archaeology is establishing how to correctly interpret structural and architectural features, so too is establishing how to reconstruct murals correctly. As mural paintings and frescos are extremely delicate and highly susceptible to wear and tear, this task is labour intensive, and requires expensive technical expertise [12]. Deterioration mechanisms include: unstable ambient conditions; structural problems affecting the building that can lead to delamination and plaster stratigraphy, which cause artworks to lose their artistic qualities; dampness; saline efflorescence; biodeterioration; and poor repairs that do not respect the building’s antiquity or its original materials. All of these mechanisms can impact the quality of such artworks by causing them to be partially obscured, i.e., whitewash, or completely destroyed.

Prior to any intervention, researchers must possess a thorough knowledge of the piece in question in order to ensure that conservator-restorers have of all the necessary techniques and instruments needed to recover as much information as possible for any damaged or missing items. As such, results from the multispectral analysis of visible and invisible spectrums, ultraviolet fluorescence photography, or infrared reflectography, help researchers to examine paintings and support structures, and select suitable techniques for revealing elements that are invisible to the naked eye [13,14,15]. Analytical techniques such as chromatography, mass spectrometry and atomic absorption, X-ray fluorescence, or electron microscopy scanning have made it possible to identify the chemical composition of mortars and polychrome paints [16,17,18], but it is also possible to go one step further and establish the extent to which pieces have deteriorated using techniques such as static infrared thermography, spectroscopy and finger-tapping [12,19,20,21].

However, the aforementioned techniques, which are viewed as high-performance techniques in their respective fields of application, are not universally applied, nor are they considered suitable for all types of projects or professionals. The reason for this lies in the fact that equipment is expensive, and extensive specialist training is required in order to know how to use them correctly. What is more, some of these methods are considered too time consuming, e.g., using finger-tapping to detect defects in large format murals in churches [19].

This paper provides details of the process used for the virtual reconstruction of a historic building in a state of advanced deterioration. The 3D reconstruction of the building has been performed using a well-known piece of software and standard topographic surveying and photogrammetry techniques. The software used to recover the deteriorated murals is DStretch^®, which is a piece of software that has not previously been used for this purpose. The processes involved and the use of the aforementioned tool have allowed researchers to recover pictorial elements no longer visible to the naked eye, and then insert these results into a 3D model.

DStretch^® is a digital imaging tool designed to record and document rock art. It works by enhancing different tones using decorrelation algorithms under different colour spaces, and thus generates false-colour composites that reveal pictorial elements that are barely visible [22,23,24]. To date, it has proven itself capable of delivering incredible results in this field [25,26], and researchers have successfully used the tool to enhance pictographs, making them easier to see.

As a result of applying DStretch^® to the historic mural paintings described in this paper, it was possible to appreciate with much greater clarity the compositional aspects and brushstrokes that were virtually impossible to see upon first inspection with the naked eye. This demonstrated that modifying the red-green-blue (RGB) channels of a digital image provides a way to view the pictorial elements of extremely dilapidated murals.

In the following sections, the authors describe the process for virtually restoring the St Augustine church, a UNESCO World Heritage Site located in San Cristóbal de La Laguna, Tenerife, Spain. Destroyed by fire in 1964, the abandoned church fell into ruin and disrepair. As a consequence, its architectural features and mural paintings have been almost entirely lost. The authors detail the processes and techniques used to (i) obtain a realistic 3D model of the building, (ii) recover dilapidated and destroyed murals, and (iii) create an interactive virtual reality environment that uses a virtual reality headset to grant users access to the majestic structures and paintings that once adorned the ancient ecclesiastical building.

2. Literature Review

Knowledge, understanding and appreciation of cultural heritage is the keystone that supports a society’s commitment towards conserving heritage and introducing cultural policies that guarantee its survival [27]. In this sense, the development of new technologies has presented the field of Historic and Artistic Cultural Heritage Management and Conservation with an invaluable resource, one that allows members of the public to familiarise themselves with cultural heritage through virtual experiences that display digital reconstructions of cultural artefacts [28,29,30,31,32]. Such models improve people’s understanding of damaged or missing artefacts and assets via the graphic transposition of the remains of conserved materials and related documentation sources. These virtual reconstructions appeal greatly to modern societies that are becoming increasingly accustomed to virtual reality, and that demand and consume digital experiences [33].

The technical simplicity involved in creating digital reconstructions nowadays means that there is a risk of a tidal wave of pseudo-cultural products that lack scientific rigor and historical accuracy. Such products are capable of generating long lasting misconceptions given just how powerful they are when it comes to disseminating information. For this reason, it is important to establish solid foundations that regulate the production of this type of material and that establish guidelines for generating models truly capable of respecting the history of the cultural assets and materials being represented in the virtual environment [34].

The possibilities offered by 3D modelling tools and virtual reality have brought us closer to a world of information in 3D, which has enabled technological companies, independent professionals and university research teams to explore new developments and conduct ground-breaking research. As a result of their efforts, it has been possible to identify precise methods for digitally documenting cultural heritage sites [35].

One of the most popular techniques for obtaining data and rendering 3D models involves using photogrammetry and digital restitution [10]. Traditionally this technique required specialised instruments and the expert knowledge of photogrammetry specialists. However, certain software can provide simple and straightforward solutions for what are complex photogrammetric calculations, which thereby makes drawing façades in 2D plans much easier [36].

Another common technique used to model historic sites in 3D involves performing scans with high-precision lasers [37]. This method creates extremely accurate models via an automated process [38]. The downside to this technique, however, is that it is a very expensive. Other more economical methods do exist to obtain 3D models. For example, the Microsoft Kinect sensor has been used to scan archaeological artefacts and historical sites, although the results lacked precision when compared against high-precision laser scanning [39]. There are also a large number of mobile apps on the market nowadays that create 3D models from photographs taken in the 3D space around an object. The most noteworthy of these is the ReCap™ Photo by Autodesk^® [40].

The established processes used to recover and restore murals and frescos require specialist techniques and equipment that is extremely expensive, e.g., gas chromatography mass spectrometry, optical microscopy, scanning electron microscopy, infrared spectroscopy, microchemical analysis and X-ray diffraction [41]. When it comes to recovering pictorial art, the digital imaging tool DStretch^® has proven to be an excellent alternative to such expensive techniques and methods [42]. Although originally designed for rock art researchers, DStretch^® has also been used to study faded murals [43], frescos [44] and even images obscured by smut and grime [45]. In yet another demonstration of its potential, it has also been used to recover information about polychrome on antique ceramic [46]. By artificially enhancing hidden or discoloured traces of pigment, DStretch^® reveals details in photographs that are hidden to the naked eye [47].

Thanks to interoperability between applications, which guarantees successful data exchange, it is now possible to faithfully document and display cultural heritage: Recovered paintings and pictorials can be conserved and then displayed by merging results obtained by 3D modelling techniques and the possibilities offered by virtual reality graphic engines.

3. Background

The city of San Cristóbal de La Laguna is located on the island of Tenerife (Canary Islands, Spain). The layout of its historic ensemble and the outstanding universal value of its urban design, which includes an original layout and fifteenth century architecture, led to the city being declared a UNESCO World Heritage Site in 1999.

San Cristóbal de La Laguna is the first example of an unfortified Spanish city, the layout of which served as the archetype for many colonial settlements in the Americas [48]. Located amongst its numerous temples, convents, chapels, public squares, old mansions and palaces is the church and former convent of St Augustine. The church is one of the oldest sites of worship in the Canary Islands, and has played an important role in the cultural and spiritual life of the city. The church was originally founded in 1506 by the Augustinian friars Andrés de Goles and Pedro de Cea. These friars arrived on the island together with Don Alonso Fernández de Lugo, who conquered the island of Tenerife for the Castilian Crown. He is known as the founder and architect of the city San Cristóbal de La Laguna [49].

Within the church itself, special prominence was assumed by majestic murals decorating the altar and presbytery. The murals depicted an allegory of the catholic founders, saints, theologists and intellectuals related to the Eucharistic miracle. These murals were painted in 1780 by Tenerife painter Félix Padrón (1744–1814) using the trompe-l’œil technique [50].

This technique employs artificial and theatrical pictorial elements to create an optical illusion that, in this instance, used swaths of marble and plant motifs to mimic the architectural setting of the classic courts and decorative styles of the eighteenth century. The trompe-l’œil technique was widely used on ceilings and walls in churches from the middle of the XVIII [17]; nevertheless, in the local context it proved to be quite the novelty when it was applied.

On Tuesday 2 June 1964 this place of worship was left in ruins following a large blaze. The disaster destroyed an extensive amount of pictorial heritage deemed to be of great artistic value (see Figure 1).

Since then, the murals have also been lost to neglect in the abandoned church. As a result it is now impossible to see any of the paintings that used to adorn the walls with the naked eye.

In recent years various initiatives, ideas and recovery efforts have been discussed concerning remodelling and renovating St Augustine church so as to turn the site into a leisure centre that could be used by members of the public for cultural and recreational activities [51]. The study presented in this paper constitutes a reference for reconstructing the site, and for all intents and purposes, a restoration of the original historic building.

4. Objectives and Hypothesis

The general objective of this paper is to propose a process work for the virtual retrieval of the architectural structures, features and mural paintings of these dilapidated historic buildings using classical and other novelty tools, so these buildings digitally reconstructed can then be displayed in a virtual environment for the enjoyment of all. One of the works detailed in this paper is to produce a digital restoration of the eighteenth century murals that once adorned the St Augustine church in San Cristóbal de La Laguna for reintegrating them within a virtual model of the church. Currently the church is quite literally in ruins, and its murals almost entirely lost, which means that the recovery efforts involved in this digital reconstruction project are very much about restoring both, the building and its pictorial issues, to their former glory. It should be noted that in reviewing the scientific literature there were no records of papers studying the use of DStretch^® to recover murals. Consequently, the authors of this paper decided to propose a work process that guarantees the trustworthy digital reconstruction of the church.

To reach this objective, a practical case study was designed in which the following specific objectives were set:

• Obtain a virtual reconstruction of the building through classical methodology (planimetric survey by topography and photogrammetry) to represent graphically its structures, its features and its materials, both indoor and outdoor.

For the digital processing and recovery of frescos in the altar and presbytery of St Augustine church:

• Propose the methodological process for digital image analysis using DStretch^® to recover hidden motifs invisible to the naked eye.
• Recover paintings and drawings that are invisible to the naked eye from the altar and the presbytery using DStretch^® digital image processing, image analysis and antique photograph records.
• Obtain a digital restoration of mural paintings.
• Include the digital restorations of the mural paintings in the 3D virtual model of the church.

For virtual interaction with the building:

• Create an interactive model that can be viewed using a (HMD) virtual reality headset that allows the user to interact with supplementary information in this virtual environment.

In light of these objectives, the following hypothesis has been formed: “As a piece of digital image analysis software DStretch^® is useful for recovering obscured artwork from worn and eroded murals”.

If this hypothesis is confirmed in this work, DStretch^® could be considered to be a tool for the digital reconstruction of murals and frescos. The contribution made to innovation would be the expansion of available methodologies for studying dilapidated murals.

5. Methodology

The virtual reconstruction project designed for the St Augustine church is grounded on detailed information compiled from historic records, art history research, photogrammetric and topographic surveys, antique photographs and digital image processing and analysis. Figure 2 shows the process used to recover and reconstruct the architectural structures and mural paintings from the ruins of a historic building, and visualize them within an interactive virtual reality environment.

5.1. 3D Reconstruction of Architectural Features

Using the work of the research team’s engineers, surveyors, and conservator-restorers, a 3D model was constructed containing the building’s architectural structures and features. In this model the church was restored to its original appearance prior to the fire [49,52].

This stage commenced with a topographic survey performed using the Leica TCRP1205 Plus R400 Total Station. This survey was used to establish the floor plan of the church, the height and position of the pillars, and the height of the walls. Data was also gathered from the arches that still remained standing in order to calculate the true shape of their curves and to record their dimensions. In the area of the altar and the presbytery, measurements were taken for the visible pictorial elements on the walls using a tape measure, and then cross-referenced against the points measured on the planimetric map. This was done so that the building’s planimetric map (Figure 3) could be used to provide references during the digital reintegration of the mural paintings.

The church has a rectangular footprint with two visible façades. The convent building adjoins the other two façades. A photogrammetric survey was performed for both façades using 50% overlap. Metashape 2019 (a new version of PhotoScan) was used to do the analysis, image processing and façade restitution, obtaining their orthographic representation (see Figure 4).

The floor plan was produced using AutoCAD and the surveyor data. The work by Blender [53] was then used to create a 3D model of the architectural structures of the church. Using the aforementioned floor plan and height measurements as the starting point, the 3D modelling process began with the inner walls and floors of the church. Once complete, the modelling process continued with the pillars and the archways that divide the interior space (see Figure 5). Next, architectural features were added, and the surveys of the church’s two visible façades were incorporated, giving them volume. This entire process culminated in the creation of the coffered ceilings and small ornamental elements, i.e., doorframes of wooden doors, roofs and ceilings, pillar decorations, and the vault crowning the side door. All of these elements were recreated using antique photographs and onsite visual inspections of existing remains, irrespective of their state of deterioration.

Having completed the detailed 3D modelling of the church, the research team then proceeded to add texture to the model in order to create a realistic appearance. The texturizing process consisted of adding colours and material textures (wood, stone, metal, etc.) to the model. For some elements, these textures were recreated using onsite photographs taken of the pillars, walls, arches and wooden doors; however, for others, such as the paved floors, researchers searched for and edited images of a similar appearance in order to obtain results as faithful to the original as possible. Finally, researchers incorporated the digitally retrieved images of the altar and presbytery frescos (see Figure 6) into the 3D model. The process through which these digital images were rendered is described in detail in the following section.

5.2. Recuperation of Altar and Presbytery Murals Using Digital Image Processing

5.2.1. Current Situation

As mentioned previously, one of the fundamental objectives of this study is to digitally render the pictorial heritage of the St Augustine church with great precision, and then insert these digital images into a 3D model of the building. However, the murals had deteriorated to such an extent that it was virtually impossible to see what was being depicted. In addition, there was a distinct lack of historic records and documents detailing its architectural composition. Unfortunately, there are very few surviving photographs of the inside of the church from which any visual information could be gleaned, and those that do exist are in black and white (e.g., photographic records of its facings).

Regarding the state of disrepair of the pictorial heritage, the murals on the facings to the right and left of the altar have disappeared completely, and the polychrome on the central arch that has been exposed to the elements has been damaged as a result of biodeterioration (see Figure 7a,b). It was noted that the second body, the presbytery, is in a better state of conservation, and the facings and murals appear to have survived the blaze better, which may be due to the presbytery being further away from the heart of the blaze itself. Regardless of the cause, this has meant that the ceiling survived the fire and the walls have not been directly exposed to the elements. Nonetheless, the extreme temperatures caused over 70% of the polychrome to be lost, and the preserved sections show severe signs of deterioration, including flaking, fading, changes in pigmentation chroma, as well as the additional problems of heavy salt efflorescence, crumbling mortar, bird droppings, graffiti and layers of dirt and grime (see Figure 7c).

5.2.2. Proposed Work Process for Recovering Highly Deteriorative Murals

The task of digitally rendering the architectural structures and features of the church faithfully and realistically was achieved thanks to extensive and thorough documentation and comparative analysis that was subdivided into three stages:

Stage one—on site data collection. This stage involved two steps: firstly, researchers recorded any alterations found whilst performing a direct visual inspection of the building, which included: flaking paint; cracked moulding; differing granulometrics in pictorial layers; physical, chemical or thermal shock as a result of the blaze; fading; grime build up (recarbonatation or soot); graffiti; and chromatic alterations of original tones caused by high temperatures; secondly, researchers performed a formal analysis of the conserved remains of the presbytery and the architectural representations. In doing so, it was possible to appreciate that the architectural representations spread out symmetrically from the central axis of the room, thus providing a conical perspective that adds depth.

This discovery meant that it was possible to recover a large amount of lost information by making it possible to project some of the visible elements onto the damaged areas to obtain the remainder of the missing information. Using detail shots, photographs were taken of elements in order to recover as much detail as possible, and thereby gain a better appreciation of the said elements. Using a colour chart, colorimetric analysis was performed on small colour samples taken from the accessible areas free of deposits in order to accurately document and reproduce the true colour of the murals.

Stage two—historical archives review. This stage involved studying historical records and antique photographs. Among these antique photos, researchers found photographic records of the altar facings (Figure 6a), previously thought to have been lost. This discovery assisted the team in recovering damaged images, correctly aligning them, and correcting distorted perspectives. However, as the photos were in black and white, it was not possible for the team to correctly identify the mural colour schemes. Researchers did not have the same luck with the presbytery, as it only appeared in photographs indirectly by way of chance.

These gaps in the historical records combined with the advanced stage of deterioration of the building meant it was difficult to correctly visualise the artistic motifs. This underscored the need for new research and analysis strategies that would ensure that as much information as possible can be recovered whilst helping to prevent guesswork, and thus ensuring false conclusions are not drawn.

Stage 3—analysis of digital records from stage 1 and digital image processing using DStretch^® (www.dstretch.com). The first step involved orthophoto generation prior to digital image processing in DStretch^®. The software Metashape was used to correct images and obtain orthogonality. This software was chosen based on its ease of use, fast calculation speed, and the fact that it also automatically corrects photographic deformities and lens distortions. An orthophotograph contains all of the elements of the image to scale, free of deformations, thus it is a digitally corrected photograph that provides an orthogonal projection free of perspective effects from which precise measurements and extract graphic information can be extracted without any type of deformity.

The corrected photo, free of distortions and perpendicular in all its points, was then processed in DStretch^®, thereby ensuring that deformities had been corrected prior to digital image processing and data extraction.

DStretch^® Digital Image Processing and Analysis

DStretch^® is a piece of software originally created for rock art research that brings out faint pictographs and unclear details using digital image processing [22]. The successful results obtained when recording rock art engravings and paintings led to the software becoming widely used throughout the scientific community in several countries [42,54,55,56,57,58,59]. DStretch^® is a plugin for ImageJ. This plugin is a piece of opensource software for digitally proessing scientific images. It is distributed by Donationware licensing, which is a licencing model that supplies fully operational unrestricted software to the user and requests a donation be paid to the developer.

DStretch^® uses decorrelation stretching, a technique commonly used for remote sensing tasks that artificially enhances the colour separation of an image, thus producing a second image referred to as a false-colour image [60]. This technique combines contrast stretching with decorrelation techniques. The primary purpose of these techniques is to enhance the colour differences of highly correlated images, thereby making it easier to visualise the different elements within an image. In highly correlated images the digital level of the different colour spaces is very close; put simply, the colours of neighbouring pixels are very similar [61], making it difficult to distinguish between the different elements within an image.

DStretch^® starts by applying the Principal Component Analysis—Karhunen-Loeve Transform transformation technique (PCA-KLT). The longest standing and probably most well-known multivariate analysis technique is PCA. It was discovered by Pearson [62] and later developed by Hotelling [63] and Basilevsky and Hum [64]. In Engineering, PCA is normally known as the Karhunen-Loève Transform, hence the abbreviation PCA-KLT. This technique provides the best linear estimate for any given vector range.

This transform is commonly used in image coding systems and even more so in shape and object recognition strategies, i.e., face recognition, finger print recognition, written characters, etc. DStretch^® allows users to enhance colours (stretch) and to even out variations. It uses inverse transform to map or correlate colours back to an approximation of the original.

DStretch^® produces a transformation matrix that is applied to all of the colours in the image. The software supports several default colourspace options for users to choose from: (a) standard colourspaces (RGB or LAB); (b) customised colourspaces created by Harman (YDS, YBR, YBK, LDS, LRE) that are based on LAB colourspaces. These are simply the three hues of RGB light measured under specific lighting conditions in combination with particular colours: L (light, black-white), A (red-green), B (yellow-blue). As such, the YDS colourspace enhances yellow pigments better, YBR and LRE the reds, and YBK the blacks, blues and yellows; (c) colourspaces that users can tailor to their own needs (YXX and LXX) [30,65].

In conclusion, DStretch^® contains a wide array of filters that can be applied to digital images, either directly or in combination, in order to enhance certain colours. The result is a false-colour image that reveals motifs that are nearly invisible to the naked eye, differences in colour pigments, and differences in superposition.

User Interface and Technical Requirements for Original Image

On the PC, the end-user firstly has to install ImageJ and then add the DStretch^® plugin file. To start processing images the end-user then needs to open ImageJ. Once opened, the program asks the end-user which image they would like to analyse. The image is then viewed in the user interface (see Figure 8). DStretch^® colour enhancements can be easily selected using the default colourspaces buttons displayed in the user interface beneath the image. When a button is selected, the corresponding decorrelation stretch algorithm for that colourspace is applied. The programme instantly displays the resulting transformation, enhancing colours in accordance with the said algorithm.

The software is designed to provide the end-user with helpful prompts when the mouse cursor is hovered over a colourspace button. A pop-up window will appear explaining the type of enhancement produced if this said button were to be selected. Among the array of options available to the user are the standard colourspaces RGB and LAB, as well as those described above (see Figure 8).

It is possible to select a particular area of an image to restrict the pixel count, and in doing so, eliminate potentially confusing information or elements that the user is not interested in. Another interesting parameter that should also be mentioned is scale. The default scale value is 15, however this can be increased or decreased as necessary to minimise noise.

Authors with extensive experience using DStretch^® in the context it was originally designed for have recommend using TIFF files rather than JPEG files (loss-less information compression), and the highest resolution possible [26,66]. Furthermore, they also recommend using low ISO to reduce pixel noise, using natural light, and using a tripod or support to ensure image stabilisation for long exposure images [54,56].

This program is very intuitive and users can quickly learn how to get to grips with the basics. It has been designed for end-users with no prior experience of image processing, allowing them to change colourspaces, modify histograms (contrast levels), saturation, etc., and in doing so, identify elements that were not visible to the naked eye.

6. Results

The research produced a 3D model of the church and a virtual reintegration of the altar facings (AL, AR and ACA—Figure 7) using the orthographic projection of images obtained from antique photographs.

Image processing analysis using DStretch^® was centred on the murals in the second body of the St. Augustine church, the presbytery. The decision to apply this technique in this section of the church was motivated by the highly deteriorated state of materials in this area and the lack of historical records.

First, to establish a general overview of the area, the researchers performed an analysis of the orthographic images obtained for the three presbytery facings (LW, CW, RW—Figure 7).

The most significant results obtained were for the right lateral facing (RW) and the corner joining the central facing. Over 70% of data was missing from the entire area, meaning a conservator-restorer might accidently draw false conclusions. In addition, there also existed a risk that small sections of polychrome fragments could be overlooked as a result of being hidden under dirt and efflorescence deposits.

In the case of the central facing (CW), no information could be recovered by directly inspecting the walls and murals with the naked eye. In addition, artistic depictions in this area had not been documented in historical archives. However, it was possible to obtain results by processing orthoimages in DStretch^®. The LDS colourspace at 15-value scale brought out the dark tones of the cornice that spread out towards the centre of the scene, and with them the small and isolated remains of this motif. This helped researchers to identify and determine the minimum length of this element. Similarly, the RGB colourspace at 20-value scale allowed researches to visualise crisp images of the partial remains of quadrangular elements that spread out across the central arch in the same direction (Figure 9). By opening these enhanced images in Adobe Photoshop^® (or a similar image manipulation program could be used), it was possible to isolate the highlighted colour using the ‘magic wand’ feature.

Significant results were also obtained for the lower sections of the right lateral facing (RW). These results served to rectify imprecise or confusing conclusions that had been drawn from direct visual inspection. Initially, inspections were hindered by the presence of large deposits of extraneous materials or efflorescence. However, by analysing these images researchers were able to clearly identify the square-shaped depictions located between the large lateral pillars and the small door providing central access. In addition to clearly demarcating the finishing on the small pillar flanking the door, the images also revealed the existence of blue decorative motifs that served as a connecting element with the lintel located above (Figure 10). The colourspaces that produced the best results in this area were LDS, LAB and YYE. When applied at the 7.5-value scale, these colourspaces revealed contemporary graffiti that was invisible to the naked eye (Figure 11a). The manner in which certain tones are highlighted based on the colourspace selected can be best understood in the histograms. In an unprocessed image, one in which all the pixels share a similar tone, the histogram contains predominantly red tones, and the blues and green are very dim and evenly matched (Figure 10a). Having processed images using different colourspaces, greater contrasts were revealed, and the spectral ranges were more concentrated in blues, greens and magentas (Figure 10b–d and Figure 11c,d). These contrasts were obtained by making adjustments directly to the images’ histograms. The best results were obtained in the red and blue colour spectrums with the green spectrum fully reduced. This analysis was not performed to obtain the tones of colour pigments, but rather to identify the forms, lines and shapes of graphic elements. Due to the specific nature of these oil-based colour pigments, high temperatures have the effect of altering their colour. Thus, following such a blaze we cannot hope to recover such information. However, by manipulating the tones of the colours it is possible to obtain the contrast needed to obtain a rough sketch of the artwork. From this, it is then possible to establish the lines (or silhouette) that define the graphic representations.

The results obtained and interpreted from image processing analysis were then extracted and symmetrically extrapolated to fill in the details of areas in which it was impossible to gather any data (Figure 12).

As well as reconstructing the central facing (CW) using the image processing tool DStretch^®, researchers also had to reconstruct the central area below the pictorial arcade. This area could be reconstructed thanks to a single historical image containing vestiges of its geometrical structures (Figure 13).

The results obtained from analysing the image of the central facing (CW) led the team to resume its study of antique photographs from the archives. On this occasion, photographs were subjected to photo enhancement processes to bring out elements that were not clearly visible. Consequently, researchers found that they were able to establish that the quadrangular elements found inside the central arcade and the blue traces located between the door and lateral wall opening depicted wainscoting and cornice that connected the altar and presbytery spaces (Figure 13).

Integration in Virtual Model

A reconstruction of the church’s pictorial elements was created by combining three aspects: (i) data gathered from direct visual inspection, (ii) results of a digital analysis of the images, and (iii) a review of historical archives and antique photographs. The reconstruction process culminated with researchers inserting the high-resolution digital renderings of the retrieved mural paintings onto the facings of the altar and the presbytery in the virtual model (Figure 14 and Figure 15).

7. Integration in a Virtual Reality Environment

The VR app was created using the video game engine Unity. The decision to use this engine over others was based on it being one of the most popular engines for creating VR environments and experiences, which is due to it being extremely versatile, easy to use, and that it is supported across many different platforms. Like other graphics engines, Unity can also be used to create interactive applications. This fact makes the engine highly appealing to developers of interactive virtual environments.

The VR app was designed to work with the headset Windows Mixed Reality (WMR). For this, researchers needed a computer with Windows 10 OS, Visual Studio programming software and SteamVR SDK. Unity includes integrated support for WMR, as well as a wide array of tools, packages and resources for creating cross-platform, interactive experiences. It is important to highlight four interesting features of SteamVR SDK for Windows Mixed Reality: (1) Visual interaction that allows the user to observe three-dimensional environments; (2) Movement capabilities that allow the user to move through the virtual environment through the use of a hand-controller, either by walking or by teleportation; (3) Scene selection that can be executed by clicking on special marks within the scene, or by pressing dedicated buttons for this purpose; (4) Presentation of text and audio data by selecting specific buttons (see Figure 16).

Acer Windows Mixed Reality is a virtual reality headset for room-scale virtual reality that is used to visualise and interact with the virtual environment that has been rendered. It is important that objects and dimensions are full-scale for virtual reality environment walk-throughs. The 3D model created by Blender software was imported into Unity in the FBX format. It is possible that during this process the scale of 3D model changes, and if so, it must be corrected.

Teleportation is used to help users overcome limitations arising from a lack of physical space in the real world (Figure 16a).

The camera position was adjusted to eye-level to allow users to view the 3D model comfortably. The initial viewpoint opens in the centre of the building at a height of 1.70 m. Users can use their hands and controllers to navigate in any direction and move closer to the finer details of the building. To make the VR application suitable for interactions, several C# scripts were implemented as features, allowing text labels to be viewed from different angles.

Pointing at murals activates the markers that indicate there is additional information available (Figure 16b). When the user interacts with a marker, a text box containing factual information opens: for the altar murals, users are presented with historical data on the depictions, and for the presbytery murals, users are presented with information on the pictorial techniques and representations. Figure 16 displays a screenshot of the VR model in use and potential interactions.

8. Conclusions

At present, it has never been easier to document cultural heritage sites using VR technologies. It is now possible to obtain reliable records of cultural heritage by combining the new technologies, equipment and computer programs with tried and tested methodologies from the fields of cultural heritage conservation that have been used for several decades. The results of this documentation process can be used for different purposes: research, tourism, exhibitions, and so on. However, documenting the current and past states of cultural heritage for conservation purposes is arguably the most important goal of the documentation process.

This paper presents a work process for recovering cultural heritage using classical and novel tools. The reliable results provided by these tools are then used to create a faithful reconstruction of a historical element.

In this study, a model for documenting dilapidated cultural heritage sites using VR technologies is presented.

The intervention was applied in the St. Augustine church in the city of San Cristóbal de La Laguna, which is located in Tenerife, the Canary Islands. The objective was to reconstruct a historical building using 3D modelling techniques and use VR technologies to visualize and interact with this said model.

The 3D model was created using traditional processes, starting with a precise topographical survey, onsite visual inspection, and a review of historical photographic archives, especially for the roof that no longer exists.

Using the surveys and processes that were completed and then optimised utilising digital image analysis tools, researchers were able to create a reasoned and logical argument as to the nature of the murals of St Augustine church.

DStretch^® proved to be an incredibly versatile analytical tool, despite the fact that the processes being performed in this study fall outside its original field of application. The tool allowed researchers to digitally visualise invisible elements with detail. This not only made it possible for researchers to recover details hidden to the naked eye, but also improved the objectivity of the formal analysis that was performed, and thus of the digital reconstruction work that was undertaken. One of the main advantages of this plugin is that users need not possess a wealth of prior knowledge of digital image processing to be able to use it. Its design and default settings help users learn quickly and easily, and it provides users with immediate feedback. Unlike other image processing software, the digital enhancements made possible by this piece of software are dependent on the operator, i.e., the results tend to vary greatly depending upon the skills and experience of the operator.

In light of the results garnered, the authors of this paper argue that DStretch^® is an invaluable tool for the formal analysis of historical frescos and murals, and thus, a useful tool for conservator-restorers and for cultural heritage education.

Having inserted a custom-built 3D model into Unity, an immersive virtual reality visit was developed for the VR system Acer Windows Mixed Reality. This VR system provides a truly immersive experience that allows users to step inside the virtual church environment without ever having seen the church itself in real life. The universal appeal of VR apps, and the importance of leaving reliable records of our cultural heritage, make remote historic monuments or ruins the ideal settings for immersive VR applications.

In terms of knowledge dissemination, the inclusion of additional information such as text, photos, multimedia products and user interactions widens the realm of possibilities. Even adding the support functions for multiple users in VR apps creates new capabilities that allow for a whole host of new applications, i.e., virtual meetings, virtual training courses, or workshops for professionals wishing to discuss the virtual reconstruction of the cultural heritage monument, or educational sessions.

The final result is a valuable virtual record for cultural heritage research and conservation purposes. It is a scalable solution, because it is possible to include more virtual information in this environment. You could even use this virtual and interactive building to host virtual exhibitions of sculptures, paintings, and so on. In this regard, Unity is a very flexible piece of software that is apt for creating VR environments.

This way of working is of interest to teachers on conservator-restorer training courses, and can provide improvements in terms of student motivation, assimilation of content, and greater autonomy during working reconstruction and conservation processes.