ROBOTIC MAPPING OF CULTURAL HERITAGE SITES

In archaeological studies the use of new technologies has moved into focus in the past years creating new challenges such as the processing of the massive amounts of data. In this paper we present steps and processes for smart 3D modelling of environments by use of the mobile robot Irma3D. A robot that is equipped with multiple sensors, most importantly a photo camera and a laser scanner, enables the automation of most of the processes, including data acquisition and registration. The robot was tested in two scenarios, Ostia Antica and the Würzburg Residence. The paper describes the steps for creating 3D color reconstructions of these renown cultural heritage sites.


INTRODUCTION
Archaeology is a historical science of high social interest.It studies the human being and its legacy, such as buildings, tools and art.Cultural heritage sites can be found all over the world and they tell us the story of humanity in different areas of the world.Remote sensing has become state of the art in modeling archaeological sites.This way of digitization of entire buildings or areas gives as a unique opportunity to preserve the current state of prehistoric buildings and to join forces of experts all over the world.Collecting the data is tedious work.It includes finding the best position for a laser scan, moving the equipment to the position and georeferencing of the scanning position.Letting a robotic system take over this work reduces the time spent in the field by 75 % and decreases the impact to the sites.We present the robot Irma3D, that was designed to create in a tele-operated fashion digital 3D models of environments.This paper describes the setup and the capabilities of the robot and the steps to create these 3D models automatically from multiple sensor sources.The entire process is demonstrated by means of experiments carried out at cultural heritage sites.
The robot was tested in two scenarios, Ostia Antica and the Würzburg Residence Palace.In this paper we describe the data collection with the robot Irma3D in these two renowned historic sites, the post-processing needed to create a full 3D color model and  present the resulting models from a garden house in Ostia Antica (http://youtu.be/sf-gq5xlaIc), the White Hall (http:// youtu.be/_wPug_So_iE)and the Imperial Hall (http://youtu.be/jKVxlLvu7Pk) at the Würzburg Residence.The results can interactively be viewed in a 3D viewer or with an Oculus Rift (cf.Bruder et al. (2014)).

EXPERIMENTAL SETUP
In this section we explain the setup for robotic mapping of cultural heritage sites.First, we describe the environments where measurements were to be taken.Second, the hardware used in both scenarios is presented.

Cultural Heritage Sites
The robot was tested in two scenarios, Ostia Antica and the Würzburg Residence.Ostia Antica is a large archeological site, close to the modern suburb of Ostia (Rome).Due to the exceptionally well preserved state of the city, Ostia is of immense interest for the study of the Roman empire.According to archeologically unverified tradition Ostia was founded during the second half of the 7th century B.C. at the mouth of the Tiber river.This region was said to be strategically important for Rome due to the there existing salt marshes.Supported by evidence is the settlement at the mouth of the Tiber as early as the beginning of the 4th century B.C. Initially only a military camp meant to defeat Rome towards the sea Ostia grew to be an autonomous harbor city of about 50 hectares with a population of approximately 50,000 in the second half of the 2nd century A.D. In the following began the decline of the city.Large parts of the town were abandoned following historical events.The last inhabitants left the city after raids of the Saracen pirates from the sea around the 9th century A.D. (Ostia Antica, 2014).
The experiments took place in one of the garden houses in Ostia Antica.Historians suspect that these garden houses were a large building project consisting of many serial prepared houses that were rented out afterwards.Of special interest is the context of architecture and accouterment, i.e., the wall paintings and the floor mosaics.The garden house in question contains of one large semi-open area and several smaller rooms that are connected with narrow hallways.Scanning the entire area completely without holes requires a large number of scan positions.
The Residence Palace in Würzburg Germany is a baroque palace in the city center of Würzburg, Germany and was labeled a UN-ESCO World Cultural Heritage site in 1981.Being built from 1720 to 1744 with the interior finished in 1780 it is now one of Europe's most renowned baroque castles.It was laboriously reconstructed after being heavily damaged during World War II.Not destroyed during the war remained the large unsupported trough vault above the main stair-case designed by architect Balthasar Neumann, the Garden hall with ceiling paintings by Johann Zick, the white hall with the impressive stucco work by Antonio Bossi and the Imperial hall with frescos by Giovanni Battista Tiepolo.With its large colorful paintings by the Venetian painter Giovanni Battista Tiepolo and fine stucco work by stuccoist Antonio Giuseppe Bossi in many of the almost 400 rooms the Würzburg Residence is a unique example of baroque style (Würzburg Residence, 2014).
Experiments were carried out in both the White hall and the Imperial hall, two large halls with impressive 3D structure.Together with the colorful paintings in the Imperial hall the environment can only be captured by the combination of two technologies, e.g., laser scanning and photography.

HARDWARE
The data was acquired with the mobile robot Irma3D (Intelligent Robot for Mapping Applications in 3D).Irma3D is a small, battery-powered, light weight three wheeled vehicle.It consists of a modified Volksbot RT 3 chassis with two front wheels.Each is actuated by an individual 150 W DC Maxon motor.The motors are powerful enough to move the robot at a maximum velocity of 2.2 m/s.The third wheel is in the back of the chassis and is swivel-mounted and thus completely passive as it follows the directions of the front wheels.The high-powered electrical twowheel drive is equipped with rotary encoders to measure wheel rotations.This information is used to provide pose estimates of the robot via odometry.The pose estimates are improved using data from the Xsens MTi IMU device that is also attached to the robotic platform.For obstacle avoidance when moving autonomously a Sick LMS 100 2D laser scanner is added to the front of the robot.This sensor can also be used to improve the localization of the robot.The central sensor of Irma3D is the 3D laser scanner VZ-400 by RIEGL Measurement GmbH.The scanner is mounted on top of the Volksbot RT 3 chassis.Attached to the top of the scanner is a Canon 1000D DSLR camera.After a 3D scan has been acquired the camera is used to acquire color information for the point cloud.
In the Würzburg Residence an iSpace sensor frame is also mounted on top of the laser scanner.iSpace is a high-precision position and tracking system from Nikon Metrology (2014).The optical laser based system consists of several transmitters.These are mounted on a wall or on tripods to cover the experimental area both indoors and outdoors.The rotating head of each transmitter emits two perpendicular fan-shaped laser beams at a unique distinguishable frequency near 40 Hz.The vertical opening angle of the laser beams is limited to 40 degrees and the detectable range lies between 2 to 55 meters.Several sensor frames can be located within the system.A sensor frame consists of at least one detector, a photo diode with a horizontal opening angle of 360 degrees and a vertical opening angle of 90 degrees.A small radio frequency module transmits the sensor data wirelessly to the base station of the iSpace system, a PC running the iSpace control software.A sensor frame with one detector is sufficient to acquire 3D position information.To measure also the rotation and to increase the accuracy of the position data the sensor frame used on the robot has a total of four detectors.The iSpace system differs from other position and tracking systems as the transmitters do not actively observe the position of the sensor frames.Instead, each sensor frame receives the laser data from the transmitters and sends the information on to the control PC.The control PC calculates the elevation and azimuth angles between all detectors for a sensor frame and each visible transmitter based on the received data defining a straight line between transmitter and detector.Given the relative transformation between the transmitters the length of the lines is calculated using triangulation.To determine the position of the transmitters a calibration procedure using a few hundred points from a special sensor frame is applied.An optimization process calculates the position of all transmitters in a self-defined coordinate system.In typical environments the iSpace system is able to perform measurements at a sampling rate of 40 Hz with a maximum error of [±0.25]mm.In practice environmental factors such as size, reflection of the surface and occlusions of the transmitters have to be taken into consideration.

ROBOTIC MAPPING
Scanning a large interconnected area consisting of many rooms requires a large number of scanning positions if one wants to cover the area completely without holes.In terrestrial laser scanning this is done by manually moving the laser scanner.To determine the exact position of the laser scanner for each position special targets are commonly placed in the environment whose positions are measured with a total station.Afterwards, the position of the scanner is either determined by placing another target on the tripod at the position where the scan is to be taken or by identifying the targets directly in the laser scan.This procedure is time-consuming and has to be repeated for each scanning position.The idea of using a robot for scan acquisition is to eliminate this time and thus reduce the data acquisition time.In this section we describe the methods used for creating precise 3D models of the collected data, i.e., the registration and calibration methods for bringing all the data from the various sensors into one common coordinate system automatically.

Sequential scan matching
To collect data at several locations the robot is moved to a scanning location and stops there for data collection.In robotics research methods have evolved over the past years that are specifically designed to register point clouds collected by a mobile robot automatically.Commonly used for this task is the Iterative Closest Point (ICP) algorithm (Besl and McKay, 1992).The algorithm takes advantage of the fact that robots usually have a rough estimate of their current pose (position and orientation).For the robot used in this paper this pose estimate is calculated from the movements measured with the rotary wheel encoders and the IMU device.Starting from such initial pose estimates the algorithm calculates effectively the correct transformation between two point clouds by means of minimizing distances between corresponding points.Corresponding points are chosen based on Euclidean distances.The algorithm is described in Algorithm 1.
Given a registered point cloud M and a point cloud D with an initial pose estimate the ICP first tries to find for each point di from D the point mi in M that is closest to di.Then one needs to solve for a transformation (R, t) (orientation R and translation t) that minimizes the error function EICP.Nüchter et al. (2010) presents several minimization methods.These two steps are iterated to find the best transformation between the two point clouds.For best results a threshold t dist is introduced and all point pairs with a distance larger than t dist are discarded from the calculations.N is the number of remaining point pairs.In practise this procedure is adopted as follows.Using the first scanning position as reference the nth scan is always registered against the (n − 1)th scan.This way all scans are sequentially transformed into the same coordinate system.

Global optimization
When applying the sequential procedure small local errors for the registration of each pair add up, possibly leading to larger errors for long sequences.To overcome this issue a global optimization was proposed by Borrmann et al. (2008).Given a graph of corresponding scans point pairs are determined for each scan pair (j, k) and the new error function Eopt requires finding transformations for all scans simultaneously: Methods to minimize Eopt are presented in (Nüchter et al., 2010).
The implementation used in this publication is freely available from The 3D Toolkit (3DTK) (Andreas Nüchter et al., 2014).

Acquiring initial pose estimates
ICP and the global optimization rely on initial pose estimates to determine the correct transformation between laser scans.Different methods to acquire pose estimates are described in the following.

Odometry
On mobile robots pose estimates are commonly attained from odometry.Odometry for wheeled robots such as Irma3D is based on calculating the distance traveled by the robot based on wheel rotations.For this purpose the relation between the count c of the wheel encoders and the wheel rotations are related to each other using a factor f .Knowing the diameter d of the tires the distance traveled by one wheel is calculated as Considering the distance B between the two wheels and the distances traveled by each wheel ∆s l , ∆sr the pose at time step n (x, y, θ) of the robot is calculated as: The quality of these measurements depends highly on the behavior of the robot on the ground.If the floor is slippery and the wheels spin uncontrolled the estimates lack precision.To increase the precision of the position estimates the xSens IMU is attached to the robot and thus the measured accelerations are integrated into the position estimates.This is especially helpful for rotations.
Odometry works sufficiently when short distances are covered and the robot follows smooth trajectories.With increasing path lengths, many rotations and frequent stops along the trajectory errors sum up.3.3.2Feature-based registration If no pose estimates were acquired during data collection the remaining option is to determine them directly from the data.Apart from range information modern laser scanners often capture the amount of light that is returned to the sensor.This information, known as reflectance value, can be used to detect features in the data.In terrestrial laser scanning it is common practice to attach high reflective markers to the environment and detect them in the data to register the laser scans.This procedure takes a lot of time to place the markers in good positions, find them in the data and register the data.Often times the exact positions of the markers are also measured using a tachymeter extending the time needed in the field even further.As an additional disadvantage the markers will be visible in the data thus adulterating the scene and the mapping.Alternative methods use natural features rather than manually attached markers.Conditioned by the buildup of the laser scanning platform the points are captured as range, reflectance and spherical coordinates.This facilitates the generation of a panorama image using the spherical coordinates and the reflectance information.The 3D data is thus projected onto an image.Different projection methods are evaluated by Houshiar et al. (2012).
The panorama generation enables the use of image based feature matching methods.These methods analyze the image and create a description of areas with high changes in intensity.The most common features are SIFT (Scale invariant feature transform) features.They also show superior performance for feature-based point cloud registration (Houshiar et al., 2012).As the SIFT feature detector works in gray-scale images the panorama images from reflectance values of laser scans are ideally suited for feature matching.For the automatic registration of point clouds using these panorama images corresponding features are detected in the panorama images of scan pairs.Feature correspondences found in two reflectance panoramas are used to calculate pairwise transformation matrices of the point clouds with a RANSAClike (Fischler and Bolles, 1981) approach.For this purpose the algorithm identifies the feature in one image that is the closest to the sampled feature from the other image based on a comparison of their descriptors (see Figure 2).Several algorithms such as the k−nearest neighbor (KNN) search and the radius KNN search are possible solutions to this problem.The ratio nearest search as presented by Lowe (2004) has shown the most promising results.The registration proceeds by testing a subset of 3 point pair matches and examining the two triangles that are defined by these points.The algorithm translates the triangles so that their centroids lie at the center of the common reference frame.The orientation that minimizes the error between the points is then computed by the closed form solution proposed by Horn (1987).
Depending on the projection method used for the panorama generation the optimal image size varies.The larger the image, the

Camera and laser scanner co-calibration
The sensors on Irma3D used for reconstruction of the environment are the Riegl VZ-400 laser scanner from terrestrial laser scanning and the Canon EOS-1000 DSLR camera.The laser scanner acquires data with a field of view of 360 • × 100 • .The camera is equipped with a Canon EF-S 18-55 mm IS II zoom lens.Thus it has a much smaller opening angle.To achieve the full horizontal field of view the scanner head rotates around the vertical scanner axis when acquiring the data.We take advantage of this feature when acquiring image data.Since the camera is mounted on top of the scanner, it is also rotated.We acquire several images during one scanning process to cover the full 360 • .
To avoid blurring and the problems that come from the necessity of synchronization we refrain from taking the images while scanning.Instead we perform a full 360 • rotation for scanning and rotate back with stops at the image positions.A further advantage of this strategy is that the camera can be connected with regular USB cables because the cable is unwound after each rotation.
To join the data from the laser scanner and the camera automatically the two sensors have to be co-calibrated.This process is described in detail by Borrmann et al. (2012).Each sensor perceives the world in its own local coordinate system and to join the perceived information we need the specific parameters of these coordinate systems.Laser scanners are calibrated to acquire precise geometric coordinates.A camera has unique parameters that define how a point (x, y, z) in world coordinates is projected onto the image plane.These intrinsic camera parameters are calculated through a process known as geometric camera calibration.Intrinsic camera calibration is typically done using a chessboard pattern because the corners are reliably detected in the images.For determining the transformation between the camera and the scanner coordinate system, i.e., extrinsic calibration, we attach the calibration pattern onto a board that is mounted on a tripod (cf.3).This way the board can easily be positioned at different locations and hangs almost freely in the air facilitating the the detection of the calibration board in the point cloud data.
The RANSAC algorithm (Fischler and Bolles, 1981) or the Randomized Hough Transform (Borrmann et al., 2011) are common methods for plane detection in point clouds.The search area is easily reduced with a simple thresholding technique leaving the board as the most prominent plane in the data.In the Residence, the scanner was tilted when mounted onto the robot.On the one hand this enables free view of the ceiling.On the other hand, it makes removal of the floor more difficult as the scanner coordinate system is not aligned to the floor anymore and the distance between the board and the floor is small.This is easily overcome by searching for the two most prominent planes and removing the floor from the search results.To calculate the exact pose (position and orientation) of the board a plane model is generated by subsampling points on a plane with the dimensions of the calibration board.This model is transformed towards the center of the detected plane facing the same direction as the plane.The model is then fitted perfectly to the data using the ICP algorithm (see Section 3.1).Since the positions of the chessboard corners on the board are known, their exact positions in 3D space can be calculated from the pose of the board.
The precise transformation between scanner and camera is determined by acquiring several pairs of data and evaluating the reprojection error of each individual transformation.The thus calculated transformation is directly used to project the scanned points onto the camera image and color the points according to the pixel color.Due to the different fields of view of the sensors, they each perceive slightly different parts of the world.A region that is visible from one sensor might be occluded for the other sensor.When mapping the color information to the point cloud this causes wrong correspondences and therefore faulty assigned values.To solve this problem a ray tracing procedure was implemented that checks whether a point in the point cloud can be seen by the camera.We connect the point P and the camera position C with a straight line P C and select all points with a distance less than a threshold t to P C, i.e., all points Oi for which holds true.If any point Oi lies between P and C, P is not visible from the camera and is therefore discarded.The threshold t accounts for small inaccuracies in the calibration.To speed up the checking procedure the points are organized in a kD-tree data structure.With a quick check those voxels are immediately discarded that are not traversed by the ray and therefore all the points within are ignored.

Calibration of sensors for localization
To achieve precise pose estimates for the 3D laser scanner the localization sensors of the robot have to be calibrated to the laser scanner.The method used for odometry, IMU and the 2D laser scanner is explained in Elseberg et al. (2012).In this paper an iSpace sensor frame is attached to the VZ-400 laser scanner to localize the robot in the iSpace coordinate system.After setting up the transmitters of the iSpace system several reflective markers were attached to objects in the environment.The centers of the markers are measured with the iSpace handvector bar, thus determining their position in the iSpace coordinate system.These markers show up nicely in the reflectance data of the laser scanner.To measure their precise position first a full scan of the environment is carried out.The RiScan Pro Software is used to detect the markers in the environment.The correct markers are manually chosen from a list of automatically detected candidates.In a second step, fine scans of the markers are performed.The software controls the scanner automatically to scan the area around the selected markers with a very high resolution to determine the precise position in the local coordinate system.Third, the coordinates of the markers in the coordinate system defined by iSpace are imported as control points and the scans registered to these control points based on the marker position.This yields the position and orientation of the laser scanner in the iSpace coordinate system at the time the scan was taken.Additionally, the pose of the sensor frame is also recorded.In the following poses will be treated as transformation matrices T, consisting of the rotation R and the translation t.Repeating this procedure for n scans gives n pairs of poses for the Riegl laser scanner Tr,i and the sensor frame Tm,i.From these poses the transformation Tm→r between the coordinate systems is calculated as: To reduce noise the average Tm→r over all transformation matrices Tm→r,i is calculated.Afterwards, for each new scan position the position of the laser scanner in the iSpace coordinate system is calculated as: 4 EXPERIMENTAL RESULTS

Ostia Antica
Scanning took place over the course of 5 days.A total of 59 scans were acquired.Every morning scanner and camera were recalibrated to avoid inaccuracies caused by the transport.Also, the exposure time of the camera had to be adjusted according to the lighting conditions.The number of scans acquired per day are listed in Table 1.The first day was mainly used to prepare the environment for data collection.Especially the floor had to be cleaned from dust to allow for perceiving the floor mosaics.The first scans in the afternoon of day 1 were taken to check the functionality of all components.On day 2 the full day was reserved for scanning and the number of scans could be increased.On day 3 a technical defect caused a break in the data acquisition phase but the defect could be resolved and the scanning resumed.At the end of day 4 the planned parts of the garden house had been captured.Day 5 was left to take some additional scans in areas that were hard to access and from the outside to complete the model.
A floor plan was created from a 2D orthogonal projection of the final model without the protection roof and is depicted in Figure 4.The scan positions are marked in the floor plan.The robot coordinate system is defined by the position of the robot at the start of the robot control program.Thus, for every run the coordinate system is different.We tried to overcome this issue by starting the robot at the same position every day.However, the inaccurate starting position in combination with the long and curvy robot path to the first scan of the run often lead to large errors in the pose estimates causing the odometry based mapping algorithms to fail.The discontinuity of the path also caused problems for the sequential ICP algorithm as subsequent scans often have no overlap.Therefore, the poses were manually improved and scan pairs for matching with sufficient overlap manually chosen to create a coarse registration.Afterwards, the global optimization was applied to create the final model using the graph shown in Figure 4.
After the problems with initial pose estimates from odometry the feature-based algorithm was implemented to automate the reg-

W ürzburg Residence
To evaluate the methods further experiments were carried out in the Würzburg Residence.The iSpace system was used to measure the robot position with high precision.The localization system was set up using six transmitters that were placed in the Imperial Hall as depicted in Figure 7.The robot was manually driven to 11 scanning positions.At each scanning position the position of the robot was recorded using the iSpace system.Figure 8 shows a top view of the resulting models.The scanning positions are connected by black lines.The points are color-coded based on the scanning position.Several observations can be made.First, it is obvious that the position estimates from odometry contain very large errors.Thanks to the fact that the environments consists of a single large hall it is still possible to correct the model using point-based registration.In more complex environments such as the garden house, however, the approach is likely to fail.Second, feature-based registration leads to really good results.In fact, when comparing the maps generated with iSpace measurements and FBR, the walls appear thinner for FBR, suggesting that the positions are more accurate.Third, point-based registration further improves the map quality.This can be seen by having a close look at the color distribution in the maps.Close to the individual scanning positions the density of the color of that scan becomes higher, while in the other maps the distribution is more random.This effect is the same, when comparing the results with start estimates from different methods.
Figure 10 explains the inaccuracies in the position estimates from the iSpace system.Shown are the uncertainties in estimating the position of the robot's sensor frame for each scanning position as calculated by the system.The high uncertainties come most likely from the reflections caused by the windows and the chandeliers.This makes clear, that registration based solely on the iSpace system does not yield the required accuracy.A further disadvantage of the method are the transmitters that act as obstacles during the data acquisition.In combination with the time needed for setup this makes the system infeasible for the purpose.
The effectiveness of the ray tracing procedure is demonstrated with an examplary scan of the Imperial Hall in

CONCLUSIONS AND OUTLOOK
The 3D reconstruction of cultural heritage sites is tedious work.
The acquisition and registration of data from multiple data sources requires skilled personal and a lot of time to create a complete virtual 3D environment.In this paper we outlined steps for 3D modelling of environments by use of the mobile robot Irma3D.The robot collects simultaneously color images and 3D laser scans.
Several processes are described that help automate not only the data acquisition but also registration step.These processes are evaluated in two examplary environments to demonstrate their effectiveness.These experiments show the limitations of the approaches but also outline the ability to overcome them.Future work focusses further on the evaluation of the different methods to evaluate the accuracies of the presented approach.The use of the 2D laser scanner to acquire initial pose estimates will also be evaluated in comparison to the methods presented here.

Figure 1 :
Figure 1: The Irma3D robot with the setup used in the Residence.

Figure 2 :
Figure 2: Illustration of a scan pair with Mercator projection and matched features from the garden house in Ostia Antica.

Figure 3 :
Figure 3: The calibration board seen from Irma3D in the Würzburg Residence as photo (left) and point cloud (right).moredetail is depicted.At the same time, however, the processing time for the registration increases.Therefore image size optimization can be carried out by calculating the optimized size of the image based on the aspect ratio of the projection to reduce the extra distortion introduced by fitting a panorama image to a fixed size image(Houshiar et al., 2013).

Figure 4 :
Figure 4: Floor plan of the garden house in Ostia Antica generated as an orthogonal projection of the 3D model without the protection roof.The scan positions are marked.Left: The red lines show the graph used during the global optimization of the model reconstruction.Right: The arrows indicate how the scans were registered with the feature-based registration.Green arrows with solid lines indicate the scans that were registered with the standard settings.Red dashed lines mark those pairs where a modification of the parameters was necessary.The outdoor scan pair connected with a black dotted line was not registrable at all.

Figure 5 :
Figure 5: Irma3D at the garden house in Ostia Antica.

Figure 6 :
Figure 6: The model of the garden house in Ostia Antica.(a) two views of the data acquired on the first day.(b) The complete model with data from all days.The white lines mark the scanning sequence.(c) Outside view of the complete model.

Figure 8 :
Figure 7: (a) Irma3D in the Imperial Hall of the Würzburg Residence during calibration.In the background is one of the transmitters for the iSpace localization system and a box with two reflective markers for the calibration procedure.(b) + (c) One scan from the Imperial Hall colored with the information from the photos without (a) and with (b) correction from using raytracing.It is obvious that the raytracing methods removes wrongly colored points.

Figure 9 :Figure 10 :
Figure 9: Reconstruction of the Imperial Hall in the Würzburg Residence.Visible in the hall are four of the six transmitters from the iSpace localization system.

Figure 11 :
Figure 11: Reconstruction of the White Hall in the Würzburg Residence.In the close-up view the impressive stucco work is clearly visible.

Table 1 :
Work progress in Ostia Antica