COMPARATIVE GEOMETRICAL INVESTIGATIONS OF HAND-HELD SCANNING SYSTEMS

An increasing number of hand-held scanning systems by different manufacturers are becoming available on the market. However, their geometrical performance is little-known to many users. Therefore the Laboratory for Photogrammetry & Laser Scanning of the HafenCity University Hamburg has carried out geometrical accuracy tests with the following systems in co-operation with the Bochum University of Applied Sciences (Laboratory for Photogrammetry) as well as the Humboldt University in Berlin (Institute for Computer Science): DOTProduct DPI-7, Artec Spider, Mantis Vision F5 SR, Kinect v1 + v2, Structure Sensor and Google’s Project Tango. In the framework of these comparative investigations geometrically stable reference bodies were used. The appropriate reference data were acquired by measurement with two structured light projection systems (AICON smartSCAN and GOM ATOS I 2M). The comprehensive test results of the different test scenarios are presented and critically discussed in this contribution.


INTRODUCTION
In recent years the measurement market has been significantly expanded in the lower to middle price segment (EUR 500 to 20,000) through development of hand-held 3D scanners.The typical application fields of these 3D scanners are limited mostly to close range, i.e. for measuring tasks with distances under one meter up to some few meters.Due to the current technological variety within the area of 3D scanning it is a challenge to select a suitable scanning system for a specific application.Tallig et al. (2015) present a concept and a verification method for selection of 3D scanning technology with a versatile catalogue of criteria.
Hand-held 3D scanners are an optimal supplement to terrestrial laser scanning.However, due to their favourable price and their simple handling these hand-held scanners also potentially represent significant competition to the expensive and precise structured light projection systems (also known as fringe projection).Therefore the question arises, how accurate are these 3D scanners compared to classical structured light systems (e.g. from the manufacturer GOM, Steinbichler, AICON and others) and what metric quality can the user expect for the acquired 3D data.In this area, some results are already available in the literature, e.g. for systems from the gaming industry as well as so-called low-cost systems (structured light system David SLS-1 and Kinect v1/ReconstructMe) for the 3D reconstruction of small objects (Hieronymus et al. 2011, Omelanowsky et al. 2013, and Boehm 2014).These investigations demonstrate, as expected, that the stability and the metric quality of these systems cannot at present compete with high-end systems.
In the following contribution geometrical accuracy tests using different hand-held 3D scanners will be presented.For these Figure 1.Reference bodies for the investigations of the handheld scanning systems -Testy (left), cross-shaped body HSBO with spheres (centre) und granite slab (right)

REFERENCE BODIES
For the benchmarking test the following reference objects were used (Fig. 1): four so-called "Testys "(height of 380 mm) from the Institute for Computer Science of the Humboldt University in Berlin (Reulke & Misgaiski 2012) and geometrically-stable reference bodies from the Bochum University of Applied Sciences: a cross-shaped body with steel spheres (max.distance 450mm of five spheres with a diameter of 65mm) and a planar granite slab (size 300×300mm²).As a reference object for larger surfaces the wall of the 3D test field of the HCU Hamburg was used.This had been scanned with a terrestrial laser scanner, IMAGER 5010 from Zoller + Fröhlich, with resolution high (6mm@10m) using three scan stations.

TESTED HAND-HELD SCANNING SYSTEMS
The following hand-held scanning systems (Fig. 2), with selected technical data summarized in Tab. 1, were available for the tests: three DotProduct DPI-7 (State Office of Criminal Investigations Hamburg (LKA), Bochum University of Applied Sciences (HSBO) and Dr. Hesse and Partner Engineers (dhp:i)), Artec Spider (LKA), Mantis vision F5 Short rank (MexConsult), Kinect v1 and v2 (HCU Hamburg), Structure Sensor (Humboldt University in Berlin) and Google Tango (dhp:i).In addition all objects were acquired with a digital SLR camera Nikon D800 and the camera of a smartphone Galaxy Note 1.

DotProduct DPI-7
A substantial component of the hardware of the DPI-7 scanner (DotProduct, USA) is a PrimeSense sensor, as is also mounted in the Kinect v1 (NIR projector as well as NIR and RGB cameras).
After a cold boot the system needs approx.20 minutes preheating time.The controlling of the sensor is made by a connected android Tablet using the software Phi.3D.For the registration of the point clouds the sensor data of the internal accelerometers and gyroscopes of the Tablet is used.If sufficient overlap is available for the scans (control via visual color information at the Tablet), an ICP algorithm (Besl & McKay 1992) performs a preregistration of scans.After scanning, the registration will be optimized by also eliminating incorrect points (e.g.mixed pixel).
The measuring range of the DPI-7 scanner is between 0.6m and 3.3m, whereby a short range version with up to 1.2m range (system of dhp:i) and a long range version with up to 3.3m (systems of the LKA and the Bochum University of Applied Sciences) is available.The instrument has the dimensions 20×24×6cm 3 .Investigations of the DPI-7 are presented by Jahraus et al. (2015), applications by Ahern & Spring (2015).

Artec Spider
Artec Spider (Artec 3D, Luxembourg) is a hand-held 3D scanner, which was developed particularly for CAD users, to scan small items with complex surface structure, sharp edges and thin ribs with 7.5 photos or with 1 million points per second.The system needs a preheating time of approx.30 minutes and works with a linear field of view between 90×70mm² and 180×140mm².The measuring range is between 0.17-0.35m.The Artec Spider uses structured light technology (speckle pattern) with blue LED as a light source and a color camera with 1.3 megapixels (24-bit radiometry) for the texture mapping of the objects.In combination with the Artec studio software the system is (according to the manufacturer) a desktop tool for designers, engineers and inventors of each art.Sample applications of this system are published by Adams et al. (2015), Friedman et al. (2015) as well as Inzerillo et al. (2015).

Mantis Vision F5 Short Range
The Mantis Vision F5 (Israel) is a structured-light hand-held scanner with a measuring range of 0.5m -4.5m (MVC F5) respectively 0.3m -0.8m (MV F5 Short Range).The sensor hardware consists of two modules: a video camera and a projector, which is integrated in a grab handle.The projector emits infrared light on the object (proprietary pattern), which is captured as coded light by a video camera.The triangulation algorithm calculates a point cloud with 500,000 points/sec.The point density in XY is 1.6mm@0.5mdistance for each image.SL -speckle pattern 0.60 -1.20 2 (@ 1m) < 1.00 5,000 DPI-7 (LKA/HSBO) SL -speckle pattern 0.60 -3.30 2 (@ 1m) < 1.00 5,000 The depth of field of the sensor-system is about 0.3 -0.8m.Because of the low sensitivity, compared to the ambient light, the system is usable both in darkness and in daylight.Wrona (2014) and Zhang et al. (2015) describe diverse applications of the scanner.

Structure Sensor
The Structure Sensor had been developed by the company Occipital (USA).The sensor has to be adapted to an iPad or iPhone6 (Structure App), which communicate via WLAN with a PC (Mac OS X or Windows) with a CUDA graphical board.The data processing is done using the PC software Skanect Pro.
Alternatively the processing can be done later, when using a PC with a graphical board which is not supported by the software (requires increased expenditure of time).Skanect Pro offers functions such as reduction of point cloud or elimination of artefacts.The sensor itself consists of a NIR laser-projector and a NIR camera (resolution 640x480 @ 30fps respectively 320x240 @ 60fps).Additionally the RGB camera and the attitude sensors of the iPad are used.The latter allows a tracing of the track during the scanning.Hence it is possible to generate textured and oriented 3D scans.The measuring range is between 0.4m-3.5m,whereat the depth accuracy reduces with rising distances: 0.15% (0.5mm) @ 40cm, 0.3% @ 100cm and 1.1% (35mm) @ 350cm.All data are transferred to the Skanect Pro application in real-time, where they are processed and stored.

Kinect v1 und Kinect v2
The Kinect v1 had been developed by Microsoft together with PrimeSense as a hardware which controls the video gaming console Xbox 360.Quasi it had been the first low-cost 3D sensor in the market.The functional principle is described in detail by Wujanz et al. (2011).Accuracy investigations are documented in Wujanz et al. (2011), Khoshelham (2011), Chow et al. (2012), Smisek et al. (2013) as well as Gonzalez-Jorge et al. (2013).
The Kinect v2 is composed of a HD wide angle camera (FOV 70×60 degrees) with a resolution of 1920×1080 pixels @ 30 fps and an IR sensor with 512×424 pixels.Three IR projectors actively illuminate the scene to be observed.The RGB camera captures the colour information, while the IR camera simultaneously acquires depth maps and also IR data in the realtime.Lachat et al. (2015) present geometrical accuracy investigations of the Kinect v2.

Google's Project Tango
Google's Project Tango combines 3D motion tracking with depth measurements, to inform the mobile device about its own position and movement in space.It is a tablet, which operates as a real-time multi-sensor system for navigation and orientation in space.The tablet uses an RGB-D camera with a fisheye lens (170° field of view) and a depth sensor to register dense point clouds.Additional integrated sensors are: an RGB-IR camera with 4 MPixel, a front camera with 1 MPixel and fix-focus, accelerometers, a barometer, a compass, GPS and gyros.Furthermore an IR projector (from Mantis Vision) and a flashlight are included.Based on this hard-and software the 3D motion is tracked via the motion-tracking camera, position and orientation are calculated, and a map of the surrounding area is created with 0.25 million 3D points per second.Loianno et al. (2015) show that the Google Tango can be used in mobile applications (as a sensor on a UAV).An indoor application is presented by Winterhalter et al. (2015) and Klingensmith et al. (2015).

Cameras -Nikon D800 und Smartphone Galaxy Note 1
Parallel to the hand-guided 3D scanners photos were provided with a Nikon D800 as well as with a smartphone Galaxy Note 1.The image blocks were processed with the SfM-software Agisoft PhotoScan (Rel.1.2.3.).The Nikon D800 is an actual DSLR camera with a sensor size of 35.9×24.0mm 2 (CMOS) and 36.3 million pixels.The smartphone camera has a sensor which is clearly smaller, probably 5.7×4.3mm 2 (8 MPixel), and a prime lens with 3.97mm.Examples of SfM image blocks of similar objects were already published by (Kersten & Lindstaedt 2012, Omelanowsky et al. 2013).

Reference systems -ATOS I 2M und smartSCAN 3D
The GOM (Company for Optical measuring technology) ATOS (Advanced Topometric Sensor) I 2M, Braunschweig (Germany) is a structured light projection system (Gray code) consisting of two CCD cameras with a resolution of 1624×1236 pixel and a structured light projector.Depending on the lenses used the field of view varies between 500×400mm 2 and 250×200mm 2 .The ATOS I 2M had been employed as a measuring and reference system in different applications (Kersten et

DATA ACQUISITION
All measurements took place on the 5th and 6th of January 2016 in the Geomatics lab at the HafenCity University Hamburg.At various stations data of the reference bodies had been derived from handheld 3D scanners, cameras and the two reference systems.For the cross-shaped HSBO test body a coating spray was used to convert the shiny surface into a matt and bright surface.
For data acquisition all scanners have to be passed manually, in a slow and uniform movement, around the whole object.The collected data is transferred to the connected tablet (Structure Sensor, Google Tango, DPI-7) or computer (Kinect, Artec Spider) in real time and displayed in the software or app.Fast or abrupt movements should be avoided because they will lead to signal loss.This can be handled different in the software, e.g. the DPI-7 has to be passed again over a collected part of the object for re-orientation.The Mantis Vision F5 control unit shows only the actual video image and not the captured data, so a possible loss of signal can be repaired by registration of segments in post processing.The experience of the operator influences significantly the speed and quality of data collection.The manual movements of the sensor have to be coordinated with the liveview on the display.
Acquisition time for all scanners is around a few minutes depending on the object complexity; two of the systems need a preheating time first (Artec Spider and DPI-7).During data collection it turned out that the Google Tango was not suitable for most of the test objects because the resulting pixel size (in object space) of about 3cm was not sufficient.Thus it was only used for the 3D test field.
The Nikon D800 and the Smartphone Galaxy Note 1 were used to take a set of images for each object around it at different heights.The D800 was used on a tripod because of the high exposure values (settings: 85mm focal length for all test objects, 3D test field 24mm, ISO 200, f-number 22).The image acquisition took between 20 and 45 minutes for sets of 40 to 80 images.The Galaxy sets consist of 28 to 68 images; their recording took only three to six minutes without a tripod.Calibrated scale bars were placed near the object for scaling in post processing.
For the measurements with the structured light projection systems ATOS I 2M and smartSCAN the objects were placed on turntables.The systems were calibrated before the recording started.The data collection is done in a number of single scans, which are registered to each other by using small targets stuck onto the object (ATOS) or matching aids (small, geometric, explicit structured objects), which are placed in object space (smartSCAN).For the ATOS system the number of scans per object lay between 24 and 120 with the registration accuracy differing between 0.023mm and 0.041mm.

EVALUATION AND RESULTS
To evaluate the data from the diverse measurement systems multiple formats had to be processed.Some systems delivered point clouds (DPI-7), some others already generated 3D models on the fly using triangulated meshes (Google Tango, Kinect v1 and v2) or subsequently using the system software (Structure Sensor, Mantis F5, Artec Spider, ATOS, smartSCAN).The cameras provided photos (image blocks) as a basis for the dense matching of point clouds and meshes (Agisoft PhotoScan).
Additional observations (three calibrated scales) were integrated in the bundle block adjustment, which resulted in accuracies of 0.1 -0.6mm.
The reference bodies Testy had been measured in detail and with high precision with the fringe projection systems and afterwards the modelling had been done with Geomagic Studio 2012.The ATOS system generated the reference data sets for Testy 1, 2 and 3, while Testy 4 was measured with the smartSCAN.The guideline VDI/VDE 2634, part 2 and part 3, is an accredited standard for acceptance tests (verifying the specified accuracy) and reverification (to ensure long-term compliance) of optical measurement systems based on area scanning (VDI/VDE 2002, 2006).Using the framework of well-defined test scenarios suitable test objects (artefacts) are employed to determine quality parameters.
Following the guidelines, tests were executed using the crossshaped body HSBO with spheres and the granite slab.The derivable quality parameters are:  The quality parameter probing error PS (size) arises from the difference between the measured diameter and the diameter of the calibrated sphere.


The quality parameter probing error PF (shape) is the range of the radial distance between the measured points and a best-fit sphere.The best-fit sphere is determined according to the least-squares method with free radius.


The sphere-spacing error SD is determined from the difference between the measured and calibrated values of the distance between the centers of two spheres.The measured distance is derived from the measured values obtained from multiple area-based probings.The limit, SD, for the permissible three-dimensional sphere-spacing error is the quality parameter sphere-spacing error.It is determined as a length-independent quantity and shall be observed within the entire measuring volume specified.


The quality parameter flatness measurement error, RE, is the range of the signed distances of the measurement point from the best-fit plane calculated according to the leastsquares method.
To evaluate the datasets and calculate the quality parameters Geomagic Studio was used.

Cross-shaped reference body HSBO with spheres
Fig. 3 shows the calculated quality parameters probing error.The characteristic curves of the reference system smartSCAN refer to a comparative measurement with a laser tracker API T3, while the other graphs are referenced to the smartSCAN system.Related to the probing error PS (Fig. 3 left) it is remarkable that some sensors (DPI7-HSBO, Mantis, Galaxy Note 1) point out systematic deviations: measurements are too large or too small, which indicates a scale problem.Compared to the reference system the probing error PS of these systems is around a factor 5-35 larger.The probing error PF (Fig. 3 right) shows the noise behaviour of the sensors.The range of the radial distance between the measured points and a best-fit sphere is particularly large with the camera systems.A reason for this might lie in the Probing Error PF (Shape) Galaxy Note 1 homogeneous grey surface of the spheres, which had been treated with lime spray for the measurements with the scanners.However this surface arrangement is less suitable for the dense matching.
The results for the DPI-7 handheld scanners are homogeneous for all systems and oscillate themselves in the case of the probing error PF from approx.10mm.With the Mantis scanner this value is below 2mm on the average.The Structure Sensor shows similar amazingly good noise behaviour; here with only 4 mm.
The sphere-spacing errors (Fig. 4) show systematic positive or negative deviations for almost all sensors.These effects are particularly pronounced with the Mantis as well as with the Structure Sensor (on the average approx.1% of the distance).The afore-mentioned scale error is to be assumed as the main reason for this behaviour.

Reference body granite slab
The charts of Fig. 5 show the results of the flatness measurement error RE.A dependence between the arrangement of the object surface (granite slab with a coincidental pattern, consisting of bright and dark areas) and the measurement principle of the particular sensor is also visible here.It is recognizable from the data of the structured light projectors (smartSCAN and ATOS), that both systems are able to measure the surface with a similar quality, although the number of acquired points differs significantly.It might be assumed that one reason lies in the different principles of the scanners' projector units (a current LED lighting with the smartSCAN respectively halogen light with the ATOS) while another reason might be the different resolutions of the cameras.
The granite slabs measured by photo triangulation and dense matching show a flatness measurement error which is comparable to the results of the structured-light projectors.The natural texture of the granite slab here surely meets the requirements of the dense matching.Systems with active projection are disadvantaged in this case.Here, in particular, the Structure Sensor supplies comparatively little measuring data (approx.15000 points), which includes extreme skew (Fig. 5 right diagram).An interesting effect also can be seen with the DPI-7 scanners: obviously there is no direct connection between the number of points in the cloud and the surface quality.

Reference body Testy
The results of the 3D comparisons are summarized in Table 2 and illustrated in Fig. 6.The average deviation shows that the two structured light projection systems have very small deviations, each of 4 and 12m, and that even the span, which is calculated from the difference between the average negative and positive deviations, of 30 to 50µm is very low.Thus, the good quality of the structured light projection systems is confirmed as a reference system with superior accuracy.The best quality, proved by the slight deviation of up to 70m and by visual inspection (see Fig. 6), was achieved with the models that were derived from image matching of the D800 data.No other 3D scanner could achieve these accuracies.Furthermore, no system could completely capture the Testy due to the complex geometry and some systems showed obvious systematic scale differences.

Reference wall of the 3D test field
The temporary 3D test field in the Geomatics lab includes about 40 black and white targets (DIN A4 paper) on a wall and two pillars in front of it.The dimension of the test field of about 3×3×2m 3 fits well to the measuring range of some of the test systems.For the comparison the planar surfaces of the wall, the floor and the pillars were used.The 3D test field had been scanned as a reference using the terrestrial laser scanner IMAGER 5010 from three scan stations.The registration of the three scans was obtained with a standard deviation of better than 2mm.Here only the sensors which promise good results due to their characteristics even for larger objects were compared; namely the Kinect v2, DPI-7, Google Tango as well as the Nikon D800.A registration of the test data sets to the reference was carried out using the ICP algorithm.However, for the data set from the Nikon D800 registration was achieved using the existing test field targets (coordinates of control points).The results of the 3D comparisons are represented in Fig. 7.The smallest deviation shows the data sets from the Nikon D800.Slight larger deviations appeared only in the close range area at a column as well as rough outliers at the transition from ground to wall surface, which were manually filtered.For this object the Kinect v2 shows a relative good result.Even the wall surface is very well represented by smoothing during scanning.Due to the flat angle of incidence larger deviations are visible on parts of the socket as well as on the ground floor.With the DPI-7 data a problem in the registration of single scans occurs as concatenated registration with stretched objects.The wall surface is represented as a cylinder surface so that the surfaces at the boundary deviate clearly from the reference surface.However, the potential that the object scanning did not occur sufficiently within the boundary regions cannot be excluded.For the comparison with Google Tango only a small section of the test field was scanned.This relative sparse data set shows large outliers at points along the ground floor as well as between the surfaces of the columns.

CONCLUSION AND OUTLOOK
In this contribution the results of the comparative geometrical accuracy tests for different hand-held 3D scanners were presented.The tests demonstrated that the evaluated scanning systems currently do not reach the accuracies and the quality of the reference data produced by high-end structured light systems.However, it should be noted, that not all of the selected reference bodies corresponded optimally to the typical range of applications of the tested 3D scanners.In principle, the handling of these systems is simple.However, the scanning by slow, homogeneous movements -around and over the object to be recorded -requires appropriate user experience for keeping a constant signal and permanent matching between the scanned point clouds.The acquisition speed of a few minutes for each object is quite high for all presented systems.Following the guideline of VDI/VDE 2634, part 2 and 3, the determined quality parameters (probing error and sphere-spacing error) proved the fact that the instrument scale was not precisely determined for some hand-held scanners and/or that the sensor is possibly not stable due to a mechanical unstable structure.Procedures for the field check and/or simple self-calibration achievable by any user are both meaningful and necessary.The results of the flatness measurement error tests document that the image-based acquisition procedures with cameras have very small deviations compared to the structured light systems, while the systems with active projection show a factor of 8-50 higher deviations than the reference system.Therefore, the signal to noise ratio of the active scanning systems needs significant improvements.
The two systems Mantis F5 and Artec Spider settled in the middle price segment could not satisfy the accuracy specifications of their manufacturers in the investigations using the reference bodies Testy.However, beside the pure accuracy values (average deviation and span), the visual quality and the completeness of the scanned test objects must also be considered as a criterion for the evaluation of the entire quality of an examined system.The visual quality of the models was better with the Mantis F5 than with the other hand-held scanners.Using the data of the DPI-7 and the Artec Spider no satisfying models of the reference bodies could be generated due to the many holes in the data.
The results of the Nikon D800 were the best of all systems; i.e. they show that this camera, in combination with appropriate evaluation software, represents a precise and inexpensive alternative with respect to the hand-held scanners.The quality of generated models from the data from the Galaxy Note 1 was also not useful due to the large holes.However, it remains to be seen how the new generation of smartphone cameras will perform in future tests.It was also shown that the quality of a model generated with a specific system depends very strongly on the experience of the operator.Further available, but not in this document evaluated, data from the Faro Freestyle is ready for extended analyses.Future investigations will be carried out in the context of alternative test scenarios, e.g. with larger reference bodies.Using those reference bodies and a test field a comparison with laser scanner measurements seems to also be meaningful, as generally the hand-held systems will be able to fill a gap between high precision structured light systems and terrestrial laser scanners.

ACKNOWLEDGEMENTS
The authors acknowledge the deployment of the 3D scanning systems by the State Office of Criminal Investigations Hamburg, Dr. Hesse and Partner Engineers (Hamburg), and MexConsult (Bredstedt).In particular, the energetic support during the data acquisition by the students of the master study program Geomatics of HCU Hamburg, the students of the Bochum University of Applied Sciences and the staff of the State Office of Criminal Investigations Hamburg is gratefully acknowledged.