BENCHMARKING THE OPTICAL RESOLVING POWER OF UAV BASED CAMERA SYSTEMS

: UAV based imaging and 3D object point generation is an established technology. Some of the UAV users try to address (very) high-accuracy applications, i.e. inspection or monitoring scenarios. In order to guarantee such level of detail and accuracy high resolving imaging systems are mandatory. Furthermore, image quality considerably impacts photogrammetric processing, as the tie point transfer, mandatory for forming the block geometry, fully relies on the radiometric quality of images. Thus, empirical testing of radiometric camera performance is an important issue, in addition to standard (geometric) calibration, which normally is covered primarily. Within this paper the resolving power of ten different camera / lens installations has been investigated. Selected systems represent different camera classes, like DSLRs, system cameras, larger format cameras and proprietary systems. As the systems have been tested in well-controlled laboratory conditions and objective quality measures have been derived, individual performance can be compared directly, thus representing a ﬁrst benchmark on radiometric performance of UAV cameras. The results have shown, that not only the selection of appropriate lens and camera body has an impact, in addition the image pre-processing, i.e. the use of a speciﬁc debayering method, signiﬁcantly inﬂuences the ﬁnal resolving power.


INTRODUCTION
During the past years aerial image acquisition by unmanned aerial vehicles (UAV) became a well-established method for photogrammetric 3D object point reconstruction.It is of special advantage for large-scale applications of smaller areas with high requests on flexibility and recent mapping.Furthermore the UAV technology is increasingly used in classic engineering survey tasks and often replaces the point-based object acquisition with tachymeters.New challenging geodetic monitoring and inspection issues demand for high image resolution in the sense of smaller ground pixel.To meet this challenge camera systems have to deliver much higher resolution requirements (even subcm resolution) compared to traditional airborne mapping.
Hence photogrammetric workflows become more and more relevant to challenging engineering geodetic scenarios, especially in terms of accessible accuracy.Besides correct image geometry and processing chains the choice of camera system itself is of major importance.The camera is the primary sensor as it delivers the observations of which later products (3D object points, 3D point clouds) are being processed.The quality of the measurements directly affects the quality of the products.The camera geometry and their calibration is one of the aspects to be considered (Cramer et al., 2017b).The (automatic) detection of tie points in order to form the photogrammetric block is another most important step in the process flow.Performance of image matching will influence the later geometrical accuracy.The matching mainly relies on the grey value information of distinct points / areas in images.Low image noise will support the matching process.Thus, radiometric performance is essential for the matching and will also be of influence for geometric performance of the block.Radiometry should not be neglected as it determines the * Corresponding author quality of imaging systems and their (effective) resolving power.
The cameras used in UAV context often are off-the-shelf solutions with main focus on total weight and the potential to integrate these cameras into the overall system.Most often photogrammetric aspects (i.e.stable geometry) did not have major priority.These currently available cameras could be divided into several groups depending on their specifications / system design (e.g.compact cameras, system-or bridge-cameras and systems specifically designed for UAV purposes).Typical representatives out of these groups have been investigated empirically in terms of their resolving power.The results are given in this paper, which can be seen as benchmark for current UAV based cameras.It is recommended to read this publication in conjunction with the work of Cramer et al. (2017a) -to be submitted to this UAV-g conference -where geometrical calibration and stability of these cameras is discussed and empirically investigated.Some of the cameras investigated from radiometric point of view here also were geometrically analysed by Cramer et al. (2017a).
The paper is structured like follows: the basic radiometric characteristics are discussed in section 2. Following this, a procedure is described in section 3 that allows the determination of the (effective) resolving power of image acquisition systems in general.Section 4 covers several methods used to reconstruct colour information of (raw) image data, in order to overcome the Bayerpattern.Section 5 introduces the experiment / benchmark that allows the comparison of different kinds of cameras which may be suited for UAV applications or for aerial-imaging applications in general.

RADIOMETRIC CHARACTERISTICS
Radiometric quality of a sensor system is causally determined by multiple factors.Light rays which are being reflected by an object and detected by a camera sensor partially traverse the atmosphere and lose some of their energy due to diffusion and absorption.In UAV applications this part could be considered very small and won't be discussed further here.Next the light passes a (complex) lens system where an aperture is integrated and limits the effective solid angles for every ray.As a consequence the aperture directly affects the amount of light which in turn determines the amount of photons that reach the sensor plane and contribute to the imaging process.The smaller the aperture is chosen the more the diffraction of light limits a sharp optical imaging.On the other hand, if the aperture is chosen too large spherical and chromatic aberrations gain influence.The amount of photons passing through the lens system and reaching the sensor at a distinct time frame directly influences the exposure time needed to create an equivalent sensor signal.In aerial photogrammetry the exposure time however affects a sharp optical imaging in terms of motion blur that is a result of the systems change of location / movement whilst the sensor is exposed.This change of location can be compensated actively and several aerial camera systems offer some techniques.But nearly all of the systems for UAV retain as additional parts increase the total weight limiting flight endurance and operation time.Still, situation is changing, when looking on the video recording.Here quite sophisticated stabilized mounts are available to minimize blur in images.The influence of image blur, comparing imaging in static (laboratory) and dynamic (operating) conditions has been shown in Kraft et al. (2016b).In order to guarantee repeatability of the benchmark approach only static (laboratory) conditions are part of this investigation.Another interfering aspect is the gain of shading (or inverse the luminous intensity decrease) starting from the principle point to image corners.This effect is often described as vignetting and is caused by the lens system itself and by the integrated aperture.The vignetting can be measured and corrected as an image processing step whilst determine the Photo Response Non-Uniformity (PRNU) (Kraft et al., 2016a).After the light rays passed the lens system they hit the sensor surface.That is the part of the camera system that creates a digital interpretable signal directly depending on the amount of collected photons during the exposure time window.The quality of that signal is affected by several electronic components (e.g.sensor read-out electronic, analog-digital converter).A measure of this quality is the signal noise ratio (SNR).The SNR also is characterized by a) the ambient noise level that unavoidably occurs when a semi-conductor is connected to its supply voltage and b) to the photo-effective area of each sensor element (pixel).The larger the effective area the more photons contribute to the signal assuming identical time frames and therefore increase the signal.The electronic ambient noise can be determined pixel by pixel as part of the Dark Signal Non-Uniformity (DSNU) (Kraft et al., 2016a).

DETERMINATION OF RESOLVING POWER
Sharpness as an image property is characterized by the modulation transfer function (MTF) which is the spatial frequency response of an imaging system to a given illumination."High spatial frequencies correspond to fine image detail.The more extended the response, the finer the detail -the sharper the image."(Mix, 2005).The effective image resolution or resolving power of an imaging device can be determined in different ways.A classic approach is the use of defined test charts (e.g.USAF resolution test chart with groups of bars).There, the (subjectively) identified image resolution corresponds to that distance where the smallest group is still discriminable.This is very similar to the Rayleigh criterion (Born and Wolf, 1999) that defines the mini-Figure 1. Designated test pattern Siemens star (left), radial modulation analysis for one circle (right) (Reulke et al., 2004).mum distance between two point sources in order to be resolved by an imaging system.To reduce the subjective influence with bar charts during the determination process some approaches use signal processing techniques to calculate the effective image resolution.The method described by Reulke et al. (2004Reulke et al. ( , 2006) ) is one of the latter approaches.There, the modulation transfer function (MTF) and subsequently the point spread function (PSF) are calculated for images with a designated test pattern (e.g.Siemens star).According to the above mentioned approaches the smallest recognizable detail or "the resolution limit is reached if the distance between two points leads to a certain contrast in image intensity between the two maxima."Using a priori knowledge of the original scene (well-known Siemens star target) the MTF and PSF are approximated by a Gaussian shape function. (1) where MTF (eq. 1) is the modulus of the OTF (optical transfer function) which is equivalent to the PSF in frequency domain (Reulke et al., 2006) and H is the space invariant 2D-PSF (eq.2), see Figure 1.There are several criteria for the resolving power of cameras.The parameter σ (standard deviation) of the PSF (a Gaussian shape) is one criterion.It directly relates to the image space and can be seen as objective measure to compare different camera performances.Another criterion is the spatial frequency where the MTF reaches a certain (minimal-) value (i.e.10%, MTF10).The reciprocal of this frequency is the approximation for the number of the smallest line per pixel.The width of PSF at half the height of the maximum is another criterion (full width half maximum -FWHM).

DEMOSAICING METHODS
The predominant majority of colour cameras use micro filter arrays (one filter evaporated onto every single pixel, so-called Bayer-pattern) to capture colour information.The array usually is arranged with alternating colour filters (e.g.green -red) for a single sensor line and then a complementary arrangement in the following sensor line (e.g.blue -green), etc.During the colour reconstruction process the missing information is determined by interpolating between the neighbourhood values.Hence, demosaicing or debayering is another factor influencing image resolution.There is a huge variety of demosaicing algorithms out of we chose several methods (Figure 2) that are widespread and assumingly popular.

Bilinear interpolation
The simplest way to restore the missing information is to interpolate each channel separately using neighbouring values.Bilinear interpolation is the most commonly used mode, but it would be possible to use nearest neighbour or bicubic interpolation instead.This method is efficient and straight forward to implement, but images will exhibit colour fringing at edges.

Variable Number of Gradients (VNG)
VNG reduces colour fringing by using edge detection (Chang et al., 1999) The gradient NE (one of eight) at position g7 is calculated by the following equation: A threshold is used to determine if the pixel lies on a smooth area and averaging can be used to approximate the missing values, or if the pixel lies on a steep gradient, where it is better to use one of two neighbouring values.The maze-like structure on the right is an effect of the algorithm alternating between horizontal and vertical interpolation (Góźdź, 2009).

Adaptive
direction of interpolation is erroneously selected (interpolation along an object boundary is preferable to interpolation across the boundary), are addressed through a nonlinear iterative process.
The image is interpolated twice -once vertically f h and once horizontally fv.The final output f is calculated by combing f h and fv based on a homogeneity matrix H f which aims to minimize colour artefacts.
AHD can create visually pleasing images, but there are cases, where it gets confused between vertically or horizontally repeating patterns close to the Nyquist frequency.(see Figure 4).

MHC (Malvar, He, Cutler)
MHC is a simpler algorithm than VNG or AHD, it has higher performance than such nonlinear algorithms and doesn't suffer from artefacts due to (sometimes wrong) assumptions about gradients in the image (Malvar et al., 2004).It works linearly in a 5x5 neighbourhood by first filling in values using bilinear interpolation.It tries to analyse local luminance changes by comparing the actual value at the current pixel position to the value arrived at by interpolating same-colour neighbours.It then factors a corresponding gain term when calculating the other two colour values at the same position.

DCB by Jacek Góźdź
Finally, the iterative DCB algorithm is included.It is largely undocumented, but an open source implementation which performs well is available.(Góźdź, 2009).

Adobe Camera Raw (as shot)
Adobe Camera Raw is a toolbox in terms of raw image converter and supports a huge variety of image file formats and more important many camera systems.Although it is a black box implementation it yields good results in past investigations but is largely undocumented.However Schewe and Fraser (2010) describe the method according to the selection parameter 'As Shot' as follows: "When you select As Shot you allow Camera Raw to attempt to decode the white balance data stored when your camera captured the image.Camera Raw may not exactly match the camera's numbers or its rendering of that white balance, but it does a pretty good job of accessing most cameras' white balance metadata."

Adobe Camera Raw (automatic)
Again, citing Schewe and Fraser (2010) the selection parameter 'Auto control' is described as: "Camera Raw has a single Auto control, which does its work evaluating the statistics of the image's histogram while attempting to produce an optimal distribution of values throughout the image."

Notes
So far, several different demosaicing algorithms have been introduced, varying in complexity and performance.It is easy to see that there is no fit-all solution for all scenarios.The method of choice will depend on processing power, whether results should be visually pleasing or geometrically correct, or other factors such as camera lens design since the resulting PSF affects the correlation between channels and thus how much information about one channel can be gleaned by analysing another.A further observation is that poorly calibrated white balance can lead to negative visual artefacts with the edge detecting algorithms for the same reason.
The influence of different demosaicing methods regarding resolving power of a camera system suited for UAV applications has been pointed out in (Kraft et al., 2016a).Figure 5 shows the parameter σ of the PSF (see section 3) for a Ricoh GXR MountA12 with Zeiss Biogon 21/2.8 lens as a representative of the bridge camera family.The images have been acquired under laboratory conditions.It can be seen that the values for σ of PSF change due to the use of different demosaicing methods.While the bilinear approach performs worse in other cases the differences are rather small.Adobe's Camera Raw (closed source) automatic technique performs best.
During the evaluation process the DCB method provided promising and nearly top-rated results.Additionally and in contrast to Adobes Photoshop Camera Raw Suite it is an open source implementation instead of a black box implementation.Therefore we used this demosaicing approach during the experiment / benchmark (see below).

EMPIRICAL UAV CAMERA RESOLUTION TESTS
Having introduced the underlying theory of resolution testing and demosaicing different camera systems will be empirically tested.By this time there is a growing variety of camera systems that either claim to be suited for UAV applications or could be considered suitable because of their specifications (e.g. in terms of sensor size, trigger event control, overall weight, interface accessibility and acquisition costs).These systems are often being grouped as a) systems specifically designed for UAV purposes b) large format cameras, c) system (or bridge-) cameras and d) single-lens reflex cameras.The camera systems compared in this paper are given in Table 1 where Sigma's DP1 is listed in the category Other.The decision was made due to the fact that this camera has a Foveon-sensor (Hubel et al., 2004) where each pixel element detects colour information instead interpolating neighbourhood pixel of a Bayer-pattern arrangement.

Benchmark layout
The experiment that could (arguably) be appropriate for a benchmark is motivated and arranged as follows.
Aim is to determine and compare spatial resolution for different sensor-lens combinations.In order to guarantee repeatable measurements with identical controlled light conditions and to prevent extraneous light a sufficiently large basement hall has been identified (see Figure 6 top).For every camera system the related distance to the designated test pattern (see Figure 6 bottom) has been calculated to ensure identical nominal ground sampling distance (GSD -according to focal length and pixel size on sensor).The GSD in this benchmark has been set to 5mm to address the aforementioned fields of application (e.g.inspection / monitoring) including their resolution requirements.Usually, resolving power is changing across the field of view.In order to analyse this effect multiple images have been taken to have the resolution target imaged at different locations in image space (e.g.image center -image half field -image corner).All images have been converted from their raw format to usable tiff format using the same demosaicing algorithm.In this case we chose the DCB method (see Sub-section 4.5) with its open source (C++) implementation of <libraw> and the primordially implementation for DLRs MACS Micro versions <mipps>.For every converted image the PSF and corresponding σ (see section 3) has been calculated.To depict the optics resolving power along the sensor diagonal the ambient noise level (DSNU) and the vignetting effects (PRNU) have not been corrected (see section 2).This should guarantee the genuine system response from object space to sensor.The results are presented in Figure 7.

Discussion
The (effective) resolving power of a specific sensor-optic (camera-lens) combination decreases as expected from imagecenter via image-half-field to the image-corner.This deteriora- Imaging performance of all DJI systems is fairly homogenous and the variation in resolving power in comparison to the aforementioned sensor-optic combination is quite low.The MACS Micro prototype system (especially the 16 megapixel version) shows top-rated results which (possibly) indicates a connection between pixel size (photon effective area) and resolving power since this sensors pixel pitch is the largest (7.4 µm) compared to other systems with Bayer-pattern.The results for resolving power of Sigma's DP1 support the assumption since its pixel pitch is the most largest (7.8 µm).Furthermore the Foveon-chip outperforms all other systems.This could be due to the Bayer-pattern and necessary demosaicing.
Ricoh's GXR offers a consistent performance and is close to the MACS Micro (12MP) and Phase One's IXU 1000.Especially the image center resolves close to top-rated value.
The Phase One IXU 1000 with its large sensor format and 100 megapixels shows very low variation in resolving power over the complete image space although it is slightly behind in terms of overall resolution power compared to MACS Micro.Canon's EOS 5DS R as a representative of the DSLR group is equal to PhaseOne's IXU but its resolving power diminishes perceptively along the image diagonal.
As it can be seen the blue channel occasionally is determined significantly worse compared to green channel and especially red channel.The Bayer-pattern arrangement consists of twice the number of pixel for green compared to red respectively blue.Therefore one would expect slightly better results for the green channel but almost equal results for red and blue.This issue is not finally solved and will be investigated in future work considering the presumptions if this problem is either caused by chromatic aberrations and / or colour temperature of the used light source.
Hereafter the results given in this paper are connected to the work of Cramer et al. (2017a).As part of their investigation different UAV camera systems have been calibrated with airborne image data in a calibration field.
Additionally, the influence of different image block configurations on the quality of in-situ calibration has been derived for DJI's Inspire and DJI's Phantom 3. The obtained quality measures are derived from 22 control points and 23 check points, respectively.Only some of the tested block configurations are cited here (see Table 2).
There, the total (mean) error considering all block configurations of the Phantom system (3,37cm) is slightly but significantly better than the Inspire system (4,27cm).The evaluation of radiometric resolving power given here (see Figure 7) provide similiar results.
The Phantom system outperforms the Inspire system slightly.
Connecting both results one could assume that radiometric resolving power directly influences subsequent workflows (e.g.geometric calibration).

CONCLUSION AND OUTLOOK
The importance of (effective) resolving power as an additional quality measure regarding high resolution requirements for UAV applications has been emphasized and investigated in this publication.A benchmark procedure has been introduced to obtain and evaluate this measure.
A further question is if and how it is possible to determine the impact of radiometrical characteristics on quality of the subsequent photogrammetric workflows (e.g.bundle adjustment, Semi-global matching).It is expected that images corrected from falsified values using DSNU and PRNU are more qualified for feature extraction than uncorrected images.Minimized sensor noise (DSNU) and corrected vignetting effects (PRNU) should provide more consistent features especially in outer image regions.
Another topic is a) to combine the experiment results under static / laboratory conditions with radiometrical correction parameters (PRNU / DSNU) to provide more fundamental information and b) to verify radiometrical and geometrical quality characteristics presented by Cramer et al. (2017a) under dynamic / operating conditions.
Yet another (technological) field of investigation is the influence of backside illuminated sensors compared with frontside illuminated sensor elements.
Homogeneity-Directed demosaicing (AHD) Hirakawa and Parks (2005) identify three different classes of artefacts: misguidance colour artefacts, interpolation colour artefacts and aliasing.They set out to minimize aliasing by using filterbank techniques.Misguidance colour artefacts, which arise when the

Figure 4 .
Figure 4. Artefact of the AHD algorithm.The maze-like structure on the right is an effect of the algorithm alternating between horizontal and vertical interpolation(Góźdź, 2009).

Figure 5 .
Figure 5.Comparison of different demosaicing methods in terms of (effective) image resolution / resolving power (PSF in pixel).

Figure 6 .
Figure 6.Controlled light conditions (top left), no extraneous light (top right).Illustration of the experiment setup (bottom)

Table 1 .
Specifications of the compared camera systems (* SigmaDP1's Foveon-chip with true 4.7Mpx has no Bayer-pattern, therefore no demosaicing necessary) Figure 7. Results of the comparison of PSF in pixel (CE center, HF half field, CO corner)