A CALIBRATION WORKFLOW FOR “PROSUMER” UAV CAMERAS

: High-end consumer quadcopter UAVs or so-called “prosumer devices”, have made inroads into the mapping industry over the past few years, arguably displacing more expensive purpose-built systems. In particular, the DJI Phantom series quadcopters, marketed primarily for videography, have shown considerable promise due to their relatively high-quality cameras. Camera pre-calibration has long been a part of the aerial photogrammetric workflow with calibration certificates being provided by operators for every project flown. Most UAV data, however, is processed today in Structure-from-Motion software where the calibration is generated “on-the-fly” from the same image-set being used for mapping. Often the scenes being mapped and their flight-plans are inappropriate for calibration as they do not have enough variation in altitude to produce a good focal-length solution, and do not have cross-strips to improve the estimation of the principal point. What we propose is a new type of flight-plan that can be run on highly textured scenes of varying height prior to mapping missions that will significantly improve the estimation of the interior orientation parameters and, as a consequence, improve the overall accuracy of projects undertaken with these sorts of UAV systems. We also note that embedded manufacturer camera profiles, which correct for distortion automatically, should be removed prior to all photogrammetric processing, something that is often overlooked as these profiles are not made visible to the end user in most image conversion software, particularly Adobe’s CameraRAW.


INTRODUCTION
As the UAV industry continues to mature, sophisticated quadcopter UAVs with features hitherto reserved for purposebuilt systems are becoming available at low price-points and, as a consequence, to a wider range of potential users.In conjunction with Structure-from-Motion photogrammetry software, the data from these UAVs can produce what appears to be a high-quality mapping product with next to no user intervention.Yet this new generation of enthusiastic users often do not have a background in photogrammetry or aerial photography and are largely ignorant of practices long-established in the aerial mapping industry (Fraser, 2013).Camera calibration, in particular, has long been the foundation of the photogrammetric workflow, but is entirely overlooked as such software performs an autocalibration using the same image set as is being used for mapping (Hashim et al., 2013;Suh, Choi, 2017).As a consequence, poor estimates of the interior orientation parameters drive the error into the external and absolute orientations, thereby diminishing the quality of downstream products like DSMs, DEMs and orthophotos.In many cases the calibration process of a project is not included in the final product (Casella et al., 2014), although this was and still is obligatory for commercial aerial mapping projects.As has been noted in the earlier literature, improper estimation of camera internals leads to the so-called "doming effect" in DEMs created from Structure-from-Motion software that uses camera parameters derived from conventional aerial blocks (Wackrow, Chandler, 2011).While novel curving flight-plans have been proposed to remedy this "doming" issue, we propose to address it simply by robust camera pre-calibration procedures that will allow conventional aerial blocks to be flown.
There have been previous attempts to pre-calibrate UAV cameras using a test-field with control points (Honkavaara et al., 2006;Pérez, Agüera, Carvajal, 2012), but such a method requires extensive and time-consuming control pick-up.Other researchers are calibrating UAV cameras terrestrially indoors, again with a large number of control points (Cramer, Pryzbilla, Zuehorst, 2017).What we propose is a workflow that does not require control points, but instead, uses automated, preprogrammed flights to collect imagery suitable for camera calibration.This method needs only a highly textured scene with good variation in height; the flight plans require a maximum of 30 waypoints and can be flown in the field in about 15 minutes or less.
Terrestrial camera calibration procedures are well documented and have been extensively published (Cramer, Pryzbilla, Zuehorst, 2017;Luhmann, Fraser, Maas, 2016).The inversepyramid configuration of camera stations is frequently used.At each station images are captured at multiple rotations --90°, 180° and 270° --about the x-axis to ensure through a standard procedure that matching points are observed through all parts of the lens, something that significantly improves the accuracy of the principal point (Xp, Yp).Although terrestrial calibration procedures have been used with DJI UAVs, they are performed indoors with a large number of control points, conditions that cannot easily be matched in the field (Cramer, Pryzbilla, Zuehorst, 2017).
A significant source of systematic error in camera calibration of "prosumer" UAVs occurs when geometric distortion models are imposed on in-camera JPEGs or are embedded in RAW files and imposed during the conversion of these files to a usable format for photogrammetry; like JPEG or TIFF.These geometric distortion corrections are generalized for the lens and camera combination and cannot take into account the manufacturing variances that are obtained between different instances of the camera model.As a consequence, if imagery is used that has such a geometric calibration applied, photogrammetry software will almost certainly misestimate, and usually significantly underestimate, the real interior parameters.These geometric corrections are frequently not apparent to many end users who even insist on the use of RAW imagery as the basis of photogrammetric processing.For instance, Adobe's CameraRAW, often used to convert RAW images to JPEG format, will apply the manufacturers geometric correction and give the user no option to do otherwise.Therefore, the foundation of our calibration procedure will be a workflow to recover the original images with no geometric corrections applied.

METHODS AND MATERIALS
The camera used for the study, the FC6310, is that built into the DJI Phantom 4 Pro UAV and held by a brushless gimbal.The photographic parameters are given in Table 1  The significant difference in our procedures from earlier studies is that we insist on using a completely automated, custom flight sequence to replicate the best indoor lab results in the field.

Site Locations and Scene Selection
Many of the image sets captured for this work were taken of excavated archaeological structures at the National Institution Stobi in the Republic of North Macedonia, particularly the socalled "Building with Arches", in December of 2018 and February of 2019.Additional image-sets with control points were taken from the summer 2018 excavation season of a structure adjacent to the so-called "Theodosian Palace".Small image sets were also taken of limestone campus buildings at Queen's University, Canada.When selecting a scene for aerial camera calibration we used the same principles as terrestrial scene section: 1) significant variations in depth in the scene, and 2) extensive texture across the scene that fills the field of view.
The former improves the estimation of the focal length (Remondino, Fraser, 2004); the latter ensures that matching points are distributed evenly across each image so that points can be compared across every part of the lens during the bundle adjustment.The "Building with Arches" at Stobi meets all of these criteria in that its partial walls vary in over 5m in places, and the entire structure is comprised of highly textured stone or brick.

Flight Plan Creation
As earlier studies have noted, image sequences for calibration should follow quite different principles than conventional aerial blocks used for mapping.The sequence should cover only a small area, image overlap should be relatively high, the same area should be seen with several camera rotations, and images should be captured from several different heights (Cramer, Pryzbilla, Zuehorst, 2017).The addition of oblique imagery can also significantly improve the accuracy of the calibration by increasing the base-to-distance ratio of points observed by converging cameras (Haala, Cavegn, 2016).Though these principles are well established in the published literature (Slama, Theurer, Henriksen, 1980;Balletti, Guerra, Tsioukas, Vernier, 2014;Remondino, Fraser, 2004), they have not been implemented in an automated flight plan to date.

Flight Planning Software
Creation of these complex flight plans is not a trivial task.Most photogrammetric/survey flight planning software only allow for the creation of aerial blocks at a constant height.Commercial flight planning software for videography does allow for more freedom and control of the UAV's position and pose but is lacking in the ability to create a flight plan based on photogrammetric parameters such as forward/side image overlap and image footprint.DJIFlightPlanner is a third-party flight planning software specifically for DJI UAVs that is designed with surveying in mind.The basic camera specifications for all DJI UAVs are included in the software (Focal Length, Sensor Size etc.), so as to allow users to simply specify a boundary, flying height, and desired overlap to generate a flight plan.The software also allows the user to trigger an image according to time, or by waypoint, where it will hover the UAV while the photo is taken.The software then generates a simple CSV file with the waypoints and actions to be executed by the Litchi app, available for Android or iOS, which will control the UAV during the mission.This intermediate CSV file allows the user to easily modify the flight plan so that it is relatively simple to rotate the UAV while it is flying to take images at 90° or 270° with respect to the direction of travel, as well as with imagery at multiple heights or at oblique camera angles.

Flight Plans
Five flight plans were created, and each flown separately to create five independent camera calibrations that could be compared.All flight plans take images at three heights above the terrain: 60ft, 75ft and 90ft (DFIFlightPlanner requires imperial values to be entered for flying height).At each waypoint the camera captured three images; one image in landscape and two images in portrait ±90° relative to the direction of flight.
The first flight plan (M1 in Table 2) is a recreation of a typical terrestrial camera calibration model as outlined in 3DM Analyst User Guide.The three image tiers form an inverse pyramid with a single strip at 60ft, two at 75ft, and three strips at 90ft.The strips have a high forward overlap of 80%, and as a result the flight contains 90 images in total.Four additional oblique images were taken at -45° to nadir in the corners of the rectangular flight area at 75ft to improve the overall robustness of the calibration.The next two flight plans (M2, M3 in Table 2) do not use the inverse pyramid configuration and instead use three full aerial blocks at the same three altitudes.In each of the blocks landscape and two portrait images were taken at each waypoint along with four oblique images at the corners.The only difference between these two plans was the overlap: M2 had 60% forward and 40% side overlap, and M3 a forward overlap of 80% and 40% sidelap.The final two flight plans (M4, M5 in Table 2) repeat the same procedure as the previous two missions except with an increased forward and side overlap of 80% and 90% respectively.

Post-Processing and Calibration
This research will use two separate photogrammetric software packages.First, CalibCam (version 2.5.0 build 1776), produced by ADAM Technology of Perth, Australia, is used for evaluating calibration accuracy.CalibCam provides reliable reporting of calibration parameters, a correlation matrix and the accuracy to which individual parameters have been solved.The second package, PhotoScan/Metashape (version 1.5.2), was used as an example of the common Structure-from-Motion approach to calibration where the interior orientation is usually not held fixed before the bundle adjustment.PhotoScan has been used extensively in archaeology for documentation at Stobi.Several flight plans from July of 2018, with 10 control points each, were processed in PhotoScan with pre-calibration according to our method, as well with the standard auto-calibration procedure where the image set being processed were used for the solving of camera internals.The residuals on the control points, from separate least-square calculations, have an accuracy of 3mm.
Adobe CameraRAW (version 11.1) and RawTherapee (version 5.5) were used for image conversion from the DNG ("Digital Negative") files produced by the UAV.While the geometric correction embedded in the metadata could not be deactivated in CameraRAW, RawTherapee allows the user to disable this correction.

RESULTS AND ANALYSIS
We will first demonstrate that the automatically applied geometric correction applied to the images from the Phantom 4 Pro have a dramatic impact on the overall calibration.By using RawTherapee we can avoid these geometric corrections and recover the images as shot, as well as a true solution for the lens distortion of the camera.
The accuracy of the flight plans described above was then established in CalibCam by generation matching points by Normalized Cross-Correlation Least Squares Matching, followed by a bundle adjustment to solve for the interior orientation parameters: Focal Length (C), Radial Distortion (K1, K2, K3), Principal Point Offset (Xp, Yp), Decentering Distortion (P1, P2) and the Pixel Scaling Factors (B1, B2).The sigma expressed in pixels in the bundle adjustment report for each of these parameters was then compared between the flight plans.Generally, we did not compare P1, P2, B1, and B2 as these parameters were almost always solved to high accuracy regardless of the flight plan.Instead, the Focal Length and Principal Point showed the most variation and were reported below, along with the three Radial Distortion parameters.

Image Conversion using Adobe and RawTherapee
When using Adobe CameraRAW for converting DNG images to JPEG for post-processing, DJI embeds a lens profile that forces a geometric correction on the JPEG.RawTherapee has the ability to deactivate the use of this geometric lens correction, maintaining the true original image.Two 18-image terrestrial calibration sets were collected on Queen's Campus using the conventional inverse pyramid calibration structure.Figure 1 shows two sample images that demonstrate the obvious effect of the distortion correction.While the image dimensions in pixels of the two images are the same, one noticed extensive cropping and stretching to compensate for barrel distortion when the geometric correction is applied in Adobe CameraRAW.While this corrected image is more visually appealing, it is inappropriate for photogrammetric processing.When we compare a visualization of the interior orientation correction generated from the image sets (Figure 2) with and without the geometric correction applied this visual difference can be quantified.When comparing the displacement of pixels in the outside of the lens, as well as overall RMS value for the amount that pixels have been moved across the entire sensor (Table 3), the effects the manufacturer's software-based lens correction has on the calibration parameters becomes apparent.Although this new method can prevent geometric corrections from being applied, it introduces the problem of vignetting on the outside corners of the images.Manual vignetting correction was applied in RawTherapee to remove this effect.This sort of vignetting correction should only modify the brightness values of pixels and does not impose any geometric correction.In order to verify that the vignetting correction did not impact any interior orientation parameters, calibration with and without vignetting correction were compared.Figure 3 shows that vignetting correction did not impact the distribution of matching points across the lens/sensor.Table 4 shows that the solutions for Focal Length and Principal Point showed no significant differences whether or not vignetting correction was applied.

Camera Calibration Assessment
Missions M4 and M5, which each consisted of over 200 images, were used to demonstrate that calibration accuracy showed greatly diminishing returns after 90 images as seen in Figure 4.
For both M4 and M5, increasing the number of images taken was not necessary to produce a satisfactory calibration with an accuracy of below 0.3 pixels for all important parameters.Instead, as flight plans M1 to M3 show, the configuration of the images is far more important to calibration accuracy than sheer numbers.They will show that the number of images required in a calibration flight-plan can be as low as 20, thereby saving significant amounts of flying and processing time.M1, which used the terrestrial inverse pyramid calibration structure, showed the importance of adding rotated images for the accurate solution of Focal Length and Principal Point (Table 5).In agreement with earlier published literature, the addition of oblique images improved the estimation of Focal Length by an order of magnitude.While the accuracy of the Principal Point was nearly doubled with the use of rotated images, the addition of oblique images had a negligible impact on these parameters.LPPO 8.850 0.070 0.024 0.020 0.061 0.128 0.084 Table 5. Sigma statistics for M1.L -Landscape, P -Portrait -90°, PP -Portrait ±90°, O -Oblique M2 and M3 consisted of images taken at three altitudes, with multiple image being captured at each waypoint with different rotations with respect to the direction of travel along with four oblique images per mission.These missions had more images than M1, but they were much easier to plan than the inverse pyramid structure.The increased number of images in both missions led to an improvement, usually two-fold, in the accuracy of the radial distortion parameters.The same influence of image rotations and oblique images was also observed on Principal Point and Focal Length estimates.A question remained.Were three flying-heights really necessary for a good interior orientation or could the variation in the height in the scene offer sufficient variation for good focal-length estimates.By reducing the number of heights, we could further reduce the number of images required.Table 7 (below) demonstrates the effect of height on the accuracy of the interior orientation parameters.We tried each height, as well as three combinations of two heights in Table 7.These results show that provided the scene used for calibration has considerable variation in height, there is no impact on Focal Length estimations when flying at only one height.The slight improvements on the other parameters with the use of two heights as opposed to one is likely due only to the increased number of images being used.60 & 75 8.849 0.026 0.011 0.009 0.034 0.066 0.040 60 & 90 8.849 0.023 0.010 0.008 0.029 0.057 0.035 75 & 90 8.849 0.024 0.009 0.008 0.028 0.056 0.035 Table 7. Sigma statistics for proposed aerial missions when comparing flying heights(ft).
If only a single height is used, then the number of images can be reduced to under 40.Image overlap remains the next variable can be adjusted.If overlap can be reduced, then even fewer images can be used to calibrate the camera.M3 was used as the basis for this analysis.Only the images captured at 75ft were used, as well as the four oblique images captured at the same height.Table 8 shows the effect of disabling either every fourth image or every second image.This operation requires some clarification.The images were divided into four groups: Landscape, 90-degree rotation, 270-degree rotation, and oblique images.The disabling of every second or every fourth image was applied to each group.The effect of this was to disable every second or every fourth combination of images taken at a particular waypoint.Because only four oblique images were collected, and their value was clearly demonstrated none of these images was disabled.Disabling images, whether every fourth or every second, had negligible impact on calibration accuracy.If every second image in each group can be disabled with no ill-effects, then the calibration image-set can now be reduced to few as 20 images overall by planning forward overlap as 60%.

In-Situ use of Calibration Model and Control Network Check
Within PhotoScan, a camera calibration was produced using an aerial camera calibration mission block that included oblique images and rotations.Two image sets from the same flight plan over a relatively flat area with 10 control points and 9 control points respectively were processed in PhotoScan, first using the pre-calibrated camera, and the second time using autocalibration.
The effect of calibration on the control point residuals were then compared.While not all residuals were improved using the precalibrated camera, overall the RMS error on the control points was reduced significantly with the use of a properly calibrated camera.In particular, and as expected, the height accuracy always significantly improved when a calibration with an accurate Focal Length was used.This fact has long been understood in the aerial mapping community, but to date does not appear to be a lesson that operators of "prosumer" UAVs and Structure-from-Motion software have taken to heart.

CONCLUSION
Prosumer quadcopter UAVs offer real advantages in photogrammetric mapping.The economies of scale in their production have reduced their cost, flexible and inexpensive flight planning software is now available, and, as we have shown, their cameras can be calibrated to a satisfactory level of accuracy if some care is taken.The first step is ensuring that the manufacturer's geometric correction is not imposed on the images.The second step is to create custom calibration flightplans that leverage well-understood calibration practices published over the past decades.As we have demonstrated, as few as 20 images taken over a scene of varying texture and height is sufficient to generate a satisfactory camera calibration.This is hardly a big investment in time or money and could potentially resolve many of the systematic errors like "doming", almost certainly due to misestimation of the interior orientation parameters, without resorting to non-standard curving flightplans.The aerial block structure, long employed for mapping missions using manned aircraft, can still be used for UAVs provided pre-calibration of the cameras is regularly done.As we have shown, this need not be a time-consuming burden on UAV operators.

Figure 1 .
Figure 1.Visual comparison of DNG images when converted to JPEG with manufacturer supplied geometric correction imposed in Adobe (left), and with no geometric correction applied using RawTherapee (right).

Figure 3 .
Figure 3. Relative only Point Density (#points/location) visualization with vignetting correction (left) and without vignetting correction (right)

Figure 4 .
Figure 4. Interior Orientation Results for M4 (green) and M5 (red) as additional images are continually added in an attempt to further improve orientation parameters.

Figure 5 .
Figure 5. Error residuals for control points of two separate missions flown over the same scene on different days; July 13, 2018 (top), July 16, 2018 (bottom). below.

Table 3 .
Distortion statistics for Camera Calibration

Table 4 .
Interior Orientation Results

Table 8 .
Sigma statistics for proposed aerial mission A (M3) comparing the removal of every 2 nd and 4 th image with oblique imagery at 75ft.