Resource Saving Approach of Visual Tracking Fiducial Marker Recognition for Unmanned Aerial Vehicle

Unmanned aerial vehicle (UAV) tracking fiducial marker is a challenging problem, because of camera system vibration, which causes visible frame-toframe jitter in the airborne videos and unclear marker vision. Multirotors have very limited weight carrying, controller, and battery power resources. While obtaining and processing motion blurred images, which have no useful information, requires much more image processing subsystem resources. The paper presents blurry image frame elimination based approach of UAV resource saving fiducial marker visual tracking. The proposed approach integrates accelerometer and visual data processing algorithms to predict image blur and skip blurred frames. Experiments have been performed to verify the validity of the proposed approach.


Introduction
Camera pose estimation is a common problem in various computer vision based applications requiring an accurate localization in the environment such as robotics, augmented and virtual reality [1], [2].Computer vision is an excellent solution for land and air robots as a low-cost, information-rich source, which supplements the sensor suite necessary for precise control.
For fully autonomous UAV, the ability of autonomous visual tracking and landing in case of losing GPS signal is essential for successfully completing appointed mission [3], [4].Obtaining the camera pose from images requires finding the correspondences between known points in the environment and their camera projections.While some approaches seek natural features such as key points or textures [5], [6], [7], fiducial markers are still an attractive approach because they are easy to detect and allows us to achieve high speed and precision.However, the design of marker tracking system for airborne platforms is a challenging problem, due to the motion of the platform, on which camera is mounted, is unconstrained, therefore frame-to-frame jitter in the airborne videos occurs [8].Also, multirotors have very limited weight carrying, controller, and battery power resources.While obtaining and processing motion blurred images, which have no useful information, requires much more image processing subsystem resources.Consequently, special attention should be paid to resource saving based image acquisition and processing algorithms optimization to work as fast as possible and save processor resources.
There are several open source software tools for visual markers available such as ARToolkit or OpenCV.
ARToolkit is an open source c/c++ library for building Augmented Reality (AR) applications by tracking a planar AR marker using pose estimation techniques [9].OpenCV is an open source computer vision and machine learning software library, which has more than 2500 optimized algorithms, which include a comprehensive set of both classic and state-of-theart computer vision and machine learning algorithms [9].However, special attention should be paid to image processing algorithms improvement for systems with extremely limited resources like UAV.This paper presents blurry image frame elimination based approach of UAV resource saving fiducial marker visual tracking.The proposed approach integrates accelerometer and visual data processing algorithms to predict image blur and skip blurred frames.Image threshold and contour detection algorithms, provided by OpenCV library, were used for marker visual recognition and tracking.Experiments have been performed to verify the validity of the proposed approach and to provide reliable, efficient, and resource-saving approach of visual tracking fiducial marker recognition for UAV.

Related Work
There are plenty of examples demonstrating fiducial marker visual tracking applications [5], [6], [9] based on valid markers and an algorithm, which performs marker detection, and possibly a correction, in images.Used black and white planar circular markers where the identification is encoded in circular sectors or rings [10].Cybercode [11] or VisualCode [12] is derived from 2D-barcodes technology as MaxiCode or QR, but can also accurately provide several correspondence points.QR codes have been designed to be easily found and to have its size and orientation determined under bad imaging conditions.Many applications are related to aiding visually impaired people [13] or to QR code processing in video and mobile devices [14].Other popular fiducial markers are the ReacTIVision amoeba markers which that are also based on blob detection, and its design was optimized by using genetic algorithms.Some authors have proposed methods to improve detection and classification in cases of bad illumination and blur caused by fast camera movement [15] or in cases of partial marker array occlusion [16].Viola and Jones proposed a boosted cascade of simple classifiers for rapid object detection in images [17].Their framework is used for the detection of finder patterns (FIPs), which is a fixed pattern located in three corners of any QR code.Belussi and Hirata proposed a post-processing algorithm to help decide if the detected FIPs are or not corners of a QR code symbol [18].Single or multiple target tracking approaches [19], [20] are widely used in the stationary tracking systems, but it can be used in robotics applications such as navigation.The abovedescribed markers can be used in visual tracking applications based on image pattern recognition.Fiducial marker recognition applications are widely used in researches to design vision systems for UAV autonomous landing applications such as novel multiple view algorithm to improve the motion and structure estimation for vision-based landing of UAV [21], real-time visionbased landing algorithm for UAV on a moving target [22], vision system for helicopter landing in complex environment based on marker detection UAV vision system [23].However, in airborne videos, due to large monitoring area, these approaches may not be able to satisfy the requirement of high processing speed.When airborne platforms vibrate significantly, the position of the marker in the airborne videos undergoes extreme changes and target marker may be lost.Other researchers deal with marker recognition improvement analysing software based blur images correction [24], which uses complex multistep motion direction estimation algorithms.These algorithms require a lot of computing resources, therefore in our case they are too slow for real-time application in UAV system.

Fiducial Marker Detection and Tracking Approach
Assuming that moving UAV takes blurry images and filtering those images requires additional controller's operations leading to enlarged energy and time consumptions.The resource-saving approach of fiducial marker visual recognition and tracking is based on the integration of additional hardware -accelerometer into image processing subsystem, to predict blurred image and ignore appropriate frames.
In this case, a custom made fiducial marker was used (Fig. 1).It is made of four finder patterns (FIP) in the corners similarly to QR codes.Since we need only finding fiducial marker position our marker does no't carry any additional information, that' is why there is no code pattern in the middle.In the middle of the fiducial marker, there are next four slightly smaller patterns, and in the middle of those are the other ones.
The whole size of the fiducial marker between biggest FIPs sides is 50 cm, which enables multirotor better recognise from the greater distances as much as 10-20 meters.In the case of flying robot will be close to the marker or need to land on it, maker size might not fit in the view of the system, that' is why we added smaller FIPs inside.
As shown in Fig. 2, at the start of fiducial marker detection and tracking algorithm, an undesirable vibration data is obtained by an accelerometer that is attached to the camera.If acceleration is high, the probability of blurred image is high and useful information extraction from it is low.The cycle is repeated over and over until obtained acceleration value is small enough and the probability that read image will be blurred is minimal.The acceleration value limit is selected in accordance the quality of image capturing equipment, i.e. what kind of camera is used, if there is another camera stabilization equipment, etc.If acquired acceleration value is low enough to get a quality image, then the algorithm goes to image acquisition and processing part.At first "grab" function is used, which sends commands to the camera to acquire a currently visible image, then "retrieve" function to retrieve image data from video camera cache to the controller.In the further research, we focus on monitoring of radio links parameters at the tens of GHz frequencies, which have a conclusive correlation weather with the signal level.The attention is going to be also focused on the weather impact on a different type of modulations.
To find the black/white marker in the environment of image, "threshold" function was used.The function applies fixed-level thresholding to a single-channel array Eq. ( 1).The function is typically used to get a binary image out of a grayscale image or to remove blurriness.
where src (x, y) is grayscale image matrix, maxval is maximum pixel value, thresh is threshold parameter and dst (x, y) is binary image matrix.
Thresholding becomes a simple but effective tool to extract objects from the background.The output of the thresholding operation is a binary image whose one state will indicate the foreground objects like printed text, a legend, a target, defective part of a material, or in our case markers black areas, while the complementary state will correspond to the background.The local threshold technique is useful when there are a strong illumination or reflectance gradients.The local method calculates min and max intensity of image part then equalize histogram.
After that, all the contours are traced in the binary image.All the contours sizes are checked, and too small contours are filtered out to reduce image noise.The remaining contours are delineated with rectangles, and their centers are found using "moments" function.The function computes moments, up to the 3 rd order, of a vector shape or a rasterized shape.In our case, raster image spatial moments are computed as: where (x, y) is binary image pixel coordinates.And mass center of one contour (x, y) is computed as: When two contours are found which share the same center coordinates, their areas and perimeters are found: where R P is large and small marker quads perimeter proportion, R S is large and small marker quads area proportion, P is large marker quads perimeter, P check is small marker quads perimeter, which will be checked, K Perror is perimeter comparison error, S check is small marker quads area, which will be checked, K Serror is area comparison error.
Since fiducial marker consists of black squares with smaller white and black squares inside, and those squares proportions are known, we calculate if those squares might belong to the fiducial: where θ is image pixel vertical and horizontal coordinates.
When all contours, which might belong to marker are found, they are grouped by size and distances are calculated between them.Then FIPs belonging to the fiducial marker are determined.Using them, marker center is calculated.This method works even when there are only three finder patterns visible.Kalman filter helps to eliminate inaccuracies and keep fiducial marker center.
The experiment was carried out in the laboratory under ideal illumination conditions.The fiducial marker was placed at 10 meters from the flying quadrotor and the quadrotor flying height was 0.5-1.5 meters.Captured images were carried out on ASUS K55A laptop computer with Intel Core i5 3210M 2.5 GHz processor and 4 GB RAM.

Experimental Results and Discussion
At first the proposed fiducial marker recognition algorithm was tested under regular flying conditions, when quadrotor movements were made with the maintaining camera facing marker and without acceleration limit set.The image capturing results are shown in Fig. 4. On the abscissa is shown an iteration of the proposed fiducial marker recognition algorithm.Dark blue colour shows measured acceleration at appropriate iteration, and light blue represents whether marker was detected (1) or not (0).At the both sides of the graph, there are shown fiducial marker images obtained at different accelerations.As you can see images are blurrier, where acceleration is higher.The graph shows that algorithm easily managed to detect fiducial marker at low accelerations.While quadrotor was lying on the ground with propellers spinning, measured vibrations were up to 0.05 g.
To find out how does algorithm detect a fiducial marker at different accelerations, quadrotor had to do various sudden movements while maintaining a view on marker all the time.Acceleration measuring step was chosen 0.016 g.After long testing, we managed to get at least 100 measurements in each acceleration interval, and the results are shown in Fig. 5(a) and Fig. 5(b).
Figure 5(a) shows fiducial marker detection percentage at different accelerations, which cause image blurriness.The graph shows that the best marker detection is at low acceleration and then drops rapidly.At an acceleration of 0.2 g and more, less than 20 % of iterations the fiducial marker is detected.Figure 5(b) shows the time of fiducial marker detection algorithm iteration at different accelerations.Blue colour shows iteration time when the fiducial marker is detected, red, when no fiducial marker was detected, and yellow, average cycle time of this experiment.
Image processing time highly depends on environment texture, which generates different amounts of contours.For example in the brick wall transition from one brick to another is taken as a contour.Due to our algorithm attribute, the less blurry image was got, the more information is needed to process.That is why in the interval froorm 0 to 0.2 g image processing time is higher irrespective of marker detection result.In the interval from 0.2 to 2 g cycle time slightly drops.At higher acceleration algorithm time bounce, because each time image is received blurred differently and the amount of data, when the fiducial marker was detected successfully, is low.
Marker detection percentage at high accelerations is low, and it requires a lot of computing resources of the system at the same time.After experiment data had been received, acceleration limits were introduced into the system.As seen in Fig. 5(a), fiducial marker detection probability drops exponentially till 0.5 g. and successful fiducial detection at 0.5 g, is only 10 %.At higher than 0.5 g acceleration, it holds at 4-9 %.To increase system efficiency and save recourses, images at higher than 0.5 g acceleration is not obtained and processed.Instead system goes back to reading accelerometer data, till acceleration lowers.
In Fig. 5(c) is shown fiducial marker detection percentage at different accelerations after acceleration limit were introduced into our algorithm and in Fig. 5(d), algorithm cycle time at different accelerations with acceleration limit.As shown in Fig. 5(c), average cycle time after 0.5 g is 20-25 ms.That does not include image acquisition time which is 2-4 ms.Using accelerometer saves that time that is used for possibly proper image acquisition.In our case, that increased algorithm efficiency up to 5 % (saving up to 5 % computation process time).

Conclusion
Our research results prove that multirotor generated vibrations and accelerations do negative impact for fiducial marker recognition.The proposed algorithm takes 20-35 ms to process each frame.Using accelerometer, we can measure acceleration and predict image quality.
During the experiment, obtained data were used to set acceleration limit into the algorithm.When acceleration exceeds 0.5 g, the image is not obtained and processed, because probability fiducial marker will be detected at higher accelerations is only 4-9 %.This way 2-4 ms of image acquisition and 20-35 ms of image processing time is saved in every iteration.While system waits for the correct image, the processor is idle and uses less power.When movements of multirotor were mild, the proposed approach increased fiducial marker detection and tracking algorithm efficiency up to 5 %.In the future, this approach will be adapted for multirotor, when images will be processed on board or remotely, in order to detect and track fiducial marker in various situations, such as marked object tracking, UAV localization in the environment or for autonomous landing on the platform.That will lead to greater autonomy and use in different fields.
Edmundas KISELIOVAS is M.Sc.student at the Department of Informatics and Statistics, Klaipeda University.He graduated B.Sc. degree in Informatics engineering, Klaipeda University (2014).His research interests include robotics, image processing and automation.