Virtual Experimental Stand for Automated Fiducial Marker Comparison in Gazebo Environment

This paper presents experimental automated approach for comparing fiducial marker systems. Previously we compared ARTag, AprilTag, CALTag systems under three types of adverse conditions: systematic occlusion, arbitrary overlap with an object and marker rotation. In effort to improve statistical significance of our previous work with manual experiments, we faced a challenge of conducting over a thousand additional experiments. Using Gazebo environment we implemented virtual robot system that performs all necessary manipulations automatically. Further, we investigate adding Gaussian noise in order to make simulations more realistic


Introduction
Fiducial marker systems (FMS) are systems of markers that are automatically detected by a camera with help of corresponding algorithms. FMS find their applications in various areas: physics, medicine, robotics, augmented reality, metrology etc. Taking in consideration only application in robotics, marker systems are used in robot navigation 1 , localization 2 and mapping 3 , camera calibration 4 and tasks that require camera pose estimation 5 . As our global goal is to calibrate Russian humanoid robot AR-601M mono camera and manipulators using FMS, we faced many FMS options. Each FMS has it strengths and drawbacks that are derived from the original area of its applications. Therefore, we need to compare markers in order to select a single suitable FMS from a large number of existing systems for our specific task. There are several marker performance criteria 6 . In our work, we are particularly interested in marker occlusion resistance since the occlusion is a rather usual and unavoidable while working with humanoid robot manipulators. If the markers are located on the robot itself, they may become overlapped with varying intensities by other workspace objects (e.g. its own manipulators) and have some inclination angle with regard to a camera, which also affects its detectability 6 .
Previously, we conducted manual experiments 7,8 to evaluate FMSs resistance to overlapping marker's area (systematic occlusion and arbitrary overlap with an object) and various rotations. Since it is time consuming to conduct such experiments with thousands of trials manually, we have decided to automate them and to start from experiments in ROS/Gazebo environment (Fig. 1).
This paper presents automated approach of conducting experiments with marker systems to evaluate occlusion resistance. Section 2 presents our related work of comparing FMS in presence of occlusions, in Section 3 we briefly overview ROS Gazebo environment and in Section 4 we describe our simulated world in Gazebo and experiment setup. Section 5 shows the results of experiments and analyze of it. Finally, we conclude and plan our future work in Section 6.

Related Work
During the selection of a most suitable FMS we have selected three systems most applicable in our field: ARTag 5,6 , AprilTag 3 and CALTag 9 . As mentioned above, initially we conducted all the experiments manually. Our experimental work consisted of two main parts: pilot experiments with web camera Genius FaceCam 1000X 7 and experiments with AR-601M humanoid 8 (Fig. 2). We outlined two experiment designs: simple and compound. First design (simple) consisted of systematic occlusion and arbitrary overlap with an object. The second design (compound) consisted of several types of occlusion: marker rotation, systematic occlusion, marker rotation with systematic occlusion and arbitrary overlap with an object. The examples of first set of experiments are showed in Fig.3. Figure 4 demonstrates the scheme of marker rotation during second design experiments. Figure 5 shows example of a marker rotation experiment. These two experimental designs initially were carried out using Genius FaceCam 1000X camera for investigating FMS's applicability using low-cost equipment. After initial experiments were completed, the same experiments were performed using humanoid robot AR-601M mono camera Basler acA640-90gc.
Analyzing the results of our experiments, we concluded that AprilTag and ARTag demonstrated high sensitivity to edge occlusions, while showing satisfactory results in the experiments with arbitrary overlap and achieved perfect score in the marker rotation tests.
CALTag system on the other hand showed high resistant to all types of occlusion: edge occlusion, arbitrary overlap with an object and tag rotation.

ROS Gazebo Environment
Robot Operating System (ROS) is the most fast-growing and popular framework for programming robots that was initially developed by Willow Garage. The main idea of ROS framework is a collaborative construction of robotics software. Everyone can use and improve ROS packages -an atomic unit in ROS system that represents one or more functionalities or ways to solve particular problem -or create a new one.
Gazebo is a popular 3D simulator for simulation of different type of robots, helping with the creation of new robots and testing algorithms in rather realistic conditions. Gazebo includes physics simulation, robot and environment models, and custom plugins. We selected Gazebo for our virtual experiments due to its integration with ROS, accessible custom joint controllers and a convenient way to design a robot with urdf-files.

Experiment Setup
We targeted to automate execution of thousands of experiments with each of the FMS families to minimize time and effort that are required by real world experiments. For this purpose, we created a Gazebo world with two robots: R2D2-like robot (observer) and the tag-managing robot (performer). The performer robot is actually several interchangeable robot models designed for each type of experiments. In this Section we discuss the design and results of pilot virtual experiments with AprilTag markers rotation around Z axes. Those type of experiments are presented in the Fig. 4 as Type 1A.
The performer was implemented using ros_control package for its controllers. We tuned a PID controller and designed a procedure to perform rotation around Z axis by a given angle (from 0 degrees to 90 degrees) in both directions (implemented as tag_ rotation_node node). Using already existing implementation of AprilTag in ROS (apriltags_ros by Mitchell Wills) we detected a tag in a set of camera frames while the performer rotated AprilTag publishing the results of the detection (ID of a tag and an angle of its rotation). As the performer rotated the tag, it published current angle of rotations. Thus, we could deduce, which angles resulted errors in detection or rotation estimation by comparing messages from aforementioned nodes. The comparison process was automated, as during launching virtual experimental stand we specified the tag family and ID, experiment type, number of repetitions, noise level and when experiments were completed we received summary of experiments (number of errors in detection, rotation estimation etc.) To make our experiments closer to real word conditions, we added Gaussian noise parameters for the camera of R2D2-like robot. We used three values for noise standard deviation (stddev) parameters: 0.009, 0.09 and 0.1. Figure 6 shows examples of camera view with varying levels of Gaussian noise. Image on the left represents stddev parameter of 0.009 and it is considered to be a common case for a real camera. In the central image the camera has 0.09 stddev, which represents moderate level of noise. Image on the right shows the camera with 0.1 stddev, representing high levels of noise, and its threshold value serves as a boundary condition of marker recognition possibility or failure.

Experimental Results
To test the virtual experimental stand we chose four randomly selected AprilTag type marker with the same IDs as in previous work 7 : ID 4,6,8,9. The stand performed 3000 experiments for each marker with the results of experiments demonstrated in Table 1. Every marker could be successfully detected with up to 41 degrees of rotation with low nose and up to 25-38 degrees with high level of noise. This factors explain the low percentage of successful experiments in Table 1, because the rotation was performed up to 89 degrees. As was expected, with noise level increase the number of successfully detected and recognized markers decreased.  Moreover, virtual experiments with noise also demonstrated that some markers are significantly more successful that others (e.g., ID4), which emphasizes the necessity of various markers performance comparison.

Conclusions and Future Work
This paper presents experimental automated approach for comparing fiducial marker systems (FMS). Previously we compared ARTag, AprilTag, CALTag systems under three types of adverse conditions: systematic occlusion, arbitrary overlap with an object and marker rotation. In effort to improve statistical significance of our previous work with manual experiments, we faced a challenge of conducting over a thousand additional experiments, and pilot experiments of such automated approach were presented in this paper. We constructed a world in Gazebo environment, which consisted of R2D2-like robot and AprilTag marker holding robot that rotates AprilTag for a given angle (from 0 degrees to 90 degrees) in both directions. We integrated it with AprilTag node tag detector in such way that we could detect the marker in each robot camera frame. To make simulated experiments close to real world conditions, we investigated three different Gaussian noise standard deviation values. Then we performed experiments applying different noise levels and analyzing AprilTag system recognition under rotation. The system detected tags with up 40 degrees rotation with a low level of noise and up to 25-38 degrees with high. Virtual experiments with noise demonstrated that some markers are significantly more successful that others, which emphasizes the comparison necessity.
As a part of our future work we plan to construct an automatic tool for marker rotation and test it with several FMSs. The most resistant for all types of occlusion marker system will be selected for further AR-601M camera and manipulator calibration.