C-ARM POSE ESTIMATION AND NAVIGATION IN SURGERIES FOR AUGMENTED REALITY APPLICATION

: C-arm X-ray imaging systems are widely applied in surgeries. Overlaying X-ray with optical images during the surgery has been shown to be an efficient approach. Moreover, overlaying needed data from different modalities in an augmented reality (AR) manner can improve the accuracy of surgical procedures, decrease the variability of surgical outcomes, reduce trauma to the critical structures, increase the reproducibility of surgeons’ performance, and reduce radiation exposure. C-Arm geometric calibration and recovering the C-arm pose are essential for surgical navigation and AR applications in operating rooms. Therefore, in this paper, existing researches for calibration and pose estimation of C-arm devices in surgical AR applications are evaluated from photogrammetric point of view. Then, a proposed marker-based method for C-arm pose estimation is introduced. For this purpose, a marker is designed to facilitate tracking and pose estimation in mixed reality based on golden section principle, and perspective invariants such as cross-ratios, collinearity, and intersection. Moreover, a procedure is also proposed for fast determination of these fiducial markers. The experiments show benefits of such a structure which has a limited occlusion with consistency to different conditions such as narrow field of view and at the same time, even in images with high projectivity. It also results that the distortion correction step is important and the effect of distortion of X-ray images can cause inconsistency in the perspective invariants.


INTRODUCTION
Mobile C-arm X-ray imaging is commonly applied in operating rooms for guiding surgeries.Accurate intra-operative information of the treated anatomy provided by C-arm X-ray images can improve the results of orthopedic operations and treatments.In this way, clinicians use this device to track the progress of the surgery and make the corrections that might be needed.But these systems have some drawbacks, which are as following: (a) The clinicians need a period of training to gain the ability to interpret X-ray images because of its special imaging principle; (b) during the surgery, the clinicians usually acquire many X-ray images to navigate which increases radiation dose; and (c) the X-ray image is displayed in a separate monitor, which means the clinicians need to look away from the surgical scene and perform mental mapping between these two spaces (Wang et al., 2016).Therefore, many researches have been done for optimizing the usage of X-ray images in surgery for solving these problems.Due to these limitations, one possible solution is the use of Augmented Reality (AR) and overlaying X-ray with optical data, and other modalities.This has been demonstrated to be an efficient way to facilitate navigation during procedures and reduce radiation expose for both patient and clinicians by requiring fewer image acquisition (Navab et al., 2010).This also improves accuracy and surgery time.

NOVIGATION AND AUGMENTED REALITY IN SURGERIES WITH C-ARM SYSTEMS
AR supplements the real scene with virtual objects superimposed upon or composited with the real world.AR has been widely used in areas such as manufacturing and repair, annotation and visualization, robot path planning, entertainment, and military aircraft guidance (Azuma et al., 1997).AR improves the reality and it provides seamless interaction between real and virtual scenes.
Medical AR technology has aided surgeons with surgical planning, and it enhances the surgeon's view of the patient with computer generated images of anatomy and tools in real-time has been successfully applied in various disciplines of surgery, such as neurosurgery, orthopedic surgery, and maxillofacial surgery (Nikou et al 2000;Shuhaiber et al., 2004).It was shown that AR is a promising solution to improve the accuracy of surgical procedures, decrease the variability of surgical outcomes, reduce trauma to the critical structures, increase the reproducibility of surgeons' performance, and reduce radiation exposure (Nikou et al 2000;Shuhaiber et al., 2004).
Combining multi-modal intraoperative data such as visual, and X-ray imaging has been employed for surgical AR, is taken to the consideration, since the images can be updated during surgery.
For surgical application such as medical augmented reality in operations, C-arm calibration and pose estimation are essential.
In the following, the state-of-the-art approaches for calibration of C-arm imaging systems and pose estimation of C-arm in operating rooms are evaluated.

C-arm X-Ray Calibration for AR
X-ray fluoroscopic images suffer from high pose-dependent distortions, particularly Pincushion distortion, S-shape Sigmoid distortion, and localized distortion.Their explanations are presented in Section 3. State of the art methods for C-arm calibration are performed in a two-step method for C-arm devices with intensifiers, because of special high pose-dependent distortion mostly caused by C-arm intensifier.However, there are few researches using one step C-arm calibration.The first step is distortion correction of X-ray images, mostly by using a grid board phantom such as in (Habert, 2018) or using a look-up table such as (Wang, 2012) or principal component analysis (PCA) such as (Chintalapani et al., 2007).The second step is the calculation of interior orientation.
The C-arm X-ray projection geometry can be mathematically approximated by a pinhole camera model (Navab et al., 1999;Mitschke et al., 2003).The intrinsic parameters of the X-ray fluoroscope are calculated using 2D/3D phantoms by methods based on projective models such as DLT.Yaniv et al. (1998) applied a phantom with three planes to perform the off-line calibration.They used Tsai's 11 parameter pinhole camera model and solution method to model the fluoroscopic camera.They computed the projection matrix by corresponding 3D coordinates of the phantom and the 2D coordinates of X-ray images and then decomposed it.Livyatan et al. (2002) and Meng et al. (2014) utilized more points on two parallel planes for on-line calibration.Shechter et al. (2002Shechter et al. ( , 2005) ) used a plastic rectangular phantom.In these researches, Tsai's method was widely applied to model the fluoroscopic camera.However, Tsai's method includes several assumptions that are not appropriate for C-arm calibration, and it is needed to use a more robust method in order to obtain higher accuracy.Also, there are other researches, proposing DLT and Tsai's method for this purpose, such as (Hofstetter et al., 1999;Kaptain et al., 2011).For surgeries of the total replacement of the knee joint, Broer et al. ( 2002) proposed a C-arm calibration method as a verification tool for the aim of accuracy assessment of the cutting gauge alignment.Their method was based on using a planar test field with bundle orientation.Jain et al. (2006) proposed a system to estimate the relative position of the C-arm between two images from stationary natural features.To recover the pose of radioactive seeds, they designed an encoded fiducial object with a unique representation at every viewpoint.Other details of their research are explored in Table 1.Currently, it is common to use calibration procedures which are applying Zhang's method (Z.Zhang et al., 2000) for calculation of interior orientation parameters.In case of C-arm calibration for operating rooms, several works have used the Zhang's method to calibrate the X-ray fluoroscopy (e.g., Groher et al., 2003;L. Wang, 2012;X. Wang et al., 2015X. Wang et al., , 2016aX. Wang et al., , 2016b;;S. Habert, 2018).Wang et al. (2016b) proposed a 3D/2D calibration method to achieve a video augmented fluoroscope.For C-arm calibration and also RGB-D/C-arm calibration, they designed the calibration phantom and calculated the projection matrix from the depth camera coordinates to the X-ray image.They captured images of the calibration board in fifteen different poses, with various angulations and heights and applied Zhang's method to obtain the intrinsic matrix of the X-ray source.There are a few works, especially in C-arm calibration.In Table 1, several important researches on C-arm pose estimation and calibration for surgical navigation and surgical augmented reality with their accuracies are explored.All of the mentioned researches in Table 1 used pinhole camera model.Moreover, a new research have been done by Ha et al., (2018) adopted a modified pine-hole model, instead of conventional pinhole model for C-arm calibration to reduce the displacement, probably caused by moving the X-ray source position for augmented reality surgical navigation.In the proposed method, the X-ray source movement was modeled as variable intrinsic parameters and represented in the pinhole model by replacing the point source with a planar source.

C-arm Pose Estimation
In the following, approaches proposed for recovering C-arm poses, such as external tracking systems, methods using additional sensors and marker-based methods are also explored.does not suffer from measurement distortions due to metallic components as electromagnetic techniques do, or from drift problems, like inertial sensors or the limitation of the size of the arm for mechanical tracking systems.OTS detect and track markers attached to the treated anatomy or the C-arm and surgical tools by using external optical cameras.However, OTS needs a clear line-of-sight between the patient, instrument trackers and the optical cameras which limit its use in Computer-Aided Surgery (CAS) procedures despite their accuracy (Koivukangas et al., 2013).These problems limit the use of external tracking systems for orthopedic surgical applications.To avoid the limitations of external tracking systems, one way is using inside-out tracking, where one or more sensors such as 2D video cameras or 3D depth cameras are attached on C-arm devices to provide tracking.Figure 1 shows a glance of some of the applied methods for recovering the C-arm pose during the operation.In the following, several other states of the art researches in this field are explained and evaluated.
Figure 1.C-arm pose estimation and navigation for surgeries Moreover, many researches have been done on attaching additional sensors on the C-arm for pose estimation and navigation.Different researches have been carried out by attaching active or passive sensors such as video cameras to the C-arm by using markers or active depth cameras without markers.Navab et al., (1996) applied a 2D video camera on fixed on the C-arm and fiducials placed on a cylinder shape following a codeword pattern around the patient head to recover the poses of several angiography images at different orientations for 3D reconstruction of vessels.Mitschke et al. (2003) used the optical cylinder-shape calibration phantom with marker codes to recover the poses of C-arm.They compared the results achieved by external sensors (optoelectronic tracking system Polaris) with those obtained using a CCD camera in terms of the accuracy of motion estimation and the quality of the final 3D reconstruction.The accuracy of their method using a CCD camera for motion estimation is shown in table 3 and it can be compared with the accuracy of Polaris optical tracking system in table 2. Attached active sensors such as depth cameras without markers are also used to track C-arm such as (Wang et al., 2016).In several studies also the Inertial Measurement Unit (IMU) have been applied to track the C-arm as marker-less methods such as (Grzeda et al., 2010;Amiri et al., 2014).
Moreover, some researches have been done on using fiducial markers for pose estimation of the C-arm as marker tracking methods for augmented reality applications.These markers can be three dimensional or planar such as multi-modal ArUco markers, or individual metal spheres in known configurations.Andress et al., (2018) applied Augmented Reality Toolkit multimodal marker for C-arm augmented reality and navigation application (cf. Figure 2).Figure 2. AR multimodal marker applied by (Andress et al., 2018) However, these methods have several problems of accuracy, time, occlusion, and space occupation.In Table 3, these methods are compared based on different metrics, and the advantages and disadvantage of each method are explored.The disadvantages are shown in grey in Table 3.The accuracy of methods using inertial sensors can be improved by using combined and advanced systems.For each method, sample researches with their accuracies are presented in Table 3.With regards to the mentioned characteristics, each method in this study has several advantages and disadvantages which should be considered for being used in the desired application with its certain limitations.We evaluated existing C-arm pose estimation and calibration methods from different aspects, and their limitations were explored.It is concluded that tracking systems are difficult to apply in surgeries because of their drawbacks such as illumination and line of sight problems for optical systems or the problem of the arm size for mechanical systems or the effects of ferromagnetic objects for magnetic tracking systems, and other mentioned limitations, beside their high prices.We also evaluated inside-out tracking methods and their advantages and disadvantages, shown in table 3.Moreover, it should be mentioned that these methods usually need additional calibrations between the sensor and the C-arm.Another proposed method in this field is applying radiopaque ARToolkit or ArUco markers.As it is mentioned in this section, these markers occlude  Sample systems with their accuracy large areas of the X-ray image captured by C-arm which has a very narrow field of view (10~20 degrees).This can cause losing valuable information from X-ray image.The occluded areas also can limit the registration processes.The aim of our research is also proposing a direct marker-based method for C-arm pose estimation and augmented reality.For this purpose, our method applies a new designed marker which has distinct features with little occlusion of the X-ray image.To design a new efficient multi-modal marker, for the first time, we combine golden section principle, and cross-ratio in a pentagram structure for fiducial marker to facilitate C-arm tracking and pose estimation with least occlusion, with high contrast, accuracy and speed in mixed reality applications and at the same time considering the C-arm limitations such as narrow field of view.

METHODS
As it is mentioned in the previous section, C-arm pose estimation is needed for surgical navigation and augmented reality applications.Marker tracking is a common method for this purpose and it is widely applied for augmented reality applications.In this way, standard markers are usually used for optical images.For X-ray images, radiopaque AR markers have been tested also.However, they have several limitations for Carm X-ray imaging such as size, high occlusion, narrow field of view of C-arm devices and distortion.In this paper, we propose a simple method based on projective invariants particularly crossratio, intersection and collinearity for marker tracking in surgical AR applications.Cross-ratio is invariant in projective geometry.It can be defined for four collinear points or for five coplanar (and non-linear) points which are respectively shown in figure 2  (This irrational number is a solution of the quadratic equation x 2x-1=0).Moreover, the cross-ratios of the harmonic points are equal to -1.We can define various harmonic points on the pentagram structure.In this way, we can create more markers with distinct numbers of points and patterns if it is needed.We consider cross-ratios between sets of points of the marker in the image.By computing the cross-ratio of a few sets of points in the pentagram, we can identify kind of each point (points on the enclosing conic, harmonic points, …) and other cross-ratios can be predicted.In this method, for the first time to our knowledge, the benefits of the golden ratio and cross-ratios to propose a direct fast marker-based method for C-arm pose estimation and augmented reality are applied.In this research, computing the marker pose is done as followed: (a) X-ray image distortion correction and calibration, (b) extraction of marker points, marker identification and identification of the fiducial IDs of the marker, (c) pose estimation.For images of C-arms with intensifier, the distortion of X-ray images is high and pose-dependent distortions, particularly Pincushion distortion, S-shape Sigmoid distortion, and localized distortion.The pincushion is a radial distortion that can be caused by the projection of the planar image onto the curved image intensifier, and the sign of this distortion is positive.Sigmoid (S-Shaped) Distortion is due to the interaction of electrons in the image intensifier and a homogeneous external magnetic field, i.e., that of the Earth (Browbank et al., 2000) and its behavior depends on the image intensifier's orientation within the magnetic field.Local distortions are due to the interaction of electrons in the image intensifier with local, inhomogeneous magnetic fields generated by equipment in the imaging environment (Browbank et al., 2000).We calculate the distortion by using a calibration board consisting of grid holes with 1.5mm radius.For this purpose, we attach a grid board to the intensifier detector rigidly and capture images.In this way, matching between distortion-free points and detected points on the X-ray image is used.We apply Bernstein polynomial as a well-known acceptable polynomial illustrated in previous studies such as (Kedgley, et al., 2012;Chintalapani et al., 2007) for X-ray distortion correction step: where and Cijs are the coefficients.We chose n=3 in this equation due to the previous researches (Kedgley, et al., 2012;Chintalapani et al., 2007) and find the coefficients.We then calibrate the C-arm using bundle adjustment for achieving the intrinsic parameters of the C-arm, mainly the distance between the X-ray source and the detector, and principal point offsets by using the Levenberg-Marquardt algorithm.For this purpose, we captured twenty two X-ray images of a three-plane phantom consisting of attached radiopaque ball bearing with radii of 0.75mm.After distortion correction and calibration step, we extract the points of the proposed marker from captured images by using a simple blob detector algorithm (Thresholding, Floodfill, Morphological filtering, and Blob detection).We used OpenCV library in C++ for real data.After extracting the points of the marker and the centroid calculation, the points on the circular/elliptical boundary are found by intersecting with the approximated elliptical boundary.For this purpose, we modify the algorithm proposed by (Moshtagh, 2009) for minimum volume enclosing ellipsoid (MVEE) in our method with known centers, and combine it with cross-ratio for pentagram structure markers.The center fiducial of the marker is achieved easily because of its distinct larger radius (P in figure 4).However, for other possible star marker shapes, it can be also calculated.The ellipse equation in the center form is given by: (xi -c) T A(xi -c) =1.The MVEE problem, for n points in 2D space X={x1,x2,…,xn} is: minimizeA: log(()) subject to: (xi -c) T A(xi -c) ≤1 i=1,…,n.A>0 Where c is the center of the ellipse and it is supposed to be known in our method.Variable Matrix A contains the information of the shape of the ellipse (the radii and orientation of the ellipse that can be achieved by the Singular Value Decomposition).This is solved by using Khachiyan Algorithm.In this way, we achieve high accuracy and speed.By applying cross-ratios and the mentioned constraints, the points of the marker and their indices are extracted automatically precisely since this method uses cross-ratio to correct displacements caused by different sources such as noise and image distortion.It should be mentioned that we can create and apply different star markers in one image and choose one or several of them for higher accuracy.For this purpose, we can use simply k-means or other clustering methods and identify each marker by the number of its fiducials and its pattern (cf. Figure 14.a).However, since we want to have the least occlusion, in this paper we concentrate to examine C-arm pose estimation and augmented reality with our proposed marker, although this can be extended to multi-marker tracking.After extracting the features identification of the points, we have the 2D coordinates of the image points of the markers and their corresponding 3D object coordinates.Therefore, the final pose of the C-arm can be estimated applying by different methods.In this research, for more accuracy we apply Infinitesimal plane-based pose estimation (IPPE) proposed by (Collins et al., 2014).Since the fiducials of the marker are circular and they are more than four, we expect to achieve more accurate localization than markers with corner features (cf. Figure 2).In the next section, our proposed method is evaluated for both simulated and real data.

EXPERIMENTS
For evaluating our proposed method, we divide our experiment into two parts.In the first experiment, we create simulated data and work with them in MATLAB.In the second experiment, we work with the real data and implement and test our method for C-arm pose estimation and augmented reality.In the first part of Experiment, we simulate data using some intrinsic calibration parameters of the C-arm.We consider a field of view of 20° × 20° and X-ray image resolution of 640 × 480 pixels for our simulation data.Using the intrinsic parameters of the C-arm, and dimensions of our designed marker, we simulated 190 X-ray images of the designed marker.Figure 5 shows the distribution of C-arm poses used in the simulation.In this simulation, the c-arm poses for capturing images are distributed on a sphere, and they are looking at the center of the marker plane.The marker is also simulated in MATLAB as shown in figure 6.We calculate cross-ratios between different subsets of points {P, A, P3, C, D, E, F, G, H, I} of the simulated marker and the simulated images with various orientations (rotations around X and Y axes from -80 to 80 degrees) and several levels of noise.shown for the first test.We consider translation noise of 1mm and angle noise of 0.05°, and the digitizing precision of 0.01 pixel which are applied for projecting the object points to the images for the simulation.The ground truth of the cross-ratio of each set of points are calculated from their corresponding points of the marker model in the object space, and it is shown in parentheses in a rounded form in Figure 7.
Figure 7. Variation of cross-ratios under various orientations for the first test.Red lines indicate the variation of cross-ratio for the rotation around the y-axis and blue lines indicate the variation of cross-ratio for the rotation around x-axis.The horizontal axis is the rotation angles in degrees and the vertical axis is the crossratio.For the two below charts, the percentage of the cross-ratio is declared.Similarly, figure 8 and 9 show the variation of cross-ratio under different C-arm orientations for similar four sets of points for two distinct levels of noise.We consider translation noise of 1cm, angle noise of 1°, and digitizing precision of 0.5 pixels for the simulation in the second test which the resulted cross-ratios for four point sets are displayed in Figure 8.  Red lines indicate the variation of cross-ratio for the rotation around the y-axis and blue lines indicate the variation of crossratio for the rotation around x-axis for the third test.We implemented our method on simulated data and our method successfully identified the IDs of fiducials of the marker from the simulated image in the rotation movements of 70 degrees and provide an acceptable estimate of the orientation of the simulated C-arm poses even in high rotations.However, as it is shown in figures 7, 8 and 9 for instances, in higher levels of noise, the variation of the cross-ratio increases especially for the rotation movements more than 60 degrees for the simulated data.For experiment 2, we apply Siremobil Iso-C C-arm from Siemens medical solution to work with real data.We also perform our method and do some tests on real X-ray images for augmented reality using the recovered C-arm pose.First, it is needed to find the coefficients of the polynomial which was explained in Section 3 for distortion correction and calibration step.After calculating the coefficients of the distortion as described in Section 3, C-arm calibration is done using bundle adjustment with a three-plane phantom consisting of radiopaque small ball bearings.The C-arm intrinsic parameters are achieved with the reprojection error of 0.21 pixels.Our marker is made by Lasercutter precisely as shown in Figure 13.a and 13.b.The proposed marker consists of a Plexiglas plate with nine small ball bearings (BBs) with the radius of 0.75mm, and one BB with the radius of 2mm in the center of the marker (point P in figure 10).Now, the real-time X-ray images can be captured from the proposed marker.The IDs of the points of the marker are shown in figure 10.As can be seen in figure 10, the marker has a very limited occlusion with the anatomical objects.
Figure 10.The X-ray image of the proximal femur and the proposed fiducial marker with IDs of the marker's points.
After capturing X-ray images of the proposed marker, and undistorting the image, we implement our method explained in Section 3 to identify the fiducials of the marker and their IDs.We tested the algorithm and the marker for different orientation.The marker fiducials are successfully identified even after high projective transformation (as shown in figure 11.b).However, when the level of noise of the image was very high or the distortion correction step was not done properly, in high rotation angles of the C-arm, the results might not be satisfied.Because of the large size of the fiducials (especially the P point in the center of the marker), for rotation angles higher than 60 degrees, the marker could not be detected automatically since the projections of the BBs of the marker would be superimposed on each other as shown in Figure 12.Therefore, if we want to work with angles higher than 60 degrees, one solution can be choosing BBs with a smaller radius to prevent superimposition.We can also apply multiple markers in proper distances and orientations.Comparing the calculated cross-ratios of point sets of the marker on the X-ray image with ground truth (calculated from corresponding points of the marker in the object space), it shows that the cross-ratios are preserved more than 93% in different orientations.
Figure 12.X-ray image of our marker with rotation angle of 60°.After identification of the points of the marker, the pose of the Carm can be achieved.In this way, we can use various methods such as homography and planar pose estimation PnP methods.We apply IPPE (Collins et al. ,2014) method to recover the Carm pose using the planar marker.The achieved accuracy by the proposed star-shaped marker for C-arm pose estimation is 5.86 mm for translation and 0.91 degrees for rotation which are acceptable for augmented reality in some surgeries, and it can be beneficial for surgeries without any guiding system and providing useful information for the surgeons.We also apply the achieved C-arm pose with the proposed method for augmenting the X-ray image with virtual data (a 3D model) in real-time.For this purpose, we apply OpenGL library beside OpenCV in C++.In figure 14 Using the setup of Intel(R) Core i7 2.80GHz processor with NVIDIA Geforce GTX 1050 Ti., the achieved time for Fiducial markers identification step is 49 ms.The C-arm pose estimation time using our direct method is 18 ms, and the augmentation time is also 6 ms.

CONCLUSIONS
In this research, recent methods for calibration and pose estimation of C-arm for augmented reality and navigation in the operating rooms are evaluated from different aspects.It concludes that because of the limitations of external tracking systems such as high costs, illumination, and line of sight problems for optical systems or the problem of the arm size for mechanical systems or the effects of ferromagnetic objects for magnetic systems, it is difficult to use them during the surgeries.Moreover, inside-out tracking methods are also assessed.These methods need additional sensors and additional calibration processes by considering their applications.Another method is applying radiopaque markers.These markers usually occlude large areas of the C-arm image.This can also cause losing valuable information from X-ray images and limits the registration processes.
Considering the limitations of existing methods, in this research, a direct marker-based method with a new fiducial marker is proposed for c-arm pose estimation.The structure of the fiducial marker is designed based on golden ratio and perspective invariants.This marker has consists of coplanar points of a starshape (pentagram) beside harmonic points.We apply the benefits of cross-ratios for the points of a pentagram which has certain values related to each other.By computing the cross-ratio of a few sets of points in the pentagram in combination with MVEE problem, we can identify kind of each point such as enclosing points or harmonic points, and predicting other cross-ratios.In this method, for the first time to our knowledge, the benefits of the golden ratio and cross-ratios of a pentagram are applied to propose a direct fast marker-based method for C-arm pose estimation and augmented reality.The experiments show benefits of such a structure which has the least occlusion and high consistency to the conditions such as narrow field of view and at the same time, being detectable even in images with high projectivity.The proposed procedure for automatic detection of the designed marker has efficient performance and high speed in the experiments and the marker is detected precisely in images with high projectivity which is important for the pose estimation.However, it also shows that the effect of distortion of X-ray images can cause inconsistency in the perspective invariants and it is important to correct it before starting the algorithm.In addition, the proposed method can be extended using other mathematical models and modification of the shape.
(a, b).The crossratio of the four collinear points A, B, C, D can be written as (based on figure 3. a): Cr(A, B; C, D) = Cr(A ′ , F; G, H) = ( The cross-ratio five non-linear coplanar points using the area of the triangles (△) can be defined as (based on figure 3. b): Cr(P, C, B, F, G) = ( Cross-ratio for: (a) Four collinear points, (b) Five coplanar non-linear points Marker design: The basic design of the proposed marker in this research is shown in Figure4(a).This marker has several characteristics.It consists of coplanar points of a star-shape (pentagram) and harmonic points.In Figure4(a), point D is the harmonic conjugate of A with respect to E, C. In addition, the cross-ratios of points in a pentagram has certain values which are related to each other and can show the kind of the point.As an instance, cross-ratios of the collinear main points of a pentagram are equal to Golden ratio which is φ= Figure 4. (a) The design of the proposed marker, and (b) the marker after the projective transformation.

Figure 5 .
Figure 5. Distribution of C-arm poses used in the simulation.The C-arm sources are shown in green, and their directions are shown in pink.On the top, the marker plane is displayed in cyan color.The dimensions are in meters.The marker is also simulated in MATLAB as shown in figure6.We calculate cross-ratios between different subsets of points {P, A, P3, C, D, E, F, G, H, I} of the simulated marker and the simulated images with various orientations (rotations around X and Y axes from -80 to 80 degrees) and several levels of noise.

Figure 6 .
Figure 6.Simulated marker in object space with certain points {P, A, P3, C, D, E, F, G, H, I}.The dimensions are in meters.Figure 7 reports the variation of cross-ratio under different C-arm orientations for four sets of points (two sets of coplanar nonlinear points P.G.D.C.F, and P.H.D.C.F on the top row, and two sets of collinear points D.C.P3.A and E.D.C.A in the bottom) areshown for the first test.We consider translation noise of 1mm and angle noise of 0.05°, and the digitizing precision of 0.01 pixel which are applied for projecting the object points to the images for the simulation.The ground truth of the cross-ratio of each set of points are calculated from their corresponding points of the marker model in the object space, and it is shown in parentheses in a rounded form in Figure7.

Figure 8 .
Figure 8. Variation of cross-ratios under various orientations.Red lines indicate the variation of cross-ratio for the rotation around y-axis and blue lines indicate the variation of cross-ratio for the rotation around x-axis for the second test.Moreover, for the third test, we consider translation noise of 4cm, angle noise of 3°, and digitizing precision of 1 pixel for the simulation and the results are displayed in Figure 9.

Figure 9 .
Figure 9. Variation of cross-ratios under various orientations.Red lines indicate the variation of cross-ratio for the rotation around the y-axis and blue lines indicate the variation of crossratio for the rotation around x-axis for the third test.We implemented our method on simulated data and our method successfully identified the IDs of fiducials of the marker from the simulated image in the rotation movements of 70 degrees and provide an acceptable estimate of the orientation of the simulated C-arm poses even in high rotations.However, as it is shown in figures 7, 8 and 9 for instances, in higher levels of noise, the variation of the cross-ratio increases especially for the rotation movements more than 60 degrees for the simulated data.For experiment 2, we apply Siremobil Iso-C C-arm from Siemens medical solution to work with real data.We also perform our method and do some tests on real X-ray images for augmented reality using the recovered C-arm pose.First, it is needed to find the coefficients of the polynomial which was explained in Section 3 for distortion correction and calibration step.After calculating the coefficients of the distortion as described in Section 3, C-arm calibration is done using bundle adjustment with a three-plane phantom consisting of radiopaque small ball bearings.The C-arm intrinsic parameters are achieved with the reprojection error of 0.21 pixels.Our marker is made by Lasercutter precisely as shown in Figure13.aand 13.b.The proposed marker consists of a Plexiglas plate with nine small ball bearings (BBs) with the radius of 0.75mm, and one BB with the radius of 2mm in the center of the marker (point P in figure10).Now, the real-time X-ray images can be captured from the proposed marker.The IDs of the points of the marker are shown in figure10.As can be seen in figure10, the marker has a very limited occlusion with the anatomical objects.
(a) The original X-ray image with zero rotation movement; (b) X-ray image with lateral rotation angle of 50 degrees.
Figure 13.(a) Our proposed multimodal marker consisting of small metallic balls colored in white as fiducials, attached on a Plexiglas by using lasercutter and glue; (b) Radiopaque marker overlaid with printout of the corresponding optical marker to increase contrast for optical imaging; (c) The overlaid X-ray image of the marker with its optical image.

Figure 14 .
Figure 14.Augmented reality on the real-time X-ray image with a virtual CT 3D model of the femur by our proposed method

Table 2 .
Evaluation of External tracking systems for C-arm pose estimation

Table 3 .
Evaluation of applying attached additional sensors on C-arm and Multi-modal markers for C-arm pose estimation and AR applications 'If initial sensors are combined with other systems, it can be accurate.*Translation **Rotation