Performance and Accuracy Analysis of 3D Model Tracking for Liver Surgery

Abstract

We are developing a liver surgical navigation system that uses a 3D liver model of the patient. As the liver contains several types of blood vessels, surgeons can sterically check the treating positions using the navigation system. For better navigation, the 3D liver model should be continuously and precisely matched with the patient’s liver state, and the liver location and orientation should be calculated in real time. The location and orientation is estimated by comparing two depth images of the liver; one is captured from the patient’s body using a depth camera, and one is rendered from the pre-scanned 3D liver model using OpenGL. The tracking is performed using gradient descent optimization algorithm. In this paper, we measure how the tracking agreement rate will change while the liver location and orientation are changing, so that we can understand the tracking performance, accuracy and the pitfall range of the navigation system.

1 Introduction

Liver contains several types of blood vessels in its soft organ, and X-ray imaging, computed tomography (CT) scanning or magnetic resonance imaging (MRI) have been widely used to check its state. Surgeons precisely examine the liver state using those data to plan the operation. For the better planning and performing the operation, medical simulation technology is rapidly developed. 3D model data such as STL (Standard Triangulated Language) data is one of the technology that is made from those scanned data and used for surgical navigation. We have developed a system for liver surgery navigation by tracking knife positions and liver states using the STL liver model [1,2,3,4]. We have also examined its performance from the aspect of the algorithm [5] and from the aspect of the inter-process communication cost [6].

In this paper, we focus on the depth matching mechanism that is used for the liver posture tracking. By rotating the virtual liver model around two axes, we observe how comparison scores will be changed and check the possibility of pitfalls. We also examine the tracking ability of the comparison score using gradient descent, so that the characteristic of the liver tracking can be properly understood for further investigation.

2 Liver Surgical Navigation System

2.1 System Overview

Two cameras above the liver, a surgical knife, and a computer that communicates with those cameras compose our liver surgical navigation system as shown in Fig. 1. Each camera sends different information to subsystems of the navigation system.

Fig. 1.

System overview of our liver surgery navigation system

One camera is a marker tracking camera, which tracks two dimensional markers in real time (illustrated as a yellow camera in Fig. 1). A surgical knife has multiple tracking markers at the bottom, and their locations are captured by the camera and sent to the subsystem called “knife position estimator”. The knife position estimator binds multiple location data to estimate the exact position of the knife tip.

Another camera is a depth camera, which captures depth images of the liver in its measuring range (illustrated as a black camera in Fig. 1). Each frame of the captured depth image sent to the subsystem called “liver posture estimator”. The liver posture estimator performs image processing over the depth images for estimating current position and rotation of the patient’s liver.

The main system then summarize the data from those subsystems using shared memories [6], for visualizing current surgical information. As the system is composed of multiple subsystems, each subsystem can be improved and maintained separately. In this paper, performance and accuracy of the liver posture estimator is focused and discussed.

2.2 Liver Surgical Navigator

The liver surgical navigator integrates the knife position data and the liver posture data from the shared memories. It also loads the polyhedrons data with STL-format independently, for separately showing sub parts of the liver such as the hepatic artery, the portal vein, lobes and so on for the surgical navigation. Figure 2 shows the experimental implementation of the navigator.

For better navigation for surgeons, surgical procedures should be stored and understanding the current procedure is required in the future implementation.

Fig. 2.

The liver surgical navigator showing knife and liver data

2.3 Liver Posture Estimator

The liver posture estimator periodically takes depth images from the depth camera, and compares the shape with Polyhedrons rendered on the memory for finding the exact location of the liver and its orientation. Figure 3 shows how the real liver model (A) is captured as a depth image (B) and how it will be compared with the depth image rendered on the graphics memory using OpenGL (C).

Fig. 3.

Real liver model (A), a depth of real liver (B), and a depth of virtual liver (C)

The accuracy of the posture estimation is important for tracking liver state and displaying navigation information over the virtual liver. To improve the accuracy, both of the performance of depth image comparison and the performance of posture tracking are important to be investigated.

2.4 3D Model Matching

It is difficult to repeat the system verification on human bodies. Therefore, a polyhedron liver model using STL that integrates DICOM based liver images captured by MRI is used in this paper. An STL polyhedron data can be used to be output with 3D printer, as well as rendered in a video memory.

In this paper, a 3D printed model of a liver STL is set below a depth camera. Kinect v2 is used as the depth camera. The identical STL data is loaded in the liver posture estimator, and OpenGL renders the data by changing its position and rotation for matching its depth data with the depth image of the 3D printed liver model.

3 Depth Matching for Liver Posture Estimation

3.1 Depth Image Matching

In the liver surgical navigation system, the liver posture estimator is the key subsystem. It performs iterative rendering of the virtual liver, calculates how much those depth images differ from the depth image of the real liver, and estimates the current posture of the liver. Therefore, the depth image matching mechanism defines the performance of the estimator.

The liver has a signature form and it easily changes its shape when looking it from multiple directions. This fact encourages us to think that the posture estimation can be accurately performed. In our previous papers, liver posture estimation has been implemented based on the idea, and actually tracking is succeeded [5]. However, for optimizing and achieving more accurate tracking in the future, we have to investigate and get the insight about the characteristic of the depth image matching mechanism itself.

In this chapter, an experiment is performed to examine the characteristic of the depth image matching mechanism. In many cases of liver surgery, a liver often rotates more than translation. Therefore, we focus on the combination of rotation around two axes, and try to observe how the comparison score will be changed according to the angles. The sum of squared errors is used as the comparison score to compare two depth images. Less value of the comparison score indicates more matching posture. In the actual case of the surgery, the region to compare to depth images should be specified in some ways, however, the region is assumed to be given in this paper.

3.2 Experiment and Result

In this section, the comparison scores are precisely plotted as 3D graphs to show the characteristics of liver model matching. By rotating the virtual liver model around two axes simultaneously and checking the comparison scores with a real liver model that is fixed and captured by depth camera, we can check how easily the liver can be tracked. At the beginning, the posture of the virtual liver is manually set to be the same as the real liver. Afterwards, angles are changed around two axes and the comparison scores are calculated.

Figure 4 shows the rotation axes in our navigation system. In our experiment, the rotation angle varies from −90° to +90° for each rotation axis and the depth images are compared between rotated virtual liver and the depth image of the fixed real liver model. Because there is the possibility that some pitfalls exist according to the shape of the liver, the result of this experiment should guide us where to focus on the liver surgical navigation.

Fig. 4.

Rotation axes for liver tracking

Figure 5 shows the comparison scores of the depth images by rotating the virtual liver around X-axis and Y-axis simultaneously. At the center of the graph that is colored as blue, the posture of the virtual and real livers are almost the same. The minimum score was 158.8. We can see that the score is gradually increasing by separating from the center, and there is not any serious pitfalls. Therefore, the tracking should be successfully performed in the rotation between X-axis and Y-axis.

Fig. 5.

Comparison scores of liver depth images with X-axis and Y-axis rotations

Figure 6 shows the comparison scores of the depth images by rotating the virtual liver around X-axis and Z-axis simultaneously. At the center of the graph that is colored as blue, the posture of the virtual and real livers are almost the same. The minimum score was 126.2. We can see that there is a hill in the graph around the point where x rotation is −70° and z rotation is −70°. This can be a pitfall for the real time tracking. We will see that the tracking using the gradient descent method will be stuck at the point in the next chapter. This combination of the rotation axes should be carefully managed in the navigation system.

Fig. 6.

Comparison scores of liver depth images with X-axis and Z-axis rotations

Figure 7 shows the comparison scores of the depth images by rotating the virtual liver around Y-axis and Z-axis simultaneously. The minimum score was 102.8. We can see the flat place around the point where y rotation is −70° and z rotation is 70°, but it is not a hill and it gradually goes down to the valley of the blue colored center point. As the experiment in the next chapter shows, there is not any pitfalls in this case and the tracking can be done smoothly.

Fig. 7.

Comparison scores of liver depth images with Y-axis and Z-axis rotations

4 Tracking Analysis of Gradient Descent

In this chapter, performance of real time tracking of the liver is discussed, especially focusing on the rotation around two axes. The experimental result of the previous chapter gave us the good valley formed transition of the comparison scores of the liver posture matching. Therefore, gradient descent algorithm is adopted in this chapter, as an iterative optimization algorithm to examine the performance of real time tracking of the liver.

In our previous paper, we examined the system performance by measuring computer clocks [6]. However, for examining precise characteristic of the liver matching, step numbers are used for comparison in this paper instead of the time.

4.1 Gradient Descent for Real Time Tracking

As an iterative optimization algorithm, gradient descent is a well known algorithm that can help the real time tracking of 3D models. Basically, given a differentiable function, gradient descent calculates the gradient of the function at the current point, and decides positive or negative direction to move. However, because the liver easily changes its form and surgeon rotates the liver many times, we cannot assume that such the function is given for tracking.

Instead of using a given differentiable function, it can be considered that rotating the virtual liver slightly from current posture and calculating the gradient. By rotating 1 to 5° around X-axis, Y-axis, and Z-axis, the tendency of the comparison score change of depth images can be used as the gradient to decide positive or negative direction to change the posture. In this paper, gradient descent is tried to perform real time tracking of 3D liver model in this way. Because of its simplicity, it clearly gives us a good insight into the characteristics of depth image matching of the liver.

4.2 Experiment and Result

In this section, the experiment about the tracking performance is explained according to the two axes rotation. At the beginning, the posture of the virtual liver is manually set to be the same as the real liver. Afterwards, angles are changed to −70° from the first rotation in each of two axes. By using the gradient descent method described in previous section, the virtual liver gradually rotate to minimize the comparison score. It can be checked that how many steps will be needed to correct the posture and how much the virtual liver posture can be the same as the real liver posture.

Figure 8 shows the result of the experiment, where orange line shows the combination of rotation around X-axis and Y-axis, blue line shows the combination of rotation around X-axis and Z-axis, and green line shows the combination of rotation around Y-axis and Z-axis. As the graph shows, orange line (X + Y) and green line (Y + Z) gradually come to the point around 350. At this point, the posture of the virtual liver is very close to the real liver as the left image of Fig. 9. However, blue line (X + Z) stays at the point around 620. At his point, the posture of the virtual liver is quite different from the real liver as the right image of Fig. 9. Figure 9 shows the final state of the depth image matching after 120 tracking steps have finished.

Fig. 8.

Comparison score change during the posture tracking using gradient descent

Fig. 9.

Depth image matching after 120 tracking steps using gradient descent (X-axis and Y-axis rotation (left) and X-axis and Z-axis rotation (right))

As the experiment in Chapter 3 showed that there is a hill around the point where x rotation is −70° and z rotation is −70°, the tracking will be failed if the liver rotates in this combination of axes. Furthermore, tracking performance for the combination of X-axis and Y-axis is better than the combination of Y-axis and Z-axis. The orange line comes to 350 points in 30 steps, where the green line takes 60 steps to be the same point. It is also important to see that the comparison score using gradient descent stays at 350 points, where the minimum comparison scores are around 120 as we described in Chapter 3. Further investigation is required to understand the reason why it does not come to the minimum point during the tracking using gradient descent.

5 Conclusion

In this paper, we examined the characteristic of the depth image matching for the liver surgical navigation. It is confirmed that there is one pitfall at the combination of rotation around X-axis and Z-axis. However, it is also confirmed that the comparison score gradually and beautifully decreases to the center for performing a good tracking of the liver in most cases. This insight should lead us to optimize the implementation of our liver surgical navigation system.

In the future, further investigation is required for understanding the difference between the minimal score and the score of tracking using gradient descent. In addition to that, comparison method other than the sum of squared errors for binary depth images should be discussed.