Real-time 3D Shape Instantiation from Single Fluoroscopy Projection for Fenestrated Stent Graft Deployment

Robot-assisted deployment of fenestrated stent grafts in Fenestrated Endovascular Aortic Repair (FEVAR) requires accurate geometrical alignment. Currently, this process is guided by 2D fluoroscopy, which is uninformative and error prone. In this paper, a real-time framework is proposed to instantiate the 3D shape of a fenestrated stent graft based on only a single low-dose 2D fluoroscopic image. Firstly, the fenestrated stent graft was placed with markers. Secondly, the 3D pose of each stent segment was instantiated by the RPnP (Robust Perspective-n-Point) method. Thirdly, the 3D shape of the whole stent graft was instantiated via graft gap interpolation. Focal-Unet was proposed to segment the markers from 2D fluoroscopic images to achieve semi-automatic marker detection. The proposed framework was validated on five patient-specific 3D printed phantoms of aortic aneurysms and three stent grafts with new marker placements, showing an average distance error of 1-3mm and an average angle error of 4 degree.


I. INTRODUCTION
Endovascular Aortic Repair (EVAR) for the treatment of Abdominal Aortic Aneurysms (AAA) involves the insertion of compressed stent grafts via the groin, advancement through the vasculature, and subsequent device deployment and exclusion of the aneurysmal segment. Blood flow is reestablished through the deployed stent graft with reduced pressure on the diseased aneurysmal segment and therefore the risk of rupture is greatly reduced. For patients where the location of the aneurysm overlaps with branching vessels such as the renal arteries, Fenestrated Endovascular Aortic Repair (FEVAR) is necessary; this involves the use of a fenestrated stent graft with fenestrations or scallops [1]. A regular stent graft used in EVAR and a fenestrated stent graft used in FEVAR are shown in Fig. 1a and Fig. 1b respectively. In addition to the location and size of fenestrations and scallops, the size and length of the stent graft are also customised according to patient aortic geometries. An increasing number of stent graft manufacturers, such as Cook Medical (IN, USA) and Jotec Inc (Hechingen, Germany), are supplying fenestrated stent grafts today [2].
FEVAR typically involves multiple steps. The first step is to deploy the main fenestrated stent graft and the principal challenge is the alignment of the fenestrations or scallops with the visceral arteries. The next steps are to cannulate and seal branch stent grafts into the visceral arteries; here the main challenge is to deliver the compressed branch stent graft through the deployed main fenestrated stent graft, arrive at the fenestrations or scallops, and then deploy it into the visceral arteries [1]. These challenges have led to the development of many robot-assisted systems for FE-VAR allowing for improved manipulation of catheters and guidewires. One commercially available system is the Megellan (Hansen Medical, CA, USA), which includes a masterslave catheter and guidewire driving system. The Magellan has been shown to provide the possibility of increased safety, improved manipulation, and less exposure to X-ray radiation than manually operated FEVAR [3].
In current robot-assisted FEVAR, navigation is still dependent on 2D fluoroscopic images (Fig. 1c), as is used in manual procedures. Both the stent and graft have poor visibility under fluoroscopy imaging. High dosage fluoroscopy may improve the visualization, however, this will increase the radiation dose to patients. To improve FEVAR navigation, markers are sewn onto the fenestrated stent grafts to indicate the position and orientation of the fenestrations and scallops (Fig. 1b); these markers are typically made of gold, have different shapes, and can be placed in various positions to aid in alignment of the device with the anatomy.
There has been previous research to improve stent graft deployment. Automatic detection and tracking of stent graft delivery devices from 2D fluoroscopic images has been proposed [4], with Frangi filtering and Robust Principal Component Analysis. Optimized stent graft sizing and placement for pulmonary artery stenosis using cylindrical affine transformation and hill climbing has also been demonstrated [5]. A registration scheme combined with a semi-simultaneous optimization strategy that is to take the stent graft geometry into account was proposed to overlay 3D stent graft shapes onto 2D fluoroscopic images for navigation [6]. However, these methods have been demonstrated on regular off-theshelf stent grafts for EVAR but do not take into consideration fenestrations or scallops. Renal arteries and commercial markers have been highlighted on intraoperative fluoroscopic images to aid with stent graft deployment [7]; however, this is only in 2D and does not provide the 3D shape of the stent graft.
It is necessary to know exactly where fenestrations or scallops are in the stent graft to enable complete deployment of all stent graft devices during FEVAR. A possible 3D navigation or robotic path is shown in Fig. 1d. The path goes along the centralline of the deployed main fenestrated stent graft (black path in Fig. 1d) and then heads to the center of corresponding fenestrations or scallops (green, blue, red path in Fig. 1d). In order to keep a minimum radiation dose during FEVAR, we aim to use a single fluoroscopic image of several markers with feasible placements and shapes for 3D shape instantiation of the deployed main stent graft body. 3D shape instantiation in this paper refers to 3D shape recovery but with only a single 2D projection as the input.
After being deployed into an aneurysm, the stent graft may experience twisting, bending, rotation, and translation with respect to its initial straight state, making 3D instantiation of its entire shape, orientation, and deformation challenging. Most of these non-rigid deformations are caused by what we term the graft gap, shown in Fig. 1b, which is only made up of the graft fabric. For the stent segments which include the metal stent and the graft attached on it, as shown in Fig. 1b , these tend towards their initial states closely due to their relative stiffness. Thus the deformations of the whole stent graft will be split into the rigid transformation of stent segments and the non-rigid deformation of graft gaps. We proposed a framework, as shown in Fig. 2, which instantiates the 3D shape of a fenestrated stent graft from a single 2D fluoroscopic image in real-time. Firstly, each stent segment of a fenestrated stent graft was placed on five markers at different positions. Then, the rigid transformation of individual stent segments were calculated by the RPnP (Robust Perspective-n-Points) method while the non-rigid deformations of the entire stent graft were instantiated by graft gap interpolation. Finally, Focal-Unet was proposed to achieve semi-automatic marker detection. The proposed method was validated on five 3D printed phantoms from AAA patient data and 91 tests of different view angles on three stent grafts with newly placed markers.

II. METHODOLOGY
Stent graft modelling, 3D stent graft shape instantiation including marker placement, rigid transformation calculation of stent segments and non-rigid deformation instantiation of the whole stent grafts, semi-automatic marker detection, the experimental set-up, and data collection will be introduced in this section.

A. Stent Graft Modelling
Previous work, i.e. [6], usually only focused on modelling the stents for EVAR. In FEVAR, the grafts are of equal or greater importance as fenestrations and scallops are on these grafts. Computerized Tomography (CT) could be used to acquire stent 3D shapes but not for grafts due to the poor visibility of the fabric under CT. For fenestrated stent grafts, all parameters including the height, radius, gap, etc. are known via the original stent graft design and hence mathematical modelling is available.
A stent graft is modelled with circles of different radii and set at different heights. Circle vertex was defined by r * cosθ , r * sinθ , h (Fig. 3a). Neighboring vertices were connected by triangles regularly to generate a surface mesh; the resolution in the height was set as 1mm while that in the radial direction was set as 1 • in this paper. The accumulation of these circles made up the graft modelling. To model fenestrations and scallops, vertices within the fenestration or scallop were removed (Fig. 3b).
rcos(2πi/N v ), rsin(2πi/N v ), h sin(2πiN s /N v )/2 was used to model the stent vertices [8], where r = r n + (r x − r n ) * (h sin(2πiN s /N v )/2 + h /2)/h , i ∈ (1, N v ) and N v was the vertex number on a stent. N s was the number of sine wave cycles describing the stent. h was the height of each stent ring (Fig. 3c). For the example in Fig. 3c, r n = 11.5mm, r x = 15mm and h for the six stents from the bottom to top are 17mm, 13mm, 13mm, 16mm, 21mm, 25mm, respectively. In manufacturing, stents cannot lie across fenestrations or scallops and are forced onto fenestration or scallop edges; we modelled these crossed stents onto the nearest fenestration or scallop edges too.

B. 3D Shape Instantiation
The non-rigid deformations of the whole stent graft was split into multiple rigid transformations of the stent segments in this paper. The 3D pose of each stent segment will be instantiated based on the 2D marker fluoroscopic projections. By using the RPnP method, which estimates the pose of a calibrated camera given a set of n 3D points in the world and their corresponding 2D projections in the image, the 3D pose of a stent segment could be instantiated. Compared to the traditional 2D/3D registration, the RPnP method has the following advantages: 1) RPnP is fast as it solves the 3D pose mathematically; 2) RPnP is less ambiguous as it solves the 3D pose based on similar triangles; 3) RPnP only needs 4 points to instantiate a reasonable 3D pose. The correspondences between 3D markers and their 2D projections are supplied by marker placement and detection in this paper.

1) Stent Graft Marker Placement:
RPnP could achieve 3D pose instantiation with minimum n = 4 [9]. We adopted n = 5 for higher robustness. Five markers were sewn at five non-planar positions at each stent segment, as shown in Fig. 3d. The marker position pattern for each stent segment is similar, which is used for later marker classification. Usually gold markers are used in commercial fenestrated stent grafts. We simulated the gold markers by printing on a Mlab Cusing R machine (ConceptLaser, Lichtenfels, Germany) with SS316L stainless steel powder. The marker size was similar to commercially used ones and the thickness guaranteed marker visibility under low radiation fluoroscopy. 2) 3D Pose instantiation of Stent Segments: For n markers on a stent segment with initial known 3D positions (via the original stent graft design): {P 1 , ..., P n }. After its deployment inside an aneurysm, these 3D positions are transformed as: {P 1 , ..., P n }. With the known corresponding marker 2D projections (via fluoroscopy): {p 1 , ..., p n }, the transformation matrix {P 1 , ..., P n } = Tran × {P 1 , ..., P n } could be instantiated by solving a RPnP problem [9].
Firstly, a rotation axis was selected to reduce the unknown variables -here the Z axis was chosen. Secondly, the PnP problem was divided into (n−2) P3P problems with equation system: Here k = D 2 /D 1 , D 1 , D 2 , D 3 denotes the triangle side lengths |p 0 p 1 |, |p 0 p 2 |, |p 1 p 2 |, as shown in [10].
→ v 2 are also as shown in [10]. x was solved by the local minimum of Thirdly, the depth of each vertex was determined by perspective similar triangles. Fourthly, the rotation along the Z axis with c = cosα, s = sinα with angle α and translation t x , t y , t z of the intraoperative markers was solved by [9]: Finally, the solved transformation matrix of the intraoperative markers was normalized by a standard 3D alignment based on Least-Squares Estimation [11]. This normalized matrix was the 3D pose of the marker set and the corresponding stent segment.
C. Non-rigid Deformation Instantiation for Whole Stent Grafts 1) Continuous Constraints for Stent Segments: In theory, the RPnP method instantiates both the position and pose accurately. In our validation, the manually sewn markers drifted and the stent segments experienced small non-rigid deformations during repeated stent graft compression and deployment. These practical issues added minor error on the instantiated position and pose. The position drift of stent segments influenced the continuity of the entire stent graft and was corrected by applying continuous constraints on the central points. It is expected that these issues will not be a problem clinically with accurate manufacturing and only a single deployment of the stent graft in vivo.
2) Graft Gap Interpolation: After instantiating the pose and correcting the position drift for each stent segment, the graft gaps between neighboring stent segments were interpolated by: where T ∈ (1 • , 360 • ) controls the twisting and rotating of a circle. θ ∈ (1 • , 360 • ) is the angle of a vertex. r i is the radius. In the above equations, c θ +T represents cos(θ + T ) and c Ω represents cos(Ω). s θ +T represents sin(θ + T ) and s Ω represents sin(Ω). Ω is the angle between the circle normal and the xy plane (Fig. 3a). α, β , δ is the cross product of the circle normal and 0, 0, −1 and controls bending.
x i , y i , z i controls translating. The circle normals and positions of graft gaps were interpolated linearly by the normals and positions of neighboring stent segments.

D. Semi-automatic Marker Detection with Deep Learning
Semi-automatic marker detection including automatic marker segmentation and manual marker classification was used to find the correspondences between the 3D markers and their 2D projections. After segmenting all markers by Focal-Unet, they were classified manually into different marker types and stent segments. Focal-Unet -Unet [12] trained with focal loss [13] -was proposed in this paper to segment the markers automatically from 512×512 2D fluoroscopic images. Unet is a widely applied convolutional neural network for image segmentation, and has demonstrated high accuracies in medical problems. It manages to formulate image segmentation into pixel-level classification, and hence to enable an end-to-end training procedure. The Unet structure in this paper is shown in Fig.  4. All convolutional layers were with zero padding and a stride of 1. All Max pooling layers were with a stride of 2.
Softmax with cross-entropy loss is usually applied for nooverlap segmentation (no overlap between classes) in Unet . For binary segmentation, it is defined as: where y = 1 is the foreground or markers, y = 0 is the background. p ∈ [0, 1] is the probability of that pixel to be foreground. Equ. 7 could be rewritten as loss cross−entropy (p, y) = loss cross−entropy (p t ) = −log(p t ), where The distribution of two classes in our case -foreground or markers and background -are extremely imbalanced, as markers are small and only take up to approximately 0.1% of the total pixels in the whole image. Weighted loss which adds much higher weights to the foreground loss was usually applied to solve this kind of class-imbalance problem: loss weight (p t , y) = −w y log(p t ). However, its underperformances in extremely high class-imbalanced problems, i.e. our case. So focal loss [13] is applied instead in this work: where γ is the pow coefficient. Empirically in our experiments, w y was set as 30 for foreground and 1 for background, γ was set as 2. Focal loss is significantly advantageous over weighted loss: instead of assigning constant weights for foreground and background pixels, focal loss separates "easy" (correctly classified) and "hard" (incorrectly classified) pixels dramatically and automatically in individual iterations, which enables the network to "gaze" at hard pixels by greatly reducing the loss contribution of easy pixels. To provide promising parameter initialization for Focal-Unet, the training procedure was divided into two steps: 1) the network was trained with weighted loss; 2) the model trained with weighted loss was used as the initialization for the training with focal loss.
The segmented markers were manually classified into the five types based on the position pattern, as shown by the colorful bounding boxes in Fig. 3d. In practice, fluoroscopy is usually scanned in a coronal or oblique plane, which enables division of the markers into the corresponding stent segment by their vertical positions, as shown by the white dividing lines in Fig. 3d.

E. Experimental Setup and Data Collection
1) Simulation of FEVAR: Five abdominal aneurysm phantoms, created from contrast-enhanced CT data of AAA patients were printed on a Stratasys Objet (MN, USA) in VeroClear and TangoBlack, one example is shown in Fig. 5a. Three stent grafts: iliac (6 − 10mm diameter, 90mm height, Cook Medical), fenestrated (22 − 30mm diameter, 117mm height, Cook Medical) and thoracic (30mm diameter, 179mm height, Medtronic, MN, USA) were used in the experiments. Each stent segment of the three stent grafts was sewn on five markers. The initial 3D marker positions were acquired by a CT scan. In a setup, a stent graft was compressed within a Captivia delivery system (Medtronic, 8mm diameter, shown in Fig. 5a), inserted into the 3D printed aneurysm, deployed at the aneurysm position.
2) Data Collection: 14 different setups were collected. For each setup, 13 2D fluoroscopic images from different view angles, varying from −90 • to 90 • with 15 • interval were obtained on a GE Innova 4100 (GE Healthcare, Bucks, UK). A cone-beam CT scan was also collected for each setup with the same scan machine. The coordinates of marker projections on 2D fluoroscopic images were transformed into the CT coordinate system, as shown in Fig. 5b. Manual marker detection was performed in 3D Slicer [14]. 11 setups with 13 fluoroscopic images for each were used as the training data for marker segmentation. 1/10 of the training data were used as the validation set, and the rest were used as the training set. The ground truth for marker segmentation was labeled in Analyze (AnalyzeDirect, Inc, Overland Park, KS, USA). The training and validation data were augmented by rotating and flipping to about 5000 images. 3D Slicer [14] was used to segment the stent 3D shape from the CT scan as the ground truth for 3D shape instantiation. The average unsigned distance between the instantiated 3D shape and the ground truth was calculated in CloudCompare [15]. 3D shape instantiation and semi-automatic marker detection were validated with errors shown in this section. Firstly, with manual marker detection for four setups (two for illiac stent graft, one for fenestrated stent graft and one for thoracic stent graft), the average distance and angle errors of 3D shape instantiation are illustrated. Secondly, with semi-automatic marker detection for three setups (one for illiac stent graft, one for fenestrated stent graft and one for thoracic stent graft), the marker segmentation results, the marker detection accuracy and the 3D shape instantiation accuracy are presented. The average distance error is the average unsigned Euclidean distance between the instantiated 3D stent graft and the ground truth. The average angle error is the average unsigned angle (θ in Fig. 3a) difference between the instantiated markers and the ground truth. Angle errors were measured, as the facing and orientations of fenestrations or scallops are important for path planning (red, green, blue path in Fig. 1d) in robot-assisted FEVAR. Lastly, the instantiated 3D shape details and a comparison between 3D shape instantiation based on manual and semi-automatic marker detection are stated. Fig. 6. 3D Shape instantiation errors (mean ± stdev) of distance (left) and angle (right) for three stent grafts (1-iliac; 2-fenestrated; 3-thoracic). Markers were detected manually.

A. 3D Shape Instantiation with Manual Marker Detection
Four setups with 13 2D fluoroscopic images for each were used. The 3D shapes were instantiated in all 52 cases, as shown in Fig. 6. The average distance error of instantiated stent grafts increased with the stent graft size but was still within approximately 3.5mm. The average angle error of instantiated markers was about 4 • which was not influenced by stent graft size.

B. 3D Shape Instantiation with Semi-automatic Marker Detection
Three setups with 13 2D fluoroscopic images for each were used. One example of the marker segmentation is shown in Fig. 7. Unet trained with weighted loss missed many markers and introduced many false positives while the proposed Focal-Unet segmented all markers correctly. The average distance error of segmented marker centers are shown in Fig. 8. It is around 0.15mm on average for all cases.  The segmented markers were classified manually and their centers were used to instantiate the 3D shapes in all 39 cases, as shown in Fig. 9. The average distance error was less than 3mm and increased with the stent graft size. The average angle error was not influenced by stent graft size and was around 4 • .

C. Instantiated Details and Static Evaluations
Examples of 3D shape instantiation coloured by distance error are shown in Fig. 10a -the light grey mesh is the proposed shape instantiation result while the coloured stents are the ground truth. It can be seen that the bending, compressing, twisting, etc. of the stent graft, the scallops or fenestrations are instantiated well. Examples of the instantiated scallop and fenestration (Fig. 10b top) are compared to the real ones (Fig. 10b bottom). The dark grey stents in Fig. 10b top are the ground truth from CT with commercial markers indicating the scallop and fenestration.
The average distance errors and the average angle errors for the seven setups are shown in Tab. I. The distance errors are influenced by the size of stent grafts while the angle errors are not. The accuracy of 3D shape instantiation with semi-automatic marker detection are similar to that with manual marker detection. Fig. 9. 3D Shape instantiation errors (mean ± stdev) of distance (left) and angle (right) for three stent grafts (1-iliac; 2-fenestrated; 3-thoracic). Markers were segmented automatically and classified manually. The computational time is less than 8ms in MATLAB for instantiation of the shape of one stent segment on an Intel(R) Core(TM) i7-4790 CPU@3.60GHz computer. The marker segmentation takes less than 0.1s in Tensorflow on a NVIDIA TITAN Xp GPU. For training Focal-Unet, the first step takes about 30 minutes while the second step takes almost 2 hours on the same GPU.

IV. DISCUSSION
In this paper, the non-rigid deformations of the whole stent graft was split into piecewise stent segment rigid transformations and then was instantiated by interpolating these instantiated stent segments. The average distance error of instantiated stent grafts at around 1−3mm and the average angle error of instantiated markers at around 4 • illustrates that this splitting is reasonable and could be used for future work on stent grafts. The average distance error of stent grafts -3mm are comparable, as the thickness of the stents and markers are approximately 1mm. Even if all the stent segments in our experiments have experienced different levels of non-rigid deformations due to the repeated compression and deployment, comparable average distance and angle errors were achieved. In practical applications, the fenestrated stent graft is in a good condition and is compressed and deployed only once, hence the accuracy is expected to be higher than the experiments in this paper.
The only input for the proposed 3D shape instantiation is a single fluoroscopic image of markers, which decreases the X-ray radiation to a minimum, as markers are always visible, albeit not always clearly, even under lowest X-ray radiation. The stents, 3D printed aneurysms, and the holders all show up in the 2D fluoroscopic images in our experiments - Fig.  7. In practice, the 2D fluoroscopic images, i.e. Fig. 1c, are much more 'clean' than our experiments due to the block of tissue. The commercial markers made of gold also have higher visibility than the 3D printed markers made of steel used in this paper. It will be easier to segment and classify the markers in practical applications.
Focal-Unet was proposed in this paper, which achieved promising results in marker segmentation without any image pre-processing, compared to the use of the Unet with weighted loss. The achieved IoU (Intersection Over Union) is approximately 0.5. This is reasonable, as the number of pixels of each marker is very small; a small area of incorrectly segmented background will decrease the IoU greatly. The accuracy of segmented marker centers (0.15mm) and the proportion of missed markers (none was missed in all 39 cases) are more important. The markers are classified manually in this paper as the marker shapes are not currently designed into different shapes. In our test, markers with obvious different shapes -the black bounding boxes in Fig.  3d -could be classified automatically by Focal-Unet. With improved marker shape design, fully automatic determination of point correspondence can be achieved.
With regards to 3D shape instantiation accuracy (Tab. I), that with semi-automatic marker detection is slightly higher than the accuracy with manual marker detection. This minor accuracy improvement was due to the different software used for labeling (Analyze was used for manual marker detection while 3D Slicer was used for segmenting the training data in semi-automatic marker detection). The experiments also demonstrated the robustness of the proposed framework to fluoroscopic view angles -fluoroscopic images from 13 view angles were tested and shown with no difference in accuracy. The running time of 0.1s per image indicates that the proposed framework can work in real-time; typical fluoroscopy acquisitions used in clinic are approximately 2-5 frames per second.

V. CONCLUSION
A 3D shape instantiation framework for fenestrated stent grafts including marker placement, stent segment pose instantiation, stent graft shape instantiation and semi-automatic marker detection was proposed in this paper. The proposed framework only needs a single fluoroscopic image and is only based on markers, which decreases the X-ray radiation to a minimum. Compared to the current 2D fluoroscopy navigation used in robot-assisted FEVAR procedures and previous work, the proposed framework instantiates not only the 3D shapes of the stents but also the grafts, fenestrations and scallops. In the future, markers will be designed into different shapes to achieve fully automatic marker detection. This work is a first step towards a complete 3D shape instantiation which predicts the 3D shape of a fenestrated stent graft after the deployment from a single 2D fluoroscopic image of its compressed state to improve robotic navigation for FEVAR.