Simultaneous pose estimation and patient-specific model reconstruction from single image using maximum penalized likelihood estimation (MPLE)☆
Graphical abstract
Introduction
Registration and reconstruction of a deformable shape are coupled problems, solving either one will appreciably facilitates solving the other. In some medical applications, aligning peri-operative 3D data with intra-operative 2D image(s) is one of the crucial steps for image-guided intervention, surgery and therapy [1]. Registration of the 3D data with respect to 2D image(s) (called pose estimation or 3D–2D registration) provides a mapping between the 3D data and patient anatomy. When a patient׳s tomographic image is used as the 3D data, pose estimation generally recovers a rigid transformation as the anatomy of interest has negligible or no deformation. Otherwise, a patient-specific model needs to be constructed from 2D image(s) along with pose estimation. This requires to optimize the shape for the patient-specific model and the pose for mapping the model to patient anatomy with sufficient accuracy. Various methods have been proposed to reconstruct patient-specific models from 2D images, based on either image intensity or image contour. The former methods [2], [3], [4], [5], [6], [7] maximize certain intensity similarity metric between 2D images and digitally reconstructed radiographs (DRRs) generated from a 3D model. Their major differences lie in the 3D model, similarity metric and optimization method used. It is known that these methods have high computational load and a statistical volumetric model is not easy to build. Moreover, additional structures (e.g., irrelevant anatomies) and foreign objects (e.g., surgical instruments) that present in 2D images but not in volumetric model may introduce big challenges to an accurate pose estimation and model reconstruction.
Contour-based methods minimize a distance metric between image contours and model׳s apparent contours. Mathematically, given unmatched 2D and 3D points, the objective is to solve the minimization problemwhere are image points, are apparent contour points, denotes a deformation with parameter , and represents a perspective projection following an affine transformation with parameter that consists of a scale and a 6-DOF rigid transformation . The assignment function assigns a paired correspondence between a 2D point and a 3D point; equals one if corresponds to and zero otherwise.
The presence of deformation renders the pose estimation ill-conditioned. To deal with this, a common approach is to regularize using a-priori knowledge represented by a statistical shape model (SSM) built from T triangular meshes. An “instance” of the SSM is approximated aswhere is the mean shape, the shape parameter, the kth mode, and K chosen based on accumulated variance.
As pose estimation and reconstruction of a deformable shape are coupled problems, previous methods [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19] proposed to solve them via consecutive stages. The first stage establishes paired correspondences and estimates pose from them, both using only the mean shape. Manually identified correspondences, variants of iterative closest point (ICP) and intensity-based registration were used in this stage. Then, fixing at the estimated pose and using the paired correspondences, the second stage constructs an instance via a SSM-regularized surface deformation by minimizing 2D contour-to-contour distance or 3D point-to-line distance. Adapted gradient descent, pattern search and preconditioned conjugate gradients in a trust region were used. To achieve a better shape reconstruction, some methods have a third stage that minimizes local free-form deformations using a first-order Gauss–Markov process or a surface fitting with thin plate splines as the smoothness constraint, still fixing at the pose and using the paired correspondences obtained in the first stage. In [13], [17], [18], contour normals or image gradient was also used to alleviate false paired correspondences. These methods require delineated image contours or a good initialization.
When estimating pose using the mean shape, deformations (morphological differences) are considered as random noise that contaminates the mean shape. However, the shape deforms under the constraint of SSM rather than randomly. Thus the two-stage procedure may result in a registration biased by deformations and a reconstruction involving an improper pose. Consequently, as the optimization does not fully take into account the SSM, i.e., (1) is not minimized subject to (2), the accuracy of the pose estimated in the first stage will be sacrificed. With this incomplete pose estimation, the reconstruction from the registered mean shape does not describe purely shape deformations. In addition, the convergence of previous methods is local, requiring an initialization near the global optimum. Hence, the coupled pose estimation and shape reconstruction should be essentially formulated as one optimization problem and solved simultaneously using a global optimizer.
The most closest method is the one for 3D–3D rigid point set registration [20], where the pose was estimated using the semi-definite positive (SPD) relaxation. However, it is designed for rigid rather than deformable objects. Most importantly, it cannot be directly extended to 3D–2D registration as the nonlinear perspective projection renders the formulation of the SPD relaxation invalid.
We present a novel contour-based method for simultaneous pose estimation and patient-specific shape reconstruction from a single 2D image. This work is a significant extension of our point-based rigid 3D–2D registration method [21]. An early version was briefly introduced in an application [22], [23], but without derivation and incorporation of image gradient. This paper provides a complete and detailed derivation on how the simultaneous pose estimation and surface reconstruction are rigorously formulated as a MPLE problem, which is unobvious, in Section 2. Then, intensive evaluations were performed under various initializations in Section 3 and results were discussed and compared with state-of-the-art methods in Section 4, followed by conclusions and future work in Section 5.
Section snippets
Problem statement
We do not directly solve the minimization problem (1). Instead, we formulate the problem as a statistical inference of the pose and shape parameters, without establishing paired correspondences.
For image contour points, we form N dyads (4-vectors) , where denotes the location of the nth image contour point and , denotes the image gradient at . For apparent contour vertices of an instance transformed by , we form M dyads (6-vectors) , where
Experiment and results
The evaluation is to examine the stability and accuracy achievable with our method under various initial conditions using DRRs. The DRRs were generated using our “virtual C-arm” software that simulates a distortion-free flat-panel cone beam CT imaging bench [37]. The source-to-detector distance dSD is 1184 mm and the image resolution is 1024×768 pixels. In practice, these parameters can be obtained via C-arm calibration.
The probabilistic correspondence
Almost all feature-based patient-specific model reconstruction methods are derived from the ICP, making use of image contours and apparent contours of SSM. They assumes that there are paired correspondences between the two types of contour, and as such, dedicate to singling out a set of “best” paired correspondences, based on which both rigid registration and SSM regularized deformation are carried out.
The ICP-based methods assume a position noise that corrupts the data, as illustrated in Fig. 4
Conclusions and future work
We propose a novel method for simultaneous 3D–2D pose estimation and surface reconstruction of a deformable object, which are two intrinsically coupled problems. Unlike previous methods that solve a simplification of the coupled problems as a two-stage procedure in two consecutive steps, the proposed method formulates the two coupled problems as one MPLE problem and solves it simultaneously. In addition, the proposed method utilizes a global optimizer whereas previous methods used local
Conflict of interest
None declared.
Xin Kang received his Ph.D. from The University of Hong Kong, Hong Kong, China. Before that, he served as a Senior Lecturer at the Department of Electrical and Electronic Engineering. His current research interests include pose estimation, image analysis, visual tracking and image fusion.
References (47)
- et al.
3D/2D registration and segmentation of scoliotic vertebrae using statistical models
Comput. Med. Imaging Graph.
(2003) - et al.
A 2d/3d correspondence building method for reconstruction of a patient-specific 3d bone surface model using point distribution models and calibrated x-ray images
Med. Image Anal.
(2009) - et al.
2D–3D shape reconstruction of the distal femur from stereo X-ray imaging using statistical shape models
Med. Image Anal.
(2011) - et al.
Statistical shape models for 3D medical image segmentationa review
Med. Image Anal.
(2009) - et al.
A mixture model for representing shape variation
Image Vis. Comput.
(1999) - J. Yao, R. Taylor, Assessing accuracy factors in deformable 2D/3D medical image registration using a statistical pelvis...
- et al.
Deformable 2D–3D registration of the pelvis with a limited field of view, using shape statistics
- et al.
2D/3D deformable registration using a hybrid atlas
- et al.
Reconstructing the 3d shape and bone mineral density distribution of the proximal femur from dual-energy x-ray absorptiometry
IEEE Trans. Med. Imaging
(2011)
Automatic inference and measurement of 3d carpal bone kinematics from single view fluoroscopic sequences
IEEE Trans. Med. Imaging
Personalized x-ray reconstruction of the proximal femur via intensity-based non-rigid 2d–3d registration
Nonrigid 3-D/2-D registration of images using statistical models
Three-dimensional biplanar reconstruction of scoliotic rib cage using the estimation of a mixture of probabilistic prior models
IEEE Trans. Biomed. Eng.
A hierarchical statistical modeling approach for the unsupervised 3-D biplanar reconstruction of the scoliotic spine
IEEE Trans. Biomed. Eng.
3D reconstruction of the human rib cage from 2D projection images using a statistical shape model
Int. J. Comput. Assist. Radiol. Surg.
Statistical shape model-based reconstruction of a scaled, patient-specific surface model of the pelvis from a single standard ap x-ray radiograph
Med. Phys.
Registration of a CT-like atlas to fluoroscopic X-ray images using intensity correspondences
Int. J. Comput. Assist. Radiol. Surg.
Statistical shape model-based femur kinematics from biplane fluoroscopy
IEEE Trans. Med. Imaging
A biplanar reconstruction method based on 2d and 3d contoursapplication to the distal femur
Comput. Methods Biomech. Biomed. Eng.
Rigid and articulated point registration with expectation conditional maximization
IEEE Trans. Pattern Anal. Mach. Intell.
Robustness and accuracy of feature-based single image 2D–3D registration without correspondences for image-guided intervention
IEEE Trans. Biomed. Eng.
Cited by (10)
Visual question answering from another perspective: CLEVR mental rotation tests
2023, Pattern RecognitionMultiple geometry representations for 6D object pose estimation in occluded or truncated scenes
2022, Pattern RecognitionCitation Excerpt :Within the field of 6D object pose estimation, traditional methods either build the relationship between object 3D models and images through hand-crafted features [8] or apply template matching techniques to search the closest pose solution [9–11]. These methods are very sensitive to cluttered background and severe occlusion [12]. In recent years, deep learning technology has been gradually introduced into this field and has shown promising performance improvements.
Fast and automatic periacetabular osteotomy fragment pose estimation using intraoperatively implanted fiducials and single-view fluoroscopy
2020, Physics in Medicine and BiologyAutomatic Tooth Segmentation and 3D Reconstruction from Panoramic and Lateral Radiographs
2020, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Joint Rigid Registration of Multiple Generalized Point Sets with Hybrid Mixture Models
2020, IEEE Transactions on Automation Science and Engineering
Xin Kang received his Ph.D. from The University of Hong Kong, Hong Kong, China. Before that, he served as a Senior Lecturer at the Department of Electrical and Electronic Engineering. His current research interests include pose estimation, image analysis, visual tracking and image fusion.
Wai-Pan Yau received his MBBS from The University of Hong Kong in 1992. He is the Chief of the Division of Sports and Arthroscopic Surgery in the Department of Orthopaedics and Traumatology, Queen Mary Hospital, where he is also a Honorary Clinical Associate consultant. He is a Clinical Assistant Professor of the Department of Orthopaedics and Traumatology in The University of Hong Kong.
He has the Fellowship of the Royal College of Surgeons (Edinburgh) (FRCSE), of the Hong Kong College of Orthopaedic Surgeons (FHKCOS), and of Hong Kong Academy of Medicine (FHKAM). He is also a recipient of numerous awards, including the Harry Fang Gold Medal conjoint examination of FHKCOS and FRCSE (Ortho), the Arthur Yau Award of the 14th Annual Congress of HKOA, the Arthur Yau Award and the David Fang Trophy of the 24th Annual Congress of HKOA, and the Japanese Orthopaedic Association Congress 2008 Fellowship Award.
Russell H. Taylor received his Ph.D. in Computer Science from Stanford in 1976. He joined IBM Research in 1976, where he developed the AML robot language and managed the Automation Technology Department and (later) the Computer-Assisted Surgery Group before moving in 1995 to Johns Hopkins, where he is the John C. Malone Professor of Computer Science with joint appointments in Mechanical Engineering, Radiology, and Surgery and is also Director of the NSF Engineering Research Center for Computer-Integrated Surgical Systems and Technology.
He is the author of over 250 peer-reviewed publications, a Fellow of the IEEE, of the AIMBE, of the MICCAI Society, and of the Engineering School of the University of Tokyo. He is also a recipient of numerous awards, including the IEEE Robotics Pioneer Award, the MICCAI Society Enduring Impact Award, and the Maurice Müller Award for excellence in computer-assisted orthopaedic surgery.
- ☆
This research was supported by Research Grant and RPg Exchange Funding of University of Hong Kong and Johns Hopkins University internal funds.