Application of Dual Modality Contrast Agent Combined with Multi-Scale Representation in Ultrasound-Magnetic Resonance Imaging Registration Scheme

To achieve the image registration/fusion and perfect the quality of the integration, with dual modality contrast agent (DMCA), a novel multi-scale representation registration method between ultrasound imaging (US) and magnetic resonance imaging (MRI) is presented in the paper, and how DMCA influence on registration accuracy is chiefly discussed. Owing to US’s intense speckle noise, it is a tremendous challenge to register US with any other modality images. How to improve the algorithms for US processing has become the bottleneck, and in the short term it is difficult to have a breakthrough. In that case, DMCA is employed in both US and MRI to enhance the region of interest. Then, because multi-scale representation is a strategy that attempts to diminish or eliminate several possible local minima and lead to convex optimization problems to be solved quickly and more efficiently, a multi-scale representation Gaussian pyramid based affine registration (MRGP-AR) scheme is constructed to complete the US-MRI registration process. In view of the above-mentioned method, the comparison tests indicate that US-MRI registration/fusion may be a remarkable method for gaining high-quality registration image. The experiments also show that it is feasible that novel nano-materials combined with excellent algorithm are used to solve some hard tasks in medical image processing field.


Introduction
Because US is high efficient, easy to operate and maintenance, it is widely used in clinical application of the city and the countryside. At present, US is greatly improved owing to the use of contrast agents [1][2][3][4]. However, because of imaging principle of US, the quality of US is very general compared its contrast and resolution with that of MRI or computed tomography (CT). MRI is another applied generally imaging modality with desired soft-tissue contrast and high-quality spatial contrast and resolution; in addition, in particular MRI can offer functional information required by the clinical diagnosis. The fatal defect of MRI is that it can't provide real-time motion-related images.
To sum up, no single imaging modality possesses all the virtues fulfilling various clinical needs, and various imaging modalities own their respective merits and defects in clinical application. Under many circumstances, it is complimentary between MRI and US. Accordingly, and it is ideal to fuse US with MRI. To fuse US and MRI together, firstly US-MRI registration is needed. For the moment, Owning to US's strong noise and unclear background, to register US with any other imaging modality is a significant challenge. Though researchers have made great efforts to improve algorithms for medical image registration, they made very little progress in this field [5][6][7][8][9][10]. We have done preliminary work on MRI-US registration/fusion based on DMCA, and have obtained some results [11][12][13][14].
At present many scholars have shown interest in DMCA and have made great progress. The abovementioned DMCA is the dual-modality contrast agent, which holds contrast function property for both US and MRI at the same time. Superparamagnetic iron oxide nanoparticles (SPIO) can be used as an efficient contrast agent for MRI, while microbubbles can be used as a powerful contrast agent for US. The combination of SPIO and microbubbles, DMCA, can be used as the dual-modality contrast agent for both US and MRI because the DMCA can enhance the merits and decrease the defects of SPIO or microbubbles, respectively. Wang et al. [15,16] developed a new class of uniform biodegradable yolkshell Fe 3 O 4 @PFH@ PMAA-DOX microspheres as US/MRI dual-modality imaging contrast agents and drug delivery system, and the multifunctional biodegradable microspheres are safer for normal tissues and more beneficial in actual clinical applications. Song et al. [17,18] prepared superparamagnetic selfassembled microbubbles consisting of "Poly(acrylic acid)-Iron oxide nanoparticles-Polyamine" sandwichlike shells and tetradecafluorohexane cores were fabricated by a template-free self-assembly approach, showing great potential as US/MRI dual contrast agents. Morch et al. [19] developed nanoparticlestabilized microbubbles for multimodal imaging and drug delivery, showing that these microbubbles can act as contrast agents for conventional ultrasound imaging. Successful encapsulation of iron oxide nanoparticles inside the poly butyl cyanoacrylate nanoparticles is demonstrated, potentially enabling the nanoparticle-microbubbles to be used as MRI/US contrast agents. Wang et al. [20] developed a dualenzyme-loaded multifunctional hybrid nanogel probe (SPIO@GCS/acryl/biotin-CAT/SOD-gel, or SGC) for dual-modality pathological responsive US and enhanced T2-weighted MRI. This probe is composed of functionalized superparamagnetic iron oxide particles, a dual enzyme species (catalase and superoxide dismutase), and a polysaccharide cationic polymer glycol chitosan gel. Wu et al. [21] adopted a premix membrane emulsification (PME) method to prepare uniform PEGylated poly (lactic-co-glycolicacid) microcapsules with superparamagnetic Fe 3 O 4 nanoparticles embedded in the shell (Fe 3 O 4 @PEG−PLGA MCs) for US/MRI. Multimodal contrast agent (MMCA) or DMCA has been used preliminarily in clinical practice. Zheng et al. [22] introduced an MMCA, which may serve as a valuable tool for cardiovascular imaging as well as image registration and guidance applications in radiation therapy. Kuznetsova et al. [23] assessed the performance of structure-guided deformable image registration (SG-DIR) relative to rigid registration and DIR using TG-132 recommendations. The assessment was carried out for image registration of operation planning CT and MRI scans with Primovist contrast agent obtained post stereotactic body radiation therapy. Piskunowicz et al. [24] presented some cases of pediatric patients treated in oncological departments, with the use of ultrasound contrast agents, the examination had a considerable influence on the diagnostic and therapeutic process. It showed that imaging with contrast agents could help solve some clinical problems when other diagnostic methods failed. Literature review shows that registration based on MMCA OR DMCA is still rarely studied.
The important contribution of the paper is the introduction of MRGP-AR algorithm combined with the above-mentioned novel nano-materials DMCA to US-MRI registration [21,25,26], and focuses on the research of how DMCA influences on registration accuracy between US and MRI. Employing DMCA prepared by Yang et al. [1], based on MRGP-AR scheme, this paper carries out the research on registration accuracy of US-MRI, and gets a conclusion that with the use of DMCA, the proposed MRGP-AR method works well. The rest of the paper is arranged as shown below: US-MRI registration scheme (MRGP-AR) is demonstrated in Section 2. Section 3 provides comparison experiments on registration accuracy with/without DMCA, and discusses the experimental results, while Section 4 summarizes our paper.

Registration Method
During registration process, it is essential for interpolation transformation of floating image, and the diagram of cubic convolution interpolation is given in Fig. 1. In the registration process, to enhance the region of interest, DMCA is injected into phantom, and US and MRI with/without DMCA are obtained. Then MRGP-AR scheme is constructed to complete the US-MRI registration process. Finally, affine transformation parameters are estimated and used to register the US, and how DMCA influence on registration accuracy is chiefly discussed by multiple comparison experiments with/without DMCA. The diagram of the proposed registration method MRGP-AR combined with/without DMCA is shown in Fig. 2, and multiscale scheme is introduced as shown in Fig. 3. The basic steps of MRGP-AR are as followed: Step 1: Build the multi-scale representation by cubic convolution interpolation at different scales (Section 2.1).
Step 2: Carry on affine registration at the most coarse scale p 0 (Section 2.2), get the solution u p 0 .
Step 3: For i ¼ 1; . . . ; n, at each scale p i , carry on affine registration and compute the similarity measure for evaluation of the registration approach, u p iÀ1 as the initial guess, then get the solution u p i .
Step 4: Letũ be the solution at the final scale, that is, the affine transformation parameters are estimated u ¼ u p n .

Cubic Convolution Interpolation Based Multi-Scale Representation
To build Gaussian pyramid, down-sampling is needed. At each scale registration, cubic convolution interpolation is introduced to transform the floating image. On the basis of grey scales of all the sixteen pixels in a small neighboring area around the reverse transformation point p, weighted mean value of p is computed according to a certain weighting coefficients, and the gray value of the inverse transform point is interpolated. Schematic diagram of cubic convolution interpolation is shown as Fig. 1.
Suppose the floating image is mapped to the reference image in reverse direction, a transformation point is obtained, and its coordinates are i þ u; j þ v ð Þ , where i and j are positive integers, u and v are pure decimals of the [0,1) interval. The value of f i þ u; j þ v ð Þcan be determined by the gray values of the 16 pixels of the p centered neighborhood in the original (or reference) image. The formula is followed as shown in (1). where In (5), s w ð Þ is a weighted interpolation coefficient function.

Affine Registration
We introduce a particular rigid-like type of affine transformation ', which is a composition of scaling, rotation and translations, defined by (6).
where ' is the solution of the following optimization problem as shown in (7).
In (6), x 0 y 0 and x 1 y 1 are initial coordinate and transformation coordinate, respectively. x 0 represents the scale, which is set to 1 and ignored in the next sections. h is the rotation angle. Dx and Dy denote the translations on the x-axes and y-axes, respectively.
At coarse scales, the corresponding input representations preserve only the main and global features of the images, and successively, at finer scales these representations contain more and more details. At coarse scales, because of down-sampling, the size of start images is relatively small, and the global registration parameters are relatively easy to be estimated, and then the solution of the registration problem at one scale is the starting guess for the registration problem defined at the next finer scale, where the representation of the data (reference and floating images) is obtained with the next scale and shows more details. This multi-scale representation is a strategy that attempts to diminish or eliminate several possible local minima and lead to convex optimization problems to be solved quickly and more efficiently [27][28][29].

Results and Discussion
Phantom and DMCA are made and acquired from Jiangsu Key Laboratory for Biomaterials and Devices. Phantom is prepared from glycerol, agar and water ratio of 3:4:90, in which a "U" shaped silicone tube is "vertically" set to sit in the agar phantom. The production process of DMCA can be consulted from relevant literatures [1,2,12], and DMCA can negatively boost MRI T2-weighted (T2*WI) imaging signal; on the contrary, it can positively reinforce ultrasound backscattering echo intensity and boost the contrast and brightness of US.
With/without DMCA, for US, the phantom is imaged by using the GE LOGIQ3 PRO equipment with a 4 MHz ultrasound transducer; For MRI, the phantom's T2*WI imaging is carried out by using a 0.3 T magnetic resonance equipment (AIRISII, Hitachi Ltd., JAPAN).
Two-dimensional MRI is used as reference plane, while an image generated by an affine transformation of a parameter a ¼ Dx; Dy; Dh ð Þbased on a two-dimensional ultrasonic plane is used as a floating image, and comparison tests of registration accuracy with/without DMCA are carried on. For the parameter a, the coordinate system is defined as shown in Fig. 4. Dy is vertical and upward on Y-axes direction, Dx is horizontal and left on X-axes direction, and Dh is anticlockwise in rotation direction, respectively. For the parameter a, the unit of Dx and Dy is pixel, and the unit of Du is degree, namely p 180 . In the following experiments, a is set to (3, 3, 3), (8,8,8), (11,11,11) and (15,15,15), respectively.

Registration Results with the Proposed Improved Affine Transformation (MRGP-AR)
Using MRGP-AR, comparison results of registration accuracy without/with DMCA are as shown in Figs. 5 and 6, respectively, and the registration accuracy comparison results, namely corresponding transformation parameters, are computed quantitatively as shown in Tab. 1.
The MRI or US imaging of a "U" shaped silicone tube in the phantom is called region of interest (ROI). Obviously, the ROI of Fig. 6a is darker than that of Fig. 5a because DMCA can negatively enhance MRI T2*WI imaging; similarly, the ROI of Fig. 6b is brighter than that of Fig. 5b because DMCA can positively boost the contrast and brightness of US. Besides, intuitively, the detail information the ROI of Figs. 6a and 6b is more abundant than that of Figs. 5a and 5b.

Registration Results with Traditional Affine Transformation (TAT)
Using TAT [30][31][32][33], comparison results of registration accuracy without/ with DMCA are as shown in Fig. 7, respectively, and the corresponding transformation parameters are computed as shown in Tab. 2.
From Fig. 8 and Tab. 3, the angle error obtained without using DMCA is obviously larger than that obtained with DMCA, which shows that DMCA is beneficial to optical flow field registration algorithm. Briefly speaking, from Tabs. 1-3, with the increase of the parameters a values from (3,3,3) to (15,15,15), no matter whether MRGP-AR, TAT or optical flow field method is used, and no matter whether DMCA is added or not, there is a growing trend of the deviations between the set parameters and the corresponding solved transformation parameters because the greater the parameters are set, the more difficult the registration is. The fundamental cause is that as the offset between the floating image and the reference image increases, search space increases during registration. The registration process is more likely to fall into local optimum, resulting in the final solution is not ideal.
Comparison between Tabs. 1 and 2, no matter whether DMCA is added or not, for MRGP-AR the better registration accuracy and less deviation can be obtained compared with that of TAT. The fundamental cause lies in multi-scale representation based the construction of Gaussian pyramid. By the above coarse-to-fine Gaussian pyramid, at the coarse level, we can roughly calculate the transformation parameters as a whole, and the computed affine parameters are used as initialization parameters at fine level. Beginning from the top story, calculation is conducted downward story by story, the computed affine transformation parameters successive approximated of real values, which can also be intuitively seen in Fig. 3. To sum up, without DMCA, for MRGP-AR, TAT and optical flow field method, the calculated registration accuracy is not ideal. However, the proposed method, MRGP-AR combined with DMCA, higher registration accuracy is achieved compare to that using TAT or optical flow field method with DMCA. It can also be intuitively seen from Figs. 5-8 that by using DMCA, the contrast and brightness of US and MRI are improved compared with not using DMCA, which is beneficial to subsequent registration.

Conclusions
Based on MRGP-AR algorithm combined with the novel nano-materials DMCA, US-MRI registration is carried out, and some conclusions are drawn as follows.
Firstly, for US-MRI registration, large amount of data, long running time, and easy to fall into local minimum need to be solved urgently. The multi-scale representation is a strategy that attempts to diminish or eliminate several possible local minima, and lead to convex optimization problems to be solved quickly and more efficiently. The multi-scale Gaussian pyramid regards the situation as a whole, and then deals with the local details, At coarse scales, the global registration parameters are relatively easy to be estimated, and then the solution of the registration problem at one scale is the starting guess for the registration problem defined at the next finer scale, etc. The multi-scale representation may be combined with other methods to handle complex optimization problems.
Secondly, for medical image registration, especially US registration involved with strong speckle noise, it is a great challenge and difficult task, and only by improving algorithm to advance registration results becomes harder and harder. Since it is hard for US de-noising, instead of only improving algorithm models to de-noise US, nano-materials are introduced to enhance the contrast and brightness of US, which is equivalent to de-noise US in a sense. In a word, it is feasible that novel nano-materials combined with excellent algorithm models are used to solve some difficult problems in medical image field. Nowadays, since it has become the bottleneck to only improve the algorithms for medical image registration, nano-materials could be introduced to enhance medical imaging, which will be beneficial to the following registration.
Lastly, for medical images processing (US-MRI registration included), algorithm modeling stage or images processing stage should not be only focused on, and attention must be paid to the imaging stage. High quality imaging is essential for follow-up work, and comparison experiments also demonstrate the above conclusions.
Funding Statement: This project is financially supported by Xuzhou's special funds for science and technology innovation (KC18008, 2018). Funding also partially comes from Jiangsu Overseas Visiting Scholar Program for University Prominent Young & Middle-aged Teachers and Presidents.

Conflicts of Interest:
The authors declare that they have no conflicts of interest to report regarding the present study.