Aided Surgery ( ACCAS 2018 ) Development of compact portable ultrasound robot for home healthcare

Robotic medical ultrasound can support diagnostics and alleviate fatigues. However, state-of-the-art ultrasound devices are too large and complex for home healthcare. Moreover, organs move in accordance with respiration. This movement changes the ultrasound image pattern and it is difficult for the operator to diagnose easily and accurately. To cope with these problems, the authors newly developed a compact portable ultrasound diagnostic robot for home healthcare, which compensate the organ motion. It can support those who find it difficult to visit a hospital for temporal, spatial, or physical reasons. A robust template matching method to servo the target was applied. Specifically, this robot moves to the target, whose position is detected by finding and identifying the image position of the target in the real-time input images. The authors also applied a multi-threading algorithm with two threads to enhance the real-time performance. One is for image processing with template matching. Another is for robot control to servo the target. Experimental results show that their proposed robot and algorithms can be suppressed to 25.2% motion in pk-to-pk for the periodic phantom organ motion (period of 3 s and of pk-to-pk 40 mm).


Introduction
When compared with computed tomography and magnetic resonance imaging (MRI), ultrasonic (US) equipment/devices can provide excellent lower cost alternatives for real-time non-invasive diagnostics and are widely used clinically on site [1,2].However, although a key characteristic of US diagnosis is the high degree of freedom in its probe maneuvering, that same characteristic requires a skillful operator.Research has therefore been highly active in recent years for the development of artificial intelligence, robotic technology, and the support of appropriate image acquisition in US diagnosis [3].
Hennersperger et al. reported that the authors present a set of methods and a workflow to enable autonomous MRI-guided ultrasound acquisitions.They proposed a structured-light 3D scanner for patient robot and image-to-patient calibration, which in turn is used to plan 3D ultrasound trajectories.These MRI-based trajectories are followed autonomously by the robot and are further refined online using automatic MRI/US registration.Despite the low spatial resolution of structured light scanners, the initial planned acquisition path can be followed with an accuracy of 2.46 ± 0.96 mm [4].
Takachi et al developed a support system for handling ultrasound probe to alleviate fatigue of physician by introducing a coordinated motion with robot.They have developed a support system using ultrasound diagnosis robot, which is able to support manual handling of ultrasound probe in echography to alleviate fatigue of physician.As a result, the system has the potential to be able to support advanced diagnosis for conventional echography [5].
For example, research has been conducted on a US diagnosis robot that would support the acquisition of appropriate diagnostic images by pressing the probe onto the object of interest with appropriate force from an appropriate position and attitude [6].However, the equipment related to those systems is large and lacks portability.This imposes major limitations on the use of such systems and the locations where they can be installed.Therefore, the development of a compact US diagnostic robot that would enable diagnosis at home is desirable for those who find it difficult to visit a hospital for temporal, spatial, or physical reasons.
Probes for general use can be used to image two-dimensional sections, but organs are moved by respiration and the observed surface varies, making diagnosis difficult.Accordingly, a system that can compensate for respiration-induced organ motion is desirable.
In light of this need, we herein outline the concept of a compact US diagnosis robot, propose the framework of an algorithm that can be used to track and compensate for the motion of a phantom modelling the kidney, and report the results of an experiment to verify its effectiveness.

Compact US diagnostic robot
Fig. 1 shows the composition of the developed small, lightweight robot, which were designed to withstand use in the home and other environments.(Table 1) Total weight is 2.6 ± 0.05 kg, including the US probe (3.5 MHz convex probe use) to be used and a spring (elastic modulus 6 N/m) located on the robot tip to maintain a certain minimum force pressing on the probe [7].

Respiration-induced organ motion
The amount of respiration-induced organ motion differs with the patient's physique and age, but kidney displacement is regarded as ranging from ∼ 10 to 40 mm [8].
The normal adult respiratory rate is 15-20/minute [9].For our experimental parameters, we moved the phantom at a constantspeed rate with total 4 (2 × 2) parameters.As for period, we have two kinds of parameters (3, 4 s).As for peek-to-peek (pk-to-pk), we also have two kinds of parameters (10, 40 mm).As for phantom size, our kidney phantom is 108.5 mm.The average size of right and left kidney is 108.5 and 111.3 mm severally for 2068 kidneys in 1040 adults [10].

Robot motion
In this paper, we applied bank motion as a control method so as to realise the continuous robot motion.We have motion parameters (which are resolution of velocity, start speed, target speed, accel time, decel time, S-speed, distance) to control the robot motion.Bank motion also has two kinds of motions (single type and loop type) and can receive a signal of motion parameters real timely and change robot motion simultaneously.

Template matching and region of interest (ROI)
Since a commonly used two-dimensional probe can only observe a single section at a time, it is motion and track the organ under examination from the information in that section.
Normalised correlation is applied in our system.As it is useful and robust for uniform disturbances.This method is less fuzzy to set the parameters but has much amount of computation for each pixel all over the range of the exploration [11,12].
For the image tracking method in this system, we used template matching, which is the process of superimposing the input image onto the template image, and then comparing and collating them to determine if they match [13,14].If the template t(x, y) position is detected from the input image f(x, y), then the detected template t(x, y) overlaps the position of point (u, v) in f(x, y) and both are compared and collated.The normalised (range 0-1) resemblance As for the template size, the tracking failure occurs when template image of the target is too small to detect the image feature and identify the target position.For this reason, we adopted the template image size of 120 × 120 pixels to track and follow a kidney of phantom to enhance the robustness in our experiments.R(u, v) takes a value in the range of 0-1, where a value closer to 1 means a closer resemblance to the template image.Figs.2a and b show, respectively, the input image (749 × 559 pixels) and the image (120 × 120 pixels) of the template extracted from the phantom used in the present experiment (period 3 s, amplitude 10 mm).As the ROI with a high probability of phantom presence, the white rectangle in Fig. 2a was set in that area.As a result, it was possible to reduce processing time and prevent delays in image acquisition caused by excessive image processing in Figs.3a and  b.

Tracking program
The excellent real-time property of the US probe in a normal system is a noteworthy advantage [2].However, when operating a small robot in the Visual C++ program language with main-thread processing alone, intermittent operation during which the motor must wait for the output of image processing results will frequently occur.When such occurs during a tracking operation, the real-time property of the US probe cannot be utilised effectively.
With our system, shown in Fig. 4, image processing is performed by Thread A, while the motor drive is constantly maintained separately in Thread B, and a multi-hread program frame (in which information in the motion direction memory is shared between these separate threads) is adopted.
Shown in Fig. 5a, with single threading, it is performed image processing and motor driving alternately and because the latter must be wait until finished image processing, motion may be delay by the time of image processing.The multi-thread program in Fig. 5b is written to maintain a flow of multi-processing data [15] in which the motor between operations continues to be responsive within 200 μs, which makes it possible to scan in concert with a   US probe at 26.1 fps.Furthermore, the direction of scanning in relation to the probe is determined using the relative displacement X rd as X tm , with the centre of the X-axis X ctr calculated by template matching, as shown by the white dotted line in Fig. 2a, is taken as the centre of the X-axis of the acquired image.The scanning direction information directs the probe to track toward the centre axis of the template.

Box for a phantom
In this experiment, the US probe scans in parallel with the direction of the phantom reciprocating motion, as shown in Fig. 6a.The specification of the phantom used is shown in Fig. 6b.The cushioned phantom housing box (α = 115 mm, β = 145 mm, and γ = 130 mm) for central phantom insertion was formed, and four casters were placed on the bottom to enable smooth contact with the surface when the box is in motion.

Experimental conditions
The robot (SMC-4DL-PCI; Contec Co.) that moves the box is called Device A and the small US diagnostic robot with the mounted US probe is called Device B (same type of system as Device A).Each was given separate movement commands.
The box reciprocating motion parameters are shown in Table 2.These were obtained by measuring the reciprocating movement of the box alone prior to the tracking experiment and by template matching that determined the amplitude and period from the amount of movement of the central axis and the frame count of Fig. 2a template image.The probe frame rate was 26.4 fps.For experimental simplicity, the reciprocating movement speed of the box was determined and applied to the compact US diagnostic robot.
The experimental procedure is as follows.
i.While recording the diagnostic image, determine the position in order to obtain reciprocating motion of the phantom in the display centre.ii.From the recorded image, form the template (120 × 120 pixels) that captures the characteristics of the phantom.iii.Within a given timing, begin the tracking program and perform measurements for 1 min (1587 frames) The experiment was performed once each for each row i-iv in Table 2.

Experimental results and discussions
Fig. 7 shows the pre-and post-tracking experimental results (Not follow, Follow) for row iii of Table 2.The relative displacement was 25.5 mm at tracking initiation and was confirmed to transition at a constant period thereafter.Table 3 summarises the measurement results of experimental parameters i-iv, excluding the 5 s spent in the initial tracking transition.In the tracking results, from rows i to iv, pre-tracking amplitude, post-tracking, peak-topeak, and relative displacement of 24.1, 20.5, 25.2, and 16.0%, respectively, were confirmed.This can be attributed to the enhancement of real-time operation by incorporation of multithread processing with separate image processing and motor motion processing.
Without multi-threading, image processing and motor driving could not work simultaneously.When the last ultrasound image is input, we measure the distance between the identified position of the target kidney phantom and the centre line of the input ultrasound image in the X direction.The robot moves based on the measured distance until the next ultrasound image is input.This algorithm follow is not continuous and intermittent only with single threading.(Fig. 8) Therefore, the robot motion is also not continuous and intermittent.
With single threading, the robot motion is not smooth; therefore, the parts of a rack and a pinion might become fragile by intermittent starting and stopping motions.
Additional reasons may be the small input image-pattern variation for the template image and the constant capture of the phantom in the probe display.The OpenCV library function matchTemplate was used here as the image tracking method.Since this function returns similarity seeking unit values in 1-pixels units, it may be necessary to mount a subpixel-level tracking system in order to further increase the tracking accuracy.
There are some methods for template matching, such as the Sum of Squared Difference method and the Sum of Absolute Difference method.We applied normalised cross-correlation method.As the adopted normalised cross correlation (NCC) method is the most suitable method in order to find the target and follow the phantom robustly and accurately.Table 3 shows that the suppression rate is achieved to be 25.2% (experimental condition is 3.00 s in period and 42.4 mm in peak-to-peak) and the minimum suppression rate is 16.0% whose experimental condition is 3.99 s in period and 40.6 mm in peak-to-peak) by our newly developed robot.It is confirmed that our proposed method and robot could servo the target kidney motion so as not to lose sight of the target, which moves with a constant speed.However, organs also move in the rotational directions in real.Therefore, it becomes difficult to servo the target only with the proposed template-matching-based method in this report.
The motion of the phantom is regular in this experiment.While, the real-organ motion, which is mainly caused by the beating heart and the respiration, is non-regular.Hence, the controller of our robot should be improved to cope with more complex organ motions in real time.
As our future tasks, we develop novel algorithms to cope with the abovementioned rotational target motions.
Since the tracking algorithm incorporated here is used for scanning towards the centre of the template, it takes on mutually reverse-directional speed when the phantom returns, and thus generates twice the distance at that time.

Conclusion
In this report, we have briefly outlined the concept of a compact portable US robot that can be used at home and other environments for simple diagnostic work.We then showed the algorithm used and the results of an experiment that involved tracking the phantom model of a kidney.From these results, we were able to demonstrate the effectiveness of the apparatus.Matters for further study include the following.i.Further reductions in size and weight ii.Combining the phantom constant-speed reciprocating motion together with the automated probe speed adjustment and tracking iii.Phantom motion prediction and tracking iv.Experiments with animal organ We intend to pursue these matters as part of our future investigations.

Fig. 1
Fig. 1 Overview of newly developed robot

Fig. 8
Fig. 8 Experimental images (a) Screen image on experiment, (b) Tracking a phantom with developed robot holding a probe

Table 1
Range of robot motion X-axis, mm Y-axis, mm Θyz-axis, degree

Table 2 Experimental
Fig. 7 Result on follow phantom experiment