An open-source real-time ultrasound reconstruction system for four-dimensional imaging of moving organs

Four-dimensional imaging is a necessary step in computer-assisted interventions (CAI) performed on moving surgical targets, such as those located within the beating heart or inﬂuenced by respiratory motion. Ultrasound (US) is often the best imaging modality for this application, as it provides good contrast and spatial resolution while remaining simple to integrate into the operating room. However, conven-tional 2D US imaging is often insufﬁcient for preoperative planning and surgical guidance, real-time 3D ultrasound imaging using matrix array probes has a smaller ﬁeld of view and lower spatial resolution, and 3D volumes created using 3D US reconstruction suffer from motion artifacts. As an alternative, gated 4D ultrasound reconstruction using a tracked 2D US probe is a promising technology. In this paper, we present SynchroGrab4D to the open-source community. SynchroGrab4D performs 3D and 4D US reconstruction in real-time while visualizing the output volume(s) within an OpenIGTLink-compliant CAI system, such as 3D Slicer. Also included are VTK classes that perform ECG-gating and that interface with several commercial tracking systems, as well as a “4D Imaging”’ module in 3D Slicer. Our open-source imaging system can be used to collect the 4D image data required for computer-assisted surgical planning and intraoperative guidance for a variety of moving organs.


Introduction
Initial development of computer-assisted intervention (CAI) systems focused on the orthopaedic and neurosurgical fields, since assuming that the organ of interest (whether a bony structure or the brain encased within the skull) is a rigid body made superimposing preoperative images onto the intraoperative patient much simpler. The success realized in these surgical subspecialties has paved the way for today's interest in more complicated procedures, including surgery performed on soft tissue and on moving organs. Focusing on mobile tissue in particular, investigators are making inroads in all aspects of CAI, including diagnosis, modeling and preoperative planning, image registration and intraoperative guidance. For example, systems are under development for minimally-invasive therapy performed on the beating heart [8] and for respiratory motion compensation during both lung tumour radiotherapy [7] and liver tumour ablation [1]. In virtually all CAI systems for moving organs, four-dimensional imaging (which yields a time series of 3D volumes depicting the organ's deformation over time) is required at some point in the surgical workflow.
Ultrasound (US) has sufficient soft tissue contrast and spatial resolution for many applications while remaining low-cost and ubiquitous within operating rooms. US is therefore often the imaging modality of choice as an alternative to magnetic resonance imaging (MRI), computed tomography (CT) or fluoroscopy, especially for intraoperative imaging. Ultrasound probes conventionally image in two dimensions, but 2D information is rarely enough to complete an interventional task. Two alternatives exist for volumetric US imaging of moving organs: real-time 3D (RT3D) ultrasound imaging with a matrix array probe [12] and performing gated 4D ultrasound reconstruction by tracking a standard 2D US probe. 4D US reconstruction extends the well-established 3D US reconstruction technique [3] and generates a time-series of 3D US volumes by using gating to collect multiple 2D US frames acquired at the same point in the organ's motion cycle and compositing them into 3D volumes using the probe's pose as measured by the tracking system. When imaging moving organs, 4D US reconstruction with gating is needed instead of 3D US reconstruction, as volumes generated using the latter will typically contain significant motion artifacts. Compared to RT3D ultrasound, 4D US reconstruction offers advantages with respect to field of view and spatial resolution, uses widely available equipment and does not suffer from difficulties in streaming 3D data from closed US machines.
An advanced form of US reconstruction is real-time reconstruction, where the 2D US frames are inserted into the output volume(s) as they are acquired, enabling interactive visualization of the incremental reconstruction and ensuring that the volume(s) contain no "holes" and cover all of the region of interest. This is especially valuable when performing 4D US reconstruction, which is more technically challenging compared to 3D US reconstruction due to the extremely slow probe speed required. For our application in minimally-invasive beating-heart surgery, we have developed and validated a real-time 4D US reconstruction system [9] based on our group's previously described real-time 3D US reconstruction approach [5].
3D and 4D ultrasound reconstruction has been integrated into commercial ultrasound systems for quite some time, but the inaccessibility of the resulting raw image data makes these systems unsuitable for research (as opposed to clinical) use.  gated ultrasound reconstruction, including the well-known Stradx system [14], but usually cannot perform subsequent image registration, segmentation, advanced visualization (such as volume rendering) or data transfer for integration into surgical guidance systems. In contrast, 3D Slicer [10] is an actively growing open-source application that bundles an extensive collection of medical image processing and image-guided therapy modules. However, despite current research interest, 4D imaging and image processing algorithms are currently limited in 3D Slicer and in other open-source medical imaging packages.
In this paper, we present SynchroGrab4D, open-source software for the interactive acquisition and visualization of reconstructed 3D and 4D ultrasound data using 3D Slicer. We take advantage of several open-source software solutions, including the Visualization Toolkit (VTK) [11], the OpenIGTLink network transfer protocol [13], and a previously described open-source 3D US reconstruction system [2]. As part of our approach, we also describe VTK classes for gating and a "4D Imaging" module in 3D Slicer. Although our particular focus is on 4D US imaging of the beating heart using ECG-gating, our software is easily generalizable to imaging other cyclically moving organs, for example using respiratory gating.

Real-time 4D ultrasound reconstruction
SynchroGrab4D performs both 3D (non-gated) and 4D (gated) ultrasound reconstruction and interactively visualizes the results in 3D Slicer. It can be used with both freehand and mechanical scanning, supports multiplanar probes (those with a rotating imaging plane) and implements both prospective and retrospective gating. As our real-time 3D US reconstruction approach has been previously described [5,2], here we focus on 4D reconstruction. An example 4D ultrasound dataset of a beating-heart phantom reconstructed using SynchroGrab4D is shown in Figure 1, while an implementation of our algorithm within the AtamaiViewer framework [6] has been used to image the beating heart in both porcine and human subjects (Figure 2a, 2b). Figure 2c illustrates the 4D US acquisition procedure. The operator moves the US probe once per cycle while SynchroGrab4D simultaneously uses gating to divide each cycle into N phases. The 2D US frames at the beginning of each phase are identified and are inserted into the appropriate output volume using their corresponding pose from the tracking system. In our experience in intraoperative imaging of the beating heart, the acquisition takes approximately four minutes and can be integrated into the surgical workflow  quite easily. The accuracy of any US reconstruction approach depends on the quality of the spatial calibration (which determines the transformation between the origin of the ultrasound fan in the 2D US frames to the sensor mounted on the US probe) and temporal calibration (which determines the lag time between 2D image frames and their corresponding transform matrices from the tracking system). We have shown that our reconstruction technique has RMS errors in object localization measuring approximately 1.5 mm for 3D reconstruction and 2.6 mm for 4D reconstruction [9], which is within range for the spatial calibration technique used [4].
SynchroGrab4D is written in C++ using VTK 5.2 (simplified class diagram provided as a supplemental figure), uses OpenIGTLink to send images and transform matrices to 3D Slicer 3.4, and is fully crossplatform (Linux, Windows and Macintosh). Seven threads perform the following tasks: (1) grabbing and timestamping 2D US frames; (2) retrieving and timestamping transform matrices from the tracking system; (3) performing automatic phase detection from an incoming ECG signal; (4) performing US reconstruction by updating pixels within the output volume(s); (5) buffering the output volume(s) for transfer to 3D Slicer; (6) sending OpenIGTLink messages of the output volume(s); and (7) sending OpenIGTLink messages of the transform matrices (optional). Synchronization between the 2D US frames and the transform matrices from the tracking system is accomplished using the aforementioned timestamps and the temporal calibration results, while the lag in receiving the ECG signal is assumed to be negligible.
Video capture 2D US frame grabbing, timestamping and buffering is based on the vtkVideoSourcederived VTK classes, which have been refactored to provide a clearer representation for a single frame, the circular buffer of frames and the frame grabbing mechanism (vtkVideoFrame2, vtkVideoBuffer2 and vtkVideoSource2, respectively). SynchroGrab4D supports frame grabbing with Video-for-Windows and Matrox (Montreal, QC, Canada); adding support for Linux and the open-interface Sonix RP ultrasound system (Ultrasonix, Vancouver, BC, Canada [2]) is in progress. Although the original VTK frame grabbing classes always output the most recently acquired frame, in the new classes we provide the ability to specify a "desired" timestamp so that the filter output is the frame whose timestamp is closest to the desired timestamp.  . We also provide a tracking simulator (vtkFakeTracker) for testing when a tracking system is not available.
Gating An integral part of SynchroGrab4D is the gating subsystem, which currently operates on ECG signals but could be easily modified to perform other forms of gating, such as respiratory gating. Our ECGgating software performs automatic R-wave detection from patient ECG signals and splits the measured cardiac cycle period into a user-specified number of equally-spaced phases. Prospective gating provides real-time numerical output estimating the current cardiac phase and heart rate, while retrospective gating can determine the true time at which each phase began in the previous cycle (Figure 3a). The base class, vtkSignalBox, is an ECG-gating simulator with a user-specified cardiac cycle period. The derived class vtkHeartSignalBox, which was originally designed for use with a beating-heart phantom (The Chamberlain Group, Great Barrington, MA, USA) that outputs a voltage pulse at the beginning of each cardiac cycle, gates based on a 5V pulse arriving at the machine's parallel port. The second derived class, vtkECGUSBBox, reads a patient ECG signal (amplified to the volt range) over a USB connection. Automatic R-wave detection can then be accomplished by applying a user-specified threshold. We have used both vtkHeartSignalBox and vtkECGUSBBox to interface with clinical anesthesiology equipment within the operating room. The following code snippet illustrates the use of these classes: vtkSignalBox* sb = vtkSignalBox::New(); sb->SetNumberOfPhases(5); // phase shift at 0%, 20%, 40%, 60% and 80% R-R sb->Start(); int phase = sb->GetPhase(); // current phase as an integer in range [0,4] float rate = sb->GetBPMRate(); // current heart rate in beats per minute double curr = sb->GetTimestamp(); // timestamp (TS) of most recent measurement double ret = sb->CalculateRetrospectiveTimestamp(3); // TS at 60% R-R for the previous cycle sb->Stop(); 4D ultrasound reconstruction Ultrasound reconstruction in SynchroGrab4D is performed by two classes: vtkFreehandUltrasound2 implements real-time 3D ultrasound reconstruction as described  [5], while the derived class vtkFreehandUltrasound2Dynamic incorporates prospective and retrospective gating for real-time 4D ultrasound reconstruction.
In the most basic configuration, vtkFreehandUltrasound2Dynamic has N image outputs (where N is the number of phases of the associated vtkSignalBox) each depicting the beginning of a cardiac phase. SynchroGrab4D can also reconstruct a user-specified subset of phases to allow greater flexibility in the points of the cardiac cycle that are represented by the output volumes. The 4D ultrasound reconstruction algorithm proceeds as follows: If gating subsystem shows a change from phase i to phase j and phase j corresponds to an output volume: 1. Get gating timestamp t g : Prospective gating -use most recent timestamp from the gating subsystem.
Retrospective gating -find actual timestamp at which phase j began in the previous cycle (Figure 3b). 2. Get 2D US frame whose timestamp t f is closest to t g : Prospective gating -decide between most recent frame and the next frame. Retrospective gating -retrieve frame with best timestamp from frame buffer. 3. Calculate transform matrix timestamp t t corresponding to t f : subtract lag from temporal calibration. 4. Interpolate the transform matrix for timestamp t t from the transform matrix buffer. 5. Multiplanar probes -identify rotation using pixels of 2D US frame and apply to transform matrix. 6. Insert 2D US frame into output volume for phase j using transform matrix, splatting with nearestneighbor or trilinear interpolation and compositing frames with alpha blending or compounding [5].
Since variations in patient heart rate can introduce additional artifacts in 4D US reconstruction, one might not want to insert frames acquired during erratic heart beats into the output volumes. SynchroGrab4D can determine an expected heart rate by calculating the mean heart rate over a user-specified duration, while ensuring that the heart rate is relatively stable over the measurement time. During the reconstruction, frames acquired when the heart rate differs too much from the expected value are rejected. If using prospective gating, frames tagged for insertion must be saved in memory until the next R-wave, at which point the actual heart rate of when they were acquired can be checked.
Once the reconstruction is finished, hole filling [5] is performed to fill in voxels that did not intersect with a 2D US frame during the reconstruction. The output volume(s), a calibration file describing the parameters used and (optionally) the contents of the transform matrix buffer are then saved. Users can also save the inserted 2D US frames as bitmap images and/or their corresponding timestamps.

User interface and communication with 3D Slicer
SynchroGrab4D is designed as a command-line application that sends reconstructed US volumes and/or the transform matrices from the tracking system to a computer-assisted intervention system over a network connection, rather than being integrated within a particular CAI system. Command-line options allow the user to specify a calibration file containing the reconstruction parameters, the type of video grabbing, tracking and gating hardware that is to be used, and the server IP and port number. During real-time reconstruction, the reconstructed volume(s) can be sent at a user-specified rate over the network connection using the OpenIGTLink protocol [13] for interactive visualization within a CAI system. Alternatively, the user can choose to delay the image transfer until after the reconstruction's completion if the transfer speed is insufficient. The transform matrices from the tracking system can also be sent using the OpenIGTLink protocol to a second port, allowing the US probe's position and orientation to be visualized by the CAI system as well.
We have focused on visualizing the reconstruction results within 3D Slicer, although SynchroGrab4D can send image data and/or transform matrices to any system supporting the OpenIGTLink protocol. in the MRML data tree are annotated with the phase that they correspond to. If the user elects to send the reconstructed volume(s) over the network connection during the reconstruction, then the incremental results can be visualized within 3D Slicer and the operator can ensure that the reconstructed volume(s) contain all of the desired image data. A new "4D Imaging" module has also been developed in 3D Slicer for the visualization of 4D image datasets. The reconstructed 3D volumes making up the four-dimensional dataset can be added to a "4D bundle" within 3D Slicer along with their relative timestamps, and the 4D Imaging module can then display the volumes as a cine loop to visualize the motion of the organ of interest.

Discussion and conclusions
Our open-source 3D and 4D ultrasound imaging system performs real-time US reconstruction and interfaces with 3D Slicer to visualize the reconstruction's progress while imaging any cyclically-moving organ, including the beating heart and organs that are influenced by respiratory motion. 4D US image data can be used as input for CAI systems for diagnosis, image-based modeling and atlas building, and preoperative planning. In addition, our 4D ultrasound reconstruction system is especially useful for intraoperative imaging, as intraoperative MR or CT is often infeasible, while intraoperative fluoroscopy and X-ray provide 2D projection images instead of 3D volumes. Within the operating room, 4D ultrasound images can be used as part of the registration process between preoperative images and the intraoperative patient and for surgical guidance within augmented reality systems. Since our software provides a clear link to 3D Slicer, any of these tasks can be performed using the variety of tools within the 3D Slicer suite.
When performing real-time 4D ultrasound reconstruction, limitations in both memory and processing speed mean that special care must be taken when deciding on the number of output volumes, their dimensions and the time difference between them. Since memory for all of the output volumes is allocated at the beginning of the reconstruction, there is a practical limitation to the number and size of the output volumes. Also, SynchroGrab4D will drop frames if the time difference between phases of interest is less than the time required to insert a frame into an output volume. This is especially important to remember when saving the inserted 2D US frames as bitmap images on the heart drive, as this is done in real-time as well.
Several components of SynchroGrab4D, namely the video grabbing, tracking and gating subsystems, are of general interest as their modular design makes them easily extensible to work with currently unsupported hardware. A variety of ultrasound machines can be used by specifying the appropriate parameters for extracting the US fan from the 2D US frames. SynchroGrab4D's automatic rotation detection for multiplanar US probes is currently customized for grabbing frames from a Philips Sonos 7500 or 2500 scanner with a Matrox Meteor or Morphis card, but once again this can be modified for different ultrasound machines.
In conclusion, we have presented SynchroGrab4D, an open-source 3D and 4D ultrasound imaging solution that can interface with 3D Slicer for real-time visualization and subsequent processing. Our US reconstruction software can be used to generate 4D image data of moving organs using equipment that is easily integrated into the operating room, while the included ECG-gating classes are of general interest. Given the current development of CAI systems for moving organs, the ultrasound volumes reconstructed using our system have a variety of applications to support diagnosis, modeling and intraoperative surgical guidance.