VV : a viewer for the evaluation of 4D image registration

We propose an open source and cross platform medical image viewer, named VV, designed for qualitative evaluation of images registration, in particular for deformable registration of 4D CT images. VV can display multiple spatio-temporal image sequences (2D+t or 3D+t) and contains several tools for comparing images using transparency or fusion, for visualizing deformation ﬁelds, for deﬁning landmarks. It is used in the ﬁeld of radiation therapy to help researchers and clinicians evaluate deformation in 4D CT images of the thorax. It is implemented in C++, with ITK, VTK and QT open source version, runs on Linux and windows and is freely available to the community.


Introduction
New advances in CT acquisition make it possible to obtain 4D CT datasets representing spatio-temporal data such as cardiac or breathing thorax images [20]. Such images are particularly useful in the field of radiation therapy of lung cancer where 4D thorax data is increasingly employed during treatment planning to take into account patient specific breathing motion [8]. Research in this field has received increased attention last years, in particular in respiratory motion modeling with help of deformable registration methods [15]. Evaluation of the result of deformable registration is still a challenging task, especially when dealing with large 4D datasets. In addition to quantitative evaluation [13], we feel it is important to provide visualization tools that allow researchers and clinicians to qualitatively appreciate the deformation, the motion and the registration results. We thus propose a graphical tool, named VV, allowing qualitative and quantitative analysis of deformable registration of 4D CT images of the thorax.
Several open source or freely available visualization tools already exists. For example : 3DSlicer [1], Med-Inria [4], Amide [9], Osirix [12]). However, none fulfill all the requirements which we esteem necessary : • be open source • be compatible with linux architecture (mostly for researchers) and window (mostly for clinicians) • display spatio-temporal data (2D, 2D+t, 3D, 3D+t) • allow to visually compare images and deformable registration results • be simple to use, be fast and able to process large files, from several file formats.
Our proposition can be viewed as a complementary tool to already available softwares. Even if not explicitly designed for that purpose, it could be also employed to perform conventional (3D) image fusion or as a simple image slicer. VV can be downloaded on http://www.creatis.insa-lyon.fr/rio/vv with sources, binaries and example data.
2 Overview of features 2.1 Image reading and writing VV can display images from various file formats. In addition to conventional 2D images (jpeg, hdr, mhd, bmp, png, tif) or 3D (hdr, mhd), it is possible to load a list of 2D image or a list of 3D images as a 2D+t or 3D+t spatio-temporal image, or a 4D image (mhd format). Note the explicit distinction between 3D and 2D+t image ; 4D image are always considered as 3D+t. nD images can be converted in several (n-1)D images with the 'split' menu available by right clicking on an image in the list of loaded images. A simple minimal Dicom browser also allows to select and load images series, or to display Dicom tag. Any loaded image can be saved as another file format through the "save-as" menu (which can be useful to convert Dicom series to a raw-based file format such as '.mhd'.

Navigation
With four panels displaying slices, the user can navigate into space and time with mouse or keyboard. Conventional orientations (sagittal, coronal and axial), grey-level windowing with presets, color map, grab & drag image displacement are available. A dynamic zoom ('control' key + mouse wheel) zooms to the current mouse position. Slices of different orientation are synchronized between panels according to the current colored cursor position. Among others, the keys 'f' (fly to mouse position), 'g' (go to cursor position), 'w/x' (zoom) can ease the navigation. A panel (top left) indicates the currently loaded images with their associated panel viewer, an information panel displays the current image data (bottom left). Image with temporal dimension (2D+t or 3D+t) can be "played" as a movie by clicking on the green play button (frame rate can be manually changed), which allows to appreciate the motion when navigating or comparing images.

Image comparison tools
We propose here several visualization tools to help the qualitative evaluation [7,2] of image registration in the context of spatio-temporal images and deformable transformations.
Synchronized navigation Maybe the most simple but still useful tool to visualize and compare two images is to synchronize the navigation. In VV, when two (or more) images are loaded, their navigation can be synchronized in the way that changing the current cursor position in one image also change the displayed slices of the other images. By combining the four different viewer panels and temporal capabilities, sophis-ticated synchronized navigation between several images orientations or time sequences can be achieved. Pixel values of each synchronized images at current cursor position are available.
Image fusion with complementary-colors One conventional approach to qualitatively evaluate the result of an image registration procedure consists in comparing the reference image with the other image deformed by the founded transformation. Overlapping the two images allows to perceive the differences between the reference and registered images. VV provides an image fusion tool which merges two images into a single one : the image differences are enhanced with two different colors (for positive and negative differences) while the pixel color tends to the original grey levels when differences are low. Such approach is similar to the one used in the XVI software from Elekta and found to be useful for images comparisons. The choice of the two complementary colors can be customized by the user with a simple slider. Both spatial (between slice) and temporal navigation remains as previously described. Pixel values and difference values at current cursor position are indicated in the left panel. This feature is for example used to overlay a reference 3D image onto a 4D image in order to appreciate the organs displacement with respect to the reference.
Vector field superimposition Deformable registration results are often obtained in the form of deformation fields which represent the displacement of each image pixel from the reference image to the other image. Additionally to quantitative analysis, we think it is important that researchers and clinicians can appreciate qualitatively such type of data. Conventional image differences such as the one described previously allow a fast evaluation of the registration result. However, it is well known that, as deformable registration is an ill-posed problem, a perfect match between the deformed image and the reference image can be obtained even though the deformation field itself is false. In order to evaluate a given registration, it is thus required to appreciate not only image differences but also the deformation field itself. However, it is not easy to visualize such a large amount of data : for example a typical deformation field encompassing the breathing motion of a 4D CT image of the thorax is a R 3 → R 4 function, which can be represented by a 512 × 512 × 141 × 10 3D vectorial image [19]. VV allows to load and display such data by superimposing on a given slice a set of 2D arrows representing the displacement vector at each point, projected according to the slice orientation. Navigation still remains similar than previously described. The sampling of the deformation field can be modified to increase or decrease the number of displayed vectors. This is for example useful when zooming (more vectors can be displayed when a large zoom factor is used), or to adapt the navigation speed according to the hardware capabilities. Components and norm of the displacement vector at current mouse position is continuously displayed in the 'Overlay' tab. In our group, this tool is used to quickly evaluate the results of a deformable registration algorithm or helps clinicians to quantitatively evaluate the displacement of a specific point in the image.
Images fusion by transparency The last image comparison tool consists in superimposing two images with a given user-defined opacity value and colormap. Such feature is for example used for image comparison in the field of registration procedure, but is also used, in the field or radiation therapy, to superimpose computed dose distributions onto an anatomical image. Multimodality registration could also benefit from this tool.
Landmarks The previously described features are mainly dedicated to qualitative and visual analysis. One of the most common ways to evaluate quantatively a registration consists in computing the distance between anatomical landmarks [5,21] defined by experts on both images to be registered. The tab 'landmarks' allows very simple landmarks selection : the physicians should press the space bar at the current mouse location to define a landmark, displayed on the image. The set of labels can then be saved in a text file for further evaluation study.
Conclusion All these tools are available for any image dimension 2D, 2D+t, 3D and 3D+t. Superposition (fusion, transparency or vector field) can be performed between a nD reference image and a nD or (n-1)D overlapping image. If the user machine is equipped with a modern but conventional graphical cards (i.e. NVIDIA, ATI ...), the navigation is very smooth and fast (real time), even with large data volumes. If no hardware acceleration is available, navigation is slower but still largely usable on conventional PC.

Architecture design overview
The software is based on several open source libraries (see figure 2). We used the Insight Segmentation and Registration Toolkit library (ITK [6]) for image processing tasks, reading and writing files. We used the Visualization ToolKit (VTK [16]) as the core library to display images and user interaction. GUI was performed with QT [18].
The architecture is based on a Model-View-Controller (MVC) pattern in order to separate data storage, representation (i.e extracted slice) and rendering management. For each dataset, a manager, called vvSlicersManager is build. It contains a pointer to data (the image and the corresponding overlays) and several visualization parameters. It manages and synchronizes four simultaneous representations (i.e. the four panels in the main window). These representations, called vvSlicer, extract slices into space and time for the reference image and compute the corresponding slices for related data (vector field or superimposed images ...). Finally, the main Qt window interacts with every managers in order to refresh all representations that meant to be displayed.

Use cases
In this section we provide three use cases. The corresponding data (images, vector fields) are available on the Insight Journal repository at a low resolution and on the web site at a full resolution.

Landmarks definition on spatio-temporal 2D+t Cone-Beam images
In the past few years in radiation therapy, there has been an increasing interest in using on-board conebeam images to improve target localization or to assess respiratory motion estimation [11,3]. Images generated from these devices are a temporal set of 2D X-Ray projections of the patient body. In order to study tumor and other organs motion according to the patient breathing, VV can be used by medical experts to identifying homologous landmarks in all 2D projections. Such landmarks can then be used as ground truth to be compared to automated motion estimation tasks. To perform this use case, open lungConeBeam.mhd using Open xD image as (x-1)D + t menu, click on landmark tab and load expertisePoints.txt. You can now move into time, remove or add new landmarks. As already mentioned before, motion estimation on 4D CT images of the thorax still requires extensive validation. VV can be used to visualize respiratory motion and computed displacement field by navigating through space and time. To perform this use case, open the 4D image Lung4D.mhd using Load images. Superimpose a 3D image with Overlays/Add overlay image to Current Image and choose Lung3D.mhd. Clicking on the green play button will start the breathing and allow to observe the organ displacement according to the 3D static image. Deformation field can be added with Overlays/Add vector field to Current Image, choosing 4DDeformation.mhd.

4D dose distribution visualization for hadrontherapy or radiotherapy
Computing the patient dose distribution due to a therapeutic irradiation is a fundamental task in radiation therapy. However, at our knowledge, no commercially available TPS (Treatment Planning Station) allows to compute and visualize a dose distribution computed by taking account breathing motion. Several researchers have shown that it is possible to compute such a 4D dose distribution [10,17]. VV can be used to display and analyze such 4D dataset. To perform this use case, open the Lung4D.mhd image and use Overlays/Add overlay image to Current Image to choose doseHadron4D.mhd. It displays the 4D dose distribution resulting from a carbon ion spot beam, computed on a 4D CT image. The computation was performed with a Monte-Carlo simulation [14]. The effect of the tumor and organs motion on the Bragg peak location is clearly visible (press the green play icon to start looping over the time dimension). Similar method can be used to display conventional photon dose distribution (add overlay doseRadio4D.mhd).

Conclusion
We proposed an open source and cross platform viewer, named VV. It is based on VTK, ITK and QT libraries and run on Linux and Window. It is currently used for qualitative evaluation of deformable image registration of 4D CT thoracic image, in the field of lung cancer treatment by radiation therapy. It is freely available to the community through the web site : http://www.creatis.insa-lyon.fr/rio/vv Future works will include advanced landmarks definition and pairing between multiple images, as well as quantitative criteria (Target Registration Error, TRE). Checkerboard-based views will also be implemented to quickly switch from one image to the other. Dicom images selector, still rather basic, will be enhanced. Other visualization tools will be implemented according to the clinicians and researchers experiences.