Free breathing myocardial perfusion data sets for performance analysis of motion compensation algorithms

Background Perfusion quantification by using first-pass gadolinium-enhanced myocardial perfusion magnetic resonance imaging (MRI) has proved to be a reliable tool for the diagnosis of coronary artery disease that leads to reduced blood flow to the myocardium. The image series resulting from such acquisition usually exhibits a breathing motion that needs to be compensated for if a further automatic analysis of the perfusion is to be executed. Various algorithms have been presented to facilitate such a motion compensation, but the lack of publicly available data sets hinders a proper, reproducible comparison of these algorithms. Material Free breathing perfusion MRI series of ten patients considered clinically to have a stress perfusion defect were acquired; for each patient a rest and a stress study was executed. Manual segmentations of the left ventricle myocardium and the right-left ventricle insertion point are provided for all images in order to make a unified validation of the motion compensation algorithms and the perfusion analysis possible. In addition, all the scripts and the software required to run the experiments are provided alongside the data, and to enable interested parties to directly run the experiments themselves, the test bed is also provided as a virtual hard disk. Findings To illustrate the utility of the data set two motion compensation algorithms with publicly available implementations were applied to the data and earlier reported results about the performance of these algorithms could be confirmed. Conclusion The data repository alongside the evaluation test bed provides the option to reliably compare motion compensation algorithms for myocardial perfusion MRI. In addition, we encourage that researchers add their own annotations to the data set, either to provide inter-observer comparisons of segmentations, or to make other applications possible, for example, the validation of segmentation algorithms.


Supplemental data files
To complement the article, three data files are provided as accompanying material: • GigasciencePerfusionTestbedVM.zip -the test bed including all the data and software needed to run the experiments presented. A description on how to use it is given in section 2.2.
• mia-2.2.0.tar.xz -the source code version of the MIA software [1] used to run the experiments.
• gigascience-myoperfdata.zip -the data and scripts to run the experiments, for its description see section 2.3.

The test bed 2.1 Description
The test bed is provided as a virtual hard disk in VMDK format [2] of 20Gb size (dynamically allocated), comprising a minimalist Ubuntu Linux 14.04 (i386) [3] installation, and the software, data, and scripts required to run the experiments presented in the main article. The installation has been tested with VirtualBox 4.2.22 [4] (Gentoo Linux, 64 bit, AMD FX(tm)-6300), Virtual-Box 4.3.6 (GNU Debian/Jessie 64 bit, Intel Core 2 Duo), and VirtualBox 4.3.14 (MS Windows 7, 64 bit, AMD FX(tm)-6300). While the following description on how to run the virtual test bed refers to VirtualBox, other platform virtualization software packages may also be used to run the test bed. Obviously, these alternatives must support the VMDK virtual hard disk format and support Linux i386 as guest operating system.

Creating and running the virtual machine using VirtualBox
In all cases, the file providing the test bed must be unpacked:

Setup using the VirtualBox GUI
In order to set up the virtual machine manually by using the VirtualBox GUI the following steps have to be taken: First create a new virtual machine from the menu Machine|New with the following parameters: • Name and operating system -Name: <at your liking> e.g. "GigaSciencePerfusionTestbed" -Type: Linux -Version: Ubuntu (32 bit) • Memory size: ≥ 1024 MB • Hard drive: (·) Use an existing hard drive, select the file perfusion-with-results-ubuntu-1404-32.vmdk, and set the following parameters in the settings panel of the newly created machine: • System|Motherboard: Enable IO APIC • System|Processor: Enable PAE/NX • Network|Adapter 1: NAT If your host system comprises a multi-core CPU, one might consider to set the number of processors to a value larger then one on the System|Processor panel. With this configuration, the virtual machine can be executed from the VirtualBox GUI and will provide a console. One can log into this console as user motion with password motion.
To run the machine headless, Host-only networking must be enabled in VirtualBox (File|Preferences|Network) with the DHCP server enabled, and the second network adapter must be enabled as Host-only adapter in the machine specific network settings panel. When the machine is start up and assuming the VM is named GigaSciencePerfusion-Testbed the IP address can be acquired by running the command vm=G i g a S c i e n c e P e r f u s i o n T e s t b e d VMIP= $ ( VBoxManage g u e s t p r o p e r t y g e t \ "$vm" " / V i r t u a l B o x / G u e s t I n f o / Net /1/V4/ IP " ) and one can connect then to the machine by using any secure shell client software, e.g. ssh: s s h − l motion $VMIP

GNU/Linux as host system
On GNU/Linux, the scripts provided alongside the virtual hard disk can be used to create and run the machine. Specifically, in order to create the virtual machine named GigaSciencePerfusionTestbed, one can run the script ./create-vm.sh from within the unpacked folder. After this script has been run successfully, the virtual machine can be started by running . / s t a r t −and−c o n n e c t . sh If successful, the virtual machine will boot up and then prompt for a login as user motion; the password is motion.

Mac OSX as host system
Given that Mac OSX comprises a fully certified UNIX system, it should be possible to create and run the virtual machine just like with GNU/Linux as a host system. Note, however, that the authors did not have access to a Mac OSX system and that tests run by the reviewers indicate that the machine might be created but the host-only based networking is not properly set up.

Providing a GUI environment within the virtual machine
In order to provide a more familiar graphical environment when running the virtual machine, like, e.g. xfce [5], one can run the following commands within the console of the running virtual machine (cf. Running these commands will require internet access. In this case one might want to review the display settings for the virtual machine, specifically the amount of video memory should be increased, and a Remote Display could be enabled. Note, however, that the experiments have to be run from a command line shell.

Data layout
The data is provided in the home directory of the user motion in the folder myoperfdata. In the root of this folder various scripts can be found to run motion compensation experiments. Then, for each patient, two folders StudyN TYPE are available, N referring to the number of the patient, and TYPE indicating whether this is a rest or a stress study. Within the study folder the images are given as consecutively numbered DICOM and PNG files. The segmentations are provided as XML files, namely • segmentN.set -empty segmentation that only lists the files corresponding to the anatomical location N ∈ {0, 1, 2}, (0 -apical, 1 -middle, and 2 -basal slice), • segmentN -dcm.set -segmentation set referencing the DICOM images, and • segmentN -png.set -segmentation set referencing the PNG images.
New segmentations are best started by using the empty sets segmentN.set. In addition, to the studies in the folder scripts various shell and R scripts are provided needed to run the experiments and evaluate the validation measures.
The git repository and the separate archive follow the same layout.

Running the motion compensation experiments
Within the virtual machine, in the home directory of the user motion the folder myoperfdata contains the data and the scripts to run the motion compensation experiments.
The output of experiments 1 as published is saved in output-compensation.published and the corresponding results of experiment 2 in output-no-movement.published .
Here, for each input data set set run with the algorithmm algo and a parameter set p 1 , p− 2 , ..., p n , the motion compensation result is written to the folder set/mia-algo-p_1-p_2-...p_n.
Within these folders a file files curvesX.txt can be found that contain the obtained time-intensity curves for slice X. For section i column 3i contains the time-intensity values before registration, column 3i + 1 the ground truth values, and column 3i + 2 the values after motion compensation. Experiment 1 is executed by running the script run-ica-quasi-combined.sh, the results will be saved in the folders output-compensation. The statistical analysis will be stored in the the files awked-* in a LaTeX friendly table format. Note, that for the stress and summary statistics, two result rows will be presented for the unregistered data, one that includes the series for which registration failed, and one that includes that series. In the paper the latter values that include all series were used to indicated R 2 and NMSE for the unregistered series. Experiment 2 is executed by running the script run-ica-quasi-no-movement.sh and the results will be saved in the folders output-no-movement. The statistical analysis representing Pearsons correlation coefficient will be stored in the file cc-Rest-all-stat-awked.txt, and MNSE in mnse-Rest-all-stat-awked.txt.
To  Alternatively, the current development version of MIA can be used, instructions how to compile and install the software are given on-line [8]. In order to run the experiments the optional software packages NLopt, DCMTK, IT++, and PNG are required to compile MIA and run the experiments. Note that only POSIX compatible operating systems are supported, and only GNU/Linux running on Intel compatible processors and PowerPC has been tested.
With the software for the testbed in place, the data and scripts can either be acquired from the GigaScience repository, or from the public git repository [9].

A note on reproducibility
Although the test bed provides a well defined environment to run the experiments, various runs may result in silghtly different results, because FastICA uses random numbers for initialization. When running the software on different machines, other factors have to be taken into account: For example, older versions of IT++ have a bug and MIA will not utilized the deflation approach of FastICA if at compile time such an older version of this library is detected. Second, MIA makes use of a lot of third party libraries that may also come in different versions and may have been compiled with different compiler flags.
Also note, that depending on the processor architecture and the optimization level results of floting point operations may vary slightly. For instance, when running software compiled for i386, floating point operations rely on the old i87 instructions set. Here floating point numbers are represented with 80 bit in the processor registeres, and only when the value is written to the working memory, the accuracy is reduced to 32 bit for single floating point values or 64 for double floating point values. Hence, since with a higher optimization level usually more operations are executed without storing intermediate values in working memory, rounding errors will differ from results that are obtained by running the same code compiled with a lower optimization level that issues code that stores more intermediate results, and hence maintains a lower accuracy on these intermediate results. Likewise, when switching from i386 to the amd64 architecture, the compiler will issue SSE instructions instead of i87 instructions, and more register are available to store intermediate results, hence, results might also differ slightly when running the code on these two different architectures (cf. [10]).
To illustrate these issues, in the appendix of this document we report additional results for experiment 1 that were obtained on different versions of GNU Linux and with a different hardware than used in the main document. In these two cases a pre-release version of MIA was used (branch: master, tag: gigascience-submission). Nevertheless, although the numbers vary slightly, the actual results -that ICA-SP performs better than QUASI-P -holds.

Closing note
Public forums for discussing the software and data are available at sourceforge.net [11,12].