Stereoscopic Volume Rendering of Medical Imaging Data for the General Public

The Ars Electronica Center (AEC) is known as the Museum of the Future. As such, one of its specialties is exhibiting multifarious hybrids and blends of various artistic genres, scientific fields and technological developments and elaborating on them. Biotechnology and genetic engineering, neurology, robotics, prosthetics and media art meet here on an equal footing as they constitute experimental arrays for testing how we might be coexisting and communicating with our physical surroundings as well as other human beings in the not-too-distant future, and indicating what these changes will mean for us and our society.


Introduction
The Ars Electronica Center (AEC) is known as the Museum of the Future. As such, one of its specialties is exhibiting multifarious hybrids and blends of various artistic genres, scientific fields and technological developments and elaborating on them. Biotechnology and genetic engineering, neurology, robotics, prosthetics and media art meet here on an equal footing as they constitute experimental arrays for testing how we might be coexisting and communicating with our physical surroundings as well as other human beings in the not-too-distant future, and indicating what these changes will mean for us and our society.
Plus, the Ars Electronica Center offers its visitors something that, in the form it assumes here, is absolutely unique worldwide: a projection space in which an audience of 150 can behold 16 × 9-meter images on a forward wall surface as well as 16 × 9-meter images on the floor in front of them. These visuals require eight projectors-four for the wall, four for the floor. This makes laser tracking and 3-D animation available to the general public ( Figure 1).

Medicine and the natural sciences for the general public
Approximately eight years ago, the authors came up with the idea of also presenting medical subjects to the general public here-for example, modern aspects of medicine (anatomy of the human brain, possibilities of examining the brain using MR imaging) and especially of radiology (e.g. functional MR imaging, tractography), the principles upon which CT and MR are based and the sequences of steps these examinations entail ( Figure 2). These presentations were originally conceived as one-shot-deal events, but their success exceeded the organizers' most optimistic expectation. The upshot: the development of a series of continuing education events now staged on a regular basis at the AEC.

Stereoscopy in deep space
Deep Space is not only a setting for high-resolution projections displayed on dual 16 × 9-meter surfaces on the wall and floor. It also features technology to present stereoscopic 3-D images, 3-D films and interactive 3-D real-time graphics. Here, visitors outfitted with 3-D shutter glasses can enjoy a truly impressive three-dimensional experience.
A year ago, this possibility inspired a fascinating idea: presenting medical imaging examinations (CT and MR data sets) in 3-D here. But the first problem was to make these data sets 3D-enabled. One of us-F.B., an Ars Electronica FutureLab staffer-finally solved this problem by writing a volume rendering program that was capable of directly importing regular DICOM data (exactly like those we generate in radiology), processing them and presenting them without having to manually process them in advance ( Figure 3).
For stereoscopic volume rendering, three different methods, all based on ray tracing (no meshes involved), were used to display different aspects of the imaging data sets ( Figure 4): 1) Density Threshold (detect surface by jump of density over a certain threshold) 2) Gradient Weighting (detect surface normal by big gradients in data set) 3) Density Filter (filter dataset for a specific density within a certain density window) The methods were implemented without any reference to existing methods. We assume that similar principles are used by other existing approaches.
For presentation a workstation is installed in Deep Space prior to an event and manned during the presentation by F.B. or another FutureLab staffer. Thus, what takes place here is live volume rendering of the image data, which makes this experience even more enthralling for the audience.
Modification of the density values (CT) or intensity values (MR) makes it possible to depict various strata of the human body. Usually, we begin with a setting for the body surface or the surface of the part of the body that is being examined.
During a smoothly adjustable online rotation, the patient is rotated 360° so the audience can see how the patient was lying on the CT table ( Figure 3).
Then, we proceed successively deeper into the interior of the body. Along the way, we point out the various anatomical structures of the human body (bones, thorax, abdomen, different abdominal organs, and vessels) (Figures 4 and 5).
The data sets can also be smoothly enlarged, shifted and edited online, which makes it possible to focus on various different parts of the body and the organs in them. It's also possible to select from among several modes of depiction and to directly fine-tune their density or surface threshold values at will during the presentation so that a wide array of structures (e.g. bones, blood vessels and parenchyma structures such as bronchia) can be optimally depicted (Figure 6).
At present, there is still hardly any literature on the subject of stereoscopic volume rendering. A Medline search using the search terms "stereoscopic volume rendering" yields a mere 39 publications; entering "volume rendering" yields 2,358 works.
The 39 works on this subject deal primarily with the advantages of three-dimensional viewing of imaging examinations for attending surgeons-for example, in the case of a pleural mesothelioma [1] for the thoracic surgeon, a neurosurgeon's preparations for an operation on an intracranial aneurysm [2], or for neurosurgery in general [3,4]. Further publications report applications in gynecology [5] and otolaryngology [6].
Our research in the literature has yet to yield descriptions of the situation as we have described it here.
The reason why the three-dimensional images from Deep Space appear blurred here is that the camera-in contrast to a human spectator-sees both images simultaneously and records them overlapping, whereas spectators in Deep Space wear special glasses that separate the image for the left eye from the image for the right eye. This is what produces the impression of depth, and also makes the images crystal-clear (See the projection in Figure 2, which is not a three-dimensional image).

Previously presented investigations
The following investigations have already been presented at various continuing education offerings: CT whole body, as described above; also, contrast-enhanced MR angiography of neck arteries ( Figure  7), and non-contrast 3D time-of-flight (TOF) MR angiography of intracranial arteries.

Special features
At a very early stage-in the wake of the first presentation-we became aware of another problem. It was impossible to use a laser pointer to show the various structures mentioned above because, for one thing, the point of light was too weak, and, for another, because a spatial shift of the point occurred when aimed at an area outside of the center (from the demonstrator's point of view). F.B. responded to this issue by writing a program that enables the demonstrator to use a commercially available tablet to exercise online control over a virtual three-dimensional arrow (Figure 8) that can be manipulated not only two-dimensionally on the x-axis and y-axis but also in the third spatial dimension, the z-axis of depth.
In the next developmental phase, F.B. implemented a labeling option that can be used to mark and label any structures the demonstrator desires (Figure 9). When the demonstrator undertakes a zoom, shift or rotation online, the markings and labels move along with the data set so as to remain spatially correct.

Future: Implications for the Medical Domain?
Until now the focus is on the topic to provide stereoscopic volume rendering of medical imaging data to the general public. For this purpose, only CT and MR data sets with normal findings are presented.
A current issue of research in medicine is the value of stereoscopical assessment of volume-rendered medical imaging data in comparison to traditional monoscopic viewing. A recent paper of Stewart et al    [7] concludes that stereoscopic viewing of images should be further investigated and may well eventually find a permanent place in procedural and diagnostic medical imaging.
In the future, we plan to evaluate different pathological findings in the different body areas, whether these can be visualized sufficiently with the stereoscopic method in the environment of the AEC Deep Space with a projection area of 16 × 9 m that we have introduced in this paper. For this purpose, different domain experts have to be involved (e.g. trauma surgeons, abdominal surgeons, neurosurgeons, etc.).

Conclusion
Ever since this project's inception, the presentation of medical and scientific subject matter to the general public in Deep Space at the AEC has been so successful that the producers have been motivated by popular demand to successively increase the frequency of these events. The highlight is the opportunity to behold CT and MR examinations in high stereoscopic quality on a 16 × 9-meter projection surface.