Spherical Panoramas for Astrophysical Data Visualization

Data immersion has advantages in astrophysical visualization. Complex multi-dimensional data and phase spaces can be explored in a seamless and interactive viewing environment. Putting the user in the data is a first step toward immersive data analysis. We present a technique for creating 360° spherical panoramas with astrophysical data. The three-dimensional software package Blender and the Google Spatial Media module are used together to immerse users in data exploration. Several examples employing these methods exhibit how the technique works using different types of astronomical data.


Introduction
Data presentation in astronomy has expanded beyond the traditional two-dimensional plots to include multi-variable analysis, immersive three-dimensional (3D) models and exploratory analysis on a multitude of hardware platforms.Many forms of astronomical data can benefit from alternative forms of data exploration.Data cubes can be rotated in 3D, fly-throughs of planetary landscapes can be explored, and all-sky maps can be examined.Exploratory 3D presentations give users a different perspective on their data.Traditional desktop and laptop configurations as well as hand-held mobile devices can be used as the hardware mechanism for these data explorations.The techniques can be employed in both research-level visualization and can deliver easy access to materials for education and public outreach (Kent 2013).
Panoramic imaging in the physical domain is accomplished with a tripod mounted camera to encompass a larger field of view (FOV) than is visible through the lens.Multiple images are obtained while attempting to minimize parallax errors between images (Derrien & Konolige 2000).In some cases, four to six ultra wide-field cameras are used to record video in every possible direction.Software can then merge these images and video encompassing a FOV into a cylindrical or spherical panorama (Ozawa et al. 2012).Computer hardware and screen resolution along with network bandwidth are now sufficient to allow 360 • spherical panoramic video: a constant stream of panoramic images obtained with multiple wide-field cameras that gives a user control over the view while the video advances.Natural movement to view the scene can be accomplished with accelerometer-enabled hand-held devices or with desktop browsers.
We present a methodology overview for creating still and video spherical panoramas using astronomical data.Section 2 gives the reader a brief overview of rendering with Blender and the Google Spatial Media module.In section 3 we describe the math and techniques behind building a spherical panorama.Section 4 outlines examples utilizing the libraries and techniques described in this paper.Section 5 summarizes the techniques.

Panoramic Rendering
Traditional data presentation revolves around graphs, charts, tables, and other forms of two-dimensional media.They have a critical role in research publications.Complex or multi-dimensional data can benefit from a more advanced data display mechanism.The ability to immerse users in their data has a number of technical challenges that can be accommodated with recent advances in both hardware and software.Data can be presented within a 3D space, including data cube volume rendering, planetary surface maps, and astronomical catalogs (Hassan & Fluke 2011;Kent 2013).This paper introduces astronomers to spherical panoramas as a data display mechanism.It is advantageous to use immersive spherical panoramas as they allow easy and natural control of the viewing space, intuitive and minimalist user interfaces, and cross-platform support across a range of hardware and display software.
Projection algorithms in astronomical FITS images have been well curated and documented (Hanisch et al. 2001).The growing size of CCD cameras required new drizzling algorithms and reprojection models that paved the way for mosaicking and fast, large, wide-field projection and coordinate transformation algorithms (Fruchter & Hook 2002;Makovoz 2004).Montage is a recent example of image mosaicking and projection code that has been very successful on this front (Berriman et al. 2007;Jacob et al. 2009).While this paper focuses on adding value to existing images and data, it is important to acknowledge the algorithms that go into making the products from large projects.Astronomical surveys produce wide-field imaging science-ready data products and catalogs (Abazajian et al. 2009;Skrutskie et al. 2006;Kent et al. 2008).Wide-field radio surveys require special algorithms where mosaicking is done in the interferometric uv -plane (Tasse et al. 2012;Offringa et al. 2014) as opposed to the imaging plane.
With wide-field mosaics across multi-wavelength regimes increasing in size, resolution, sensitivity and detail, it is useful to create immersive spherical panoramas where the user is inside the data.Panoramic photography and imaging play an important role in interactive user interfaces, data analysis, and 3D reconstruction (Peleg et al. 2001;Gledhill et al. 2003).More recently, stereoscopic vision, virtual reality, and augmented reality applications can combine to supplement data viewing in geographic and mapping software (Azuma 1997;Azuma et al. 2001;Azuma 2004;Wither et al. 2011).This has applications in haptic feedback devices, 3D caves, and other forms of virtual reality (Sanchez-Vives & Slater 2005).Displaying 3D data on traditional two-dimensional (2D) displays (a computer or a tablet) requires certain considerations when projecting images (Gilbert et al. 2012).Further applications include stereo vision in 3D environments with headgear apparatus (Gasparini & Bertolino 2013).We focus here on the simplest use of easy to obtain hardware: desktops, laptops, and mobile devices.Used in conjunction, the software package Blender combined with the Google Spatial Media Library make a powerful combination for a researcher to animate and display their data for other users.
We outline spherical panoramas described in this paper that can fit into one of three scenarios (Figure 1).An example from each of these scenarios will be described in section 4.
• Static spherical panoramas.This type is best exemplified by a user stationary in the middle of the celestial sphere.All-sky maps work well with this scenario.It requires a single frame and can be executed with short render times.
• Single pass fly-throughs.Imagine this scenario as moving through a 3D dataset on a predetermined track.As the user moves along the track at a fixed speed, they are free to look about in any direction.
• Orbit flyovers.The scenario is geared toward data where one item is in focus, such as a particular area or object in a simulation or map feature.

Blender
Blender supports a wide range of 3D visualization and animation scenarios1 .It is maintained by the Blender Foundation as open-source software.The software has been adapted for use in a number of astronomy applications with easily accessible primers and tutorials (Kent 2013(Kent , 2015;;Taylor 2015;Naiman 2016;Gárate 2017).The novice user is referred to those works and the references therein when using Blender for the first time with their data.Blender versions 2.64a and later will work with the examples and techniques outlined in this paper.
Three-dimensional graphic suites like Blender require a workflow of data selection, modeling, and animation.Other software packages referred to in Kent (2013) including Lightwave 3D, Maya, and 3D Studio Max are popular in the graphics industry.While the internal working models and graphical user interfaces may differ between packages, Blender supports data import from all of those packages.As with any data reduction or graphical display package and depending the complexity of data, it is best to begin learning the package with a simple dataset such that the developed data model can be built upon for reuse.The user's proficiency can quickly improve with the tutorials and examples provided at the end of this paper.To aid in the ease of scripting, the Python application program interface (API) allows interfacing with popular scientific astrophysics libraries (Astropy Collaboration et al. 2013) such that a variety of astronomical data (ASCII catalogs, FITS images, etc.) can be brought into the environment.We choose to use Blender because its built-in engine capabilities allow for fast panoramic rendering of a 3D scene.In addition, the new Cycles rendering engine allows for a node-based compositor control of many of Blender's graphical user interfaces (GUIs) and features (Table 1).The camera controls in Blender give a 360 • ×180 • FOV that can be used as a still image in a static spherical panorama (Figure 2).

Google Spatial Media
The Google Spatial Media module is a collection of specifications and tools for spherical 360 • video and spatial audio.The module can take rendered output from Blender and generate spherical panorama video.This is accomplished with a metadata injector that adds information on the file type and FOV to the header.Audio can also be included in the metadata, but it will not be discussed in this work.The module source code is available from a GitHub repository2 .The module is available as a stand-alone executable for Windows and Mac OS or as a Python module for Linux/Windows/Mac OS.By adding to the video metadata header, the output can then be exported to YouTube3 .Within YouTube the player becomes aware of the data content and gives the user a set of controls with which to navigate the environment.Users can use a mouse to click and drag or move their tablet or mobile device naturally to view the scene.The preprocessing by Blender to create a full spherical view covering 4π steradians is critical before running it through this module.It is assumed that the metadata injection carried out by the module is being applied to to a full spherical panoramic view.
The social media website Facebook also has the capability of supporting 360 • video, but the metadata injection must be inserted by the user if the video has not been already created by a spherical camera system.This creates a complication that is solved for the user by using the Google Spatial Media module with the added benefit of the Python interface.Other 3D VR/AR (virtual reality/augmented reality) applications such as Oculus Rift or Vive require rather steep hardware requirements.The scheme presented in this paper is aimed at a researcher utilizing hardware already available as well as open-source software for the processing.

Hardware and Software Requirements
The Google software interface coupled with Blender and YouTube allows for an easy implementation of a spherical panorama video on a variety of hardware platforms.The results can be viewed with a mouse-desktop/laptop setup with the Chrome4 or Firefox browser5 .Tablets and mobile devices with three-axis accelerometers are where this technique's forte really lies.Using this technique on such devices allows for an immersive data discovery session.A user can move within the dataset as the video plays or be located in a static position (Figure 3).The device itself can be moved or the user can navigate with their hands.If the camera is moved during a spherical panorama video, users can track a particular object by pointing their device in different directions.User device rotation is the controllable element in this scheme.No translation of the mobile device will have an effect on the viewer's perspective.The technique works equally well on both iOS and Android mobile operating systems.

Methods
Images can be combined and projected from an equirectangular, cylindrical equal area or Hammer-Aitoff projection.The first two may be necessary in certain cases where data is not available to encompass the entire sphere of visibility.An equirectangular projection conversion is simply defined with Cartesian coordinates x,y, and angles φ, θ as A cylindrical equal area projection can be defined as where φ is the azimuthal angle and λ is the radius of the projected cylinder.We will assume a scaling of λ = 1 in this work's examples for simplicity (known as a Lambert's equal area projection).A simple cylindrical panorama setup with a Lambert's equal area projection is shown in Figure 4.
For all-sky maps, image reprojection is the most computationally intensive part of image processing.Tools such as Montage assist with this (Makovoz & Khan 2005;Jacob et al. 2009).All-sky maps are readily available from a large number of astronomical surveys.The Hammer-Aitoff to spherical formalism is given as where These maps and projection definitions are well defined by Calabretta & Greisen (2002).These transformations will be employed in section 4.
Once a FITS image map is generated, it can be imported into Blender by way of AstroPy and its associated FITS manipulation modules.Planetary surface maps and 3D catalogs can be brought into the Blender environment as described in Kent (2013Kent ( , 2015)).Here we focus on the methods and geometry of virtually filming these data for spherical panoramas.
We can consider the method of virtually photographing our scene with data in a similar fashion to the way a camera is set up to image in a panoramic mode (Figure 5).While the internal Blender mechanics automate this process with the built-in camera object, it is useful to understand some of the underlying mathematics.The projective mapping between two planes is given by  which is summarized as where H is the homography matrix that transforms p to our plane p ′ .The formalism for panoramic homography is reviewed in Hartley & Zisserman (2004).When projecting images for viewing, it is important to understand that we are not rotating the FOV about the focal plane, but around the nodal point for the exit pupil that sits in front of the plane (Figure 6).The nodal point for a physical camera is the correct pivot point in panoramic imaging and allows us to minimize parallax errors (Zhang & Liu 2014).A virtual camera in a 3D scene can be thought of as a point from which rays originate constrained by a FOV.The rays in a virtual camera's 360 • FOV are unconstrained (see Glassner (1989) for a comprehensive review on ray tracing).Stereo imaging is not within the scope of this paper, but it will utilize two origin points for the rays separated by an inter-pupillary distance, also refereed to as a stereo baseline (Peleg et al. 2001).
With this information in mind, the broad workflow for creating our examples will work as follows.Each example will give details specific to that given scenario and will utilize the following seven steps.The seventh stage will be omitted from the examples as it is uniform to all three, where we use the Google Spatial Media module.4. Set camera path.The path is configured with a Bézier curve if required.The camera can be moved freely as determined by the user or restricted to initially focus on a particular area or object of the data.
5. Configure camera.The camera settings can be configured to maximize the FOV depending on how much coverage is yielded by the data (for instance, only a partial set of data in an all-sky map).This requires a panoramic camera selection in either the default rendering engine or the new Cycles engine.
6. Render Scene.The scene should be rendered at 4K or higher since the spherical panorama image must cover 4π steradians (Silva et al. 2016).The best output resolution should be used at 3840 × 2160 pixels.
7. Configure metadata.The rendered video can be imported into the Google Spatial Media module as a stand-alone binary or as part of a Python script.

Examples
The examples described in this section follow data import and manipulation procedures outlined in Kent (2013Kent ( , 2015)).All data manipulation examples start with the default file that opens with Blender upon startup; no changes need to be made to the default configuration.Most workflows in these examples require a number of mouse/keyboard binding combinations. 6  The first example uses a static view inside a celestial sphere.The second and third examples both utilize a moving camera with two different data sets, an astronomical catalog where the camera flies through with a linear path and a 3D planetary surface with a circular path where the camera can focus on a particular geographic feature.

All-sky maps
All-sky map data lend themselves very well to the spherical panorama paradigm.While any astronomical FITS image can be used when properly placed on the celestial sphere, wider sky coverage produces a more pronounced effect.All-sky maps cover a complete 4π steradians FOV of spatially well-sampled data over a wide range of wavelengths.For 6 References for keyboard entry and model manipulation can be found at http://www.cv.nrao.edu/∼bkent/blender/tutorials.html this example, we use a static camera view where the user does not translate.The user is essentially placed inside the center of a celestial sphere and can rotate to view everything around them (Figure 8).Data from Skyview can be used, obtaining images centered on the galactic center (l = 0 • , b = 0 • ) in a Cartesian or Hammer-Aitoff projection (McGlynn & Scollick 1994;Mellinger 2009).Oversampling the image in a 2:1 or 16:9 aspect ratio with a 360 • ×180 • FOV is best; the end product will be viewable on different resolutions with various hardware.The workflow for this example is as follows: 1. Importing data can be accomplished with astropy.io.fits.openor with a high-resolution JPEG, PNG, or TIFF image.Images can be added via the UV/Image Editor (see the lower panel of Figure 7).The data scene requires that a uv sphere mesh be added to the scene by pressing the key combination SHIFT-A.
2. Pressing the TAB key puts the main viewer in Edit mode.
The A key followed by the U key is used to map the image onto the mesh model.

3.
A material and texture can be added to the mesh sphere with an Emission shading value of 1.0, Image set to the loaded map and Coordinates set to UV.
5. The Cycles engine can be used to our advantage here, setting the FOV to 180 • .
6.The resolution output should be set to 3840 × 2160 pixels, the highest resolution currently available on YouTube7 .The video output should be set to the MPEG with MPEG-4 encoding and a bitrate of 8000 kilobits per second.

Catalog fly-through
In this example, we tour a galaxy catalog (Tully et al. 2009) where the user is on a predefined moving track through 3D space.This automated constant speed translation is coupled with the user controlled rotation as the video visualization plays.Translating schemes like this are good for scenarios in which various aspects of the data need to be given a closer look.It also may be the case that the dataset covers a large parameter space, thus traversing a particular axis is advantageous.
1. Data can be imported from ASCII text files, XML or binary tables using standard Python modules.
2. 3D catalogs are best rendered as a single mesh with disconnected vertices in Edit mode.
3. Halo Materials are best used with a Emission shading value of 1.0 to create point sources for each catalog object.
4. A linear path is set up such that the camera will traverse 75% of the full width at half maximum (FWHM) of the width of the catalog's radial distribution.This allows reasonably uniform coverage of the viewing sphere as the user moves through the 3D catalog space.
5. Single Halo materials are not supported in the Cycles engine; we use the standard Render engine provided with Blender.
6.The resolution output should be set to 3840 × 2160 pixels (Figure 9).The video output should be set to the MPEG with MPEG-4 encoding and a bitrate of 6000 to 8000 kilobits per second, depending on the length of the translation.A trade off can be made between a longer video and a slightly smaller bitrate.

Planetary Terrain Surface
With this scenario, we can render a panoramic view of the Martian shield volcano Olympus Mons.The camera path will encircle the geographic feature, allowing the user to pan the camera in any direction during the animation.The data discussed in Kent (2015) and Smith et al. (1999) can be used as an example, with layers used to build the terrain maps8 (Figure 10).As the camera revolves around the volcano, the altitude is changed along a predetermined track of revolution.Any kind of solid 3D model viewed externally is well suited to this visualization scenario.
1. Loading planetary surface images requires both satellite imaging as well as some form of altitude ranging.These can be layered as a displacement modifier and texture for uv -mapping.
2. Surface data can be rendered as as a subdivided surface plane in Edit mode with a deformation modifier.
3. A regular surface mesh material with an Emission shading value of 1.0 works best in this example.
4. A Bézier circle can be used as a path animation combined with an Empty mesh (null vector) that will allow the camera to point toward the surface during the video.The camera is attached to this animated path.The time frame of the path animation can determine the length of the visualization.
5. The user can choose between Cycles and Render; either will work well in the scenario.
6.The resolution output should be set to 3840 × 2160 pixels.The video output should be set to the MPEG with MPEG-4 encoding and a bitrate of 6000 to 8000 kilobits per second.

Summary
We have introduced a method for creating 360 • spherical panoramas with astrophysical data.The technique employs the 3D software package Blender and the Google Spatial Media module.The outlined principles of panoramic projection allow us to use these tools in concert for immersive data exploration.By moving within a full spherical view, a user can interact with astronomical images and 3D catalogs, models, and maps.The interaction works equally well in a traditional browser environment as well as physical movement with mobile devices.We provide examples for three scenarios that a user could build upon: a static celestial sphere, a linear fly-through, and a revolving orbit focusing on a single area or object.
Demonstration videos as well as sample files, Python scripts, and basic tutorials of Blender principles are available at http://www.cv.nrao.edu/∼bkent/blenderWe gratefully acknowledge the careful reading by Tom Bridgman and Kel Elkins in improving the content and clarity of this work for the astronomical community.

Table 1. Rendering Engine Comparison
Engine Name Support Notes

Render Default
All mesh material modes supported.

Cycles
Halo materials for catalogs not supported.Benefits include using node compositor.

Game Engine
Physics-based simulation and particle generator.
Note. -The Blender Game Engine is mentioned here for completeness.While it has many important simulation properties that readers are encouraged to explore, it does not play a role in the panoramic rendering at this time.

Fig. 1 .
Fig. 1.-The three spherical panorama scenarios described in this paper.(a) A static view inside the celestial sphere where a user can move their browser or mobile device over all 4π steradians.(b) A straight path of a camera flying through a 3D catalog of astronomical objects.(c) The camera revolving around a central object.The camera tracks the object with its virtual focal plane always perpendicular to the object normal vector.

Fig. 2 .
Fig. 2.-Blender graphical user interface (GUI) setup for panoramic rendering.This particular view is using a 3D catalog of galaxies from Tully et al. (2009).The display gives the user a view along all three display axes.When rendered with panoramic settings, the result of a 360 • ×180 • FOV is shown in the upper right panel.Reproduced from the Blender GUI under the GNU General Public License (GPL) (https://www.blender.org/about/).

Fig. 3 .
Fig. 3.-Mobile device or tablet can be rotated in 3D to view any direction in the 3D space.In addition, during spherical panoramic videos, the camera can be translated in the x, y, or z directions along a predetermined path in Blender.The astronomical image subtending the sphere shows the device's FOV.

Fig. 7 .
Fig. 7.-Diagram from the Blender interface shows the uv -wrapping of a Hammer-Aitoff all-sky map onto the celestial sphere when creating a spherical panorama.Reproduced from the Blender GUI under the GNU General Public License (GPL) (https://www.blender.org/about/).

Fig. 8 .
Fig. 8.-Spherical geometry setup of a virtual camera inside the celestial sphere, showing R.A. and declination.All-sky maps are projected onto the sphere from a Cartesian or Hammer-Aitoff map and filmed into a spherical panorama by the camera.

Fig. 9 .
Fig. 9.-360 • ×180 • image showing the output from Blender with its built-in panoramic camera The data shown are a 3D catalog of galaxies from Tully et al. (2009).The resulting image will be processed by the Google Spatial Media module.

Fig. 10
Fig. 10.-Camera and its circular orbit path are shown in a simulated Martian terrain flyover.The camera default will point to Olympus Mons in this scenario.