Game-Engine-Assisted Research platform for Scientific computing (GEARS) in Virtual Reality

The Game-Engine-Assisted Research platform for Scientific computing (GEARS) is a visualization framework developed at the Materials Genome Innovation for Computational Software (MAGICS) center to perform simulations and on-the-fly data exploration in virtual reality (VR) environments. This hardware-agnostic framework accommodates multiple programming languages and game engines in addition to supporting integration with a widely-used materials simulation engine called LAMMPS. GEARS also features a novel data exploration tool called virtual confocal microscopy, which endows scientific visualization with enhanced functionality. © 2019TheAuthors.PublishedbyElsevierB.V.ThisisanopenaccessarticleundertheCCBYlicense


Motivation and significance
Data visualization plays a key role in scientific discovery. Though quantitative analysis is indispensable, researchers are often forced to apply summary metrics blindly. Unfortunately, some of these statistics are limited in their ability to describe the system under test and can be misleading, as illustrated by Anscombe's Quartet [2].
Visualization software such as VisIt [3], ParaView [4], Visual Molecular Dynamics (VMD) [5], and OVITO [6] provide a straightforward interface to create three-dimensional (3D) images and observe patterns in the output of simulations. Though often used among experienced researchers, it remains challenging to extract information from datasets that consist of complex geometries or contain a large number of entities because of the intrinsic limitation of two-dimensional (2D) representations of 3D objects for traditional displays. Immersive 3D visualization technology like ImmersaDesk, Tile Wall, and CAVE2 [7] provide unique spaces for collaboration and scientific communications [8]. These solutions, however, require specialized knowledge of the respective systems   [1]. The skeletal hands are processed by the LeapMotion library in the Unity 5 Game Engine. Using finger pinches and gesture controls, the user can rotate, translate, and scale the structure intuitively in immersive VR. The surrounding graphical user interface is the Unity editor. and clear expectations of the simulation's outcome. Furthermore, the high costs of these systems make them prohibitive for most researchers.
An affordable alternative is an easily-accessible, agnostic software platform designed for the increasingly available headmounted displays (HMDs) developed for consumer virtual reality (VR). VR is a powerful visualization tool that has grown steadily in popularity over the past two decades, fueled by the success of HMDs like Oculus Rift [9] and HTC Vive [10]. Despite its widespread adoption for video games and media applications, as well as its potential as the most immersive and intuitive method for viewing data, VR has been underutilized by the scientific community. To transform VR into a common scientific tool, we have developed a software framework, called Game-Engine-Assisted Research platform for Scientific computing (GEARS) that facilitates the adoption of VR technologies and allows researchers to take advantage of the unique analytical advantages that the medium offers ( Fig. 1).

Interactive data visualization
GEARS makes use of commodity game engines, like Unity [13] and Unreal Engine [14], to simplify access to VR headsets. The most straightforward application of VR to data visualization is interactive viewing of pre-computed results for data exploration. To realize this feature, GEARS employs the iBET workflow [15], in which external 3D modelers, such as VMD and Blender [16], are used to create a 3D object that can be added to the scene along with the appropriate script (for example, LeapRTS.cs for the Leap Motion controller [17]) (see Fig. 2).
This aspect of GEARS allows users a quick, straightforward outlet for immediate visualization of snapshots of data from materials simulations or molecular structures. Though the current interactive visualization mode supports single frame data, we plan to expand this feature by creating multiple scenes containing different simulation time steps for more dynamical data exploration. Fig. 3 showcases several examples of GEARS interactive visualizations used in computational researches.

Real-time simulation visualization
GEARS also takes advantage of the programing capabilities provided by game engines, such as C# and JavaScript supported by Unity and C++ for Unreal Engine, to facilitate reuse of existing simulation codes. This mode of GEARS is suitable to explore simulation results in real-time, rendered entirely within the game engine.
To realize this real-time rendering, it is critically important to design an efficient data-bridging method between the game engine and user-developed simulation program. GEARS employs two approaches, called Run-when-Ready and Render-when-Ready, depending on the size of the data and the complexity of the simulation engine. Run-when-Ready calls the simulation engine upon a frame update to advance the state of the simulation (for example particle positions) of the frame by one timestep. Render-when-Ready makes use of the multi-threading optimization technique to offload the simulation engine computation onto a new thread while the main thread is only responsible for handling the render state of the game engine. In this approach, when the simulation engine finishes one timestep computation, the main thread either updates the frame state or stores the simulation states for rendering later while the worker thread continues to produce new states as background. Using this technique as well as other optimizations, like impostors and GPU instancing [18], we have demonstrated real-time simulation sizes reaching up to 500,000 particles. To minimize the amount of coding necessary for GEARS users, we provide two demos -Lennard-Jones molecular dynamics (MD) [19] and electron transfer simulation by kinetic Monte Carlo [20] -implementing the Run-when-Ready and Render-when-Ready approaches in the GEARS GitHub repository. See Demo3 of UnityGEARS on the GEARS repository.
In an effort to make our immersive scientific computing suite accessible to a broad research community, we have also interfaced GEARS with one of the most widely used MD simulation engine called LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) [21]. LAMMPS was developed at Sandia National Laboratory and supports a variety of interatomic potentials, statistical ensembles, and flexible simulation setups. The LAMMPS interface enables users to visualize their MD simulations in VR environment without any coding. A ''How-to'' for setting up the LAMMPS integration environment and an example demo are documented in detail in the Lammps Compliation section of UnrealGEARS on the GEARS repository.

Virtual confocal microscopy
To fully realize the promise of immersive scientific computing, we have developed a new tool to enhance GEARS' visualization capability, called virtual confocal microscopy. Confocal microscopy has become an indispensable biomedical tool, and it has proven useful in many other scientific fields [22][23][24][25]. It allows for increased resolution, contrast, and optical sectioning while minimizing photodamage and bleaching concerns. The technique uses point illumination and a pinhole to scan the sample and reject outof-focus light. As a result, the image only contains information very close to the focal plane, offering greater resolution [24]. An analogous approach in VR could enable more methodical investigations of simulated complex structures.
Virtual confocal microscopy utilizes Unity's surface shader capabilities to control how each vertex on the structure is rendered in order to highlight certain areas or planes of the material in the simulation. Our solution seeks to generate a viewing plane that sits in front of the user's head, follows their head movement, and always maintains a set distance from the user. This distance, as well as the thickness of the highlighted viewing plane can be specified In the Data Acquisition step, a researcher supplies data to the engine by either converting pre-computed results to a particular format (e.g. .obj, .fbx) or computing data on the fly via simulation. In the Interface Selection step, a GEARS user selects the method or interface through which they desire to interact with their pre-computed data or real-time simulation. In the Post-Processing step, a researcher selects a post-processing technique, which can either utilize the control scheme they chose in the Interface Selection step or make use of custom code. (b) An example of visualizing a chemical vapor deposition (CVD) simulation [11,12] through GEARS' LAMMPS integration. (c) Simulating and exploring data related to a molybdenum disulfide (MoS 2 ) fracture via the LAMMPS integration. Using the controllers indicated by white lines, researchers can freely navigate through the structure, pause and resume the animation, and even highlight regions of interest with the provided laser attachment (shown in green). and changed dynamically by the user. The rest of the simulated system will stay mostly transparent (with the opacity also dictated by user input), except for the vertices that intersect with this plane. In this way, the user can scan through complicated geometries using just their head movements -a unique experimental advantage only possible in VR.

Control schemes
A carefully designed control scheme is essential to facilitate interaction between the user and various data representations within the immersive simulation. These interactions range from object manipulation and spatial translations to temporal scaling. It is critical that these interactions mimic natural motion as much as possible to limit user discomfort, especially as it relates to user motion within the environment [26][27][28]. GEARS supports input from a variety of sources that includes the Leap Motion controller, Oculus Touch, and HTC Vive motion for immersive environments, as well as keyboard and mouse for small and precise adjustments to aspects of the simulation.

Illustrative examples
Recently, researchers performed MD simulations of desalination membranes to understand and characterize how local, atomic structures contribute to macroscopic behavior [31]. The relevant dimensions of the polyamide membranes of interest range from a tenth of a nanometer to 100 nm, making it an excellent candidate for computational studies to extract optimization information not easily accessible experimentally. They have found that water molecules permeated the membrane through benzene rings and  [29]. (d) Strain-induced structural transformation in an atomic telluride layer [30]. that the degree to which these monomers were cross-linked governed the speed of permeation along various paths [31]. However, the construction of these membranes is very complex, making a quick identification of preferential paths difficult with traditional 2D visualizations. The ability to section such a complicated super-structure in VR environment with simple head motion helps researchers identify prevalent substructures of interest in these simulated materials, and will greatly accelerate the research production cycle (see Fig. 4).

Impact
Physical simulations, like MD, inherently exist in 3D space, yet modern, desktop viewing platforms tend to only support 2D visualizations. GEARS, as an open-source, commodity platform, makes exploratory visualization and immersive data analysis more accessible for researchers. Using our workflow, outlined above, and our provided vertex shader, one can apply 3D virtual confocal microscopy to any discrete data set. Alternatively, should a researcher need to create their own customized shader (depending on their investigative needs), they can make use of the sample environments we have provided to productively bring their own data to an interactive, immersive environment. Consequently, this manuscript sets a foundational codebase for scientists to make use of modern graphics shaders for experimental data analysis.
Even if shader technology or virtual confocal microscopy is not a primary goal, researchers can still make use of the immersive playback capabilities in GEARS. For pre-computed simulations, this generally means pausing animations then exploring and analyzing sequestered regions. However, not only have we provided an immersive visualization engine for researchers, but also a simulation platform that encapsulates much of the complicated logic necessary for real-time data analysis. Researchers can choose to either port their existing data (e.g. LAMMPS DUMP files) or run entirely new simulations on-the-fly. With our framework, LAMMPS scripts can be written with VR visualization accounted for by design. Researchers can even pause a simulation mid-run, giving them opportunities to make adjustments to their simulation before supplying new commands to the linked LAMMPS library through our code interface. Afterwards, they can continue running the simulation and note the effects of their adjustments to the system. This option for immersive simulation steering is just one example of how our software also acts as a platform for VR-oriented extensions in LAMMPS. By integrating LAMMPS with Unreal, much like Atomify [32] integrated the MD simulator with mobile operating systems, we have improved the framework's extensibility as a research tool. Additionally, build upon our sample environments, researchers can more productively take advantage of the engine's rich features -for instance, its native server framework for multiuser LAMMPS VR collaboration, built-in blueprint system for interactive UI, and plugin controls for immersive navigation of a 3D simulation.
GEARS in Unity and Unreal act as an extensible hybrid simulation and visualization engine. It has already seen use in classroom and research settings, acting as a new medium for presenting data. This medium allows researchers to take advantage of humanity's innate spatial awareness to improve its user's productivity as well as a new avenue for scientific communication. It has facilitated the exchange of simulation data to both experts and non-experts, most notably at U.S. Department of Energy (DOE) Materials Genome Innovation for Computational Software (MAGICS) workshops and at a DOE conference in Washington DC.

Conclusions
We have developed a hardware-agnostic visualization framework called GEARS that takes advantage of the unique possibilities and power associated with the use of VR environments and rapidly evolving game engine technology to explore scientific datasets. VR has become a powerful new method of engaging with users in a more tactile, visceral way. The resurgence in hardware to support VR has resulted in a rich software environment that allows developers access to these tools with minimal investment. While VR technologies have a myriad of applications, they could be particularly useful for scientific data visualization and exploration, collaboration with domain experts, as well as communications with non-scientific audiences. With GEARS, we provide various approaches and modalities for researchers to interactively explore their simulation data, from visualization of pre-computed datasets in VR to the integration of external simulation engines such as LAMMPS. GEARS even allows for researchers to take advantage of immersive data visualization and simulation without requiring additional coding. Driven by the multimedia and gaming industries, it is likely that VR platforms will continue to enhance user functionality and interactivity at lower costs, making them even more approachable to researchers. Therefore, the novel data exploration and collaboration capabilities GEARS offers will be broadly beneficial for scientific community.