Commentary: Augmented Reality Scientific Phone Apps – making the APD AR Holistic Review app and using existing AR apps for scientific publications

© APD SKEG Pte Ltd 1 www.scientificphoneapps.com/ Commentary: Augmented Reality Scientific Phone Apps – making the APD AR Holistic Review app and using existing AR apps for scientific publications Jun-Jie Poh1,3, Ser-Xian Phua1, Kwok-Fong Chan1, Samuel Ken-En Gan1,2,3,# 1 Bioinformatics Institute, Agency for Science, Technology and Research (A*STAR), Singapore 138671 2 p53 Laboratory, Agency for Science, Technology and Research (A*STAR), Singapore 3 APD SKEG Pte Ltd, Singapore 529757 #To whom all correspondence should be addressed to: samuelg@bii.a-star.edu.sg Received 26 September 2018, Published: 2 October 2018. DOI: 10.30943/2018/28092018 INTRODUCTION The average scientific publication is not the most palatable of reading materials, especially to those not in the relevant disciplines. Yet, conveying scientific concepts easily is precisely what scientific publications are meant to do. Imagine what the use of video pictures/paintings as depicted in the Harry Potter movie series can do to make things easier!


INTRODUCTION
The average scientific publication is not the most palatable of reading materials, especially to those not in the relevant disciplines. Yet, conveying scientific concepts easily is precisely what scientific publications are meant to do. Imagine what the use of video pictures/paintings as depicted in the Harry Potter movie series can do to make things easier! While the fantasy of moving photos/pictures does not exist physically in the real world, just as Santa Claus cannot travel the world today without being shot down by anti-missiles, there are some shadows of reality in this idea. Apart from the popular Instagram app where short video clips can be used as video pictures, other technology in the form of Augmented Reality (AR) can make this "magic" a reality in communicating challenging scientific concepts in an interesting and attractive manner.
Utilized in popular game apps like "Pokemon Go", AR can also be used in scientific publications to illustrate biological structures and concepts. With animated AR, the typical scientific paper can masquerade as published research from Harry Potter's Hogwarts School of Witchcraft and Wizardry.
While the use of AR had been gaining traction, mostly towards video games, education and entertainment, AR has yet to penetrate fully into scientific publications where it can play an important role to address the difficulty of squeezing three-dimensional (3D) ideas into the traditional two-dimensional (2D) graphics on screen/paper.

AR APPLICATION TO SCIENTIFIC PUBLISHING
Scientific publishing is the bread and butter of academic research for the sciences, and the onus is on the authors to convey their work to the scientific community and the general public (Gan, 2018a(Gan, , 2018b. Beyond academic research, it is also the responsibility of the educators and presenters to have their materials easily accessible and comprehensible. This often involves the use of graphics embedded in slide presentations in lectures. In chemistry and biology, structures are difficult to convey in even the best descriptions. Even pictures, which is said to be more than "a thousand words", are limited. This is especially so given the emergence of complex protein crystal structures that had augmented the application of drug design and discovery (Carvalho et al., 2010;Chiang et al., 2018;Kang et al., 2018).
In the example of protein models, visualization had traditionally relied stereoimages (McConathy and Owens, 2003). Today, we have software such as PyMOL (Schrödinger, LLC, 2015) that is widely used to visualize 3D structures. Nonetheless, stereoscopic views of protein structures remain the best option on printed material and on non-interactive platforms on screens (e.g. webpage, PDF, etc), but also requiring trained viewers. These stereo-images (see examples in Ban, 2000;Henderson et al., 1990), are limited in their 3D information, requiring trained crossing of eyes, and are now rarely used. This thus opens the possibility of AR.
AR apps have the potential to be companion scientific apps to scientific articles and are likely to be the next revolution of digital scientific publishing. In addition, these are likely to be a common type of scientific phone apps of the future. While it would be ridiculous to have every scientific publication having its own mobile AR app to burden the reader, we wish to illustrate here the potential use of such AR apps in simplifying the communication of complex scientific concepts and ideas.
Coupled with animation, AR apps can better communicate concepts and ideas, providing an additional dimension without changing the graphic figures, past (previously published), present, and future.
To demonstrate how AR apps can be incorporated into the process of scientific communication, we take the example of our previously published paper (Figure 2 from Su et al. 2017) and a perspective review paper that we have submitted titled "Perspective: The Promises of a Holistic View of Proteins -Impact on Antibody Engineering and Drug Discovery". Both articles incorporated many structural biology illustrations. To demonstrate how the proteins interact with each other, we incorporated AR animations to show protein binding and movement. This was performed using Unity and MAXST.AR software to create the "APD AR Holistic Review" apps and also to utilize existing AR app -HP Revealed (formerly known as Aurasma).

Figure 1. Screenshot of APD AR Holistic Review
App with AR depiction of immunoglobulin G antibody. The publication graphic is hidden under the AR circular plate.

Development and Usage of APD AR Holistic Review
We used UCSF Chimera 1.11.2 (Pettersen et al., 2004) to generate static 3-D protein models from PDB file format trajectories before exporting the X3D file format for Blender 2.79 (https://www.blender.org/) which was used to convert the X3D file to FBX file for the Unity3D (https://unity3d.com/) engine. Unity version 2017.3 was used to further animate the 3D models to project a fourdimensional view of the protein with movement Using the MAXST.AR software development kit ver. 3.5, an Image Tracker GameObject was created, with a 2dmap file format generated from MAXST Developer (http://www.maxst.com/) website using the original graphic from the publication attached to the Image Tracker. The 3D animated model is then attached to the Image Tracker which would then be projected whenever when the smartphone camera detects the publication graphic. This process was built using the Unity software. Both iOS and Android apps were build using the Unity software before uploading to the respective app stores.
To use the APD AR Holistic review app, simple download and install the application, activate it, and point the smartphone camera at the publication graphic. The 3D animated AR projection of the figure will appear. The camera can be moved to view around the animation so long the image remains detected.

Development and Usage of HP Reveal AR models
As was performed above for the APD AR Holistic App Review app with the difference that the X3D file was converted to a DAE file. Decimation (reduction of triangles in a model) was performed on the 3D protein structures to fit approximately 15000 triangles for an optimum performance as suggested in HP Reveal 3D guidelines.
The publication graphic was uploaded as a "Trigger Image" into HP Reveal Studio Aura Creation. The thumbnail, texture image files and protein DAE file were compressed and uploaded as an "Overlay". The 3D protein model is linked to the publication graphic as an Aura and can be viewed on the application as soon as the model has been saved on the Studio platform.
To use the application, simple install and access the HP Revealed app that will open the HP Reveal scanner. Simply point the smartphone camera at the publication graphic and the AR model will appear. Move the camera or the image target to view the 3D protein structure from more perspectives.
For private sharing of the Auras, access "My Account" in HP Reveal Studio. Users may Share Campaign to obtain a link for sharing the AR models for other mobile devices that has the installed application.
HP Reveal is available in Google Play Store (https://play.google.com/store/apps/details?id=co m.aurasma.aurasma&hl=en), and Apple App Store (https://itunes.apple.com/us/app/hpreveal/id432526396?ls=1&mt=8) While the review we have written is currently under review. The title could be searched for in time for viewing the AR, but in the meantime, Figure 2 from Su et al., 2017 may be used for viewing the AR.

MAIN CONSIDERATIONS
One of the disadvantages of the APD AR Holistic Review is the size of the application package. Currently, the 3D models and the image tracker library are embedded in the application, which makes the app file sizeable. The future may include uploading models onto cloud storage which would then be fetched when the image tracker detects the publication graphic, reducing significantly the download file size of the application. Nonetheless, this may incur delays as the files would then be downloaded via the Internet live. This then is a balance between a one time large file download of the app or an ad-hoc download of the animations. With the inevitable increase in computational power in the future, an AR of a full molecular dynamics simulation using its trajectories may be possible.
In the utilization of the HP Reveal App, we had to reduce the protein model quality by flattening the surface model in order to allow a reasonable experience using the HP Reveal App.
The accuracy of the protein surface representation is thus lowered. In addition, the text orientation is fixed to the model, which makes it difficult to read at certain angles.
In the development of our own AR app, we had issues in listing the app in the Apple App Store which initially found the app to contain "minimal functionality". In overcoming this, we had to add instructions and other features for it to be approved.
Our work is a proof in concept to the possibility of incorporating more AR technology as scientific apps into scientific publishing in which is exciting due to its ability to convey graphic information better. Using the example from Figure 2 of Su et al., 2017, structural proteins crammed into a two-dimensional graphics are often intimidating and technical for readers not familiar with the field. With AR, concepts and ideas can be visualized by the reader with minimal technical knowledge. In addition, we believe the effort to better complemented with animation that would also convey key time-based events revealed in molecular simulations of proteins.

FUTURE WORK
There is still great room to grow for scientific phone apps in scientific and medical applications (as previously reviewed in Gan, 2018c;Gan and Poon, 2016) where they can replace laboratory equipment and make scientific research more convenient (in examples as shown in Ng et al., 2017;Sim et al., 2015;Wong et al., 2016), AR can be incorporated into these equipment replacing apps in time. With the development of mobile phone and cloud computing technical capabilities, tapping into more computational power to generate coarse grained and atomistic simulations in the future would be possible.