MEGA Vision: Integrating Reef Photogrammetry Data into Immersive Mixed Reality Experiences

Coral reefs and submerged cultural heritage sites are integral to supporting marine biodiversity, preserving human history, providing ecosystem services, and understanding drivers of ecosystem health and function. Despite the importance of these submerged underwater habitats, accessibility to these environments remains limited to specialized professionals. The MEGA Vision mixed reality application integrates photogrammetry-derived data products with augmented reality (AR) technologies to transcend this barrier, offering an immersive and educational platform for the broader public. Using high-resolution imagery from SCUBA expeditions, the app presents users with realistic and spatially accurate 3D reconstructions of coral reefs and submerged archaeological artifacts within an interactive interface developed through Unity and Vuforia. The applications’ instructional design includes multimedia elements for enhancing user comprehension of marine and historical sciences. This mixed reality tool exemplifies the convergence of scientific data visualization and public engagement, offering a unique educational tool that demystifies the complexities of marine ecosystems and maritime history, thereby fostering a deeper appreciation and stewardship of underwater environments. By enabling accessible, interactive, and immersive experiences, the application has the potential to revolutionize the way we interact with and contribute to marine sciences, aligning technology with conservation and research efforts to cultivate a more informed and environmentally conscious public.


Introduction
Coral reefs are productive marine ecosystems that support high levels of biodiversity and provide many important ecosystem services to humans (Komyakova et al., 2013, Hughes et al. 2002).The calcium carbonate structure created by living corals contribute to ecosystem functions including coastal protection, habitat partitioning and carbon sequestration (Moberg & Folke, 1999, Graham and Nash 2013, Komyakova et al. 2013).Coral reefs are intertwined in the livelihood and well-being of humans, but their importance is often poorly understood due to their isolation from many human societies around the globe.
While coral reefs provide important biological resources (Grafeld et al., 2017), underwater cultural heritage sites hold significance across historical, archaeological, scientific, educational, and cultural dimensions, serving as irreplaceable windows into past civilizations, maritime activities, and cultural exchanges.These submerged sites, including shipwrecks, are vital for scientific research, offering unique preservation conditions that help archaeologists uncover details about historical periods and providing scientists with information about coral resilience to disturbances.Beyond their research value, these sites also play a crucial role in education and cultural identity, providing dynamic platforms for public engagement and learning about maritime history and the importance of heritage preservation (Burns et al. 2023).Through their exploration and study, coral reefs and underwater cultural heritage sites foster a deeper connection to nature and our collective past.
Access to these underwater sites has been limited to professionals with specialized training and equipment, restricting wider public engagement.The limited accessibility to submerged habitats has hindered scientists' capacity to communicate the significance of these underwater environments to public audiences.This project integrates photogrammetry data and mixed reality tools to make these sites accessible to broad public audiences.By using accurate three-dimensional (3D) reconstructions rendered from imagery collected during SCUBA expeditions, we generated detailed representations of these environments and artifacts into a user interface to enable immersive virtual exploration of these study systems.The technical approach used to create precise 3D reconstructions of both natural and historical features (Burns et al. 2015, Burns et al. 2023) creates high resolution and spatially accurate digital models of various reefs and archaeological features.Integrating these 3D data products into a mixed reality virtual interface facilitates meaningful educational and scientific exploration of scientific data products.All 3D models were generated using photogrammetric reconstruction methods, which is a useful tool for generating accurate 3D models from overlapping 2D photographs (Ferrari et al. 2022).This approach enables the creation of spatially accurate representations of a range of marine environments and submerged archaeological features.The process involves highresolution photographs taken by SCUBA divers during methodical underwater surveys.The photogrammetric technique processes these images to produce detailed point clouds and texture maps that capture the fine details of coral reefs and shipwrecks at resolutions approaching sub-centimeter levels (Burns et al. 2015, Burns & Delparte 2017).This technology provides a reliable means to study and monitor complex underwater structures and ecosystems, contributing significantly to the field of marine research.The models produced from the various studies utilizing this approach were exported as Wavefront Object files.The resolution and texture of the models were reduced to make them more compatible in a mobile mixed reality environment.The 3D models were then integrated into an innovative mixed reality application, MEGA Vision, to provide an immersive user interface for viewing and exploring these data products.
The MEGA Vison mixed reality application is built on the Unity game engine, which integrates the photogrammetry-derived 3D models into a virtual environment.This platform also supports the development of an interactive user interface (UI).With the assistance of the Vuforia AR engine, the app overlays the models onto the real-world camera viewer finder, offering a coherent and engaging experience.Users can navigate and interact with the virtual representations of underwater sites, bridging the gap between virtual and physical realities.This application, leveraging Unity and Vuforia, enables virtual exploration of coral reefs and historical wrecks, providing an accessible and educational tool that transcends the barriers of direct access and supports research and conservation initiatives.
The mixed reality application is designed not only for visual engagement but also for the dissemination of knowledge.It integrates interactive annotations and overlays with multimedia elements to provide relevant ecological information pertaining to coral ecosystems, contextualize maritime archaeological sites, and highlight ongoing research and preservation initiatives.The integration of instructive material with mixed reality technology is designed to advance environmental consciousness, foster multidisciplinary collaborations, enhance public involvement in marine science, and generate awareness of the importance of maritime heritage research.
The primary goal of the present research is to deploy and assess the utility of a mixed reality tool that offers users an experiential and pedagogic interface, concurrently demonstrating the practical utility of photogrammetry-derived data products for educational experiences.Utilizing Unity and Vuforia platforms, and leveraging existing datasets created using photogrammetry, the project successfully produces immersive digital experiences for underwater environments.This provides users an innovative mechanism to view and understand the biodiversity and historical values of underwater habitats.Beyond facilitating virtual exploration, the application contributes to wider objectives encompassing ecological preservation, scientific investigation, and the value of underwater cultural heritage sites.
We also believe that mixed reality tools can serve a valuable purpose for scientific research by facilitating detailed cataloging and temporal monitoring of marine ecosystems.The highresolution digital models serve as precise baselines for tracking ecological changes, assisting in the evaluation of reef health and the impact of anthropogenic stresses.This technological approach offers a non-invasive means to repeatedly assess reef conditions, supporting conservation efforts with up-to-date data that is critical for informed decision-making.Mixed reality applications have the capacity to create streamlined digital libraries that can collate multiple datasets and increase access for scientists and conservationists.This can allow for more data sharing and ability to track temporal changes in ecosystem health and function.Sharing 3D reconstructions through these applications can enhance scientific collaboration and streamline access to baseline data, enabling swift review and analysis that may catalyze further research activities.
Incorporating mixed reality into educational programs will directly enhance science education and outreach.By providing an interactive platform for students to explore marine environments, it promotes engagement with scientific content and may encourage interest in marine science careers.This tool exemplifies the potential of digital technologies to supplement traditional classroom resources and to offer novel experiences that contribute to the understanding of complex ecological and archaeological subjects.
Figure 1.Examples of the application interface instructions guiding the users how to use the mixed reality application and download physical targets for 3D reef exploration.

Methods
The application was primarily developed in the Unity Engine (2018.4.20f1), and the user interface is composed of several distinct scenes corresponding to different interfaces: a splash warning screen, tutorial screens, main menu, the primary application interface, and an about section.The user interface was engineered to be intuitive across a wide array of device screens, ensuring responsiveness from small smartphones to larger tablets.UI components, including buttons and text, were dynamically anchored and scaled to adapt to varying screen dimensions.The textual content and graphical elements within the application are dynamically scaled to accommodate the diverse screen dimensions of user devices.
Upon launch of the mixed reality app, the user interface presents a compulsory advisory notice, followed by a visual tutorial that users may bypass (Figure 1).To address potential challenges in understanding mixed reality and three-dimensional (3D) modelling concepts among a heterogeneous user base, a tutorial section has been incorporated.The graphic assets for the tutorial and interface elements were developed Adobe Photoshop (version 24.0.0) and Illustrator (version 27.0).Interactive features and the application's navigational structure, including UI components such as informational dialogues, were programmed in C#.Additionally, an informational icon, denoted by an "i," was implemented to activate a pop-up window upon being pressed.This modal provides contextual data pertinent to the 3D model and includes the map location where the photos used to generate the model were captured.
The application utilizes the Vuforia Engine (version 10.15.4) to facilitate the mixed reality capabilities used in the app.Image targets, which the Vuforia Engine can recognize and track, were created with a rudimentary design consisting of gray-hued triangles and lines.Nine distinct patterns were produced and uploaded into the Vuforia Target Manager.The novel triangle and line pattern was chosen based on the high rating in terms of both detectability and trackability.A target database compiled by Vuforia was subsequently imported into the Unity environment, establishing a linkage between each target and its corresponding 3D model.The models were scaled and positioned to align with the real-world scale of the mixed reality targets (Figure 2).Upon target detection, the corresponding 3D model was programmed to be overlaid on the target in the predetermined orientation and scale.To optimize system performance, the tracking functionality was constrained to a single target at a time.In addition to the triangle and line targets, 19 targets associated with tangible stickers, logos, and graphics created by the MEGA Lab were incorporated into the application.For these targets, a VideoPlayer component was overlaid on the image target (Figure 2).The video content associated with these targets included short 1080p clips of eductation videos, documentary segments, and digital animations.The VideoPlayer was programmed to start video playback upon target detection and to pause in the event of target loss.This allows users to resume interacting with videos without restarting them from the beginning.Utilizing an array of interactive targets provides a unique explorative experience for the user.The content displayed on the screen is controlled by where the user is directing the camera.The mixed reality interface provides access to a multitude of media products which provide valuable information about science, reefs, and underwater cultural heritage.This innovative approach allows the user to choose their own adventure and explore information and content based on their personal interests, maximizing the capability and efficacy of the mixed reality application.
Before publication, the mixed reality application underwent an evaluative study with 20 individuals comprising marine biologists, diving enthusiasts, and laypersons.This evaluation aimed to gauge the application's user-friendliness and functional efficacy.The application's performance metrics, including render speeds, tracking precision, and system resource demands, were systematically measured on an array of mobile devices.
Comprehensive benchmarking procedures were performed to quantify frame rate, response time, and energy expenditure, to confirm the application's adaptability to varied device capabilities.The participants' input and constructive critiques were integral to the subsequent optimization of the application.
The application was developed for iOS using Unity Engine and was compiled with Microsoft Visual Studio 2022 (version 17.6) and Xcode (version 14).The application was then signed and uploaded to the iOS app store using an Apple Developer account.Following approval, the application was made accessible for download.A beta version for Android has been developed but is currently undergoing performance and compatibility testing.The Android version aims to match the capabilities of the iOS version and utilizes the Android Software Development Kit (SDK) and Google's Android App Bundle for the management of large-scale applications.The cutaways indicate how large the augmented reality animation or model will be viewed in relation to the real-world target.

Results
The MEGA Lab mixed reality application launched on August 2, 2020 for iOS.This application serves as a pioneering outreach tool for the MEGA Lab.It has achieved notable success, evidenced by several thousand downloads and a 5-star rating for the iOS version on the Apple App Store, and hundreds of downloads for the Android beta version on the Google Play Store.The application has significantly impacted visibility, with tens of thousands of online impressions across the iOS platform.It has undergone 7 major updates for iOS and 6 for the internally released Android version, demonstrating a commitment to improving and expanding its features.The iOS app, at 1.4GB, offers nine high-resolution models and 13 short animations and videos in 1080p.Its integration into public events across all age groups and the addition of video target functionality underline its value in educational and cultural contexts.The application supported the "He'e Nalu: The Art and Legacy of Hawai'ian Surfing" project at the Heard Museum and was showcased in the "He Honua Ola: A Thriving Coral Reef" exhibit at the ʻImiloa Astronomy Center, amplifying its reach and impact.The application is used daily at the public Mokupāpapa Discovery Center, a federal Museum highlighting ocean conservation in downtown Hilo (Figure 3 and 4).The application was also used at large events including the World Surf League Pipeline Masters on Oahu, HI, and Camp Shred in San Diego, CA.Thousands of visitors engaged with the application to learn coral reefs, underwater cultural heritage, and the value of science to our daily lives.This success motivates the MEGA Lab to continue enhancing and developing mixed reality tools to stimulate exciting and immersive educational opportunities for public audiences.
Figure 3. Real world examples of the mixed reality application being used for Astronomy Exhibits, school education, and scientific outreach events for public audiences.

Discussion
The MEGA Lab mixed reality application effectively translates scientific data from underwater photogrammetry research into accessible, interactive experiences for the public.Its design enables users, including children, to explore 3D models and videos through smart devices, enhancing educational outreach.The application's interface and immersive capabilities engage a broad audience, fostering educational engagement, particularly among students.By enabling students to engage with their physical surroundings and allowing them to interact with digital 3D models, it deepens their understanding and connection to the content beyond traditional teaching methods.The expansion of its target library to include various logos and images has significantly increased user interaction, making everyday environments interactive and educational.Incorporating video targets adds a layer of content diversity, appealing to a wider audience and enhancing the app's utility for knowledge dissemination.
The deployment of the application has yielded a notable increase in student engagement.By integrating the creation of 3D models via photogrammetry and their exploration through mixed reality, the application supports an immersive pedagogical strategy.This method enhances students' understanding of the conversion from tangible objects to digital analogues.Such interactive learning experiences transcend the constraints of conventional teaching techniques, offering a robust medium for the conveyance of scientific information.
The expansion of the application's target database to incorporate logos, graphics, and imagery has markedly amplified its interactive capacity.This development permits the incorporation of a multitude of 3D models and multimedia content, allowing for the transformation of standard environments into engaging educational settings.The introduction of video-based targets provides an alternative modality for content delivery, appealing to users who engage in more observational forms of learning.This adaptability significantly extends the application's reach and potential for adoption across diverse user demographics.
Analyzing user interaction with the mixed reality app does indicate diminished capacity to sustain user engagement over prolonged periods.Initial interactions are characterized by high levels of engagement; however, the finite number of targets leads to a rapid diminishment in user activity.This trend suggests that future versions of the application could benefit from the integration of an expanded repository of targets, encompassing a diverse spectrum of 3D models and multimedia content that can be accessed via internet connectivity.Future enhancements include downloadable content modules featuring renowned surfing locations, historical shipwrecks, and tools for coral species identification and disease diagnosis, which would serve to augment the user experience and ensure a renewable supply of content.
A primary technical constraint of the application is its large app size of 1.4GB.This size necessitates considerable device memory, high-speed internet for downloading, and user patience.
The file size has notably impeded adoption, particularly in areas with constrained internet infrastructure, as evidenced by collaborative efforts with institutions like the Heard Museum in Phoenix, Arizona, and the ʻImiloa Astronomy Center in Hilo, Hawai'i.To address this, future versions of the mixed reality app will concentrate on minimizing the core application size while offering modular, downloadable content aligned with individual preferences.Such a design would mitigate storage limitations and facilitate the inclusion of extensive content without exceeding the data thresholds imposed by app stores.The MEGA Vision application serves as a digital platform that bridges the gap between scientific data curation and communitybased participatory research.By involving users in the process of data collection and interpretation, the application fosters a sense of ownership and responsibility towards marine conservation efforts.This participatory approach not only enhances public scientific literacy but also contributes to a larger dataset, which can be invaluable for researchers tracking environmental changes and assessing conservation strategies.
The application also has simulation capabilities that can use used to conduct precise modelling of coral reef dynamics within a virtual setting, offering a consistent and replicable medium for testing ecological hypotheses.These simulations can help ecologists project the impacts of protective measures on coral health and resilience, as well as forecast the repercussions of threats like climate change, ocean acidification, and overfishing on reef ecosystems.Employing this technology for scenario analysis can inform policy development and the design of marine protected areas, by elucidating the potential long-term effects of different management approaches on coral reef survival and recovery.

Conclusion
The MEGA Vision mixed reality application represents a synthesis of technological innovation, scientific investigation, and educational outreach.Utilizing Unity and Vuforia alongside Structure-from-Motion (SfM) photogrammetry, the platform offers users an interactive and instructive virtual exploration of Pacific coral reef ecosystems and maritime archaeological sites.Despite its achievements, the application can still be improved and refined to optimize maintaining user engagement over time and managing large data requirements.
For future versions, it is planned to extend the library of targets and enhance the diversity of 3D models and multimedia content, which could be made available via online download.Refinement of the application size, through a streamlined base application with optional content packages, is anticipated to mitigate storage issues and foster broader adoption.Furthermore, the exploration of advanced AR methodologies, such as Simultaneous Localization and Mapping (SLAM), may present opportunities to refine the user experience by reducing the dependence on predefined image targets.
In advancing these enhancements, the MEGA Vision mixed reality application aspires to evolve into an influential instrument for science, education, and citizen science by allowing users to contribute to marine research by uploading geo-referenced imagery.This tool aims to narrow the divide between scientific knowledge and public awareness, thereby altering the paradigm of engagement with marine habitats and catalyzing worldwide participation in marine conservation efforts.The overarching goal is to encourage environmental responsibility, crossdisciplinary cooperation, and a more profound comprehension the importance of healthy marine ecosystems and maritime heritage.

Figure 2 .
Figure 2. Development view in Unity with (left) an illustrated cutaway showing the augmented reality target beneath the 3D model (left), and an illustrated cutaway showing another augmented reality target beneath the VideoPlayer pane (right).The cutaways indicate how large the augmented reality animation or model will be viewed in relation to the real-world target.

Figure 4 .
Figure 4. Examples of the mixed reality application targets.The coded target (top left) displays a high-resolution 3D reef reconstruction.The lab logo (top right) displays cartoon animations used to communicate MEGA lab research activities.The still image (bottom) displays a video from an educational documentary about mapping coral reefs in Fiji.Each target provides unique forms of immersive digital media for the user to engage with through the application.