SYNTHESIS: A Platform of Platforms for Integrated Management, Curation, and Visualization of Digital Cultural Experiences through VR and AR Technologies

SYNTHESIS is a platform of platforms (PoP) for the management, curation, and creation of digital cultural experiences related to cultural heritage material (artifacts), and their visualization through mobile applications and virtual and augmented reality (VR/AR) technologies. The digital cultural experiences may relate to specific cultural heritage monuments or exhibits of museum collections and may reside either on site or remotely. The platform provides a comprehensive supervisory tool that allows curators to select digital artifacts from different databases around the topic of interest, associate them through narratives that translate them into a sequence of semantic correlations that can be visualized using VR/AR technologies. The SYNTHESIS of platforms contributes to the creation of a new condition for the promotion of cultural heritage sites, museums, and exhibits, and facilitates the management and interaction of the user-visitor with them. The platform contributes to the creation of an innovative system for the management and curation of large volume of digital material related to an exhibit and consequently its promotion by unifying the stages of: (a) production of digital content; (b) con-nection of additional digital multimedia material related to the digital content; (c) curation of the digital content and transformation to a digital experience in accordance to a desired narrative; and (d) visualization of the digital experience in-situ or remotely with the use of mobile apps and VR/AR technologies.


Introduction
The SYNTHESIS PoP has emerged from the aggregation of three digital platforms that have been developed at the Integrated Systems Laboratory (ISL) over a period of seven years in the context of DICE (Digital Immersive Cultural SYNTHESIS: A Platform of Platforms for Integrated Management, Curation, and Visualization… DOI: http://dx.doi.org /10.5772/intechopen.98213 In the context of a digital experience (be it at a natural exhibition or museum, an archeological space, a virtual space, or a hybrid environment), it is possible to connect the digital exhibits semantically and thematically digital exhibits to a physical or virtual exhibition, museum, or archeological site, through the use of graphs and graph databases (Figure 2). In this process, the content reporting platform wayGoo [10] is used to visualize the georeferenced narrative in an easily to understand way in a physical, virtual, or hybrid reality space.
The easy understanding of the content by the end user in the context of a narrative is achieved through the transformation of the narrative into an immersive interactive experience capable of being consumed by common mobile tablets and smartphones using AR technologies in the exhibition/museum space/archeological site, Figure 3a, as well as through more integrated solutions such as VR headsets and/or projection imaging technologies for remote experiences in virtual space, Figure 3b. A narrative can be created by artists and people in the field of culture, using an accessible and friendly graphic environment. Storytellers will not be required to possess advanced and specialized knowledge of the underlying technologies used to implement it. SYNTHESIS is accompanied by two applications available for use by the public (Figure 4).  1. An augmented reality application for mobile devices that offers possibilities of interactive exploration of an exhibition and consumption of all available digital material associated with it as an immersive storytelling experience during the physical presence of the user in situ.

2.
A virtual reality application that allows the interactive exploration of an exhibit remotely and the consumption of rich 3-dimensional models and interactive scenarios with realistic graphics directly from a VR-enabled device.
The SYNTHESIS PoP also offers advanced tools based on artificial intelligence (AI) and natural language processing (NLP) techniques to assist the discovery of contextual correlations between digital content, digitized assets, and digital   artifacts and exhibits [13,14]. By analyzing the structural elements of the exhibits alongside with all known connections among them, the system identifies possible undiscovered correlations and provides a multimodal and user-friendly representation and visualization of the results. AI and NLP tool and technologies are also used to implement a system of personalization and personalization of the experience in accordance to the user preferences. The personalization system recommends relevant exhibits and narratives based on user interests, in order to maximize the in-depth understanding of the content of the narrative associated with the curated experience in accordance with the curation and visualization approach of the digital content curator.
Finally, using blockchain technology, it is possible to implement smart contracts to control the use of digital content by its creator, and to track the ways of its utilization. This will safeguard the integrity of the digital content and protect the copyright and financial rights of its creator, Figure 5.
In summary, the main goal of the SYNTHESIS PoP is to contribute to the promotion of cultural heritage items and places of cultural interest to a global audience by visualizing the results of recording and research with VR/AR technologies and the possibility of interactive and personalized experience of the relevant content. situ and/or remotely. It also aims to highlight the work of artists and people in the field of Culture, and to promote culture to the public through tools to create innovative and engaging experiential experiences using cutting-edge technologies. It will offer visualization of an exhibition through personalized interactive narrative experiences of virtual and augmented reality (VR/AR) and the georeferencing of its individual exhibits in places of cultural interest (art exhibitions, museums), allowing the creation of educational and educational experiences, the mobilization of tourism and social interest, and the preservation, promotion, and curation of its exhibits. In Figure 6, we give an example of using the wayGoo platform [9] to visualize an interactive experience of digital representations of cultural treasures from the Minoan palace of Phaistos in Crete created with the help of the SYNTELESIS platform by the Integrated Systems Laboratory with the use of digital immersion technologies, such as content geo-reporting (wayGoo), AR/VR, and 2D/3D animation.
Additional objectives of the SYNTHESIS PoP include: 1. The further promotion and editing of cultural heritage items through the visualization of the results of recording and research and the creation of complex digital objects.
2. The promotion of the work of artists and people in the field of Culture, and the promotion of areas of cultural interest, and the promotion of culture at large to the public, through the provision of tools for creating innovative and attractive mixed-reality experiences using cutting-edge technologies.
3. The transfer of know-how on and the experience from research and development of state-of-the-art products from research centers and professional researchers to start-ups through dissemination of knowledge and networking, to assist them generate profit, develop human resource skills, and attract new and supporting extroversion and competitiveness. The SYNTHESIS PoP seeks to strengthen the activity in the field of Culture, Tourism and Creative (CTC) Industries, in terms of the production of digital content (entertainment, educational, cultural, commercial or other interest), and the organization of big scale activities and events. The PoP and user applications are expected to be marketed either as separate products or as a single system. The architecture and modular system of the platform, as described by its main components (wayGoo, NARRATION, iGuide VR/AR component), allows future adaptation to content and requirements of different organizations. The modularity of the SYNTHESIS PoP allows to accommodate different markets and different players in the CTC Industries in the SYNTHESIS ecosystem in a way that is expected to lead to maximizing market penetration and benefits for the constituencies of the ecosystem by enabling the efficient management of digital content, the facilitation of the conversion of digital content into experiences, and the protection of IPRs and profit flows through blockchain and smart contracts.

SYNTHESIS and the DICE 1 ecosystem
The adoption of the SYNTHESIS PoP is expected to lead to the following benefits in each one of the five (A, B, C D, and E) sectors of the DICE Ecosystem, Figure 7.

A. For businesses
• Facility of specialized users in the design and editing of large-scale interactive experiences, through a narration tool that allow easy and creative workflow. • Creation of a perfect product, developing with scientific methodology and dynamic commercial presence.
• Strengthening the technological base, gaining a technological lead in the field of production and supervision of interactive producers, which sets the specifications for the expansion of the developing platform.
• Enhancing innovation and competitiveness in the field of artistic curation.
Internationally, there is no system that provides a complete solution for largescale interactive events.
• Opening to new national and international markets. Penetration in the market through the possibility of highlighting and reusing places of cultural interest: Museums, Art Galleries, Archeological sites, preserved Industrial sites etc.

B. For cultural institutions
• Strengthening cultural tourism by increasing the tourists' interest in visiting cultural sites as a result of upgrading their on-site experience, but also through a strategy of diffusion of the results of the implemented activities.
• Strengthening the networking of cultural organizations through the creation of business networks with the cultural and creative industry for the joint promotion of the innovative products of the project.

C. For the Creative Industry
• Improving the productive potential of digital technology and digital content companies.
• Approach of new markets in the field of culture and tourism, inaccessible until today, through their activity in designing interactive audiovisual content.

D. For research centers
• Utilization of research for the development of market-oriented solutions, through the conversion of the research product into a commercial product. Added value that results through innovative interventions and interdisciplinary cooperation, in order to create a final product that promotes Greek tourism and culture.
• Distribution of research results to the general public, through the commercial product but also through presentations at conferences.

E. For the Economy and Society
• Improvement of the user experience and enrichment of the means of receiving the artistic/creative result, through his active participation in the interaction in natural and virtual space.

8
• Promoting cultural spaces through the implementation of technologically and artistically perfect cultural events and enhancing the interest of the general public for them.   • Promoting the collaboration of artists with audiovisual events.
The SYNTHESIS PoP is expected to act as a catalyst in the articulation of a digital creativity ecosystem for enabling the creation of DICE (Digital Immersive Cultural Environment) [1], Figure 7. The incorporation of smart contracts and blockchain technology in SYNTHESIS is expected to stimulate the process of crowdsourcing digital artifacts, narratives, and experiences alongside with their visualization, from professionals, as well as qualified non-professionals, in the fields of art, culture and creativity. The embedded protection of the Intellectual Property Rights (IPRs) of the digital creator build in SYNTHESIS, and the use of blockchain-enabled Smart Contracts to speed up the contracting and outsourcing process, track revenue streams form the use or sales of digital content, and protect the agreed upon revenue sharing with the creators of the digital content, narrative and experience, are expected to spearhead the process of digital creativity across the entire digital creativity ecosystem as shown in Figure 8.

Methodology of flexible design and development of technological solutions
The SYNTHESIS PoP aims to create a coherent platform for the creation of advanced digital assets, the curation of digital objects and their semantic and thematic interconnection through the creation of narratives, and the creation of digital experiences in virtual and real space through the georeferencing of edited digital content and related storytelling, with a range of virtual and augmented reality technologies using tablets, smartphones, virtual reality glasses, and projection imaging technologies.
The SYNTHESIS PoP is based on the coherent integration of existing platforms, services, algorithms, and applications that have been developed in the Integrated Systems Laboratory (ISL) and their effectiveness has successfully been evaluated. The timely and effective accomplishment of SYNTHESIS objectives is largely ensured through the validation of the existing know-how and the possibilities of adopting it in developing solutions for other use cases, similar ot completely new. Technological solutions already developed will be adapted to new requirements and tools already available will be leveraged, adapted as required, and integrated in a coherent SYNTHESIS PoP.
The three platforms SYNTHESIS PoP is made up of, namely wayGoo, SYNTELESIS and AFIGISSI (Narration), are briefly describe in the following. All three platforms have been developed in the Integrated Information Systems (ISL) Laboratory of the Institute of Informatics & Telecommunications at NCSR "Demokritos" through a series of funded projects that are described in the publications [1-10] and acknowledged at the end of the Chapter. The three platforms are: a. wayGoo -wayGoo is a platform for Geolocating and Managing indoor and outdoor spaces and content with multidimensional indoor and outdoor Navigation and Guidance. Its main components are a Geographic Information System, a back-end server, front-end applications and a web-based Content Management System (CMS). It constitutes a fully integrated 2D/3D space and content management system that creates a repository that consists of a database, content components and administrative data. wayGoo can connect to any third party database and event management data-source. The platform is secure as the data is only available through a Restful web service using https security protocol in conjunction with an API key used for authentication. To enhance users experience, wayGoo makes the content available by extracting components out of the repository and constructing targeted applications. The wayGoo platform supports geo-referencing of indoor and outdoor information and use of metadata. It also allows the use of existing information such as maps and databases. The platform enables planning through integration of content that is connected either spatially, temporally or contextually, and provides immediate access to all spatial data through interfaces and interactive 2D and 3D representations. wayGoo constitutes a mean to document and preserve assets through computerized techniques and provides a system that enhances the protection of your space, people and guests when combined with wayGoo notification and alert system. It constitutes a strong marketing tool providing staff and visitors with an immersive tool for navigation in indoor spaces and allowing users to organize their agenda and to discover events through the wayGoo event scheduler and recommendation system [10]. Furthermore, the wayGoo platform can be used in Security applications and event management, e.g., CBRNE and fire incidents, man-made and natural disasters, pandemics, etc., to document and geolocate information and sensor data (offline and real time) on one end, and offer navigation capabilities in indoor and outdoor spaces. Furthermore, the wayGoo platform can be used for the creation of immersive environments and experiences in conjunction with VR/AR (Virtual & Augmented Reality) technologies [9,10].
b. SYNTELESIS -A platform designed to provide a combined web application with a mobile application and a unity desktop application to support immersive digital experiences with digital cultural assets in cultural heritage sites or in virtual counterparts. The platform features an extended exhibit object model, geo-referencing of places and exhibits with for digital physical support, and a variety of mobile and AR capabilities to support an immersive experience in-situ or remotely. A case study of an immersive digital experience of the archeological site of Phaistos, Crete, has been designed and implemented with the use of SYNTELESIS [1, 2, 7].
c. AFIGISSI (NARRATION): An integrated system for the management and editing of digital content and the production of personalized individual and collaborative narratives considers the objectives of the Faro Convention [15]. AFIGISSI aims at the development of a platform for the support, management, and promotion of digital content for infrastructures of cultural and tourist interest through the creation of personalized individual and collaborative narratives [7]. Digital collections offer the opportunity to examine objects not only as individual cultural elements but as an integral part of the cultural heritage as defined by the European Union [16].
Access to the narratives is possible either during the physical visit to the infrastructure during their regular operation and in extraordinary, periodic or temporary events or remotely. Traditional narrative practices address the development of narrative by placing objects on a linear axis. Instead, the work highlights cultural content with new narrative structures, operating with less strictly prescribed scenarios. In fact, according to new media theorist L. Manovich [17], the database is generally the way to tell a story today. The creation of narratives is achieved in an automated and semi-automated way, by reversing the time axes and with a variety of thematic grouping options, making a theoretically infinite number of combinations [18], in order to respond to modern perceptions of object multiplicity and the possibility of multiple interpretations. Personalization is done both for individual users and for groups of users who work together to create a narrative. According to the current level of collaborative platform technologies, there are possibilities to connect multiple users for the purpose of a common task [19]. The system allows specialized and non-specialized users to produce stories of various sizes based on the content available and share them with the public. The aim is a holistic approach to storytelling that will lead to mature technology and an innovative tool aimed at the market for (a) the design and implementation of digital collection management software platforms for cultural operators, and (b) web applications, mobile and VR/AR applications for organizations of digital exhibitions and creation of narratives from a registered collection of museums and other cultural exhibits.
In the SYNTHESIS PoP, the three building-block platforms are redesigned, expanded, and integrated into a single controllable and coherent digital creativity environment with the addition of blockchain technology for the protection of IPRs and the introduction of smart contracts for the facilitation of signing contracts for the creation of digital assets, content, context, and experiences, and the verifiable of equitable sharing of revenue streams across the entire creativity chain involved in the creation and visualization of digital experiences, Figure 9. ii. innate to SYNTHESIS databases, such as mongoDB, Neo4j, PostgreSQL, …;

SYNTHESIS PoP architecture
iii. a repository of digital assets; and iv. a blockchain module for the protection of IPRs and the support of Smart Contracts. iv. an AI (Artificial Intelligence) component that consists of two modules:  Moreover, the SYNTHESIS PoP architecture and the accompanying applications support the following: • Enrichment of support and management of geo-reported data related to natural areas, together with services for the consumption of this information with GIS/Tile services. • End users can consume the digital content of the platform through a mobile application. The application can be used: (a) either during the physical presence of the user in an exhibition, where is guided in the tour of the exhibition, taking into account user priorities, or (b) remotely, where the user is guided to explore the exhibition and delve into the digital content that the platform provides. Using GPS technologies and/or Bluetooth beacons, the application can locate the user physical location, suggest relevant exhibits and narratives, and assist in exploring and navigating the area. Users can locate exhibits and places of interest through an interactive map, from a list through a narrative, by scanning their own QR code, or through a search with free text which is interpreted lexically and semantically by the back-end AI module. Information about the exhibit is then presented, accompanied by its correlations with other exhibits, narratives, dates, and natural sites. Finally, it is possible to view 2D/3D models directly from the device, as well as play accompanying multimedia material in the form of images, videos, panoramic images, and sounds [7-9].
• Improving the process of creating narratives and their representation using graphs, as well as the general management of digital content and repositories [2, 4, 5].
• Design and development of Blockchain infrastructure for the use of smart contracts. To preserve the integrity of each element and to track the ways of its utilization, SYNTHESIS utilizes blockchain technology. This technology forms the basis for creating an unchanging event log. This file through the security properties of Blockchain will provide a high degree of protection of data integrity as well as their continuous availability. Within the SYNTHESIS PoP, the blockchain will record the interfaces between nodes of the graph as well as the changes in these interfaces. In addition, it will store metadata and metadata related to digital data as well as record which users of the "SYNTHESIS" platform utilize each component. Additionally, blockchain will record information about derivative projects that utilized a digital component as well as project location data in both physical and virtual exhibitions. The above functions make it possible to use the smart contracts executed on the blockchain. Smart contracts will allow creators to control which users (creators or curators) will be able to use the digital data they have created and for what purpose. Finally, smart contracts enable any interaction (e.g., data entry) with the blockchain.
• SYNTHESIS allows the establishment of correlations between different narrative elements (recommendation process) enriched with audiovisual material if required [13], and offers personalization services and personalized functions [10]. Based on the categorization of the narratives and the corresponding evaluations by the users of the application, machine learning and automatic classification techniques are applied to identify similarities between the narratives and their individual structural elements. These similarities are multi-layered, reflecting similarities that are based on the preferences of the users but also based on the content, and are determined conceptually and lexically, after processing natural language from any descriptions/tags/categories that accompany each narrative and each structural element. In addition, where possible, from each element, depending on its format (audio file, image, video, text), additional metadata/features are extracted using deep machine learning techniques. The additional metadata/features represent each object based on its components (e.g., color choices and image contrasts, rhythm/style of music, etc.). Through correlations with the respective user ratings, similarities are calculated in greater depth allowing the discovery of new correlations and features that determine user preferences. The result is a multidisciplinary graph that represents the narratives and objects, and the similarities between them at all the levels of similarity considered [11].
• With the aim of visualizing different types of experiences, the SYNTHESIS PoP supports the visualization of digital content of various types and formats (2D/3D) through a virtual reality application for mobile devices and VR headsets. Content can come from 3D modeling tools, e.g., Building Information Management (BIM) objects, or gaming machines, e.g., interactive experiences in 3D environments, (https://vimeo.com/isldemokritos). Each type of content offers different possibilities and therefore it is necessary to support all types of 3D content. The visualization of rich 3D experiences is realized with a usercentric approach [17][18][19] and through a (separate) application implemented in the Unity game machine and provides the possibilities of 3D visualization and navigation. The wayGoo application opens the Unity application and loads the corresponding experience by downloading the corresponding Unity asset bundle from the platform. Communication between the two applications can be implemented with mobile URL schemes that allow the use of parameters when opening applications [12].

SYNTHESIS PoP use in different use cases for pilot tests
The three platforms SYNTHESIS is based on, namely iGuide/SYNTELESIS, Narration/AFIGISSI, and wayGoo have been tested in different use cases, environments, and applications as pilot tests. Below we make reference to three such use-case scenarios by summarizing the use case environment, the scenario used and the targeted audience and objectives.
The developed solutions in the three use case pilot test scenarios aimed at analyzing the of SYNTHESIS and the three platforms within it, in terms of the efficiency and effectiveness of creating immersive experiences in the context of DICE. The three carefully selected use cases demonstrated the ability of SYNTHESIS, and the three platforms is made up of, to create: a. iGuide Knossos VR: An immersive 3D VR tour guide for the palace of Knossos [22], a blueprint paradigm for touring and experiencing archeological sites that are partially or totally ruined, with the help of mobile and VR/AR technologies, either in situ or remotely [2-4], Figure 11.
In March 1900, on the Greek Island of Crete, Arthur John Evans, a British archeologist, unearthed what he called the Minoan Civilization; a colorful world that seemed forgotten by time. A blend between reality and imagination, the reconstruction of the Palace of Knossos has become an iconic symbol of the past, triggering endless debates and providing a rich canvas for creative reconstructions.
iGuide Knossos is a HD/3D VR reconstruction of the Knossos Palace that merges Evans' vision of the site with contemporary archeological evidence. It is Figure 11. iGuide Knossos is a HD/3D VR reconstruction of the Knossos palace. meant to be used as a VR edutainment environment, providing a fertile ground for future development of educational, entertainment and action games such as Fruzzle, Knosseau and Knossos Labyrinth (thumbnails provided), which are using the Minoan civilization and myths as a foundation for their action play.
a. wayGoo in-situ AR/VR tour of the archeological sites Knossos and Phaistos, a blueprint paradigm for in-situ touring and experiencing archeological sites, that are partially or totally ruined, with the help of mobile and VR/AR technologies, that allow in-situ visitors to experience rich digital multimedia content (text, audio, video, including 360 o , 2D/3D models and animations, games, etc.) and associate it with relevant locations and PoI's 2 in the physical site through wayGoo's geolocative and automatic navigation capabilities, Figure 12.
a. AFIGISSI (NARRATION) The AFIGISSI platform has been used in a number of pilot project to demonstrate its functionalities in curating digital cultural assets into experience that can be visualized using different technologies. We present two use cases next. One from the Teriade art museum of contemporary paintings in Lesvos, Greece [24]; and the other a curated audiovisual experience based on an imaginary narrative leveraging 3D assets from iGuide Knossos VR and visualized using projection mapping technology [25].

Digital curation of the Teriade museum exhibits using the AFIGISSI platform
The AFIGISSI (NARRATION) platform is developed to support, manage and promote digital content of cultural and tourist infrastructures (tourist sites, museums, archeological sites, exhibition spaces, etc.) through the creation of personalized individual and collaborative narratives. Access to narratives is possible either during a physical visit of an infrastructure during its regular operation (such as permanent exhibitions) and/or its temporary events (such as periodic exhibitions) 2 PoI: Point of Interest  or remotely. The project reveals cultural content by implementing new narrative structures, less strictly defined scenarios, automated and semi-automated modes, by overturning time sequences and by offering a variety of filtering and grouping options to respond to modern perceptions of versatile objects and their multiple interpretations. The platform allows specialized and non-specialized users to produce narratives based on the available content and to share them with targeted audiences. The curation of the (digitized) exhibits of the Teriade museum has been used as a pilot use case for demonstrating the ability of the AFIGISSI platform to store, manage, curate digital content in accordance to a narrative, and transform the curated outcome into an experience that can be visualized with smartphones and VR/AR technologies, Figure 13a.

From i-guide Knossos to the journey of Ariadne
An audiovisual projection mapping installation inspired by the world of Knossos and its digital interpretation and constitutes a blueprint for the creation of an immersive digital experience (Ariadne's Journey) from 3D digital assets and animations created within iGuide Knossos for the reconstruction of the ancient Minoan Palace of Knossos in Crete, Greece, an original narrative that addresses the process of knowledge discovery through excavations of historic monuments, and history itself, by a young contemporary girl, and the visualization of the outcome of this curatorial effort with VR technology and projection mapping [5,25]. The narrative, artistic and spiritual field of Ariadne's journey explores the construction of memory and the meeting points of the historical and mythological constructions that experience the unexpected journey of a young modern heroine. Through esthetic narrative, the work aims to create an experience for listeners of all ages, translating, or perhaps more accurately, the interpretation of CH images and historical documentation into a living esthetic vocabulary, Figure 14 [23].

Conclusions
In this chapter an extended description of the SYNTHESIS Platform of Platforms (PoP) has been given. The architecture and the three platforms SYNTHESIS is based on are described in detail. SYNTHESIS results from the integration of the three platforms iGuide, Narration (Afigissi), and wayGoo, in a coherent environment that allows the seamless creation of augmented digital assets, their management and curation in the context of a narrative, and the creation and visualization of digital experiences with mobile applications and VR/AR technologies. SYNTHESIS is the outcome of an over a decade research and development at the Integrated Research Laboratory at NCSR Demokritos towards the development and implementation of the Digital Immersive Cultural Environment (DICE) concept. Examples of three different digital immersive experiences that have been developed as DICE applications to demonstrate the capabilities of SYNTHESIS as PoP for creating digital experiences, have been given. SYNTHESIS supports blockchain and smart contract technologies that are used to protect IPR's and enable smart contracts to safeguard the use of digital content and encourage the creation of new digital content, its curation and distribution by protecting the IPR's of the creators and honoring contractual obligations and revenue streams agreed upon between creators of digital experiences and distributors of these experiences,