Developing a Data Model for an Omnidirectional Image-Based Multi-Scale Representation of Space

: One of the major challenges that existing spatial data is facing is the fragmentation of its representation of indoor and outdoor space. As studies in the use of omnidirectional images in representing space and providing Location-based Services (LBS) has been increasing, the representation of the different scales of space, both in indoors and outdoors, has yet to be addressed. This study aims to develop a data model for generating a multi-scale image-based representation of space using omnidirectional images based spatial relationships. This paper identifies the different scales of space that are represented in spatial data and extends previous approaches of using omnidirectional images in providing indoor LBS towards representing the other scales of space, particularly in outdoor space. Using a sample data, we present an experimental implementation to demonstrate the potential of the proposed data model. Results show that apart from the realistic visualization that image data provides, basic spatial functions can be performed on the image data constructed based on the proposed data model.


Introduction
Cities worldwide are facing numerous challenges due to rapid urbanization and population growth.To address these issues, local governments are turning to data-driven approaches, leveraging technologies such as big data, cloud computing, and wireless networks (Silva et al., 2018).The concept of digital twins (DT) as virtual representations of physical systems has emerged as a promising solution to tackle urban challenges effectively.Such DTs play a crucial role in formulating management strategies based on real-time data and realistic representation of the physical world.
However, digital twins (DTs) go beyond merely mirroring the physical attributes of the real world using spatial data; they also encompass the relationships between entities and the underlying processes.Given the diversity of data types and formats involved in these systems, integrating and ensuring interoperability becomes crucial during their development (OGC (Open Geospatial Consortium), 2021).As the complexity of the real world is increasing, there is also an increasing demand to develop methods to represent this complexity in the corresponding spatial data (Qi et al., 2021).The challenges present in producing GIS datasets that represent the physical world must be addressed in order to successfully implement DTs.
One of these major challenges in existing spatial data is the fragmentation of space domains.As in Figure 1, indoor and outdoor spatial data is still separately constructed due to differences in positioning methods, data sources, and applicability of data construction techniques (Giudice et al., 2010).To ensure that Location-Based Services (LBS) meet the requirements of their users effectively, the spatial data employed in their development must accurately depict the continuous reality of the flow of spaces and human activities in the indooroutdoor space.For instance, an LBS application designed for navigation purposes should be built using data that reflects the uninterrupted flow of spaces, enabling seamless human mobility.While there are more common sources of spatial data used for representing both indoor and outdoor space, studies have used omnidirectional images for representing spaces (Ahn et al., 2020;Jung & Lee, 2017).These images provide a 360-degree horizontal field of view at the point of capture and are commonly used in providing street views (Google, 2022).Its rich visual content, small data size, and simple data structure make it a preferred alternative compared to laser scans or other 3D datasets.Moreover, to extend approaches such as geotagging or image overlay for providing spatial entities portrayed on the images, studies have proposed methods to supplement these with topological information, since pixel data is insufficient to enable spatial analysis and subsequently, provide LBS (Claridades et al., 2023).
However, existing studies have only utilized the concept of integrating omnidirectional images and topological data for representing indoor space and indoor features.As in Figure 2, omnidirectional images can similarly express spaces and features in other scales of space, such as in the outdoor space.With this, this paper aims to propose a data model for utilizing omnidirectional images to represent the different scales of indoor and outdoor space.We aim to describe the characteristics associated with each scale to represent them using image data, to identify spaces and features portrayed by image data, and explore how the integration of image and topology data can be leveraged to illustrate corresponding spatial relationships.Moreover, to achieve a seamless representation of space, this study also aims to define the relationships between the scales of space and apply this to the corresponding image-based representations.

Figure 2. Varying spatial objects expressed by omnidirectional images according to scale
The paper is organized as follows: the subsequent section provides a brief overview of related studies, followed by a section defining the concept of a multi-scale image-based representation of space and introducing the proposed data model.The fourth section details an experimental implementation using sample data.Lastly, the concluding section summarizes the findings and discusses future directions for the study.

Related Work
Despite the prevalence of point cloud datasets and other 3D geometric models, images have garnered attention as an alternative means to represent space due to their lower cost and quicker collection process (Jung & Lee, 2017).Image datasets offer a visual depiction of spaces, including the features within these spaces, but they do not express these objects discretely because they are made up of pixels.Consequently, various methods have been employed in numerous studies and applications to convey such spatial entities to users, with or without attribute information.Previous studies utilizing image datasets for expressing spatial information can be categorized into two approaches: those relying solely on visual input, as seen in geotagging or augmented reality applications, and those supplementing image data with topology data to facilitate spatial queries.
Geotagged images have been used to display interactive information to users alongside 3D models for decision-making systems (Ham & Kim, 2020).Images have also been used in "augmented reality" and "mixed reality" studies, both in navigation and in gaming (Chheang et al., 2020;Liu et al., 2021;Ruta et al., 2016), with evidence in an increase in cognition and acceptance by users.However, the role of image data in these applications has been limited to being a background, or base, in the visualization of space.
In order to use these image datasets in spatial analysis directly, topological information must be provided.Two major approaches have been used to supplement image data with topology data in previous studies.Reference data, such as vector datasets, can be used to match a pixel's position in the image data to real-world coordinates based on the coordinates of the shooting point (Jung & Lee, 2017).On the other hand, an approach that defines a region of interest based on the extension of the 9-intersection model, called the Spatial Extended Point (SEP), can be a direct approach in integrating image and topology data.Claridades et al. (2023) proposed a method to integrate the topological relationships from NRS data on indoor omnidirectional images and implement spatial analysis functions directly on the image-based representation of indoor space, such as identifying indoor spaces and features using the SEP method.While this method allows realistic visualization while enabling spatial analysis, the presented approach is applied to indoor space only.In this paper, we extend the concept of an image-based representation using a spatial relationship-based integration approach for representing other scales of space.

Development of a Model for a Multi-Scale Image-Based Representation of Space
This section discusses the concept of a multi-scale image-based representation of space.The first subsection discusses how image datasets are used to represent spatial entities in spatial data using topological relationships.Then, the following subsection discusses the different scales of space, and how relationships between these scales are defined.Additionally, we formalize the proposed data model using a UML diagram.

Image-based Representation of Space and Spatial Relationships
In this study, we use an omnidirectional image to represent and visualize a space, as shown in Figure 3.As described in Claridades et al. (2023), these images capture a snapshot of a specific location, known as the shooting point.An omnidirectional image captured at a shooting point depicts a segment of space, which is referred to as a subspace (Claridades et al., 2023;Open Geospatial Consortium, 2020;Zlatanova et al., 2014).

Figure 3. Using omnidirectional images to represent space
Typically, an omnidirectional image in a standard raster format adopts an equirectangular projection, encompassing the entire horizontal Field of View (FOV) centered at the shooting point location.When conceptualizing the omnidirectional image as a representation of the subspace in 3D, it can be projected into a fisheye view, providing a panoramic perspective akin to being encapsulated within a "sphere" wrapped with this image.This projection technique, coupled with the image's capture angle, enables an immersive visualization of space.This representation of a subspace using an omnidirectional image is referred to in this study as a "Scene".
Even though a single Scene discretely represents a subspace, the real world is continuous.Therefore, it's necessary to establish how these discrete Scenes can portray continuous space by defining their connectivity to other scenes.Multiple scenes captured successively along the subspaces can depict this continuity, but the relationship between scenes must also be specified.We refer to this relationship as a "Linkpoint," as in Claridades et al. (2023), and this signifies the connection between one scene and another.Illustrated in Figure 4, a Linkpoint corresponds to the pixel location within a scene that aligns with the shooting point for the connected scene (Claridades, Kim, et al., 2023).Consequently, it serves as an abstraction of connectivity in the image space, represented by an edge in the Network Representation Structure (NRS).It is important to note that the spaces that an overlap between the spaces that the Scenes portray is necessary, so that a Linkpoint can be established between such Scenes.

Scales of Space Represented in Spatial Data
In this study, the definition of each scale is based on the concept of portraying spatial entities at each scale through omnidirectional images, along with their spatial relationships, which can be expressed through NRS data.This embedding is achieved by defining these images as Scenes along with their corresponding Linkpoints.Consequently, each image data incorporates topological relationships for its respective scale, contributing to the image-based representation of spaces.Since NRS is network-based data, these representations for each scale can be integrated using methods for linking network-based datasets.This integration facilitates the creation of a multi-scale seamless representation of space.
The Building Scale pertains to the level of detail at which various agents, including humans, navigate indoor spaces, often termed the micro-level of space.Within indoor spaces, different levels are delineated, such as floor levels, zones, rooms, and units of rooms, depending on the application and navigation agent.In the context of using omnidirectional images for application development, subspaces at the room level are considered, encompassing the spaces that rooms define and the features they contain.
Transitioning to the outdoor environment, the Block Scale is defined from a human perspective, focusing on ground-level perception rather than abstracting space from a higher viewpoint as in traditional maps.This scale provides users with a more realistic navigation experience and facilitates the connection between indoor networks and outdoor spaces.Subspaces at the Block Scale include street-level portions where buildings are viewed as individual features, with spatial units such as rooms composing the building space.At the Urban Scale, which represents the macro-level of outdoor space, spatial analysis covering wider areas is feasible due to the increased coverage resulting from a higher viewpoint.Here, an image represents a subspace comprising a block, with buildings viewed as spatial units represented by footprints.
NRS representations at each scale depict spatial relationships among corresponding entities.In the Building Scale, nodes represent indoor subspaces, and features correspond to indoor facilities, while edges denote indoor navigation networks.At the Block Scale, nodes signify road spaces outdoors, features include building features, and edges represent navigable road spaces.In the Urban Scale, block nodes contain multiple buildings, with features represented by building footprints, and edges denote connectivity relationships between block spaces.These scales of space and their corresponding NRS representation is illustrated in Figure 5.

Defining the Relationships between the Scales of Space
The NRS is based on the Combinatorial Data Model (Lee & Kwan, 2005) which employs the concept of Poincare duality to define the topological relationship of 3D features.This model has served as the foundation for various applications and data standards such as IndoorGML (Open Geospatial Consortium, 2020).In the CDM, the concept of Master_node is defined as a node representing connections between a building's floor levels, abstracting the network paths within a floor network.Originally, in the NRS, hierarchical relationship establishes a connection between two scales: the space scale, where the building is conceptualized as composed of floors, and the larger scale, where the floor is composed of rooms.This concept served as the basis for constructing geometric NRS data from logical NRS data through subspacing, where a master_node representing a hallway space represents a set of nodes that represent the sub-units within this hallway space (Claridades et al., 2021).
In this study, we extend the concept of master_node in order to integrate indoor and outdoor NRS data across different spatial scales to enable analysis and provide spatial services across continuous space.Figure 5 in the preceding section illustrates how a feature in a smaller space can be decomposed into a space that can be represented as its own NRS in a larger space.We define the relationship between such a feature and the NRS it represents as a master_node relationship, as depicted in Figure 9.At the Urban scale, the subspaces that each of the nodes in the NRS express are blocks.These blocks, correspondingly, can equivalently be represented as entire NRS data at the block scale.
We define the hierarchical relationship between the urban scale NRS and the block scale NRS through a master_node.In this scenario, where each node of the urban scale NRS denotes a block, at least one of these nodes must serve as a master_node.This master_node represents the hierarchical relationship between the NRS in the Block scale to that one node in the Urban scale.
Similarly, indoor subspaces and road subspaces were defined at different scales in the previous chapter, namely the building scale and block scale, respectively.At the Building scale NRS, one node corresponds to one sub-unit of a building, while at the block scale NRS, one node represents one building.Hence, we can designate one node in the scale as master_node, which is equivalent to the NRS representing the spatial relationships of sub-units within a building space.A master_node at a higher hierarchical level abstracts a network at a lower hierarchical level.In this study, we designate one node in the block scale NRS as a master_node, representing the entire NRS at the building scale.Connectivity among spaces and containment between a space and a feature are denoted by LinkPoints on the image data, represented in the UML as the LinkPoints class.Additionally, certain features, particularly at the block scale, are identified using building footprints.In the UML representation, this is illustrated by the ReferenceData class having an association termed isReferenced with the Cellspace class, and another association with the Surface class from GML, serving as its geometric representation.

Data
The hierarchical relationship between different scales of space is captured in the UML model through the Master_Node class, which inherits from the Node class.This class is associated with the SpaceFeatures class, indicating that the master_node abstracts the NRS of another scale of space.In the transition between the building scale and block scale, the master_node representing this hierarchical relationship corresponds to transitional spaces, which are expressed through the Transfer_Link model (Claridades & Lee, 2021).This relationship is reflected in the UML model through the TransitionGraph class, which connects the building scale NRS and block scale NRS via the Transfer_Link.Similar to the Transfer_Link model, the Transfer_Link class is valued zero for the weight, indicating that it is merely an association link, not an additional geometric link between a transition graph node and a block scale node.
The hierarchical structure of the space scales is further delineated in the UML through the StructuredSpace class, which inherits from the CellSpace class.The UrbanScaleSpace, BlockScaleSpace, and BuildingScaleSpace classes inherit from the StructuredSpace class and are associated with each other, signifying the decomposition of the urban scale space into subspaces representing one block and the block scale space into subspaces representing buildings.

Experimental Implementation
In this section, we demonstrate the potential of the proposed data by building a multi-scale LBS platform prototype based on sample data.The study area for the experimental implementation is a university campus site, where omnidirectional images are collected using a handheld RICOH Theta V 360-degree camera for indoor and street-level images, and a DJI Mavic Pro drone for sky-view images.Topology authoring, which includes establishing the Scenes and Linkpoints is performed using Kolor PanoTour 2.15, and the output is packaged through Electron.Figure 11 illustrates the user interface of the image-based navigation platform prototype, and the experimental set-up is summarized in Table 1.In order to demonstrate the hierarchical relationships established on the multi-scale image-based representation of space, we extended the algorithm implemented in indoor space for identifying features in indoor space using reference data (Jung & Lee, 2017) to a method to detect a building on Urban scale images using its footprint, shown in Figure 12.For identifying building objects and indoor objects on Block scale and Building Scale images, we implement the object detection method on images based on the Spatial Extended Point (SEP) (Park et al., 2018).On Urban scale images, the function illustrated in Figure 12 and detailed in the algorithm in Table 2, is enabled to identify building features.We use the point-in-polygon method based on a ray-casting algorithm based on the Jordan Curve theorem (Hales, 2007) to determine whether a point is contained or not within a polygon.This pseudocode uses a reference data, which a separate vector file represents the building footprints, to identify building features within the image-based prototype.By extracting coordinates from the building footprint, a table is constructed to store building positions and attributes.To allow for a tolerance in the clicking action of the user, we allow for specifying a buffer that expands the polygon size prior to checking whether the point is contained within the polygon.On the other hand, on Block scale and Building scale images, because objects on the images are not building footprints, the SEP approach is used for identifying objects, as in Claridades et al. (2023).
The results of the implementation are shown in Figure 13.A long press action on a pixel that expresses a building will trigger the algorithm in Table 2, in this case, to display the attribute of a building named "21 st Century Building."In this sample dataset, where the building is expressed differently on the images according to scale, we demonstrate the master_node relationship between the Urban scale space and Block scale space by prompting whether the user wants to move to the larger scale of space.This action will display the Block scale image, which shows the street view in front of the said building.In this scale, the same building object is expressed as its façade, rather than a footprint, so a long press action on this image will also display the same building object's attributes.
Similarly, since this object is also represented in the Building scale, we can also demonstrate the master_node relationship between the Block scale and Building scale by prompting whether the user wants to navigate to the next scale of space.In this case, the Building scale image is displayed to the user, which is an omnidirectional image captured inside the building.In this Scene, indoor objects represented as indoor POI may also identified using the same user action.
The experimental implementation is simple demonstration of how different scales of space may be represented using image data and topological relationships.Topological relationships between images, implemented as Scenes, are implemented as Linkpoints that represent how the Scenes are connected to each other.The relationship between the scales of space is defined by expressing the hierarchical relationships of the spaces through the concept of master_node.These concepts are implemented on a prototype that shows how a single building object may be expressed on image data across the different scales of space.

Conclusions and Recommendations
This paper proposes a data model for a multi-scale image-based representation of space based on omnidirectional images.This model extends existing approaches of using spatial relationships to supplement image data in order to not only express space, but also spatial analysis on the images.This paper defines the various space of scales that may be represented using the image data, namely the Building scale, the Block scale, and the Urban scale, based on the types of spaces and features that are expressed on the images.This paper also discusses not only how topological data that expresses the connectivity relationships of spatial entities can be defined for each scale, but also how hierarchical relationships can be defined between the scales for a seamless representation of space using image data.We formalize the data model using a UML diagram, which is based on an extension of the IndoorGML core module.Using sample data, we demonstrate the potential of the proposed data model by illustrating its use in the multi-scale representation of a real-world object, i.e., a building, using image data, and how spatial services may be provided using an example on object detection.
This study has several limitations that future studies may address.First, one of the advantages of using image data is its small data size and simple structure, which makes it easier to collect and hence update.Therefore, there is a need for a method to update the images on the image-based platform.Second, the spatial analysis functions presented in the implementation are only basic attribute display capabilities.Additional functions which are applicable to real-life use cases must be developed for the integrated image and topology data.Moreover, this study presents an initial study of integrating image datasets with topological data.Future studies may explore integrating these images with other types of vector datasets, such as point cloud or mesh data, for a more realistic representation of space.

Figure 1 .
Figure 1.Fragmentation in space representation faced by existing GIS data

Figure 4 .
Figure 4. Representing spatial relationships in an image-based representation of space

Figure 5 .
Figure 5. Spatial features represented in spatial data for each space scale and their respective NRS representation

Figure 6 .
Figure 6.Establishing scenes on the Building Scale

Figure 7 .
Figure 7. Establishing scenes on the Block Scale

Figure 8 .
Figure 8. Establishing scenes on the Urban Scale

Figure 9 .
Figure 9. Conceptual diagram of the multi-scale hierarchical indoor outdoor NRS data

Figure 10
Figure 10 expresses the data model proposed in this section through a Unified Modeling Language (UML) class diagram, which extends the core model of IndoorGML.We extend the IndoorGML IndoorFeatures class using the SpaceFeatures class, which represents the abstraction of real-world space, whether indoors or outdoors.This then aggregates the PrimalSpaceFeatures class and the MultiLayeredGraph class from IndoorGML.The MultiLayeredGraph class captures topological relationships, including connectivity among spaces and containment between a space and a feature.These relationships are depicted in the Network Representation Structure (NRS) via a graph comprising nodes and edges, which are represented by the Node and Edge classes, respectively.These classes are associated with the GML Point and Curve classes, serving as their geometric representations.The PrimalSpaceFeatures class delineates primal 3D spaces and aggregates the CellSpace class from IndoorGML, which encompasses all spaces, including those physically occupied by features.The CellSpace class maintains a duality relationship with the Node, representing this entity in the NRS through the principle of Poincare Duality.This paper represents subspaces across the scales as omnidirectional images, so the CellSpace class has an isExpressed association with the Scene class.Attributes of each Scene are expressed through the

Figure 10 .
Figure 10.UML data model of the multi-scale image-based representation of space

Figure 11 .
Figure 11.Interface of the image-based navigation platform prototype

Figure 13 .
Figure 13.Results of detecting multi-scale expression of features on the image data

Table 1 .
Summary of the experimental set-up

Table 2 .
Detecting building objects using footprints on Urban scale images Figure 12.Identifying a building object using its footprint