Direct Use of Indoor Point Clouds for Path Planning and Navigation Exploration in Emergency Situations

This study investigates the feasibility of directly utilizing 3D indoor point clouds for real-time indoor navigation, particularly to enhance emergency response processes. Traditional indoor navigation research primarily focuses on creating navigation systems from pre-existing indoor models, resulting in a graph representation that simplifies spatial relationships, requires post-processing, and delivers results only afterwards, often overlooking real-time obstacles and complex layouts such as those in modern office floors. This research proposes an original approach by leveraging real-time generated 3D models using HoloLens 2 sensors, which combine RGB images and depth sensor output to create a comprehensive point cloud. The study explores path planning directly within these point clouds without the need for extensive preprocessing or segmentation, aiming to provide immediate navigation support with minimal delay. Utilizing the Rapidly Exploring Random Trees (RRT) algorithm, the research seeks to minimize preprocessing and swiftly visualize navigable paths, evaluating the system's performance in terms of processing time and path viability. This approach addresses the limitations of traditional graph-based methods and the challenges posed by outdated or unavailable indoor models, offering a promising solution for real-time emergency navigation assistance.


Introduction
Emergency response scenarios demand urgency and precision, as the nature of such events often involves saving lives and minimizing damage (Kapucu and Garayev, 2011).Navigating an unfamiliar location such as a high-rise building without prior knowledge, can be a significant challenge for emergency responders who need to quickly locate resources like fire extinguishers, safe spots, and alternative exits.Even a few seconds saved through an improved navigation system could have a substantial impact on the outcome of an emergency.Current indoor navigation systems primarily rely on satellite signals (such as GNSS) and local Wi-Fi or Bluetooth signals for positioning, which might offer great accuracy under normal circumstances.However, these systems often fail in emergency scenarios where power outages or structural damage can disrupt local networks and block satellite connections.Furthermore, these systems typically depend on pre-existing building data like floor plans or Building Information Models (BIM), which may be outdated or poorly maintained (Nikoohemat et al., 2020), leading to inaccurate and potentially dangerous guidance during emergencies.Most indoor navigation solutions represent the environment in 2D and simplify the spatial complexity into nodes and connecting lines, which is efficient for everyday use but inadequate for the intricate layouts and unexpected obstacles found in emergency scenarios (Boguslawski et al., 2022).This study aims to leverage the capabilities of an existing system (Morlighem et al., 2020;Smit et al., 2021;van Schendel, 2022) where 3D indoor point clouds are generated and expanded in realtime as field agents navigate through the environment.We utilize visual HoloLens 2's internal SLAM system for positioning, which does not depend on external connectivity other than local network connections provided with the system for data transfer.This research emerged from an effort to find a methodology that could utilize available 3D point cloud data with minimal processing to keep up with the real-time data influx.An unconventional approach was adopted, using a methodology popular in the robotics industry that focuses on speed and iterability.Additionally, the study uses data obtained from three different scanners: one is the Microsoft HoloLens 2, used to evaluate what could be achieved with the current system, and the other two are survey-grade mobile laser scanners (MLSs), namely GeoSLAM ZEB Horizon RT and Leica BLK2GO.Even though their workflows do not fit with the real-time mapping system, they are employed in this research to see what is achievable if sensors with higher point density and wider fields of view become available.As sensor technology continues to advance rapidly, the potential for integrating higher-quality sensors into real-time navigation system becomes increasingly feasible.Notable developments, such as the introduction of LiDAR sensors in consumer devices like Apple's mobile phones and tablets starting in 2020, highlight the trend towards more accessible and powerful sensing technologies (Díaz-Vilariño et al., 2022).The recent release of Apple's Vision Pro during the course of this research further underscores the progress in wearable sensor technology.These advancements suggest that features typically reserved for highend mobile laser scanners could soon be available in smaller, more affordable, and wearable formats.Employing the modified bi-directional Rapidly Exploring Random Trees (RRT) algorithm for pathfinding, this method treats the points in the point cloud as obstacles, using any available empty space to establish a path from the start to the endpoint.The RRT algorithm is chosen for its efficiency in rapidly processing and adapting to new environmental data, crucial for navigating dynamically changing indoor environments during emergencies.Unlike traditional navigation systems that simplify environmental data into 2D maps, this solution maintains the three-dimensional complexity of the environment.This approach allows for effective navigation through and around physical obstacles within complex indoor settings, which is essential in emergency scenarios where environments are prone to sudden alterations such as obstructions from debris or changes in accessible paths.For example, in emergency scenarios requiring navigation through complex and altered indoor layouts, traditional graphbased navigation systems might not swiftly adapt to such changes.These systems often require time-consuming reprocessing of environmental data to reflect new conditions.In contrast, the use of a real-time 3D model in this research enables the immediate adaptation of navigation paths to the current state of the environment, providing responders with up-to-date and reliable routing.The proposed navigation system can swiftly adapt to the dynamically changing environment of an emergency.By enhancing decision-making processes and providing efficient paths to safety or to other agents in distress, the system optimizes the use of real-time data with minimal processing.This navigation solution thus offers an alternative to conventional methods by leveraging advanced scanning technology and pathfinding algorithms, facilitating more effective and reliable navigation in complex indoor environments during emergency responses.

Related Works
Indoor navigation is often considered under few main topics: indoor positioning, and pathfinding, mapping (modelling), and communication of the found path.The indoor positioning aspect of this project has been addressed by preceding studies (Morlighem et al., 2020;Smit et al., 2021;van Schendel, 2022).Therefore, our study will focus on the latter.This section follows the natural progression of indoor navigation: first, modelling; second, pathfinding; and lastly, navigation support/communication.Each of these subsections will be considered from the perspective of emergency response.

3D Indoor Mapping Techniques
The importance of accuracy in indoor modelling has been previously emphasized; there is a need for an up-to-date model, whether it be a floor plan, BIM, or 3D model.An up-to-date 2D floor plan would still represent a complex office floor as a single room.Therefore, the requirement is an up-to-date, 3D indoor model.Building Information Models (BIMs) could be an option.However, a case study by the European Commission (Carbonari et al., 2020) indicated that out of 21 building logbook initiatives, only 8 require updates after renovations, and only 3 are digitalized and accessible.Consequently, even if an up-to-date 3D model exists, it might not be available.To summarize, 2D models are inadequate for representing complexity, and existing 3D models could be inaccessible or outdated.Therefore, realtime 3D reconstruction appears to be the most viable option for emergency response scenarios.

3D Path Planning
Using a 3D model for navigation is often achieved with a navigation graph.This method requires segmenting and classifying the 3D model to extract the navigable surface of the floor plane.An example is the work of Balado et al., (2019), where they use a 3D point cloud to calculate the route.While this is a robust methodology, it takes computational time.Flikweert et al., (2019) proposed a methodology that automates the process from point clouds to navigation graphs, which can perform even better with high-capacity hardware.However, simplifying the obstacles and navigable surface into a 2D representation is not feasible in a dynamic environment such as emergency response.Another approach by Broersen et al., (2016) proposes extracting the 3D empty space and structuring it into a graph to achieve pathfinding.This also requires time and processing, but their implementation of an octree approach allows them to quickly index their graph, even though it is a 3D structure.Lastly, the work of Fichtner et al., (2018) combines the use of octree and navigation graphs to streamline the process even more.However, these methodologies would require reprocessing as the dataset expands in real-time and depend on a secondary product.Therefore, we sought a method that could work with an unstructured, unprocessed point cloud.Path planning methodologies for humans, in contrast to robotics, require much more structure and classification to be understood by the end-user, which does not fit with our objective.Therefore, we adopted a methodology that emerged in the field of robotics: Rapidly Exploring Random Trees (LaValle, 1998).This simple algorithm is not computationally demanding yet effective.In addition to the base algorithm there are various studies that proposed improvements such as bi-directional search, target oriented exploration and adaptive extension (Wang et al., 2022;Zheng et al., 2022).

Visualization and Path Communication
The final part of this section involves the visualization and communication of the path.We use the term "communication" because the output of our pathfinding methodology is a list of coordinates, which might be adequate for an autonomous drone but not for a field agent.Our end user, a field agent, has limited time and resources to comprehend complex data.Patel and Grewal, (2022) conducted a user study comparing augmented reality (AR)-based indoor navigation with traditional 2D maps.Most users found the AR-based method to be less mentally and physically tiring and were able to reach their objectives faster.The study noted that a limitation was the need for users to hold a mobile phone in front of them for navigation support.The authors proposed the use of an AR-headset as a further improvement, which could enhance the user experience by allowing hands-free navigation.Additionally, preceding studies related to this project (Morlighem et al., 2020;Smit et al., 2021) utilized the Unity game engine for visualizing 3D graphics, and the current workflow is optimized around this system.Therefore, we decided to visualize the path in Unity in a way that can be overlaid onto the real world using the HoloLens's mixed reality view.This approach ensures that the path is easily understandable and can be followed by the field agent in real-time, enhancing their navigation efficiency during emergency responses.

System Overview
Diagram and overall explanation of the system is seen in Figure 1 below.Point cloud processing and coordinate picking actions are only applicable for the proof of concept and are due to the limitations of mobile laser scanners and the development environment.The 3D modelling system that this project is designed for can provide point clouds as well as the location of field agents (initial coordinates) and allows for the selection of goal coordinates.Therefore, the pre-processing section will not be required.

Data Collection Equipment and Environments
Our study employed the HoloLens 2 and two mobile laser scanners, the GeoSLAM Zeb Horizon RT and Leica BLK2GO, across three environments: an office building, an apartment complex, and a basement.The office environment was selected for its complex layout and reflective surfaces like glass and monitors, which pose challenges for scanning accuracy.The apartment complex and basement, chosen for their simpler structures and minimal pedestrian traffic, facilitated a focused evaluation of the system's routing and visualization capabilities under controlled conditions.All environments were scanned with the appropriate permissions to comply with data privacy regulations, ensuring GDPR (General Data Protection Regulation) compliance.

Scanning Workflows and Data Preprocessing
The HoloLens 2 was used to represent real-time 3D modelling system, capturing RGB and Depth images along with the scanner's pose.This data was stored in an SQL database and processed at the edge server into point clouds, though specific details remain outside the scope of this document due to proprietary restrictions.The mobile laser scanners required initial preprocessing using their respective software suites to convert proprietary data formats into standard files (.ply, .las),necessary for subsequent visualization and analysis.This preprocessing included noise filtering and SLAM processing, streamlined to mirror potential future scenarios where more advanced sensors might be used without extensive manual intervention.Furthermore, preprocessing included coordinate picking prior to path finding.This is an additional step required in the development process in contrary to the 3D modelling framework where initial location will be the field agent's location and goal location will be determined from the system interface (Figure 2).

Data Handling and Processing
While this research did not focus on sensor fusion, it emphasized the individual capabilities and data processing workflows of each scanner type.The HoloLens 2 and mobile scanners operated independently, capturing distinct datasets that were used to evaluate the system's performance across different technological capabilities and environmental complexities.

Data Structuring:
As mentioned in previous chapters, high computational demand of 3D point clouds is often a bottleneck.In this study we addressed this in a similar method to Fichtner et al., (2018) and Broersen et al., (2016).An octree was used to structure the search space, enhancing the steering and collision checks, while the search trees of the RRT algorithm were structured with a KD-tree to achieve feasible iteration speeds.

Pathfinding and Route Planning:
Pathfinding was conducted using a modified bi-directional Rapidly Exploring Random Trees (RRT) algorithm.Inputs included initial and goal coordinates, accompanied by the point cloud file either directly exported from the 3D modelling system or acquired with mobile laser scanners and pre-processed.The following sections explain the functions involved in path planning.The generalized process is illustrated in the flowchart below (Figure 6).

Collision Check:
One of the most crucial functions of the algorithm is collision checking.In most RRT implementations, obstacles are depicted as 'regions,' so the collision check involves determining if the sample coordinates fall within these obstacle limits.However, in this implementation, obstacles are the points of the point cloud, representing an indoor environment.Therefore, the aim is to stay 'in' the obstacle.To achieve this, an axis-aligned bounding box that roughly represents the size of an average adult human is used to check if there are 'points' in the path.Another challenge is defining the threshold for a collision.Since the data was not cleaned or filtered, noise in the scans could result in false collisions.Thus, we calculated the density of the obstacles to decide on this threshold.This density is sensor-specific and should be adjusted according to the data source.

Target-Bias:
In standard RRT, sampling is uniformly random across the search space, which can result in inefficiencies in large spaces or narrow openings.A performance-enhancing solution is target-biased sampling.The algorithm still randomly explores but, with a probability of , the tree is expanded from the nearest node to the goal, in the direction of the goal.

3.4.5
Step Size: Defined as the distance between current node (n) and sample node (nsample), step size is set as the width of the bounding box used for collision checking.As the algorithm checks for collisions around nsample, which is at the center of the bounding box, the previous check for  ensures the first half of the step is obstacle-free.Therefore, checking for nsample, results in a clear path.Bounding boxes and step size is seen in Figure 3Error!Reference source not found..

Bi-Directional Search:
Search process involves two RRT trees: Tree A (TA) and Tree B (TB).TA starts the search from initial coordinates and expands towards the goal coordinates (due to bias).Meanwhile, TB starts at the goal coordinates and expands to the latest node of TA in a sequential search process.

Path Optimization, Normalization & Export:
When a connection is established between initial and goal points, rewiring is executed which then checks if there is a path with lower cost.Furthermore, as the path represents a connection between crucial points of collision check bounding boxes, the coordinates are normalized to the floor level by reducing the Z component by 0.7 meters (Figure 4).This process is done to optimize the visualization results.Lastly, the coordinate list of nodes is exported in a plain text file.

Visualization:
The user interface of the real-time 3D modelling system is developed with a game engine, which is also used for visualizing the 3D model.Consequently, the path visualization is done within the same environment.A C# script that accepts the resulting text file from the path planning is used to visualize the path.Since the path shares the same coordinate system with the point clouds, the final path, after normalization, appears approximately 0.2 meters above the surface and provides visual guidance that does not require additional directions (Figure 5).The system's effectiveness was primarily measured by the speed and success of the pathfinding process, with the RRT algorithm tailored to prioritize rapid, safe route generation over finding the shortest path.This approach was chosen to reflect the urgent need for emergency navigation, where conditions can change unpredictably.The current implementation serves as a proof of concept, with plans for future tests to validate and refine the system under a broader range of real-world conditions.The aample results for the longest test path is given below.

Sensor limitations:
On one hand, it is possible to generate a point cloud in real-time.However, the resulting point cloud often has gaps where the algorithm ends up leaving the point cloud, or there are obstacles that are not completely scanned, hence considered as viable openings.On the other hand, some sensors can quickly scan the environment with much higher coverage, resulting in a dataset that is perfect for this methodology.However, it is not yet possible to access the point cloud without preprocessing.A comparison of the same environment with different scanners is shown below (Figure 8).Due to field of vision limitations, the ceiling is not covered by the HoloLens scan.

4.2.2
Step size: As mentioned in the methodology (3.4.2), the step size is set to 0.5 meters to accommodate the bounding box for collision checks.While this is effective for collision checking, a small step size negatively affects performance in large areas (e.g., the basement dataset).Variable step size implementations have proven to be useful in such scenarios (Zhang et al., 2019).However, using a variable step size would cause the collision check method to fail.

Target bias:
Another method to overcome the challenges of large areas is introducing a target bias.This improves performance in simpler layouts; however, the bias probability also introduces a failure probability in complex layouts.As seen in Figure 9 below, both trees quickly advanced to their objective yet resulted in a failure due to iterating towards opposite sides of a wall and not being able to connect.The red and yellow dots represent the nodes that are closest to each other (0.2 meters apart).This challenge is especially evident in scenarios with multiple floors.
Figure 9. Faile due to biased sampling.

Discussion
Results show that it is possible to achieve efficient path planning by direct use of 3D indoor point clouds.However, it is not feasible with the current 3D modelling system due to limitations of the sensors.First of all, coverage of the HoloLens 2 FOV (field of view) and Range is inadequate for this methodology as HoloLens 2 data has gaps and unscanned obstacles which is then considered as empty space and used for path planning by the RRT search.Second, data acquired by mobile laser scanners were suitable for search.On the other hand, streaming the point clouds in real-time with these devices is not yet possible.Another aspect to mention is the existence of drift in the scans.Especially with the HoloLens 2 and BLK2GO scans.Loop closure is practiced with Zeb Horizon RT scans while BLK2GO scans conducted without it to observe the effects.There is visible drift even in comparably short scans (>1 m drift in scans less than 10 minutes).For our methodology it does not effect the results as the path finding and visualization share the positioning and coordinate system of the point cloud.However, it could result in inaccuracies if georeferencing is required.

Continuous and Variable Collision Check:
In our research bounding box hence, the collision check is limited to a large rectangle box.In a scenario where there is an opening that only allows the user to crawl through, our method will fail.However, using a smaller bounding box that represents the minimum space required to pass could result in an unfeasible path (e.g. a path that requires to crawl under tables when there is an opening large enough to walk next to it).Implementing a variable method that changes the size after certain amount of failed iterations could solve this problem.In addition, a method that can check around the proposed path constantly instead of bound based controls (without hindering the performance) can facilitate variable step size and provide performance increase.

Visualization Improvements:
Implementing gaussian splatting to the point clouds could result in a visual that is much easier to comprehend for the observers (commanding agents) compared to the unstructured and semi-transparent look of the point clouds.

Figure 2 .
Figure 2. POI and Field agent view from 3D modeling system

Figure 3 .
Figure 3. Bounding box and step size.

Figure 5 .
Figure 5. Path visualized over the dataset in game engine.
Processing the search from both trees simultaneously instead of the current sequential methodology could yield greater improvements in performance.