EXPERIMENTAL EVALUATION OF LIDAR DATA VISUALIZATION SCHEMES

LiDAR (Light Detection and Ranging) has attained the status of an industry standard method of data collection for gathering three dimensional topographic information. Datasets captured through LiDAR are dense, redundant and are perceivable from multiple directions, which is unlike other geospatial datasets collected through conventional methods. This three dimensional information has triggered an interest in the scientific community to develop methods for visualizing LiDAR datasets and value added products. Elementary schemes of visualization use point clouds with intensity or colour, triangulation and tetrahedralization based terrain models draped with texture. Newer methods use feature extraction either through the process of classification or segmentation. In this paper, the authors have conducted a visualization experience survey where 60 participants respond to a questionnaire. The questionnaire poses six different questions on the qualities of feature perception and depth for 12 visualization schemes. The answers to these questions are obtained on a scale of 1 to 10. Results are thus presented using the non-parametric Friedman’s test, using post-hoc analysis for hypothetically ranking the visualization schemes based on the rating received and finally confirming the rankings through the Page’s trend test. Results show that a heuristic based visualization scheme, which has been developed by Ghosh and Lohani (2011) performs the best in terms of feature and depth perception.


INTRODUCTION
Light Detection and Ranging (LiDAR) has become a standard technology for procuring dense, accurate and precise three dimensional topographic information in terms of data points in short time.Digital Elevation Models (DEM) and Digital Terrain Models (DTMs) generated through such information can be used for applications which need precise geographic information.In scenarios like disaster management and mitigation, situations might arise when data are procured in short time and required to be visualized for further assessment.It is to be noted here that even for very small areas, the size of a LiDAR data set is considerably large.MacEachren and Kraak (2001) pointed out four different research challenges in the field of geovisualization in the ICA research agenda.Out of these, the second agenda item concerns the development of new methods and tools as well as a fundamental effort in theory building for representation and management of geographic knowledge gathered from large data sets (Dykes et al., 2005).Three dimensional models for visualization of geospatial data have been brought forward by Zlatanova (2000), Zlatanova and Verbree (2000), Lattuada (2006), Lin and Zhu (2006) and van Oosterom et al. (2006).Several approaches for processing and visualizing LiDAR data have been found in the literature, which indicate the following approaches for preparing 3D maps: (a) direct visualization of LiDAR point cloud (b) through manual process of classification which is time consuming, or (c) through the process of segmentation.In the cases of segmentation and classification, the extracted features are first generalized and then visualized.Out-of-core computation algorithms have also been reported which handle and visualize large point cloud datasets.
The authors have observed from the literature that the process of classification of LiDAR datasets require many thresholds, substantial a priori knowledge, and is also time consuming.Therefore, the authors have worked in the direction of development of methods and schemes for visualization which bypass this process of classification.Visualization of LiDAR data using sim-plices has been reported in (Ghosh and Lohani, 2007a,b).They have also developed a heuristic to process LiDAR data to extract and generalize features (Ghosh and Lohani, 2011).This paper develops a methodology to statistically rank the various visualization schemes for LiDAR data in terms of the visualization experience through specifically designed experiments, and select the best scheme.The various schemes of visualization that have been studied by the authors are presented in section 2. Section 5 presents the design and methodology for conducting the experiment and also the statistical background for analyzing the results.The results are presented and discussed in section 6. et al. (2008) visualized large point data sets using a head tracked and stereoscopic visualization mode through the development of a multiresolution rendering scheme that supports rendering billions of 3D points at 48-60 stereoscopic frames per second.This method has been reported to give better results over other analysis methods, especially when used in a CAVE based immersive visualization environment.Richter and Döllner (2010) developed an out-of-core real time visualization method for visualizing massive point clouds using a spatial data structure for a GPU based system with 4 GB main memory and a 5400rpm hard disk.Ghosh and Lohani (2007a) attempted to develop a methodology for stereoscopic visualization of the terrain using Delaunay Triangulation and the OpenInventor visualization engine.The Triangulated Irregular Network (TIN) obtained from the LiDAR data is draped with the geocoded texture data and the two are directed into the visualization engine via an Inventor file.The TIN generated through the Delaunay Triangulation was found to have triangles with small area but very large edges.Such triangles are removed using a certain threshold τ.The remaining triangles along with the point data and geocoded texture are sent to the visualization engine (Ghosh and Lohani, 2007b).Ghosh and Lohani (2007b)  for processing LiDAR data for visualization.The tetrahedrals generated by the process were broken down to their respective facets.The triangular facets containing long edges were removed using a threshold τ.The remaining triangles, the geocoded texture, and the points are sent to the visualization engine.

Kreylos
In case of the Delaunay triangulation and subsequent trimming, it is noted that certain dome shaped terrain features e.g very sparse trees get deleted, whereas planar features were well represented.
In the case of Delaunay tetrahedralization, the sparse trees were better represented.Owing to the complexity of the tetrahedralization process, the planar features are over represented and the rendering is comparatively slow.Thus, Ghosh and Lohani (2011) expressed the need of a hybrid mode of processing and developed a heuristic after clustering the data using data mining algorithms.
The results from all the processing methods described in this section may be visualized either in 2.5D or 3D.The notations and the schemes are shown in

OBJECTIVE
The aim of this paper is to rank the various processes listed in Table [1] in the order of their effectiveness in terms of feature recognition and depth perception.In this light, the paper addresses the following research objectives: (a) design an experiment to evaluate the various schemes listed in Table [1], and (b) find the statistically best method for processing and visualizing LiDAR data.

DATA AND TOOLS USED
In 2004, Optech Inc, Canada conducted a LiDAR flight over the environs of Niagara Falls.The flight was conducted at an average height of 1200m above ground level.Five different subsets of 100m×100m each were cut out from the datasets.The number of data points in each of these datasets is given in Table [2] (Barber et al., 1996).The OpenScene-Graph Engine with its C++ based API is used for the visualization of the LiDAR data sets and derived products (see Table [1]).The OpenSceneGraph engine is able to render in 2.5D and 3D, where 3D perception was made possible through anaglyph glasses.The programming is done on an Ubuntu Linux 11.04, 64 bit platform.

METHODOLOGY
This section presents the experiment designed in order to obtain the ratings from the participants for the 12 visualization schemes presented in Table [1].

Design of Experiment
The experiment was designed to examine the user perception of various features like buildings, trees, roads and vehicles in addition to the capability of 3D perception.It was necessary that the participants were brought to the same level before participating in the experiment.Therefore, the conduction of the experiment was divided into three phases: (a) orientation, (b) acquaintance with perception of the third dimension, and; (c) perception of features from the various schemes presented to the participants for rating the schemes.

5.1.1
The rating scale Researchers have opined differently on selecting the scales of ratings to be adopted for surveys and experiments.While Worcester and Burns (1975) argued that scales without mid points pushed the respondents to the positive end of the scale, Garland (1991) found the opposite.Dawes (2008) found that scores based on 5 point or 7 point scales might produce slightly higher mean scores relative to the highest possible attainable score, compared to a 10 point scale.Dawes (2008) also showed that this difference was statistically significant.In this study, therefore, a 10 point scale was selected.

Phases of the experiment
The orientation phase introduced the participant to the experiment through a presentation designed on LibreOffice Impress.The scheme of rating and the meaning of 3D perception was explained to the user in this phase.
It was emphasised that the rating of the scenes presented were to be done absolutely on a scale of 1 to 10, where 1 means "very bad" and 10 means "very good".
The second phase of the experiment was designed to acquaint the participant with 3D perception through anaglyph glasses.Three different anaglyph photos were presented before the participant in an increasing order of complexity in terms of the depth cue.The participants were asked objective questions which involved either counting the number of objects in a scene, ordering the objects in terms of distance from the participant or identifying the type of the objects.The orientation phase and the acquaintance phase were designed to bring the participants of the experiment to the same level.
The third phase involved the rating of the features perceived from the different visualization schemes presented to the participant in the experiment.Since the focus is on the perception of the various features on the terrain (buildings, trees, roads and vehicles) and depth cue, the following questions were formulated for obtaining the ratings: 1. How well do you perceive depth in the presented scene?
2. How well is your overall perception of features in the presented scene?Features mean buildings, trees, roads and vehicles.
3. How well do you perceive buildings in the presented scene?
4. How well do you perceive trees in the presented scene?
5. How well do you perceive roads in the presented scene?
6. How well do you perceive vehicles in the presented scene?Since there are 12 visualization schemes which are to be evaluated by the users, there could be 12! ways of presenting the outputs from the schemes to the participants.The process of confounding is used to limit the number of possible ways in which visualization schemes are presented to the participant (Finney, 1980).Vajta et al. (2007) have proposed a method of determining the order of presentation of the visualization schemes using Hamiltonian paths.A Hamiltonian path will ensure that all the visualization schemes are presented before a participant.It is observed from Table [1

Design of the user interface
A user interface using Qt/C++ and OpenSceneGraph was designed which would read a file corresponding to the processed or unprocessed dataset and the texture and also present the participant with visualization schemes and the rating questionnaires alternatively.Each participant was to be presented with results from a different dataset and a different order of schemes.Rotation, zooming in and zooming out were possible for each of the visualization schemes.The ratings obtained from a participant were to be stored in a binary format in the hard disk.Screenshots of the various schemes presented to a participant are shown in Figure [2] for one of the data sets.The user interface was designed such that the interaction with the keyboard and mouse was minimum.

Conducting the experiment
Sixty third year undergraduate students of the Department of Civil Engineering, Indian Institute of Technology Kanpur agreed to participate in the experiment through appointment slots of 45 minutes each.During each of the experiments, it was ensured that the lighting and cooling were kept constant, and the noise level was kept to the minimum possible.

Statistical analysis of the scores
Based on the previous discussion, there are 12 schemes to be compared (see Table [1]) and there are 60 participants in the experiment.This subsection describes the statistical methodology to compare the 12 methods using the scores obtained from the 60 participants.
To simplify the explanation of the analytical process, the number of processes are denoted by k and number of participants are denoted by n.Each of the n participants gives a score on each of the processes.The score given by the ith participant for the jth process is denoted by s i j (see Table [3 The alternate hypothesis is to prove otherwise.In order to check for the validity of the null hypothesis, the Friedman Test (Friedman, 1937(Friedman, , 1939) ) is conducted.The following notations are therefore introduced: R The formula for Q-statistic for the Friedman's test is given by (Gibbons and Chakraborti, 2010) Q denotes the test statistic for the Friedman's test.For large values of n and k, the probability distribution of Q is approximated by the χ 2 distribution.If the p-value, given by P(χ 2 k−1 ≥ Q), is smaller than the level of significance α then the null hypothesis is rejected.

Post-hoc analysis and hypothetical ranking
In case the Friedman's test indicates that the null hypothesis cannot be accepted i.e. results from the k processes are not similar, a posthoc analysis has to be conducted.The processes i and j are termed as different for a significance level α if (Kvam and Vidakovic, 2007) Thus all the processes can be given hypothetical ranks from 1 to k.In case two processes are found similar (see equation [2]), they are allotted the same rank as per the standard competition ranking scheme.
5.4.4Page's test for ordered alternatives Page (1963) developed an L-statistic to help rank the processes.Since the column sums are real numbers, the R • j 's can be arranged in a decreasing / increasing order.If Y j denotes the hypothetical rank of the jth column, obtained through some process, then the Lstatistic is given by The value χ 2 L is defined as where α is the level of significance, then we reject the null hypothesis and accept the hypothetical ranking of the processes.

RESULTS AND DISCUSSION
The experiment intends to present all the visualization schemes from different subsets.In order to avoid biases in the experiment, all the scenes are not presented sequentially but randomly.However, confounding is done in order to avoid the large number of experiments.With the nodes and edges as defined in Figure [1], 64 Hamiltonian paths are calculated using Mathematica.However only 60 participants agreed to take part in the experiment.

Data preparation
The user interface designed for the experiment stored the scores received from each of the participants in the experiment in separate files.Each of these files were read into a corresponding Li-breOffice Calc sheet.A LibreOffice BASIC code was then used to collate the results for each of the questions into a single sheet.Since there were six questions, six different sheets were created through the BASIC code.Each of the sheets thus contained a 60 × 12 matrix.

Data analysis
It was verified through Pearson's χ 2 goodness of fit tests and histograms that the distributions for the scores received for each of the visualization schemes were not normally distributed.It was therefore clear that parametric methods like ANOVA or MANOVA cannot be applied for these ratings.The Friedman's test (Friedman, 1937(Friedman, , 1939) ) and the subsequent post-hoc analysis (Kvam and Vidakovic, 2007) are therefore selected to determine performance differences in terms of depth and feature perception through the different visualization schemes.With respect to the formulas and tables described earlier, we have k = 12 different processes and n = 60 participants.The results of the Friedman Test for each of the questions are given in Table [5].If the level of significance α is taken as 0.05, it is clearly seen that the null hypothesis of Friedman's test cannot be accepted for all the questions.The methodology adopted for hypothetically ranking the schemes has been explained earlier in this paper.The column-wise rank sums are added up and the post-hoc analysis was conducted for each of the LibreOffice Calc Sheets corresponding to each of the questions.The column wise rank sums R • j , j = 1, • • • , 12 are laid out in the increasing order for each of these sheets and ranks are given from 1 to 12, according to the standard competition ranking scheme.The ties in this scenario are determined using equation ( 2).The values of the χ 2 L statistic and the corresponding P values are presented in Table [ The Friedman test was performed again on the ranking's obtained from the earlier post-hoc analysis (see Table [7]).The Q-statistic is found out to be 116.3224and the corresponding pvalue is found out to be 9.9362E-020.Thus at a significance level of α = 0.05, the hypothesis that all the visualization schemes are equally effective cannot be accepted.A post-hoc analysis is thus conducted and a hypothetical ranking is performed.Page's test for alternative hypothesis confirms this ranking as the values of χ 2 L is seen to be 57.0204 and the corresponding P-value is 4.3131E-014.It is therefore seen that the heuristic-based model viewed in the anaglyph mode (AN-MDL) gets the highest rank.It is therefore seen that AN-MDL (described in Ghosh and Lohani (2011)) has performed the best.

Findings from the research work
The experiment was conducted with 60 participants who were third year undergraduate students of Civil Engineering.The participants were not exposed to visualizing LiDAR data prior to the experiment.
It can be seen from Table [7] that anaglyph based visualization schemes have generally received better rankings.Almost every participant expressed appreciation for the realistic representation of the data in the anaglyph mode of visualization.Table [7] also reveals that the point based visualization in the anaglyph mode has received lesser rankings.Since laymen are not used to see the features of the terrain in the point format, low ratings have been received by the point based visualization schemes namely PL-PTS and AN-PTS.It was also observed during the experiment that the participants usually zoomed out while visualizing the point based schemes.The Delaunay triangulation only considers one of the points which have the same values of x and y coordinates and different z coordinates.As a result, there is a loss of 3D information when a triangulation is generated from the raw datasets.The structures of the trees are therefore not well represented, especially for the sparse trees.The buildings have rough and sloping walls and non-smooth edges.Further, when narrow triangles are trimmed using a threshold, sparse trees are lost in the process.Buildings turn to roofs hanging in the space when the triangles are trimmed.As a result, buildings have poor scores in AN-TDTRI and PL-TDTRI.However, the terrain and the vehicles are both well perceived in the triangulation and the trimmed triangulation schemes.The tetrahedralization based processing schemes (PL-DTET and AN-DTET) were developed for preserving the tree structures.However, the thresholding process creates "holes" in the representation of the terrain objects.Therefore good ratings are obtained for the trees but poor ratings for the buildings.The heuristic based visualization schemes (PL-MDL, PL-PMDL, AN-MDL and AN-PMDL) were designed to take the best of triangulation and tetrahedralization based schemes.Further, the buildings in these algorithms are represented with smoother walls compared to their triangulation based counterparts.In the PL-PMDL and AN-PMDL algorithms, the trees are represented by points and therefore they have received a poorer rating in the experiments compared to the other schemes.On the contrary, both trees and buildings received high ratings in AN-MDL.PL-MDL receives high ratings amongst the 2.5 modes of visualization.Roads and vehicle features are represented well in the AN-MDL and they have received "best" ratings for Questions 5 and 6.AN-MDL received the best overall ranking in terms of depth and feature perception followed by AN-DTET and AN-DTRI on the second and third places respectively.This ranking has also been verified statistically using the Page's trend test.

Remarks on the statistical method
The Friedman's test and post-hoc analysis were used repeatedly to determine the hypothetical ranking of the various visualization schemes presented in this paper.The Page's trend test confirms the hypothetical ranking.It is therefore possible to select the "best" visualization scheme in terms of depth and feature perception.Since the experiment requires a participant to evaluate a visualization scheme on more than one factor, the data is actually multivariate.Therefore, a more robust method of ranking based on permutation MANOVA or non-parametric MANOVA should be used.Further research in this direction would be presented elsewhere.

CONCLUSION
The aim of developing a visualization scheme for LiDAR data is to enable a user to perceive the various features on the terrain i.e. buildings, roads, trees and vehicles.Researchers have been working in this area of extracting features from LiDAR data using the methods of classification.The authors of this paper have been working towards the development of methods which can enable the user to distinguish between the various features available on the terrain without using the method of classification.Literature suggests that several methods of visualizing LiDAR data and derived products have been proposed by the scientific community.This paper has presented a method for designing an experiment and data analysis to select the best visualization scheme from the various available schemes.The experiment design and subsequent data analysis have proven that the methodology developed by Ghosh and Lohani (2011) performs the best in terms of depth and feature perception.Since the data obtained from the experiment is multivariate, studies using multivariate non-parametric methods for ranking and selection of visualization schemes would be presented in the future.

Figure 1 :
Figure 1: Layout of nodes and edges for creating Hamiltonian Paths ] that the presentation methods are either in 2.5D on 3D (stereoscopic) and the pre-processing schemes are namely point based, triangulation based, tetrahedralization based, and heuristic based.The pre-processing schemes are laid out in the x-axis and the presentation schemes are laid out in the y-axis.The confounding is done by ensuring that any movement is made one step either in the x-direction or the y-direction.Thus, the valid paths from one point to the other are shown in Figure[1].The Hamiltonian paths are then computed through the Mathematica package.

Figure 2 :
Figure 2: Various visualization schemes presented to the participant.Screenshots are presented for a single data subset here.
also used Delaunay tetrahedralization ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume I-2, 2012 XXII ISPRS Congress, 25 August -01 September 2012, Melbourne, Australia

Table 2 :
. Each of these subsets contains buildings, trees, roads and vehicles.For algorithms using Delaunay triangulation and tetrahedralization Description of subsets.Points are called redundant if they have same coordinates we use the Quickhull tool
Scores given by n participants 5.4.1 Pre-analysis For each of the participants, the scores in a row are ranked in an increasing or decreasing order.Tied ranks are replaced by averaged ranks.In Table[4], r i j denotes the rank of s i j amongst s i1 , s i2 , . . ., s ik .

Table 4 :
Ranks for each of the n participants 5.4.2Null hypothesis and the Friedman's test As a first step, the null hypothesis is framed as H 0 : −the processes are all similar.

Table 5 :
Q-statistic and P-value for all the questions 6].The ranks obtained are presented in Table[7].Here, equal ranks have been replaced by average ranks.

Table 7 :
Question-wise rankings for all the visualization schemes.Lower values of ranks are better.