HIERARCHICAL APPROACH FOR DETECTING CHANGES WITH THE USE OF DIFFERENT PYRAMID LEVELS IN DENSE IMAGE MATCHING

Many cities order spatial data systematically, in particular aerial nadir images and orthophotomaps. However, only the orthoimages and orthophotomaps are usually used by the city administration, particularly in spatial planning. Some of the users are not aware of the possibilities as to how the aerial images can be used. Spatial data users, who may not be specialists in photogrammetry, are sometimes not aware that it is possible to obtain 3D information from 2D images as a point cloud. The idea of dense image matching (DIM) is well-known and described in the field of photogrammetry. Although dense image matching is a timeand memoryconsuming process, this does not present a major drawback with modern computing. Images for the test area – Warsaw – are characterised by Ground Sampling Distance (GSD) equal to 8 cm. These images can be successfully used in change detection processes, comparing the dense image matching point cloud from two different dates. What is important while considering land cover change detection, is that it is not necessary to generate a detailed and high-density point cloud, e.g. in order to detect changes in buildings. The main idea of the article is to present the possibility of using higher levels of images pyramid in dense image matching within the change detection process as a way to optimize the processing time and point cloud accuracy. Which level of pyramid is needed to detect different changes in urban land cover will also be discussed.


INTRODUCTION
Many cities and countries order aerial nadir images systematically, mostly to produce orthophotomaps. Some of the users are not aware that it is possible to obtain 3D information from 2D images as a dense point cloud. The idea of dense image matching (DIM) is a well-known idea of acquiring 3D information from images (Gruen, 2012;Remondino et al., 2014). DIM point cloud generation may be a time-consuming process, especially concerning the high number of highresolution imagery. The duration of the DIM processing is one of the most critical aspects, while another technical aspect concerning DIM seems to be solved (Gruen, 2012). It is worth noticing that nowadays that the standard overlap between images has grown, especially for big cities, where there are ever-increasing densities of high buildings (Lemaire, 2008). Bigger overlaps mean lower occlusion, but this also requires more memory and becomes more time-consuming with respect to image processing. The point clouds generated from DIM are mostly used for producing digital surface models (DSMs), which represents the elevation of the ground, as well all objects on the ground. The DSMs from at least two different dates can be successfully used in the change detection process by height comparison, and they can be generated not only from the aerial images, but also from satellite stereopairs (Guerin et al., 2014).
The quality of point clouds from DIM and the resulting DSM depends on a few factors, namely on the quality of the images, their orientation accuracy and the overlap between them. In 2013 the European Spatial Data Research Organisation (EuroSDR) conducted a benchmark on image-based DSM generation (Haala, 2013), which proved that the number of software applications, as well as the quality of DIM points clouds is growing. Hirschmüller and Bucher (2010) discussed DSM accuracy, which was generated with the use of Semi-Global Matching (SGM). There are also other studies in the literature which focus in detail on the accuracy and the DIM workflow. Dominik (2017) presented the idea of taking into account the base-to-height (b/h) ratio of stereo pairs during DIM matching, when the point cloud is dedicated, to deliver the DSM. What is more, studies presenting the results of DIM in Inpho MATCH-T DSM can also be found in the literature (Lemaire, 2008). A number of factors can be identified that positively influence the application of the aerial image in terms of change detection. These factors are: growing spatial resolution (GSD -Ground Sampling Distance), growing overlap and automation of the processes. The automation of DIM and DSM generation may also lead to popularisation of using the data among national mapping agencies. Some of them, such as the Ordnance Survey in Great Britain, implement some solutions in order to perform analyses and products which are useful from their point of view (Gladstone at al., 2012;Holland at al., 2012). Another advantage of change detection from aerial images is the possibility of including archival images in building change detection and monitoring (Nebiker et al., 2014). Using DMSs in large-area analyses may be used, not only in urban areas, but also in forest inventory (Ginzler and Hobi, 2015), where the canopy height model is derived from a combination of digital surface models from images and existing terrain model. In some studies DIM and airborne laser scanning data are used in order to detect changes (Stal at al. 2013).
What is important concerning land cover change detection, is that it is not necessary to generate a detailed and high-density point cloud, in order to detect changes in buildings. The main idea of the article is to present the possibility of using higher levels of image pyramid in the DIM within the change detection process as a way to optimise the processing time and point cloud accuracy. The level of pyramid needed to detect building changes in urban land cover will also be discussed. A short presentation of DSMs generated from different pyramid levels is included. The time of point cloud processing and the level of detail of the models is provided. Moreover, some problems concerning the comparison of two DSMs from different dates are described, as well as the possible way of solving this problem.

DATA PROCESSING AND METHODOLOGY
Image pyramids are well-known in digital image processing and computer vision (Adelson et al, 1984). Image pyramid is a series of the same image, with a gradually degraded resolution. The bottom, i.e. level 0 of the pyramid, is equal to the original image. Further images are subsampled by a factor of two to obtain the next pyramid levels. Image pyramids are willingly used in photogrammetry, e.g. in dense image matching in order to make the computations more efficient.
The hierarchical approach with the use of image pyramid is proposed in this article. The idea is that in detecting changes of large areas (e.g. changes or partial changes in buildings), higher levels of the image pyramid can be used in image matching and thereby shorten the processing time. In the proposed methodology the process of matching starts at higher level, e.g. level 9, and can be stopped at different levels, e.g. levels 0, 1 or 2. What isś more, such a preliminary change detection may be efficient in large-scale change detection in order to find areas where the change occurs and then perform more detailed analyses with the use of a more detailed point cloud and DSM.
While generating a dense point cloud, the date of image acquisition plays an important role or, more precisely, the presence of leaves on the trees. Mostly the photogrammetric flights in urban areas are performed in the non-leaved period so that the objects are visible under the trees. However, it may happen that the images obtained during the full vegetation season are necessary, e.g. for vegetation analyses. During the leaf-off season, trees are practically invisible on the DSM from DIM, or visible in the form of small objects representing the tree trunks. On the other hand, in the leaf-on season, when there is foliage on the trees, the height model will deliver the full tree crowns. As a result of subtraction of such two models, which were created from images taken at different seasons of the year, the occurrence of trees will result in significant differences in altitude at the site, which does not prove that the trees were felled. Similar problems were faced in the described experiment. Images in May 2017 were acquired during the leafon season, while images in 2018 were acquired during leaf-off season. This difference resulted in changes in land cover being detected, where they have not really occurred. Therefore, in the methodology, a technique is proposed to exclude vegetation from the difference analysis of height using an infrared channel.
In Figure 1, the proposed methodology is graphically presented. As a first step, the DIM from images obtained in two different dates is processed. Then, DSMs are generated and a differential DSM (dDSM) is calculated by subtracting the newer DSM from the former. Further, the dDSM is reclassified, applying assumed criteria. In this methodology, the negative changes lower than -2 m are assigned to one class and positive changes higher than 2 m are assigned to another class. Meanwhile, the NDVI (Normalized Difference Vegetation Index) is calculated based on the near-infrared orthophoto from 2017 in order to remove the vegetation effect: (1) where R is the value in the red channel NIR is the value in the near-infrared channel From the NDVI raster, non-vegetation objects were distinguished and assigned to one class. Lastly, the reclassified dDSM and the non-vegetation raster are subtracted to obtain the final changes.
The given images were already oriented and external orientation parameters were provided. During the methodology development, dense image matching was performed in Inpho Trimble 9.2. Initially, the dense clouds were generated using different final image pyramid levels: from level 0 to 4. Then, the DSMs for each pyramid level were compared. For the two different dates, DSMs from the same pyramid level were subtracted in order to create a differential DSM. DSM generation and analyses were performed in ArcGIS 10.6.

DATA AND TEST AREA
As a test object, Warsaw was selected. As a test area, three different regions were selected ( Figure 2). Images from 2017 and 2018 were used in the analyses. The images were collected with an overlap 60/60%. For one test area, which was located in the centre of the city, the overlap was 80/80% (Fig. 2a). Additionally, selected regions differ from each other with respect to the type of housing. For example, one of them used to be an industrial area, but now blocks of flats are being built in the area (Fig. 2c). The images were already oriented and external orientation parameters were given, thus the aerotriangulation step is not included in the experiment. This is also caused by the methodology assumptions, namely when authorities in Warsaw order nadir images, the aerotriangulation project and results are also provided. The main idea of the article is to create an understandable methodology, which will be possible to be implemented in the city administration and be used by specialists in spatial planning who work with photogrammetric data, but are not specialists in this field. Apart from the images, the near-infrared orthophoto from 2017 was used to calculate the NDVI and exclude vegetation areas from the analyses.

Differential DSM calculation
In Figure 3 an example of a differential DSM for the test area, Kamionek, is presented. According to Figure 3, one can observe that the changes in height occur not only on the buildings, but also on the areas, where high vegetation grows. Other problematic objects are buildings' edges, where changes also occur. These changes are in most cases, from 1 pixel up to 3 pixels wide. This may result from imperfect overlapping of individual pixels of two DSMs, as well as a result of DIM processing. Therefore, in the methodology, some tools were included which help to discard the undesirable changes. In Figure 4, the differences between shaded models of leaf-on and leaf-off DSM are presented. The differences in tree canopy representation can clearly be seen. For some of them, especially for single-growing trees, in leaf-off DSM there is sometimes no height information, or just a few pixels representing the tree trunk. What may also be important, in the selected area, trees sometimes grow near to buildings, which results in linking the trees with buildings on the final leaf-on DSM, and the edges of the building cannot always be clearly identifiable.
Thus, in order to remove the vegetation effect, the NDVI was calculated and non-vegetation objects were assigned to one class. The threshold for determining the class was equal 0.05. Further, to remove building edges from the change detection analysis, an area criterion was chosen, i.e. objects with an area larger than 25 m 2 were indicated as interesting areas. However, the area criterion may depend on the area type or minimum area included in the building definition.

Analyses of different image pyramid levels
In the next step, differential DSMs for each pyramid level were analysed. For selected areas, the DIM in Inpho Trimble 9.2 was conducted. The final pyramid level of point cloud generation was set to: 0, 1, 2, 3 and 4. Then, the DSMs from the point clouds were generated in ArcGIS. For each pyramid level, the DSMs resolution was also lower. For level 0, the spatial resolution of the DSM was 0.25 m, for level 1: 0.50 m, level 2: 1 m, level 3: 2 m and level 4: 4 m. The DSM resolution was also lowered twice for each level, similar to the image pyramid.
The main goal of the experiment was to detect buildings' change using the DIM from higher pyramid levels. According to the processing time (Tab. 1), as could be expected, the image matching process for higher pyramid levels shortened by approximately four times comparing to the lower one.  As a next step, DSMs were generated and compared. In Figure  5, shaded models of DSMs generated from point clouds with the use of different image pyramid levels are presented. According to the DSMs, a difference in detail representation can be noticed. There is not a big difference between the level 0 and level 1 DSM, where the DSM from level 1 seems to be smoother, but some technical elements on the roofs are also visible. The DSM from the level 2 point cloud seems to be more generated, however, it is still possible to interpret what is presented on it. Levels 3 and 4 are very smoothed, neighbouring objects are merged into one, and it is almost impossible to point out where of the edges of the building are.
Thus, analysing the processing time and the resolution of the DSMs, level 2 of the image pyramid was chosen as a level which is sufficient for change detection in buildings. However, in further steps (i.e. dDSM calculation, height changes reclassification and final change detection) all levels were still included. Figure 5. Comparison of DSMs generated from dense point clouds using different pyramid level

Final change detection
After the DSM generation, for each pyramid level, the differential DSM was calculated. In further steps, the dDSM was reclassified in order to make the calculations faster. At the beginning all the interesting changes, e.g. changes lower than -2 m and higher than 2 m were assigned to one class. However, this was not the best solution, particularly for the changes of smaller areas. When the incidental changes, both negative and positive, were located near each other and were assigned to one class, then they were merged into one polygon, have bigger area and it was more difficult to remove them using the area criterion. Thus, it was decided to assign positive and negative changes to two different classes.
In the next step, the changes indicating vegetation had to be removed. In order to achieve this, the raster file representing changes was subtracted from the raster representing nonvegetation areas. The raster was also reclassified and nonvegetation was assigned to one class. Such a subtraction delivered the common part of the change and the non-vegetation raster, i.e. potential changes in buildings. In Figure 6, an example of reclassified changes before and after vegetation exclusion is provided. According to the figure, it can be noticed how the percentage of the changes in this part could be attributed to vegetation, and how effectively the vegetation was removed. The only incidental changes that were left resulted from changes on the building edges. Figure 6. Comparison of reclassified differences (changes) before and after vegetation exclusion After removing the vegetation, final changes were obtained. In Figure 7, detected changes from dDSMs from different pyramid levels are provided. According to Figure 7, for levels 0 and 1, the changes are still very detailed, and most of them are irrelevant, even after removing changes of area smaller than 25 m 2 . For levels 3 and 4, there is a risk that the areas of the detected buildings are too big. Thus, level 2 of the pyramid seems to be the most appropriate concerning building change detection from DSMs from aerial images. Finally, the percentage of changes which meet the area criteria (changes bigger than 25 m 2 ) and are aimed to indicate the building changes is presented in Tables 2 and 3. Similar correlation can be noticed as for changes after removing vegetation, namely for pyramid levels 0 -2 the selected changes because of area is quite similar, however not so similar as for changes without vegetation. According to last column in the tables, i.e. percentage of the final changes to the initial area, it is worth mentioning that results for levels 0 -2 are very similar. The percentage for levels 3 and 4 is growing rapidly, what may be also a result of growing spatial resolution of DSM with pyramid level. Therefore, according to the results presented in Tables 2 and 3, as well as in above-presented Figures confirm that level 2 of the image pyramid may be sufficient for building change detection using high-resolution aerial images.

CONCLUSIONS
In the article the idea of a hierarchical approach in land cover change detection in urban areas is presented. Based on nadir images and dense image matching, is it possible to create a highly-accurate DSM and detect changes, e.g. destroyed or newly built buildings. The idea seems to be effective, because many cities order nadir images systematically, but the images are mostly used for generating orthophotos. Additionally, the images are acquired more often than ALS, also because of the costs.
The main idea of the article is to create a methodology of change detection in urban areas. In the presented experiment images from 2017 and 2018 for Warsaw were used. The spatial resolution of the images is 8 cm. From the images for selected areas, dense point clouds were generated, with the use of different levels of image pyramid. As a result, the DSMs from different point clouds were generated and compared. A higher pyramid level of images is sufficient and can take less time than image matching on full resolution images. Thus, pyramid level 2 was chosen to detect changes in buildings.
During the calculation of the differential DSMs, some problems occur. The images were acquired during leaf-on (2017) and leaf-off (2018) seasons, which resulted in irrelevant changes in vegetation area. Thus, NDVI was calculated to remove the changes.
In summary, the presented methodology seems to be easy to implement and use by non-specialists in the field of photogrammetry, This article also shows the potential of the images, which can be exploited in an uncomplicated way. For building change detection, level 2 of the image pyramid seems to be sufficient. The buildings are correctly detected, and this level of the image pyramid does not produce many details, which are not always necessary.