Counting trees - methods of automatic analysis of photogrammetric data in forests of the continental region

The paper is devoted to the methods of automatic analysis of photogrammetric data in forests of the continental region. It also discusses how automatic tree counting can be used to manage forests. Experimental research was conducted to verify two methods: Faster R-CNN and Template Matching to automatically detecting tree objects in the continental region characterized by mixed forests with a large predominance of conifers. The research was done based on photogrammetric data taken in four areas belonging to forest districts subordinate to the Regional Directorate of State Forests in Zielona Góra. Data was collected from drones and small airplanes with a photogrammetric container. The results show that both methods can be used for analyzes in specific cases. Moreover, the level of Recall shows the advantage of Faster R-CNN methods for the photogrammetric data collected during the flights in various weather conditions.


Introduction
The role of using aerial photos in forest management is getting bigger every year since the fifties [1]. The consumption of wood grows exponentially year by year [2]. Forests management is essential, as Foley [3] and Trumbore [4] noted that across the world many forests are becoming increasingly susceptible to fire, drought, insect outbreak, and disease due to climate change. Materials obtained as a result of aerial photogrammetry, as well as satellite imagery, are used to update digital surface model (DSM), in analyzing the state of forest vegetation, determining the extent of natural disasters, in forest engineering, as well as in forest cultivation [5,6]. Determining the forest abundance, especially necessary to determine wood resources, requires the determination of the number of trees. In the traditional way, this value is determined by the inventory method consisting of taking samples of the number of trees in small areas in the field, and the obtained results refer to the area of the entire forest. The problem becomes in the case of unavailable areas. Thanks to aerial photogrammetric measurements or satellite imagery, the counting can be carried out on the recorded image faster and easier. The literature reported several supports for the automated location of individual trees and crown delineation: e.g. template matching, valley following, radiance peak filtering, clustering, or Faster R-CNN [7][8][9][10][11]. Application of automatic location of trees and crown detection are used mainly in the oil palm tree, fruit trees, tropical rainforest, citrus orchards, or commercial forest-tree nurseries [12][13][14][15][16][17]. However, there is a research gap in this regard, as there is no literature report, which confirms the use of automatic methods of conifers detection in the continental region. IOP  Hence, the aim of the article is to examine the effectiveness of few methods, applications based on automatic conifers detection on aerial photogrammetric images processed to orthomosaic. Such an aim will be fulfilled based on a literature review, which will be a basis for the development of best methods and experimental research, which will be a basis for verification of such a method. The theoretical model will be presented at first, next to the research methodology will be described together with experimental research results.

Photogrammetry in forests
Chandler [18] identified and compiled key considerations for the application of digital photogrammetry in geomorphology. He observed that automated techniques implemented in readily available software were enabling the application of digital photogrammetry by non-experts for the first time, at a relatively low financial cost, with great potential to deliver "primary data necessary for morphological representation at all scales, using the digital elevation model" [18]. Chandler [18] focuses attention on that non-expert can successfully use photogrammetry to provide a much higher sampling rate than can be achieved realistically from conventional methods such as total stations or digital tacheometers. The increase in popularity of this tool was emphasized by Carbonneau [19], noting the growth of the market of data collection tools, e.g. drones and inexpensive software for data processing. Szymczyk [20] emphasized that photogrammetry is a tool that helps in forest management.
To effectively accelerate operations on the data obtained, it is advisable to use automatic data classification. It was mentioned before the use of automatic classification data are used in the oil palm tree, fruit trees, tropical rainforest, citrus orchards, or commercial forest-tree nurseries [12][13][14][15][16][17]. This article is focused on two methods of that were used in similar task to count and localize trees and crown to improve the process of forest management [12][13][14][15][16][17].
Therefore, the following hypothesis can be formulated: H1. Usage of photogrammetry in the continental region results in more effective forest management.

Template Matching
Template Matching is used to identify image elements. It consists of finding detail in an image with the predefined pattern. This method uses two matrices: the pattern matrix, which is a digital representation of the shape being searched, and the matrix which is a digital record of the analyzed image. Matrices are subjected to the comparison of the pixel intensity, giving as a result of the objective function, which corresponds to the probability of matching. García Torres [21] extracted the trees by clustering pixels with values within a range defined using supervised classification. The algorithm is performed for a given tolerance threshold, which means that the matched areas correspond to the probability level higher than the threshold value [20].
Pollock [10] emphasizes that template matching techniques in forestry employ mathematical renderings of crown typologies to match the image intensity values used to locate the trees and then determine their crown size. This methodology requires a library of tree model, and omission errors are given for those tree crowns that are smaller than the smallest defined radius in the template library, or for irregular crowns [22].

Faster R-CNN
Faster R-CNN is based on Convolution Neural Network. Convolutional Neural Networks (CNN) have been widely used in image recognition, and researchers manifested their high efficiency in remote sensing for object detection. [23]. In combination with images taken with UAVs, CNNs have already been used in tree species classification [24], or animal detection [25].
The neural network models used for object detection in images can be divided into two categories: Single-stage models, which are based on regression. They determine class membership probabilities and object locations in a single run of the algorithm. An example of a widely used single-stage algorithm is the YOLO (You Only Look Once) model [26] and its newer versions, including YOLOv3 [27]; Two-stage models, work by determining Regions of Interest (RoI) from an image that includes potential locations of object occurrences and then classification by convolutional neural networks. These types of algorithms include R-CNN models: R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN. In the R-CNN method [28] and the improved version Fast R-CNN [29], the Selective Search algorithm is used to determine the RoI. Ren S. with other scientist established that The Faster R-CNN method replaces the selective search algorithm with the Region Proposal Network (RPN) formula, which can speed up the explore process for areas of interest and improve the detection accuracy compared to the previous two R-CNN methods [30] Both YOLOv3 and Faster R-CNN were tested for their usefulness in remote sensing (recognising objects at images taken from a far distance). Faster R-CNN indicated a higher value of mAP (Mean Average Precision) compared to YOLOv3. However, YOLOv3's process was faster than R-CNN's effort. In the task of determining the location of bark beetle-infested trees, the precision of location determination by the network is a more significant parameter than the time taken by the network to process the image, so it was decided to use the Faster R-CNN network in the implementation of the described milestone [31].
Therefore, the following hypothesis can be legitimately formulated: H2. The Faster R-CNN or Template Matching method can be used to detect tree objects in the continental region

Methods
This research work focuses on the use of two methods of automatic analysis of photographic images for tree detection -Template Matching and Faster R-CNN. Studies were carried out in the individual case of the continental region, which is characterized by mixed forests with a large predominance of conifers. In 2019, conifers covered over 70% of the total forest areas in Poland [32]. Table 1 shows the species share in the exemplary countries of the continental region. Due to the cooperation during the research with the Regional Directorate of State Forests in Zielona Góra, four areas belonging to forest districts subordinate to RDSF in Zielona Góra were selected for the research. The selected areas along with their location and the studied area are presented in Table 2. In order to verify the hypotheses, the following research activities were carried out: collecting photogrammetric data, processing of photogrammetric data into orthophotomosaic, application of algorithms for automatic data analysis, verification of the results.

Collecting photogrammetric data
Raw photogrammetric data were collected by using BZB UAS unmanned aerial vehicles ( Figure 1) and an aircraft equipped with the bEye photogrammetric container manufactured by BZB UAS (Figure 2). The flights were performed in various weather conditions -in order to check the influence of this parameter on the automatic analysis of photographic images of forests in the continental region. The flight dates were taken September-October. Parameters of flight and collected data are presented in Table 3.  Table 4. Steps of processing the photogrammetric data. Steps

Initial processing
The purpose of this step is to align collected photos. This step utilizes images, camera's interior and exterior orientation parameters and additional inputs such as GCPs to perform following tasks: a) Keypoints extraction -Specific features on the image are identified as keypoints b) Keypoints matching -Images are matched together based on the same keypoints c) Camera model optimization -Internal (focal length, lens distortion coefficients, …) and external (orientation, …) parameters of the camera are calibrated d) Geolocation -Locate the model in space if information about image's geolocation is provided Results of this processing step consist of a sparse point cloud and of calibrated interior and exterior camera parameters. The sparse point cloud is made out of triangulated matched keypoints

Point cloud and mesh
In this step the software creates a dense point cloud and mesh based on the results from the previous step. a) Point Densification -Calibrated camera parameters and the sparse cloud are used to create a Densified Point Cloud. b) 3D Textured Mesh -Polygonal mesh is reconstructed based on the densified point cloud. Additionally using information from collected photos, created mesh is textured. Results of this step consists of a Densified Point Cloud (can be classified and used in further steps) and a 3D textured mesh.

DMS, orthomosaic and index
This is the last main step of the workflow. It allows to create: a) Digital Surface Model (DSM) -It represents natural and artificial features of the Earth's surface. This model allows to compute volumes and also other products such as: orthomosaics or index maps. b) Orthomosaic -Orthomosaic is created with the use of orthorectification. It projects images onto the surface and transforms them into selected projection. Such prepared images are combined to create a seamless image without perspective distortions c) Reflectance Map -Is a map, where each pixel represents the exact reflectance of the object d) Index Map -Is a map, where each pixel's value is calculated using a specific formula that utilizes different bands of the Reflectance Map.

Template Matching -automatic tree counting
In order to conduct experimental studies of automatic analysis in the case of using the Template Matching algorithm, a template was prepared for each developed orthophotomosaic listed in Table 2, which is a digital representation of the image searched. Trials were performed for different levels of standard matching -variable threshold. At this stage, the most important thing was to define the certainty threshold for which the algorithm was to mark objects in the images. The following areas are presented: Lipinki 9 and Cybinka 2.

Faster R-CNN -automatic tree counting
In the case of the Faster R-CNN algorithm, there was no need to prepare additional patterns. The Faster R-CNN algorithm used in the application developed by BZB UAS prepared in advancedattributes of the tree. The following areas are presented: Sulechów 1 and Lipinki 8.  Lipinki 8 is a fragment of a dense mixed forest with a fragment of a water reservoir. The tops of trees are clearly visible, but due to their dense arrangement, some trees appear to be merged with others, which makes it difficult to distinguish individual trees

Verification of the results
The hypotheses Verification was based on two algorithms Template Matching and Faster R-CNN: Tamplate Matching was measured based on three variables: • Precision -the ratio of the number of correctly identified trees to the number of all detected trees, • Recall -the ratio of correctly detected trees to the number of all trees actually in the area, Faster R-CNN was measured based on two variables: Precision, Recall.
The comparison of both algorithms will be done by comparing the parameters Precision and Recall. These parameters will be averaged. For Tempate Matching, the best results from each analyzed area will be averaged.

4.
Results As expected, Template Matching and Faster R-CNN automatically detect trees in the continental region.
For areas where data were collected in worse weather conditions -haze, uneven lighting, shadows, or where the orthomosaic contained artifacts or was blurred, both methods showed a much higher mistake Recall with high Precision. In the case of the Template Matching method, during the analysis of these areas, changing the threshold parameter does not improve recall, it only slightly increases incorrect detections. Table 5 shows the evaluation of Template Matching method effectiveness results and effect of the threshold parameter on recall.
In the case of areas whose data presents a good quality of taken photos and fairly uniform size of trees, both methods give a high result of recall and precision. With Template Matching, high scores are obtained with a very low Threshold parameter. Table 6 shows the evaluation method effectiveness results.         Figure 13. In this case threshold parameter do not effect on recall.

5.
Discussion Template Matching and Faster-RCNN can be used to detect tree objects in the continental region.
Detection accuracy for both methods was high over 40%, compared to areas where the orthomosaics had large differences in the size of trees in selected fragments, blurred and heavily shaded areas, large non-forested areas.
Properly matched Template Matching and Faster R-CNN methods can be used to count trees in the continental region on orthomosaics with a resolution of about 20 cm/pixel.
The selection of a orthomosaics fragment has the greatest influence on the result. Incorrect markings occur in places where there is no forest -on paths or fragments of fields. The tree pattern matches the shadows or the texture of the field because it lacks characteristic shapes -every bright and round element is detected like a tree pattern. An image preprocessing algorithm could be used to eliminate areas of images not covered by forests, before the images are passed to the tree-counting neural network.
In the case of the Template Matching method, the Threshold depends on the analyzed area. Areas varied in terms of the level of plant development required taking into account a greater tolerance for matching than those covered with trees of similar size. This method is using only one pattern of the tree -the pattern is compared in terms of pixel colour and arrangement on orthomosaic. Pixel colour and arrangement in each area is different, so it is determined to select for each area a new pattern. Applying this method requires more time and skill from the person who has to perform the automatic analysis.
An application developed by BZB UAS that used the Faster R-CNN method had predetermined attributes of the tree and did not require additional work in selected areas. This method is recognizing trees at different resolutions of the photo maps. The area of varied plant development is not a problem. Precision of Faster R-CNN is slightly smaller than the Template Matching method. However, Recall the ratio of correctly detected trees to the number of all trees actually in the area is greater than with Template Matching.

6.
Conclusions The aim of the paper was to prove that methods of automatic analysis of photogrammetric data can be used in forests of the continental region to counting trees. The aim was fulfilled using experimental research, which verified the assumed hypotheses. The obtain results allowed to confirm that usage of photogrammetry in the continental region results in more effective forest management. Moreover, it was also confirmed that the Faster R-CNN or Template Matching method can be used to detect tree objects. Such conclusions allow to fulfil the existing research gap concerning the lack of detecting trees in forests of the continental region. The performed research has limitations, as the hypotheses verification was performed on areas belonging to forest districts subordinate to the Regional Directorate of State Forests in Zielona Góra.
Automatic tree counting methods are constantly evolving and are a perfect tool supporting forest management. The dynamic development of satellite imagery could cause, that in the future it will not be necessary to use drones or airplanes to collect high-resolution photogrammetric data. However, there will still be a need to use algorithms that will automatically improve the work related to the management of forest areas. The presented research is an introduction to a further continuation on the usefulness of algorithms and methods based on artificial intelligence and machine learning to improve management methods and quick decision-making on forest management in the intercontinental region.