The Fusion of the Visual and Thermal Images on the Basis of Determining the Image Fragments which Contain Essential Details

The aim of the following study was to develop a procedure which guarantees the data fusion of thermal and visual images. The first stage of the proposed algorithm consisted of images acquisition which guaranteed that the same parts of images represented the same parts of the observed terrain. The second stage depended on previous information about the searched object features. Two different situations were considered herein. In the case when we had the searched object’s feature vector for both representations of a searched object, we could conduct the pattern recognition for each image. It was conducted separately for visual and thermal images. In this way, we obtained the important parts of the images which should be represented in a fused image. The other case examined in the paper, considered the situation in which we did not have the formalised information about the object. In this case, it was necessary to analyse whole images in order to define the potential parts of the images where the object could be found. This analysis should be helpful for an operator to indicate the parts of the images where there are some artefacts which can be the elements of the searched object. Therefore, in this case, the second stage of the algorithm consisted in calculating the local features of the images. These features constituted grey scale gradient computed for the pixels inside the aperture. This study presented the examples of the fused images obtained by means of the developed


Introduction
The fusion of various nature images is particularly valuable when it is used for a support of terrain observations or surveying the objects by the video terminal operator.The images fusion may be conducted in order to emphasise some essential details which are difficult to find in two images simultaneously.The other goal may be to observe some relationship between information contained in these images.
One has to bear in mind that we need to solve the following initial problems during designing the system of images fusion.Firstly, we need to answer what is the goal of the fusion; secondly, what parts of the image or what kind of the objects are interesting.
We may use the algorithms of pattern recognition to emphasise the parts of the images which contain the important objects when we know some features of these objects (e.g. in the form of a pattern vector).
Otherwise we can use the methods of an edge detection or gradient computation assuming that greater homogenous areas do not contain any important objects.
However, even in this case, it is necessary to define what we mean by a greater or smaller area.Therefore we have to know something about an object which we want to observe, for example in order to set the size of the aperture used for image processing during images fusion.
There are many publications which consider the problem of visual and thermal images fusion [1,2,3,4,5].The comparison of seven well known image fusion methods is presented in [6].The security area can be also supported by the images fusion.The examples of such systems are presented in [7,8] for detection of unattended packages and for helping drivers with driving at night or in bad weather conditions [7].Moreover, the detection and recognition of clouds at night [9] may be improved by using the fusion of visual and thermal images.
In the following paper, we will consider the problem of the visual and thermal images fusion.The proposed methods will base on two different approaches.Firstly, we will use information about observed objects, and secondly, we will analyse the images without any previous information.The first stage of the proposed algorithm consists of images acquisition which guarantee that the same parts of images represents the same parts of the observed terrain.

The algorithm for visual and infrared images fusion
The second stage depends on the previous information about the searched object features.In the case when we have the searched object's feature vector for both representations of a searched object, we can conduct the pattern recognition for each image.It is conducted separately for visual and thermal images.The pattern vectors consist of moment invariants J 1 to J 8 defined in [23].In order to assess the distance between vectors we will use Euclidian metric.
On the other hand, in the case when we do not have the formalised information about the object, it is necessary to analyse whole images in order to define the potential parts of the images where the object can be found.This analysis should be helpful for an operator to indicate the parts of the images where there are some artefacts which can be the elements of the searched object.Therefore, in this case, the second stage of the algorithm consist in calculating the local features of the images.These features can be, for example, grey scale gradient, the edges, mean value for the pixels inside the aperture, the local value of temperature or temperature gradient.
In the case when we may use pattern recognition, the last stage of the proposed method is to mark in the visual image the objects recognised on the basis of visual and thermal images.
In the case when we have no precise information about some objects, the last stage consists of inserting into a visual image these parts of a thermal image for which we observed high value of gradient.The Fig. 1 presents the block scheme of the algorithm of the visual and thermal images fusion for the case when we use pattern recognition.The Fig. 2 presents the block scheme of the algorithm of the visual and thermal images fusion for the case when we use the gradient function.
The proposed algorithms guarantee that the video terminal operator will obtain the fused image which connects the most crucial parts of the visual and thermal images in a clear and comprehensive way.

Examples
In order to exemplify the presented above algorithms of the fusion, we will examine two pairs of visual and thermal images.The first experiment will be conducted with the help of a pattern vector calculated for people treated as observed objects and the second experiment will be conducted with the use of gradient calculation.
The following cases correspond to the following situations: 1) In this case, the part of the visual image is disturbed by smoke.The thermal image is not disturbed.According to Fig. 3., we can observe that thanks to using the pattern recognition for the thermal image we can see where are the people from the area which is covered by smoke.
2) In this case, the part of the visual image is disturbed by smoke similarly as in the first case.However, we will not define any pattern vector in this case, but we will calculate the value of the gradient for both images.According to the Fig. 4., we can see that the visual image is filled in by the thermal image in these places where the value of gradient is low -in places disturbed by smoke.
The presented examples illustrate the usefulness of the proposed algorithms for supporting a person who is observing or supervising a terrain.Undoubtedly, more precise information about the observed objects results in better results of such an inspection.
We have developed two procedures which allow to conduct the data fusion of thermal and visual images.These methods concern two different tasks of terrain observation and supervision.The first method was developed assuming that we can formalise information about the observed objects, for example in the form of pattern vector.In the second one there were not such a limitation.For both methods, the first stage was the same and consisted of images acquisition which guaranteed that the same parts of images represented the same parts of the observed terrain.
The second stage was different for the abovementioned methods.It depended on previous information about the searched object features.In the first method, when we had the searched object's feature vector for visual and thermal representations of a searched object, it was possible to conduct the pattern recognition for each image.The visual and thermal pattern vectors were used for object recognition respectively for visual and thermal images.In this way, we obtained the significant parts of the images which should be represented in a fused image.The second method considered the situation in which we did not have the formalised information about the object.In this case, we had to analyse whole images in order to define the potential parts of the images where the objects important for the supervision could be found.Therefore the second stage of this algorithm consisted in calculating the local features of the images.We used the gradient as an examined feature.
Both methods of the image analysis resulted in the fusion images which may be helpful for an operator to indicate the important parts of the images.
This study presented the examples of the fused images obtained by means of the developed method.

,Figure 1 .
Figure 1.The block scheme of the algorithm of the visual and thermal images fusion for the case when we use pattern recognition.

Figure 2 .
Figure 2. The block scheme of the algorithm of the visual and thermal images fusion for the case when we use the gradient function.

,Figure 3 .
Figure 3. Images for the case 1: a) the visual image, b) the thermal image, c) the locations of the people recognised in the visual image, d) the locations of the people recognised in the thermal image, e) the fused image.

Figure 4 .
Figure 4. Images for the case 2: a) the visual image, b) the thermal image and c) the fused image.