DATA FUSION OF PANCHROMATIC AND MULTISPECTRAL IMAGES BASED ON OPTIMIZATION USING LAGRANGE MULTIPLIER

The remote sensing satellites provide complementary images of different resolutions which need to be integrated using the techniques of image fusion. In this paper, image fusion using the IKONOS satellite data is discussed. Unlike other models which are based on sensor model, our approach is data centric including the effects of the sensor as well as the reflectance characteristics of the imaged object. A linear relationship is built between the panchromatic channel and the multispectral channel data. We then formulate a minimisation function in terms of Lagrange multiplier to optimally maximise the spectral consistency and minimise the error in variance. The variances of the downsampled multispectral channels are observed and compared with the original multispectral data. A chi-square goodness of fit test is performed to evaluate the data computed based on our algorithm. Simulation results are presented using the IKONOS 1m resolution panchromatic and 4m resolution multispectral data.


INTRODUCTION 1.1 Introduction
In the last few decades, geospatial technologies have taken a new dimension with the advent of satellite and computers.Satellite provides immense information about the earth's surface particularly in the form of different types of images depending on the sensor's capabilities.Along with the availability of computers, it allows transfer and storage of data and more importantly analysis of the satellite data.Remote sensing is one of the geospatial technologies which is paving the way in this aspect.The space borne or airborne cameras give high quality data which is useful for a wide range of applications.For example, the commercially launched satellite sensor like the IKONOS produces 1m panchromatic and 4m multispectral imagery (blue band, green band, red band and infrared band).The imagery provides complementary information which is not available in a single image.The high resolution panchromatic gives a more refined detail of the object on ground, which sometimes is used for object detection.On the other hand, the low resolution multispectral contains the colour information and sometimes is useful for accurate object identification and classification between similar objects.These two types of imagery can be combined to accommodate a wide range of high resolution imagery applications.The combination or integration of information from single source or multisource to obtain a single output which has higher information content from the input signal is referred to as data fusion.Data fusion has served an important role in the geospatial society, where data from different sources having complementary information can be fused together.Data fusion has been defined as a "formal framework in which are expressed means and tools for the alliance of data of the same scene originating from different sources.It aims at obtaining information of greater quality; the exact definition of greater quality will depend upon the application" (Wald, 1998).Image fusion is another form of data fusion where the data is in the form of images.Image fusion is defined as the "combination of two or more different images (of the same scene) to form a new image by using a certain algorithm" (Pohl and Genderen, 1998).Image fusion can be done for various reasons.One such reason can be to increase the spatial resolution of low resolution multispectral using high resolution panchromatic and sharpen the images.The fused image results in a single image and is expected to be of good quality in terms of spatial and spectral resolution.A new image fusion methodology for satellite images is proposed in this paper.

Related work
There exist many image fusion techniques in the literature.Let us start with some of the state of art image fusion methods.The most commonly used methods involve the intensity-huesaturation (IHS) (Chavez and Bowell,1998), (Edwards and Davls,1994) and (Tu et al., 2004) , the Brovey method (Gillspie et al., 1987) and the principal component analysis (PCA) (Chavez et al., 1991).Though these component substitution methods provide high spatial quality they also result in spectral distortion.As an advancement to these traditional methods (Hong et al, 2009) propose an image fusion method applied to SAR and multispectral image, integrated IHS method with the wavelet transform to retain the spatial detail.Also, (Zhang and Hong, 2005) combine IHS and wavelet transform to reduce the colour distortion of the fusion results.(Ourabia and Smara, 2016) combines PCA and nonsubsampled contourlet transform to overcome the drawback of PCA for spectral distortion.(Andreja and Kristof, 2006) combine HIS, Brovey methods with multiplicative (MULTI) method to preserve spectral and spatial resolution.(Demirel and Anbarjafari, 2010) proposed a fusion method based on dual-tree complex wavelet transform (DT-CWT) and showed that their fusion method was superior compared to the conventional methods.(Nikolakopoulos, 2008) made a comparison amongst nine image fusion methods including those of the sate of art methods and few other popular techniques.Based on the results, the visual quality was examined and correlation between the original multispectral and the fused images was also performed.
Further, there exist some variational models developed by a few researchers.(Ballester et al., 2006) was the first to build a variational method for fusion of panchromatic and multispectral data called as "P+XS".(Fang et al., 2013) also proposed a variational model for fusion of panchromatic and multispectral images based on certain assumptions.(Moller et al., 2012) proposed a model called as "variational wavelet for pansharpening method (VWP)".(Ning et al., 2013) presented a variational model for panchromatic and multispectral image fusion where an advanced minimisation problem is proposed based on Socolinsky's contrast model.(A.Vesteinsson et al.,2005) formulated a new fusion technique called "spectrally consistent pansharpening" which focusses on preserving the spectral consistency between the original multispectral and the fused multispectral data.Moreover, image fusion has been applied to different types of data.It can be applied to data provided by a single source or multisource.Satellite sensors such as SPOT, IKONOS, QUIKBIRD, WORLDVIEW etc. produce images of different resolutions.The single channel panchromatic usually comes with a higher resolution than the multispectral channels.Therefore, fusion of these two images is performed resulting in a high resolution multispectral data.Such fused images are widely used for various applications such as urban planning, agricultural monitoring, forestry.and also, for military purposes.(Zhang and Hong, 2005) worked on fusion methods using the IKONOS and QUIKBIRD (Ning et al., 2013) applied their fusion method to IKONOS and QUICKBIRD dataset.(Helmy and El-Tawel, 2015) reported working on IKONOS, QUICKBIRD and GEOEYE satellite images.(Svab and Ostir, 2006) reported their fusion method based on data produced by IKONOS, QUICKBIRD, LANDSAT.There are also fusion techniques applied to multisource data.(Ourabia and Smara, 2016) reported a fusion method using data provided by the SPOT and ALSTAR-2A sensors.(Chavez and Bowell, 1998) performed their IHS methods on SPOT and LANDSAT TM datasets.
In spite of the above developments, we believe that there is still room for further development in this area.For example, most of the work reported so far have dealt with developing model based on the sensor physics.But during the fusion process maintaining spectral consistency poses a great challenge and is addressed in most of the literature.The problem in many cases, reduces to solving a set of equations involving a number of unknowns, which can be more than the number of equations, in some optimal way subject to spectral consistency.The problem is termed 'pansharpening' in the literature.Our primary objective is to provide a solution to this ill posed problem.Unlike most of the models which depend on the sensor imaging physics and the model, our proposed method deals with the reflectance data (pixel values) of the satellite image.We believe that our data centric approach is more comprehensive than the sensor based approach in the sense that it not only includes the data generation characteristic of the sensor but also includes the reflectance characteristic of the object being imaged.For our work we have considered the data provided by the IKONOS satellite sensor which are available free of cost and downloadable from earth explorer 1 .
Our approach consists of developing a linear regression relationship between the panchromatic and the multispectral images.During the image fusion process, spectral distortion is likely to happen.Considering this fact, we formulate a convex optimization problem in terms of Lagrange's multiplier.Our objective function consists of minimising the deviation error and minimising the error in variance between the actual data and the downsampled fused data.We derive two strategies based on our objective function.Results are presented for both the cases using the IKONOS2 satellite imagery.Lastly, the Chi-square rest is performed to check the goodness of the fit.
The flow of the paper will be as follows.Section 2 provides a brief concept on the proposed approach where a linear relationship is assumed to exist between the panchromatic and the multispectral data.We also formulate in section 2 a minimisation function in terms of Lagrange for two different strategies.Simulation results are presented and discussed in section 3. Section 4 provides a brief summary and conclusions.

Regression model
In this section we provide a regression model relating the panchromatic and multispectral band data based on the available data.We downsample the panchromatic (P) data to bring it to the same resolution as multispectral channels.We will consider only the first three bands viz.red (R), green (G) and blue (B) channels.For better stability of computation, deviation variables are used.R, G, B deviation with respect to respective sample average is calculated.The deviation is calculated for P with respect to its sample average.A linear regression relationship between the pan and the XS channel is formulated as where R, G, B and P are the deviation from their respective sample average.a, b and c are the regression coefficients.The regression coefficients are computed using maximum likelihood solution as follows.
Considering a range of data for R, G, B and P and forming respective column vectors, we formulate where the matrix inverse is assumed to exist.

Lagrange objective function
Considering a N × N points panchromatic sample data, we have where the symbols jk, Gjk and Bjk represent deviation of spectral channel data with respect to low resolution counterparts in location (j,k ).We express the objective function to minimize spectral inconsistency and tracking error in the variance of the spectral channel data for an assumed normal distribution as min considering the red band.
Rewriting using Lagrange multiplier, we have min[ Two cases are considered in the following sections.A brief proof for the two cases are provided in the appendices.
In this case, the projected high resolution multispectral for the red channel as a solution to Equation (3) can be obtained as Case 2: When μ r = 0 The projected high resolution multispectral for the red channel as a solution of Equation ( 7) can be obtained as Remarks: The solution to Equation ( 8) is circuitous.That is, the error ratios (er/eb) and (er/eg) on the RHS of Equation ( 8) depend on the solution Rjk , Gjk and Bjk given by the LHS of Equations ( 8)-( 10).It can be shown that a solution to the circuitous problem is obtained when the ratios and so forth for the remaining ratios.

SIMULATION RESULTS
We illustrate our proposed approach on two different datasets; (i) comprising of 16 ×16 pixels panchromatic and corresponding 4 × 4 multispectral and (ii) 32 × 32 pixels panchromatic and 8 × 8 multispectral datasets.The coding platform and illustration of simulation results is performed using MATLAB (2014a) software.

Illustration on first dataset
The high resolution panchromatic 16 × 16 pixels data is shown in figure 1 and corresponding 4 × 4 multispectral data is shown in figure 2. The objective is to extrapolate multispectral data in the same resolution with the panchromatic while still maintaining the spectral consistency of the multispectral channels.The regression coefficients for this particular dataset is found to be [a b c] t = [-1.2960.5519 -0.989] t .

Results based on Case 1:
Based on equations ( 4), ( 5) and ( 6) presented in Case 1, the projected high resolution multispectral for red, green and blue channels are shown in figure 3a, 3b and 3c respectively.

Results based on Case 2:
The high resolution multispectral based on the second case is computed using Equations ( 8) -( 10) for red, green and blue channels respectively.Table 1 shows the comparison of variances between the original multispectral data and the downsampled multispectral data for the two cases.In table 2, we present the results of the chi-square test, which is performed to check the goodness of of the data between the observed data and the computed data.Chi-square test can be performed by using the following equation where Oi is the observed data and Ei is the expected data over range of sample data.
In our case, the data given in figure 2 is referred to as the observed data and the data depicted in figure 4 and figure 6 are assumed as the expected data and accordingly the chi-square test is performed on these data.

Illustration using the second dataset
Now let us proceed with the other dataset of 32 × 32 pixels panchromatic and 8 × 8 multispectral datasets.Figure 7 shows the 32 × 32 pixels panchromatic and figure 8 shows the 8 × 8 multispectral datasets.The regression coefficients for this particular dataset is found to be

Results based on case 1
Based on the first strategy, the high resolution 32 × 32 pixels multispectral for the red, green and blue channels are shown in figure 9a, 9b and 9c respectively.For the second dataset of 32 × 32 pixels panchromatic and 8 × 8 multispectral datasets, Table 3 shows the comparison of variance between the actual and the computed downsampled multispectral data for each channel.Table 4 presents the chi-square goodness of fit test using equation ( 10).

Discussion of Results:
In this paper, we have worked on two datasets:(i) 16 × 16 pixels panchromatic and 4 × 4pixels multispectral datasets, and (ii) 32 × 32 pixels panchromatic and 8 × 8 pixels multispectral.Illustration of our data dependent approach for pansharpening is shown using these two datasets.For the first dataset, Table 1 shows the variance of the original multispectral channels and the computed variance of the projected high resolution multispectral data after downsampling.Overall, for all the three bands Case 1 seems to give a small difference in variance between the original and computed multispectral channels.In Case 2, though the difference in variance is small for red channel, the green and blue channels give variances 0.4 and 6 respectively against the original respective variances of 3 and 1.Table 2 shows the results of the chi-square test performed on the first dataset.For both the cases the critical values (CV) are small and accordingly their respective p values are quite small as well.Therefore, we conclude that the fit is satisfactory.
For the second dataset, comparison of variances between the original and computed multispectral data is shown in Table 3.As observed form the table except for the green channels, the red and blue channels variances have a closer value to that of the original value.The variance value for the green channel gives almost the same value in both the cases.However, for the other bands viz.red and blue channel the second case provides a smaller difference in the variances.For the second dataset, we conclude that the fit is also found to be satisfactory (considering an acceptance p value upto 0.5).

CONCLUSION
In this paper, we emphasize the need of data fusion in the geospatial technologies particularly focussing on the remote sensing satellite images.In this context, we have developed an image fusion model which is based on the pixel values using a data centric approach.We believe our approach is more comprehensive compared to the existing sensor based approach since both the effects of sensor and the reflectance characteristics of the imaged object are included in our approach.We formulated a minimization function and discussed two separate scenarios for the fusion process; one to maximise spectral consistency and the other to track the variance of the supplied data.We illustrated our proposed approach using two different datasets of different sizes.
We conclude that our model is simple and easy to be implemented.Statistical test on the goodness of fit is satisfactory.Though variance tracking approach (Case 2) does not seem to produce the expected results, one other case could be considered.
The third case could be the convex combination of the two cases as in (3) and minimizing in terms of the parameter µ r for the red channel and similarly for the other channels.This constitutes our future work.
above expression also holds for the other two colour channels (green and blue) as follows

Figures
Figures 5a, 5b and 5c show the projected high multispectral data for red, green and blue channels respectively.

Table 1 .
Comparison of variances between the data for the first datasetMultispectral bands

Table 2 .
Chi-square test for the first dataset

Table 3 .
Comparison of variance between the data for the second