Improved reflectance reconstruction for multispectral imaging by combining different techniques

In multispectral imaging system, one of the most important tasks is to accurately reconstruct the spectral reflectance from system responses. We propose such a new method by combing three most frequently used techniques, i.e., wiener estimation, pseudo-inverse, and finite-dimensional modeling. The weightings of these techniques are calculated by minimizing the combined standard deviation of both spectral errors and colorimetric errors. Experimental results show that, in terms of color difference error, the performance of the proposed method is better than those of the three techniques. It is found that the simple averaging of the reflectance estimates of these three techniques can also yield good color accuracy. ©2007 Optical Society of America OCIS codes: (330.1710) Color measurement; (330.1730) Colorimetry; (110.4190) Multiple imaging. References and Links 1. J. Y. Hardeberg, “Acquisition and reproduction of color images: colorimetric and multispectral approaches,” Ph.D. dissertation (Ecole Nationale Superieure des Telecommunications, 1999). 2. H. L. Shen and J. H. Xin, “Spectral characterization of a color scanner by adaptive estimation,” J. Opt. Soc. Am. A 21, 1125-1130 (2004). 3. H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N.Tsumura, and Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. 39, 6621–6632 (2000). 4. V. Cheung, S. Westland, C. Li, J. Hardeberg, and D. Connah, “Characterization of trichromatic color cameras by using a new multispectral imaging technique,” J. Opt. Soc. Am. A 22, 1231-1240 (2005). 5. M. Shi and G. Healey, “Using reflectance models for color scanner calibration,” J. Opt. Soc. Am. A 19, 645656 (2002). 6. F. H. Imai and R. S. Berns, “Spectral estimation using trichromaitc digital cameras,” in Proceedings of the International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives (Society of Multispectral Imaging of Japan, Chiba, Japan, 1999). 7. M. Soriano, W. Oblefias, and C. Saloma, “Fluorescence spectrum estimation using multiple color images and minimum negativity constraint,” Opt. Express, 16, 1458-1464 (2002). 8. V. Cardei and B. Funt, “Committee-based color constancy,” in Proc. IS&T/SID Seventh Color Imaging Conf.: Color Science, Systems, and Applications (Society for Imaging Science and Technology, Virginia, 1999), pp.311-313. 9. H. Stokman and T. Gevers, “Selection and fusion of color models for image feature detection,” IEEE Trans. Pattern Anal. Mach. Intell. 29, 371-381 (2007). 10. K. Barnard and B. Funt, “Camera characterization for color research,” Color Res. Appl. 27, 152-163 (2002). 11. W. K. Pratt, Digital Image Processing, 2nd ed. (Wiley, New York, 1991). 12. L. Maloney, “Evaluation of linear models of surface spectral reflectance with small numbers of parameters,” J. Opt. Soc. Am. A 3, 1673-1683 (1986). 13. R. McDonald and K. J. Smith, “CIE94 a new colour difference formaula,” J. Soc. Dyers Colour. 111, 376379 (1995).


Introduction
Nowadays multispectral imaging techniques have been intensively studied for the applications of digital recording, archiving, and display.One of the most important tasks of multispectral imaging is to accurately recover the spectral reflectances of the object surfaces from system responses [1][2][3][4][5][6][7].In the literature, three spectral characterization techniques, i.e., wiener estimation [2,3], pseudo-inverse [1], and finite-dimensional modeling [4][5][6][7] have been widely adopted for reflectance estimation.In the wiener estimation and finite-dimensional modeling, the spectral sensitivity or responsivity needs to be instrumentally measured or mathematically recovered.The pseudo-inverse technique computes the transform between system responses and reflectance without any a priori knowledge about the imaging system.
To meet the rigorous requirement in industrial applications such as color measurement and color quality control, researchers are still working for new spectral characterization techniques.This study tries to investigate whether the accuracy of spectral reconstruction can be improved by combining the currently available techniques.In the color constancy area, Cardei and Funt found that better illumination estimation could be obtained by combing the results of different color constancy algorithms [8].Based on statistical analysis, Stokman and Gevers investigated the optimal strategy of color model selection and fusion to improve the discriminative power of image feature detection algorithms [9].This study is mainly inspired by these two works.

Formulation of multispectral imaging process
When the response of the multispectral imaging system is proportional to the intensity of light entering the camera, the response vector v of L channels can be formulated as [2]: where the subscript N=31 denotes the number of samples in the visible wavelength range, r denotes the spectral reflectance, M represents the spectral responsivity incorporating the spectral power distribution of lighting source, the spectral transmittances of narrowband filters, and the spectral sensitivity of digital camera.b denotes the bias response vector caused by camera dark current, and n denotes zero-mean imaging noise.It is noted that if the system does not behave linearly, the optoelectronic conversion function needs to be further considered [2,4].
In spectral characterization of imaging system, it is feasible to mathematically recover the spectral responsivity M and bias b from training color samples, subject to the constraint of non-negativeness [10,2].It is noted that, as the filters used in this study are narrow-banded, the condition of smoothness is not needed.

Wiener estimation
In general, the purpose of spectral characterization is to estimate the reflectance vector r ˆ from the response vector u=v-b through an N×L matrix W, such that (2) In wiener estimation [2,3,11], W is calculated by using the spectral responsivity M: where superscript T denotes transpose, K r denotes the N×N covariance matrix of r, and K n denotes the covariance matrix of noise.In this study, K n =0 is assumed.

Pseudo-inverse
In the pseudo-inverse technique [1], the transform matrix W is directly solved as ) where the superscript + denotes pseudo-inverse, R denotes the matrix of reflectance vector r, and U denotes the matrix of response vector u.

Finite-dimensional modeling
Due to the smooth property of spectral reflectance, r can always be represented by the linear combination of J (<N) basis functions b j [12]: where a j is the coefficient of b j .b j can be calculated using principle component analysis of reflectance data.By combining Eq. ( 5) and ( 1), the response u can then be represented as [4][5][6][7]: Thus coefficient a j can be estimated by pseudo-inverse of the L×J matrix [Mb j ].The reflectance r ˆ can then be obtained by substituting the estimated a j into Eq.( 5).In this study, J=10 basis functions are used, as they are generally adequate for spectral reflectance construction [12,5].

Statistical properties of different observations
Suppose a quantity is measured using K (K=3 in this study) different techniques, and x k is the observation of the kth technique.To obtain an improved estimation of that quantity, it is feasible to combine these observations as the following: where w k is the weighting of the observation of the kth technique.The estimate and standard deviation of x can be expressed according to Eq. ( 8) and ( 9), respectively.
where E(x k ) is the averaged observation of the kth technique, and σ( x k , x l ) is the covariance between observations of the kth and lth techniques.

Combination of different spectral characterization techniques
Based on the statistical properties discussed above, a new method for reflectance estimation by combining different techniques is proposed in the following.Let k r ˆ be the estimated reflectance of the kth technique, the spectral root mean square (rms) error can then be calculated as and the combined rms error can be expressed as Similarly, the CIE 1994 color difference [13] error is represented as ) and the combined color difference error then becomes It is noted that the spectral error δr is linear to reflectance r, while the colorimetric error δe is nonlinear to r, due to the third-root-square transform between CIEXYZ space and CIELAB space.To account for both spectral and colorimetric accuracy in reflectance reconstruction, these two error terms can be merged into a single error term δ.
Consequently, the estimate of δ can then be expressed as where 0≤c≤1 is used to adjust the proportions of spectral and colorimetric errors, E r and E e denote the K×1 vectors of the average spectral and colorimetric errors, respectively, and w denotes the K×1 weighting vector.Accordingly, the standard deviation of δ can be expressed where Σ r and Σ e denote the K×K covariance matrix of the spectral and colorimetric error, respectively.Considering that the spectral and colorimetric errors are always of different magnitudes, E r , E e , Σ r , and Σ e are normalized in the calculation of weighting w.
The appropriate weighting w can be obtained by solving the following objective function: minimize or, alternatively, minimize (17) Furthermore, the following two constraints should be imposed: The first constraint ensures that the weightings of the K techniques sum to 1, while the second constraint forces positive contribution of each technique.
The objective functions ( 16) and ( 17), together with the constraints (18) and (19), can be solved using quadric programming.When the weighting is obtained, the reflectance of the proposed method can be calculated as It is noted that the weightings obtained from the objective functions are suboptimal, due to the nonlinear transform between reflectance and color difference error.Nevertheless, it will be shown in the experiment section that this strategy can produce considerable color accuracy improvement, when compared with the three techniques.

Experimental results and discussion
In this study, a QImaging monochrome digital camera model Retiga-EXi, as well as a liquid crystal tunable filter (LCTF) made by Cambridge Research and Instrument Co., were used to acquire multispectral images.The LCTF contains a series of electronically-tunable narrowband filters, with the nominal bandwidth of 10nm Full-Width at Half-Maximum (FWHM).The color target used was GretagMacBeth ColorChecker DC (CDC), and its reflectance data were measured using a GretagMacBeth spectrophotometer 7000A in the visible spectrum of 400 to 700 nm, with sampling interval of 10 nm.Totally L=16 multispectral images of CDC were taken using the filters with center wavelengths at 400 nm, 420 nm, 440 nm, …, 700nm, under an approximate D65 lighting condition.The spatial non-uniformity of the lighting field was corrected by using a white paper [4].In total, 198 color patches on CDC were used for reflectance estimation, excluding the duplicated most dark ones and the glossy ones.These color patches were divided into one training set and one testing set, with each set containing 99 colors.The training set contains the color patches with odd sequence numbers, while the testing set contains the color patches with even sequence numbers.The magnitudes of spectral error and colorimetric error obtained through the objective function are related to the c values, as shown in Fig 1 .These two kinds of errors do not change monotonically with c.However, as a whole trend, increasing the c value will produce larger color difference error, while produce smaller spectral rms error.The magnitude of color difference error keeps in the range of 0.850 and 0.855 when c increases from 0.0 to 0.6, which indicates that the color accuracy of the proposed method is not very sensitive to c value.Therefore, we simply let c = 0.2 in the following calculation.For comparison, the spectral and colorimetric error statistics including mean, standard deviation, and maximum of 6 different methods (wiener estimation, pseudo-inverse, finitedimensional modeling, simple averaging, and the proposed methods with two different objective functions) are given in Table 1.In the averaging method, w 1 =w 2 =w 3 =1/3.Nonparametric statistical test indicates that the color difference errors of the proposed methods and the averaging method are smaller than those of the other three methods at a significant level p=0.05.The performance of the averaging method is close to that of the proposed method 1 using objective function (16).The advantage of the proposed method 1 over the averaging method lies that the former can balance the colorimetric accuracy and spectral accuracy by adjusting the c value.The reconstructed spectral reflectance curves with minimum and maximum color difference errors are shown in Fig. 2. The proposed method 2 using objective function (17) does not yield better performance, indicating that it may be not feasible to minimize both average color error and standard deviation.Actually, minimizing standard deviation will also decrease the average color error to a certain extent.

Conclusion
A new reflectance reconstruction method for multispectral imaging has been proposed, by combining three different spectral characterization techniques.The weightings of the estimated reflectances of these techniques are calculated based on the minimization of spectral and colorimetric errors.In terms of color difference error, the performance of the proposed method is better than those of the three techniques, while is close to that of the simple averaging method.The proposed method is applicable in industrial applications such as textile color measurement and color quality control.

Fig. 1 .
Fig. 1.Distribution of colorimetric errors ΔE 94 * under D65 and spectral errors obtained using the objective function (16) with respect to different c values.The green square-symbol curve represents colorimetric errors, and the blue circle-symbol curve represents spectral rms errors.

Table 1 .
Spectral and colorimetric error statistics of 6 methods when c=0.2: wiener estimation, pseudo-inverse method, finite-dimensional modeling method, simple averaging method, and the proposed methods.