Compressive recovery of smartphone RGB spectral sensitivity functions

: Spectral response (or sensitivity) functions of a three-color image sensor (or trichromatic camera) allow a mapping from spectral stimuli to RGB color values. Like biological photosensors, digital RGB spectral responses are device dependent and significantly vary from model to model. Thus, the information on the RGB spectral response functions of a specific device is vital in a variety of computer vision as well as mobile health (mHealth) applications. Theoretically, spectral response functions can directly be measured with sophisticated calibration equipment in a specialized laboratory setting, which is not easily accessible for most application developers. As a result, several mathematical methods have been proposed relying on standard color references. Typical optimization frameworks with constraints are often complicated, requiring a large number of colors. We report a compressive sensing framework in the frequency domain for accurately predicting RGB spectral response functions only with several primary colors. Using a scientific camera, we first validate the estimation method with direct spectral sensitivity measurements and ensure that the root mean square errors between the ground truth and recovered RGB spectral response functions are negligible. We further recover the RGB spectral response functions of smartphones and validate with an expanded color checker reference. We expect that this simple yet reliable estimation method of RGB spectral sensitivity can easily be applied for color calibration and standardization in machine vision, hyperspectral filters, and mHealth applications that capitalize on the built-in cameras of smartphones.

The knowledge of smartphone camera's spectral response functions becomes more important as numerous mobile health (mHealth) technologies rely on the built-in cameras for pointof-care diagnostics and individual health monitoring, including dermatology, ophthalmology, cardiology, and quantification of bioassays [33][34][35][36][37]. Obviously, device dependent RGB spectral response functions introduce significant errors, requiring different types of color calibration and standardization. For example, quantification of paper microfluidic assays [38], mHealth heart rate monitoring [39], quantitative urinalysis [40], and mHealth assessments of blood hemoglobin levels [41,42] need a color calibration to achieve consistent responses across different smartphone models. If RGB spectral response functions are readily accessible and estimated in a simple setting, color canonicalization in a variety of smartphones can be simplified and standardized. Unfortunately, the information on spectral response functions is often not shared by smartphone manufacturers nowadays (Table 1). In general, direct measurements of RGB spectral response functions are not an easy task. RGB spectral response functions can be recovered using directly measured narrowband stimuli in the broad range of visible light. The standard method is to record the spectral response against monochromatic light produced by a monochromator or laser light source. Specifically, the previously developed direct measurement methods use narrowband stimuli covering the entire visible spectral range, including monochromators [43,[50][51][52][53][54][55], LEDs [56], or quantum efficiency measurements [12,57]. Importantly, such direct measurements require a priori information on the spectral profile of chosen light sources and the spectral response functions of calibration equipment, all of which are often challenging to quantify in a similar manner of measuring the spectral response functions of a specific device of interest, forming a vicious cycle of uncertainty. In other words, the spectral sensitivity of reference detectors and the spectral profiles of calibration light sources should be predetermined.
Alternative approaches for estimating RGB spectral response functions are statistical and machine learning methods based on a relationship between the RGB values and reflection spectra of color samples. Typically, commercially available color reference charts and cards are used. In brief, Table 1 compares representative mathematical methods of estimating RGB spectral response functions using color references. Additional summary is also available elsewhere [16,17,48,58,59]. In particular, several optimization frameworks combined with constraints on RGB spectral response functions have been successfully introduced, including quadratic programming [45], regularization [43,46], principal component analysis (or subspace method) [8], convex optimization [44,60], Wiener [43], and rank-based spectral estimation [48]. Some advanced methods, such as rank-based spectral estimation, have also shown that rendered images (e.g., JPEG) can be used for reliably predicting RGB spectral response functions. Recently, machine learning approaches have been applied based on the assumption of Gaussian functions for RGB spectral response functions [13,16].
Except for the machine learning methods, the previous mathematical methods can be recapitulated as l 2 -norm minimization in a broad manner. In general, l p -norm is expressed with an n dimension vector [61]: When p = 2, ||•|| 2 is l 2 -norm. Indeed, l 2 -norm minimization is formulated as least squares regression. Least squares regression, which supports stable computation and minimizes overfitting, is a widely used computational scheme for parameter estimation. On the other hand, such methods require a relatively large number of colors and depend on the selection of colors. As a result, direct applications of RGB spectral response functions for color calibration and standardization are rarely available.
An alternative mathematical method is to take advantage of l 1 -norm minimization in the frequency domain [62]. When p = 1, ||•|| 1 is l 1 -norm. When p = 0, ||•|| 0 is l 0 -norm. l 0 -norm means the sum of absolute values of the elements or the number of the nonzero elements. Theoretically, l 0 -norm is the limit as p → 0 of l p -norm. l 0 -norm minimization problems can be treated to the corresponding l 1 -norm minimization problem [63,64]. As directly related to compressive (or compressed) sensing, l 1 -norm minimization has received considerable attention [62,[65][66][67][68]. Specifically, when significant information is contained in a few elements (i.e., sparsity or compressibility), l 1 -norm minimization has shown reliable and enhanced performance. The recovery of the whole information is possible with a few measurements in a variety of imaging applications. Indeed, compressive sensing allows for reconstructing signals at a rate far below the Nyquist/Shannon sampling theorem. Thus, compressive sensing in the frequency domain is an alternative candidate to recover RGB spectral response functions.
In this paper, we report a simple compressive sensing framework for estimating RGB spectral response functions of a three-color image sensor (or trichromatic camera) on the basis of a small number of primary colors. To solve an ill-posed problem, we use compressive sampling in the frequency domain to overcome the limitation of low dimensionality in conventional color reference standards, resulting in a reliable recovery of RGB spectral response functions. First, we validate this method with respect to the ground truth RGB spectral response functions of a scientific three-color camera. In this validation, the ground truth of the RGB camera is directly assessed with a corresponding mono camera and sunlight measurements. Second, as numerous mHealth applications rely on the built-in cameras, we test several smartphone cameras and validate the estimated spectral response functions using additional diverse feature colors from an expanded color reference checker. The novelty of this reported method includes the use of only 12 primary colors and the simple, yet robust, inverse calculation, all of which can simply be conducted without sophisticated and expensive calibration equipment.

Definition of RGB spectral response functions
The measured RGB intensity values from an object of interest acquired by a three-color camera can be expressed with the RGB spectral response functions S R (λ), S G (λ), and S B (λ) in the R, G, and B channels, respectively: where λ is the wavelength of light, L(λ) is the spectral shape of the illumination light source, C(λ) is the spectral response of all optical components in the imaging system, D(λ) is the spectral response of the (mono) image sensor, and O(λ) is the spectral intensity reflected from the object.
To compensate for L(λ), C(λ), and D(λ), the RGB intensity from a white reflectance standard that has a reflectivity of 99% in the visible range (i.e., O(λ) = 0.99) is measured: The RGB values of the object solely determined by S R,G,B (λ) is calculated by normalizing The true RGB values of the object is further obtained by subtracting dark noise from the measurements: where I dark is the measurement with the light source off in the dark. In other words, the true RGB values of the object are not affected by the illumination light source, the optical components, and the ambient stray light. The true RGB values are determined only by the RGB spectral response functions (i.e., S R,G,B (λ)) of the corresponding image sensor. The relationship between the true RGB values and spectral intensity of n multiple objects (n different spectra) is described when λ is discretized in the visible range (λ = [λ 1 , λ 2 , . . . , λ m ]): where Y corresponds to the true RGB values, S is a m × 3 matrix of the RGB spectral response functions, and F is a n × m matrix that consists of the spectral intensity of n different objects (i.e., different color patches in the color checker). Equation (6) can be expressed explicitly: Theoretically, some direct laboratory measurement methods can be understood such that if a set of light sources with a narrow bandwidth (e.g., laser light sources) at all discretized λ are used, F is the identity matrix with n = m, returning the recovery of S.

Spectral compressive sensing framework
We propose that a compressive sensing framework can be ideal for solving S. As shown in all of the previous studies [8,13,16,[43][44][45][46][47][48], the recovery of the RGB spectral response functions is not straightforward, given this inverse calculation is an underdetermined problem. Simply, solving S can be considered an ill-posed problem. In a compressive sensing framework, S is an unknown matrix to solve, F is a measurement matrix, and Y is an observation matrix. Compressive sensing intends to solve this underdetermined problem such that S is represented by a certain orthonormal basis Ψ that transforms S into sparser signals s: One of the simplest forms for Ψ to enhance sparsity is to use a linear combination of Gaussian basis functions and the FWHM of the basis Gaussian function is 50 nm in our case [69].
Importantly, the performance of compressive sensing recovery is determined by the properties of the sensing matrix Φ, which requires an incoherent (or uncorrelated) condition (also known as the uncertainty principle in compressive sensing) [62,[65][66][67][68].
We recover the RGB spectral response functions S by using l 1 minimization that finds a minimum l 1 -norm solution of the underdetermined linear system (Eq. (8)): where ε is a noise level in the observations. Because l 0 -norm is non-convex, it is computationally impossible to solve (also known to be NP-hard). Thus, an l 0 -norm minimization problem is treated to the corresponding l 1 -norm minimization problem. In this case, the majority of unnecessary components (weights) are forced to be zeros, yielding only a few non-zero components and avoiding overfitting. l 1 -minimization is completely different from l 2 -norm minimization, in which all of the data points are used to minimize the sum of squared residuals. For practicality, we solve Eq. (8) as an unconstrained basis pursuit denoising problem by employing a weighted value γ: which is also known as l 1 regularization or Lasso regression. It should also be noted that this regularization is different from conventional regularization with l 2 -norm minimization (i.e., least squares method). Specifically, among other commonly available l 1 -norm solvers, we use CVX that is a MATLAB-based open source based on a convex optimization algorithm [70].
Owing to the characteristics of compressive sensing, we focus on using a small number of colors for constructing the sensing matrix Φ that can satisfy the incoherence condition. Specifically, we take advantage of a commercially available color reference standard, ColorChecker Classic Nano (or Macbeth ColorChecker), to estimate RGB spectral response functions ( Fig. 1(a)). This color reference standard consists of 24 patches of colors that have chromatic importance for general photography and cinematography. We exclude featureless or monotonous colors, such as white, gray, and black, because these achromatic colors have flattened spectral profiles. As a result, 18 signature color patches of distinguishable colors and spectra serve as color bases for the proposed method ( Fig. 1(a) and (b)). We further reduce the number of color patches that can still result in reliable performance. It should be noted that the size of ColorChecker Classic Nano is small (22 mm × 36 mm) so that spatially uniform illumination and single shot imaging can easily be achieved.

Evaluation with ground truth RGB spectral response functions
We first evaluate the proposed method of estimating RGB spectral response functions against the ground truth values of a scientific machine vision camera. For this validation purpose, we use a scientific machine vision RGB camera (GS3-U3-120S6C-C, Point Grey and Fig. 2(a) installed with an image sensor (Sony ICX625). We directly assess the RGB spectral response functions in this trichromatic camera; the ground truth of the RGB spectral response functions can be obtained with a mono version (GS3-U3-120S6M-C, Point Grey and Fig. 2(a)) installed with the same image sensor. The only difference for the mono camera is the absence of the color filter array on the image sensor. In an alternating manner, each camera is mounted to the same imaging spectrograph with a diffraction grating (groove density of 150 g/mm). In other words, each camera serves as a detector for the imaging spectrograph. As an ideal broadband light source, sunlight is incident on the spectrograph. During the measurement, the azimuth and elevation of the sun are 184.26°and 27.1°at the location coordinates of 40.42242°, −86.92039° (Fig. 2(b)). Sunlight contains abundant patterns of natural elements that can also be used for the wavelength calibration of the imaging spectrograph ( Fig. 2(c)). In this case, the RGB spectral response functions S R (λ), S G (λ), and S B (λ) in the R, G, B channels can be calculated: where I R,G,B (λ) is the measured spectral intensity of R, G, and B channels of the RGB camera, I mono (λ) is the measured spectral intensity with the mono camera. As I R,G,B (λ) and I mono (λ) can be measured sequentially with a time interval of 60 seconds under the clear sky, changes in atmospheric transmission can be ignored. Figure 1(e) shows the measured RGB spectral response functions of the scientific machine vision RGB camera. It should be noted that this method of directly assessing the RGB spectral response functions is not generally possible for common smartphone cameras because a mono camera without a color filter array is not available.
Using the compressive sensing method, we then estimate the RGB spectral response functions of the trichromatic camera. In a laboratory setting, we use a Xenon-arc lamp that has a broad spectrum in the full visible range, similar to sunlight (Fig. 2(d)). The illumination from the Xenon-arc lamp is collimated and is incident onto the color checker ( Fig. 1(a)) via an obliquely positioned mirror. The light reflected from the color checker is imaged with the RGB camera via a lens (focal length of 25 mm and F-number of F/1.4, Navitar). A white reflectance standard (AS-01160-060, Labsphere) that has a reflectivity of 99% is used to obtain I reference (R, G, B) as described in Eq. (3). Conventional white LED light does not cover the entire visible range due to the low intensity at short (400-450 nm) and long (650-700 nm) wavelength ranges (Fig. 2(d)). The recovered RGB spectral response functions are discretized from 380 nm to 720 nm with a spectral interval of 0.5 nm.
We further explore if the number of colors can be reduced. Figure 3(a) shows the root mean square error (RMSE) between ground truth and recovered RGB spectral response functions as a function of the number of colors selected among 18 color patches. When a small number of colors is selected, possible combinations of selecting colors are extremely large. For example, when 9 colors are chosen among 18 colors, the number of combinations is 18 C 9 = 48,620. In such cases, we randomly select 100 different possible combinations and obtain an averaged RMSE. As expected, RMSE decreases with the number of colors selected (Fig. 3(a)). Figure 3(b)-3(e)) show four representative cases when the number of colors is 9, 12, 15, and 18, respectively. When the  Fig. 2(e)). (f) The reflection spectra of 12 primary colors (#1 -#12 in Fig. 1) serve as a sensing matrix in the compressive sensing framework. The pairwise spectral comparison map shows an average correlation coefficient of 0.579. This level of spectral uncorrelation is sufficient to guarantee the incoherence requirement for compressive sensing theory. number of colors is greater than 12, the RMSE value converges into the minimum value ( Fig. 3(a)). As a result, we use only 12 primary colors (#1 -#12 in Fig. 1). In this case, the RMSE values of the R, G, and B channels are 0.0285, 0.0220, and 0.0337, respectively (Fig. 3(c)). In pairwise cross-spectral analyses (Fig. 3(f)), the off-diagonal areas represent the correlation coefficients compared with the reflection spectra of other colors. The corresponding 12 different reflection spectra of primary colors are relatively uncorrelated with an average correlation coefficient of 0.579. In other words, only 12 color readings enable the reliable recovery of the RGB spectral response functions with absolute values in our compressive sensing-based method. This result also supports the idea that the set of selected 12 primary colors serves as a spectrally incoherent sensing matrix.
In addition, we compare the RGB spectral response functions using typical methods that are based on l 2 -norm minimization (i.e., least squares regression). Specifically, we use QR decomposition and the Moore-Penrose pseudo-inverse [71] both with a constraint of nonnegative values with the 12 primary colors. Figure 4 shows that the RGB spectral response functions estimated by these conventional methods are not in excellent agreement with the ground truth values, given that a small number of primary colors are used. The error from l 1 -norm minimization (i.e., compressive sensing) is significantly smaller than those from the l 2 -norm minimization methods; the mean RMSE values of the R, G, and B channels are 0.3557, 0.1511, and 0.0281 for QR decomposition, the Moore-Penrose pseudo-inverse, and compressive sensing, respectively ( Fig. 4(c)). Indeed, because l 2 -norm normalization is ideal for an overdetermined problem, the conventional methods often rely on a significantly large number of colors. The RGB spectral response functions of the RGB camera (GS3-U3-120S6C-C, Point Grey) are estimated with 12 primary colors (# 1 -# 12 in Fig. 1) using QR decomposition with a constraint of nonnegative values (a) and the Moore-Penrose pseudo-inverse with a constraint of nonnegative values (b). The estimated RGB spectral response functions (solid lines) are compared with the ground truth RGB spectral response functions (dotted lines and Fig. 2(e)). (c) Mean RMSE values of the R, G, and B channels between the ground truth (dotted lines and Fig. 2(e)) and the recovered RGB spectral response functions (solid lines). The RMSE of compressive sensing is the same as Fig. 3(c).

Testing with multiple smartphones
We further estimate the RGB spectral response functions of several Android smartphone models, including Samsung Galaxy Note8, Galaxy S9+, and Galaxy A21. When smartphone cameras are used, it is necessary to use the RAW format because conventional RGB images (JPEG) from smartphones are rendered and nonlinear to the light intensity [72]. RAW is an image format that captures unprocessed image data directly from the image sensor. Some high-end smartphones offer direct access to RAW images with "Pro Mode" or "ProRAW" in the default camera application. In the JPEG format, RGB color information is significantly compressed and often unrecoverable [72]. Thus, after reading RGB values in the RAW format, we recover the RGB spectral response functions. Figure 5 shows the RGB spectral response functions of the smartphone cameras recovered with the set of 12 primary colors. Unfortunately, the ground truth or directly measured RGB spectral response functions of the smartphones are not available as the manufacturers do not share those specifications in the public domain. In this respect, we establish a simple validation method by comparing the measured and synthetic RGB values of a large number of feature colors. As an expanded color checker reference standard, we use 96 colors in ColorChecker Digital SG ( Fig. 6(a)) that include additional representative colors of natural objects, such as human skin tones and various sky colors. First, we measure the true reflection spectrum of each color patch, not being affected by the light source spectrum and the ambient stray light in a similar manner of Eq. (5) (Fig. 6(b)). Second, we apply the estimated RGB spectral response functions to the reflection spectra of color patches and generate the corresponding synthetic RGB values for the specific smartphone camera. Third, we compare the measured and synthetic RGB values calculated by the estimated RGB spectral response functions of the smartphone camera (Fig. 6(c)-6(e)). To quantify errors, we use the root mean square relative error (RMSRE) defined as: where I T R,G,B is the measured true RGB values, I S R,G,B is the synthetic RGB values, and k(= 96) is the number of feature color patches. The RMSRE values of Note8, S9+, and A21 between the measured and synthetic RGB values of 96 colors are 7.88%, 6.95%, and 10.30%, respectively, on average. The small RMSRE values support the validity of the estimated RGB response functions in case of the high-end smartphones (Note8 and S9+). The low-end smartphone (A21) has a relatively high RMSRE value, due to the higher signal-to-noise ratio (high dark noise). Overall, the small differences between the measured and synthetic RGB values support the reliability of spectral response function recovery and potential applications for color calibration and standardization for a variety of machine vision and mHealth applications. , and Galaxy A21 (e-2) generated by applying the estimated RGB spectral response functions to the reflection spectra, following Eq. (6).

Discussion
We have introduced an l 1 -norm minimization framework for estimating the RGB spectral response functions of machine vision and smartphone cameras. As the estimation of RGB spectral response functions is typically an ill-posed problem, most of the previous color checker-based methods are framed to solve an overdetermined problem with l 2 -norm minimization (i.e., least squares method). Such methods are ideal for reliably recovering RGB spectral response functions with a small number of colors because the dimensionality (variation) of reflection spectra from color targets is lower than that of RGB spectral response functions. For example, even with a set of 1269 colors and their corresponding spectra, typical optimization frameworks require constraints on the illumination power spectrum or the RGB spectral response estimation [8,73]. In our study, we have demonstrated that the set of only primary 12 colors and their reflection spectra are sufficient to recover the RGB spectral response functions, owing to the compressive framework.
The previous studies on the mathematical estimation of spectral response functions are often available for digital single-lens reflex cameras (also known as digital SLR or DSLR), in part because the previous generations of smartphones are limited with the compressed format (e.g., JPEG) [74,75]. Nowadays, state-of-the-art smartphones allow end-users to control camera modes with access to the RAW format [76]. In this study, high-end Android smartphones (Note8 and S9+) offer "Pro mode" so that end-users can manually set the camera properties with the unprocessed RAW format data of images. It is even possible to extract images in the RAW format from A21, which is a considerably low-end smartphone. Third-party applications (e.g., Adobe Lightroom and Halide Mark) can also be used if smartphones allow "Pro Mode" or "ProRAW" in the default camera application.

Conclusion
We have developed a simple spectral compressive sensing framework for recovering RGB spectral response functions of smartphones with access to RAW images only using a set of 12 primary colors. This method is validated against the ground truth values via direct measurements obtained with the mono and RGB versions of a machine vision camera under sunlight. The predicted RGB spectral response functions of low-and high-end smartphones are supported by the significantly low error rates between the measured and synthetic color values of a comprehensive color reference checker. Different smartphone cameras equipped with diverse image sensors often produce distinguishable RGB images even when taken under the same conditions. Indeed, such a discrepancy has been a challenge in numerous mHealth applications where the built-in cameras are used for color or spectral quantifications. The reported spectral sensitivity estimation method can facilitate color and spectral canonicalization across different smartphone models for non-medical and mHealth applications. We also expect that the compressive sensing-based estimation of spectral sensitivity could be applied for characterizing machine vision systems and hyperspectral imaging components.
Funding. National Institutes of Health (R21TW010620, NIH Technology Accelerator Challenge).