Minimising ambient illumination via ambient subtraction: smartphone assessment of jaundice in liver patients via sclera images

Using smartphone images to quantify color presents a non-invasive way to assess jaundice and other color-related biomarkers of the human body. Here we focus on assessing jaundice through accurate bilirubin measurement in adult liver patients, the ﬁrst time optical imaging has been applied to this cohort. These patients can suffer from very high levels of bilirubin, in-dicating their severity of liver disease. A smartphone assessment technique for jaundice based around the color of the sclera (white of the eye) extracted from images is being developed, as smartphone imaging enables cheap, non-invasive and quantitative readings. Variations in ambient light cause large changes to recorded pixel values so must be accounted for to ensure that any changes detected are due to changes in jaundice level. Here we suggest the use of an ambient subtraction approach to min-imise the effects of ambient light. Pairs of ﬂash/ no-ﬂash images are captured and the extracted values subtracted to yield data as though under a pure ﬂash illumination. We present data demonstrating the technique with a group of healthy adult volunteers. We also present data from a patient study involving adults with liver disease. Images were captured and the bilirubin (jaun-dice) level predicted from these images before and after subtraction was compared to the ground truth value obtained via blood test. The linear correlation coefﬁcient increased from 0.47 to 0.85 (p < 0.001 in both cases) upon application of subtraction, demonstrating the effectiveness of the technique.


Introduction
There are many fields that seek to quantify color via images. For example, detection of agricultural disease [1] and crop ripeness [2], monitoring of marine life [3], animal coloration [4], water quality [5]. Within medicine, there is a large body of work around quantifying the results from strip tests [6,7], and of quantifying biomarkers from the human body such as anaemia [8,9], eye conditions such as anterior blepharitis [10], and the rate of wound healing [11].
Here we focus on the particular application of assessing jaundice, a yellowing of the skin and sclera (white of the eye). This condition is caused by a build-up of bilirubin, a normal breakdown product from red blood cells, in turn caused by issues with the liver in processing it correctly. Jaundice is commonly observed in neonates, however it is usually transient and the severe cases can be treated with phototherapy. By contrast, jaundice in adults is often a reflection of severe liver dysfunction. As their jaundice progresses, the bilirubin level is often used in clinical scoring systems as a prognostic index and to gauge response to therapy [12,13]. Focussing on jaundice as a visible indicator of liver disease enables the use of a non-invasive method to assess patients and determine when to intervene.
The known qualitative link between observed yellowness of a patient and bilirubin level suggests that it should be possible to obtain quantitative measures via images. The sclera is selected as the imaging site since it is free from the complicating presence of melanin. Healthy human sclerae are white, irrespective of a person's ethnicity [14]. Optical imaging is a very powerful tool within healthcare since it is non-contact and can be used outside of a hospital by individuals with minimal training. With over 8 billion subscriptions by 2018, smartphones have become globally ubiquitous [15]. The quality of smartphone cameras has also greatly improved in recent years, with increasing numbers of manufacturers providing access to raw images. Smartphones are therefore ideally placed for applications such as this, because they combine the ability to gather high quality raw images with low cost, portability and on-board processing.
In order for image data to be quantitative, the results must be processed to ensure that they are not dependent on the imaging device and lighting conditions at the time of capture. This is a non-trivial problem, and there are many ways to tackle it. The most thorough approach would be to obtain the camera spectral sensitivity of each phone and measure the spectrum of ambient light at each capture in order to generate device-independent results [3]. This approach is clearly not appropriate for our concept of a convenient, low-cost, portable method of assessing jaundice. An alternative approach would be to include a color chart in each scene, and so generate mappings on a per-image basis [8,16]. This would tackle both issues, however when imaging human subjects the inclusion of a color chart in the scene is challenging and can lead to unreliable image quality.
Instead we propose a two-step process. First, a one-time device specific calibration is carried out by imaging a color chart under the flash illumination of the phone. This produces a mapping specific to the phone and its in-built illumination. Second, the image capture process is modified to collect flash/ no-flash image pairs. The data from these two images is then subtracted to obtain results as though only under the flash illumination, providing stable results for a given phone under different lighting environments [17,18,19,20,21]. The device-specific mapping can then be applied to all subtracted data to produce illumination and device-independent results.
We aim to show that ambient light can be accounted for without the need to include a color chart in every image. Here we focus on ambient subtraction, highlighting its applicability to applications like this which involve imaging human subjects. We first present a new demonstration of subtraction for sclera images of healthy subjects under different illumination sources. We then present subtraction data for the first time from a patient study on adults with liver disease, comparing the correlation of the predicted bilirubin level extracted from sclera images before and after subtraction with the blood test value.

Theory
When we consider monitoring the color of a human sclera over time we must be confident that any changes observed are due to changes in the patient not in the imaging conditions or device. The standard image formation equation explains the interplay between the components of an image where x x x represents space, λ represents wavelength, ω represents the visible spectrum, m(x x x) gives the geometric dependence of the reflectance, s(λ , x x x) relates to the spectral reflectance properties of the surface (here the sclera), e(λ ) is the illumination of the scene, and ρ c (λ ) is the spectral sensitivity of the camera in each color channel [22]. It is clear from Equation 1 that it is not a simple task to extract information about only the sclera. To obtain device and ambient light independent color information about the sclera, we propose a two-step process with more detail in the following sections. Note that in this paper, we focus on monitoring jaundice in adult liver patients, but the approach is applicable to other fields with a similar desire to quantify color.

Ambient subtraction
First, we use an ambient subtraction approach [17,18,19,20,21] to remove the effects of ambient light. A flash/ no-flash pair of images is captured of the subject rather than the usual single image. Subtraction of the two images provides image data as though under a pure flash illumination. More mathematically, where the captured flash image f F+A is posited to be the sum of a pure flash image and an ambient image ( f F + f A ). This means that subtraction of the actual captured no-flash (ambient) image yields data as though captured under flash illumination in a dark room with zero ambient [17]. For this ambient subtraction approach to be valid, a number of conditions must be met. Firstly, the ambient light must remain constant between the capture of the two images so that its contribution is the same. Since the flash and no-flash images can be captured in quick succession, this condition is met in most common imaging situations. Secondly, the camera sensors must be linear -when the light level doubles, the recorded pixel values should also double (up to the saturation point). This crucial condition is commonly the case for smartphone camera sensors, and was carefully verified for the phone used in this study by imaging the central neutral patches of a Macbeth ColorChecker DC chart and plotting the RGB values against the known reflectance factor for each patch [23]. Note that raw images should be used for all analysis as they avoid the unknown, manufacturerspecific enhancement which produces aesthetically pleasing images which are however no longer linearly related to the scene. Thirdly, image pairs must either be captured with the same exposure time and ISO settings, or must be scaled to account for differences [19,20]. We suggest capturing images with the same settings to avoid the introduction of error due to the scaling. These settings must be selected such that both images are within the exposure range so that no clipping occurs in the region of interest.
Finally, for a pixel-wise subtraction of images, the alignment between images must be extremely good to avoid introducing error. In a lab, the use of a tripod naturally ensures image alignment. However, when developing a portable monitoring system for patients, the use of a tripod is not possible. Flash/ no-flash pairs will therefore not be perfectly aligned. One option would be to use image registration techniques to align the images, however this is not ideal as it introduces computational complexity. As we are not interested in the scene's spatial detail but only in a color indicator, our proposed alternative is to select a region of interest in each image, extract the median RGB values, and then subtract this single triplet of values.
The additional consideration for human subjects is the need for the image capture process to be comfortable. The intensity of the flash should therefore not be too high, especially given the proximity of the phone to the face (typically only a few inches). With smartphones, there are two options for image capture with the subtraction method. The front-facing camera could be used, enabling subjects to capture images themselves. In this case the phone screen can be used as a large-area illumination source, with an appropriate brightness level for use near the eye. Alternatively, the rear-facing camera could be used in combination with the flash of the phone. In order to use the flash in this context it is necessary to lower its intensity, which we achieved by means of a custom diffuser.
For the subtraction method to work well, there must be sufficient difference between the flash and no-flash images. To enable data to be checked at the time of capture, a simple metric has previously been developed based around the signal to noise ratio of the resulting pure flash data, modelling photon noise. The so-called subtracted signal to noise ratio (SSNR) is defined as where I F+A and I A are the mean intensity values for the regions of interest in the flash and no-flash images respectively [17]. An SSNR value for a captured image pair below the experimentally determined threshold of 3.4 triggers a warning suggesting that images be re-captured with lower ambient light levels or with the phone closer to the subject [17]. The inclusion of this on-the-fly metric means that the resulting images should be of a higher quality and less data has to be excluded from analysis.

Device-specific mapping
Post subtraction, the results are standardised for a given phone. However, for the technique to have general applicability, it must work on different phones. The standard approach of color management systems is to transform the image data from phone RGB space to a device-independent space, in which the relationship can be established between measured color and blood bilirubin concentration.
The need for a device-specific mapping has been previously demonstrated [24]. This means that it is necessary to calibrate each phone individually. The most thorough approach would be to measure the camera spectral sensitivity and flash spectral power distribution for each phone. But since these measurements can be time-consuming and require comparatively expensive specialist equipment this approach is not feasible here. The chart method is much more appropriate. Since the data for a given phone after subtraction is effectively always under pure flash illumination it is possible to carry out a one-time calibration to develop a mapping specific to the device and its flash illumination. Images of a Macbeth ColorChecker Classic chart are captured using the flash illumination of the phone with no ambient light. To avoid the need to capture an aligned grey card image, here an alternating least squares technique is used to correct for the spatial non-uniformity inherent to using the flash illumination [25]. The resulting mapping can be applied to subtracted image results from further capture sessions to obtain device-independent data. In this paper we focus on the ambient subtraction step so will re-main in phone RGB space, but we note the ease with which the method can be extended to give device-independent values.

Yellowness metric
Once ambient light-independent values have been obtained, the link from the color value extracted from sclera image-pairs to the blood bilirubin level can be developed. In order to evaluate the impact of the ambient subtraction step alone, here we will compare results before and after subtraction in phone linear RGB space. For simplicity, we use a yellowness metric based around the blue chromaticity of the subtracted data [26]. Chromaticity is defined as the ratio of one image channel to the sum of all channels [22], with blue chromaticity defined as The normalisation of brightness information in the subtracted data, varying exposure settings and distances from phone to patient all make a chromaticity metric the natural choice.

Methods
Unless stated otherwise, all data processing was carried out using MATLAB (MathWorks r2019b).

Image capture
The data presented in this paper is derived from images captured using a Samsung S8 phone, making use of the rear-facing camera and flash with a custom diffuser to lower the flash intensity. A custom app was developed for image capture. When the app was opened, the flash turned on to enable the subject to acclimatise to the light. When positioning the phone, care was taken, while watching the phone display, to avoid major specular reflection on the sclera due to light sources or the flash itself. When capture was initiated, the app automatically selected an ISO and exposure level based on the total illumination. This same combination of ISO and exposure level was used for both the flash and no-flash images which were automatically captured in quick succession (approximately one second apart). After capture, the two images were displayed on screen and the user was able to assess image quality visually, as well as zoom and pan to choose a test region of interest. The SSNR was calculated and a warning suggesting re-capture was displayed on screen if the level fell below the threshold for usable data.

Sclera subtraction experiment
In order to test the efficacy of the subtraction method for handheld images of a human subject, a series of flash/ no-flash images was captured of the sclerae of five volunteers aged 22-69. For each volunteer, images were first captured with no ambient light (in a dark room) to provide a ground truth sclera color under the phone illumination. Image pairs of the same scleral region were then captured in the presence of daylight, fluorescent, filtered tungsten, and two forms of LED ambient illumination. Figure 1 shows an example image capture session with ambient LED room lighting. The spectrum of each ambient light was also measured using a white standard and spectroradiometer. Three repeat images were captured, and the extracted results averaged. Regions of interest within the sclera were manually selected for each image pair, and the median RGB values extracted directly from the raw images. In order to isolate the subtraction stage of processing, the results under ambient light were compared with the ground truth before and after subtraction in the phone RGB space. To discount the effect of shading (caused by non-uniform illumination intensity) and varying exposure time, chromaticity values were used. The rg chromaticity values for the extracted sclera values were calculated before and after subtraction, and the distance to the corresponding ground truth rg value (GT ) was calculated. The distance was defined as the Euclidean distance where r and g are the red and green chromaticities, and the test and GT subscripts refer to data under ambient light (before or after subtraction) and no ambient light respectively. This volunteer sclera data set enables a test of how feasible high quality image capture is with healthy (i.e. non-jaundiced) human subjects, and how well the ambient subtraction approach is able to remove the effects of ambient light.

Patient study
A patient study was carried out on inpatients admitted to the Royal Free Hospital, London, to investigate the correlation between scleral yellowness and the blood bilirubin level for patients with chronic liver disease. Ethics for this study was granted as part of an on-going biomarker study (DASIMAR amendment 6). This was approved by the local regulatory approval (London -Harrow; REC Ref: 08/H0714/8) and all patients provided informed consent in accordance with the 1975 Helsinki Declaration. For each patient, a set of flash/ no-flash image pairs under unknown ambient illumination was captured. The ground truth bilirubin level measured via a standard hospital blood test was also recorded. Figure 2 depicts the standard processing pipeline for patient image data. As for the volunteer study, regions of interest were manually selected for each image pair, and the median RGB values extracted from the raw images. For comparison, the yellowness, based on a blue chromaticity metric, was then calculated both before and after subtraction. A link between yellowness and bilirubin level was developed for both cases. The correlation between the predicted and measured bilirubin levels was compared before and after subtraction to assess the efficacy of the subtraction method.

Sclera subtraction experiment
Before using the ambient subtraction technique on patients, the feasibility and effectiveness of the technique under more controlled conditions was tested. The subtraction method had been previously demonstrated for the Macbeth ColorChecker DC chart [17], however controlled data had not yet been collected for healthy human subjects. Here we present subtraction data collected for five volunteers under six different ambient illumination conditions, including no ambient illumination, to provide a ground truth. All data was collected using a single Samsung S8 phone.
An example cropped pair of images captured under ambient LED room lighting is shown in Figure 3, reproduced with permission from the subject. The first thing to note is the high image quality, which is more than enough for our purposes. The full size of images from this phone was 3024x4032 pixels. Second, where possible, specular reflection has been positioned on the iris rather than the sclera to avoid influencing the extracted values. Since the phone camera settings are fixed between the two images, there is visible dimming between the flash and noflash images. Along with the calculation of the SSNR, here with a value of 4.1, this dimming enables an additional visual check at the time of image capture. Note that there is some camera movement between the flash and no-flash images in Figure 3 which is inevitable with a handheld camera. This kind of shift motivated the use of separate ROI masks for flash and no-flash images and the subtraction of median values rather than a pixel-wise subtraction.
As described in the Methods section, median RGB values were extracted for each subject under each lighting condition. Figure 4 shows example results before and after subtraction for one subject. The results are presented in phone RGB space to  isolate the ambient subtraction process, without introducing errors from mapping to another space. Chromaticity values are presented here to standardise for varying exposure settings and different phone to subject distances. It is clear from Figure 4 that there is a sizeable variation in recorded sclera values under the different ambient illuminations. The ground truth value for the sclera with this technique is marked with a black square, and was determined by capturing images purely under the phone's illumination in a dark room. The data for all ambient lighting conditions after the ambient subtraction has been performed is shown in Figure 4 in yellow. The variability has been hugely reduced, and the values are closely clustered around the no ambient light ground truth value. Similar trends were observed for the five participants. The average rg distance data across all participants for the five different lighting conditions is shown in Table 1 before and after subtraction. The data in Table 1 confirms what is visually shown in Figure 4: that carrying out ambient subtraction greatly reduces the rg distance from the ground truth, to less than a tenth of the uncorrected distance for all lighting conditions. This confirms the ability of the subtraction technique to provide stable results under varying room lighting conditions, even for handheld capture of human subjects. The exact chromaticity co-ordinates of the ground truth value for the different participants varied, likely due to varying quantities of visible blood vessels and age [27]. An algorithm excluding blood vessels from analysis will be implemented for future work, however the aim here was simply to demonstrate the ability of ambient subtraction to produce stable results for a given subject.

Patient study
Having validated the subtraction technique for both a color chart and for human subjects, a patient study on adults with liver disease was carried out. Data was collected for patients across a very wide range of bilirubin levels, and in all cases the gold standard blood test bilirubin level was collected along with the image data. Since it is not possible to control the lighting in a hospital, there is no ground truth scleral color for the patients. Instead, the predicted bilirubin level was computed via a metric based around the blue chromaticity value of the sclera before and after subtraction, and the results compared as a way to demonstrate the improvement from the subtraction technique.
The results are shown in Figure 5 as a function of the bilirubin level measured via a blood test . The range of bilirubin levels is significantly higher than that seen for jaundice in neonates. This is the first time that an attempt has been made to assess bilirubin in adults with liver disease via imaging.
Whilst there is some correlation between predicted and measured bilirubin for the data in Figure 5 before subtraction, it is 0.47, so rather weak. After subtraction, the linear correlation coefficient rises to 0.85 (p<0.001), a strong correlation for biological data. This demonstrates that the ambient subtraction technique is producing values which are closer to being independent of the ambient light, since the measured yellowness is more strongly linked to the measured jaundice level. The strong correlation of the dataset also demonstrates the feasibility of the use of the modified image capture necessary for ambient subtraction in a real-world situation, and its applicability up to very high levels of yellowness.
As discussed in the Theory section, subtraction alone produces results which are stable for a given phone, but for the method to be scalable data should be processed in a deviceindependent color space. The proposed one-time device-specific calibration with this data is being used to enable development of a more advanced model for predicting bilirubin level in a deviceindependent color space. This will allow data to be collected on different phones and bilirubin predictions generated without recollecting the whole clinical dataset.
We also plan to obtain ground truth data directly from the sclerae of patients suffering from jaundice using a telespectroradiometer. This would enable a deeper understanding of the way a build-up of bilirubin affects the color of the sclera. Knowledge of the spectral reflectance could inform future developments, such as designing a filter to make the camera more sensitive to a specific region or tuning the color of the flash. We also plan to collect data for healthy volunteers over a wide age range in order to investigate the baseline variability of the color of the sclera and any correction that may need to be included in our model.

Conclusions
In this paper, we have highlighted the potential for using an ambient subtraction technique based on flash/ no-flash image pairs to standardise data from different ambient lighting conditions. The use of ambient subtraction means that it is not necessary to include a color chart in every image, which makes the image capture process significantly simpler. The subtraction technique is appropriate for a wide range of applications aiming to obtain quantified color measurements from images. We have focussed on the particular application of quantifying the jaundice level of patients with liver disease from images of the sclera. The sclera is selected as the imaging site since, as with skin, the buildup of bilirubin causes a visible yellowing. However, unlike skin, it is free from the complicating factor of melanin, meaning that healthy adults of all racial types have a similar baseline color.
The ambient subtraction technique has been demonstrated here for a group of healthy adult volunteers. Flash/ no-flash image pairs were captured of the sclera under a range of lighting conditions, and the results compared to a ground truth value ob-tained under no ambient light. Before subtraction, the results are very variable. After subtraction, however, the results are more closely clustered around the ground truth value. This dataset demonstrates the ability of ambient subtraction to standardise results for a given phone under different illuminations, even for handheld image data of human subjects.
Data has also been presented for patients with liver disease. Image capture was found to be feasible and patients reported no issue with the flash light. Bilirubin predictions were generated via the extracted scleral yellowness from image data. The correlation of these predicted bilirubin levels with the measured blood bilirubin levels was compared before and after subtraction. A significant increase in the correlation was observed after subtraction, implying that the results are less affected by ambient light. The high quality of this dataset also demonstrates that the modified image capture technique of flash/ no-flash images is achievable in the real-world, even with patients who are seriously unwell.
The technique can be combined with a device specific mapping generated via images of a color chart under the phone flash illumination. Since after subtraction the data is as though captured under a pure flash illumination in a dark room, this mapping need only be generated once per device. This two-step process has the capability to produce ambient-independent and device-independent results simply and effectively. Here, we have demonstrated that ambient subtraction is a technique able to produce stable results for a given phone under different lighting conditions in real-world situations.