Criteria to evaluate the fidelity of image enhancement by MSRCR

: Image fidelity refers to the ability of a process to render an image accurately. As image enhancement algorithms have been developed in recent years, how to assess the performances of different image enhancement algorithms has become an important question. Some objective image quality assessment (IQA) methods have been proposed, but there is little research on the image fidelity evaluation when comparing the performances of enhancement algorithms. Therefore, the authors proposed a new image fidelity assessment framework consisting of three components: the information entropy fidelity, constituent fidelity and colour fidelity. To verify the rationality of the fidelity criteria, they used the popular IQA database (LIVE), and the results indicated that the method matched better with the subjective assessment. Then, they verified the effectiveness of their method with the famous technique of image enhancement: multi-scale Retinex with colour restoration (MSRCR). The experimental results demonstrate MSRCR can improve the image quality, but it gives rise to obvious distortions. It is necessary to keep a moderate balance between image fidelity and image quality when they assess the enhanced images. Their results showed that the proposed objective fidelity index could provide an additional objective basis for the quality evaluation of image enhancement algorithms.


Introduction
Image processing is expected to contribute substantially to the development of communication media that must lead human observers to believe what they actually see. However, a common problem with image processing is the unsuccessful capture of the information seen in the original image. Therefore, to guarantee better image quality, image enhancement is necessary for many image applications and services [1]. So far, various image processing methods have been proposed for image enhancement such as histogram equalisation, wavelet transform theory, bilateral filtering theory, dark channel prior theory, Retinex theory etc. [2,3]. Histogram equalisation is one of the most common methods of image enhancement; it enhances the contrast and the detail of the image, but the image colour is prone to distortion after being processed [4]. The dark channel prior theory is one of the latest enhancement theories, but its results are sometimes slightly dark, and some details may be lost [5]. Multi-scale Retinex with colour restoration (MSRCR) is believed to be the best image enhancement method for all types of Retinex theories [6][7][8]. Although these image enhancement methods can improve the image quality [9][10][11][12][13], they have recently been challenged regarding the issue of fidelity. In most cases, image enhancement performed after image processing tends to bring out artefacts injected into the data due to the quality improvement, i.e. the image quality is enhanced at the expense of fidelity. Image fidelity refers to the ability of a process to render an image accurately, without any visible distortion or information loss. The opposite of fidelity is distortion. If the enhanced image is distorted, faulty conclusions will be reached. Image enhancement should avoid distortion, especially in medical images enhancement, as misdiagnoses may otherwise occur [14].
There are very few studies regarding the fidelity of enhanced images, and the distortion issues created by image enhancement have not been carefully studied as of yet. When we began to study the fidelity of enhanced images, we found that most image enhancement processes are sensitive to distortion creation and that there is still no public fidelity criterion for image enhancement method assessment. Two problems should be explained here. The first problem is that we cannot know whether the redundant information in an enhanced image is retained after transformation processing. The other problem is the lack of quantitative calculation for the fidelity of enhanced images. Therefore, it is necessary to develop computational measures of image fidelity in image enhancement.
There is also a natural tendency to confuse image quality with image fidelity. Although they are often assumed to be directly related, image fidelity refers to the ability to discriminate between two images, whereas image quality assessment (IQA) is inferred by the preference for one image over another. Researchers have recently proposed multiple visual factors to make judgements regarding image quality [15][16][17][18][19][20]. Usually, they can be classified into three categories: full-reference (FR) IQA, reduced-reference IQA and no-reference (NR) IQA. As for enhanced images, on the one hand, the information of the original image is rarely available, and on the other hand, the definition of a 'perfect image' is very difficult because the original image usually has bad quality and the distortion types of enhanced images are not easy to define and these approaches cannot be simply adopted. For example, a quality score may be very good, but it cannot be representative of fidelity. On the basis of these considerations, the two terms of image quality and image fidelity are often used interchangeably, but they are not the same. An image can sometimes be 'enhanced' by a distortion [21,22]. In view of the above discussion, the most significant difference between our approach and other IQAs is that we focused on the fidelity between the enhanced images and the corresponding original images. To achieve this goal, we proposed three fidelity criteria and a computational framework, and then, we verified the rationality of the fidelity criteria with the popular IQA database and the effectiveness of the method with the famous technique of image enhancement: MSRCR. The results showed that the MSRCR method of image enhancement may cause obvious information distortion, especially in dark images and over-bright images. This paper will discuss the fidelity and distortion of enhanced images based on the three fidelity criteria that we have proposed. We believe that, regardless of how the image is processed, certain features will be invariant. First, entropy can reflect the image information, and the information entropy (IE) cannot be increased before or after the image process [23]. The goal of image enhancement is to remove noise or other uncorrelated information to obtain image quality enhancement, so image enhancement belongs to the category of inverse-entropy processes. According to the theory of thermodynamics [24], the IE of the enhanced image should be constant or decreased compared with the original one. That is, an image enhancement method which the IE is invariable may be a good method. If the entropy of the enhanced image is increased, this means that other redundant information has been brought in. Second, according to the theory of histograms equalisation [25], the number of grey levels stretched should be related to the frequency of its occurrence, if the image is enhanced, it should remain unchanged. The enhancement method may stretch the spectrum of the image to enhance the image quality. However, components of the spectrum in the enhanced image may be added [26]. On the basis of this, we proposed the second criterion, which is regarded as the constituent fidelity. Finally, in the literature [27], we found that the colour of different enhanced images obtained through different enhancement methods are not consistent for a same original image listed in Fig. 1. The colour of the enhanced images shows obvious variation. Therefore, we proposed the third fidelity criterion and stated that the colour relationships of the enhanced image cannot be changed compared with the original image.

Multi-scale Retinex with colour restoration
According to the Retinex theory, the human eye perceives brightness based on ambient lighting and the irradiation light of surface reflection [8,28]. The words can be translated into the basic formula of Retinex as follows: That is where I x, y denotes a matrix of the reflection light intensity, L(x, y) is a matrix of the incident light intensity and R x, y is a matrix of the reflection coefficient of the surface of the illuminated object. The above formula is the reflection law of light intensity in the photometry and is written in matrix form. For a colour image, it can be written in the following form: where i = 0, 1, 2 denote the three channels of red, green and blue.
In the above formula, I(i, x, y) is only a measurable amount; the Gaussian filter is then used to estimate the incidence of a component L(i, x, y) of the image [29,30]. The expression of the method is represented as follows: where G x, y is called the Gaussian surround function and is denoted as follows: Jobson further presented that, on the basis of the non-linear characteristics of human vision [31], the logarithmic operation should be taken on both sides of formula (3), and thus resulting in Thus, it could be said that the processed result by means of Retinex is a logarithm of the reflection coefficient of an object surface rather than the reflection itself. Practically, the key of the Retinex method is the taking of the logarithm. We know that illuminance L i, x, y is an estimated parameter, which may introduce additional information during processing. Therefore, the information may be variable and may result in IE distortion, component distortion or colour distortion in MSRCR image enhancement processing. For example, as shown in Fig. 2a [32], pedestrians, roads and vehicles are not visible in the original image. When enhanced by MSRCR, the image quality is good, but the colour is not like that of the original one, which may be distortion. The data of Table 1 indicates that the IE, constituents and colour information of the enhanced image are changed compared with the original image. Additional components may be introduced during image processing, which leads to image distortion. Therefore, it is possible to develop computational measures of image fidelity based on certain image features.

Computing the fidelities and distortions of image features
On the basis of the three fidelity criteria that we have proposed, the computation framework of the fidelity of an enhanced image consists of five major sections as shown in Fig. 3: (i) image input: input the original image and the corresponding enhanced image; (ii) features extraction: extract the parameters, which are relative to the fidelity of the image; (iii) fidelity calculation: calculate the fidelity of the image; (iv) fidelity normalisation: obtain the normalised fidelity and distortion value; and (v) fidelity pooling: combine the fidelity values from step (iv) to provide a scalar measure to assess the fidelity of the enhanced image.

Computing IE fidelity and distortion
Entropy can denote the information of the image. Generally, the larger value of the image IE is, the more information the image contains. If IE increased, this means that there is more information in the image [23]. In the original image, many features of the image become blurred due to being covered. The purpose of image enhancement is to remove noise or other uncorrelated information to obtain good image quality, which is an inverse-entropy process.
According to the theory of thermodynamics, the IE of the enhanced image should be constant compared with that of the original image. If the IE increased, redundant information has been introduced. Thus, the first fidelity criterion is regarded as the IE fidelity. The information fidelity rate represents how much information included in the enhanced image belongs to the original image. The information distortion rate represents how much information included in the enhanced image does not belong to the original image. The IE is computed as follows [33]: where i = 0, 1, 2 denote the three colour channels of red, green and blue. p i, g represents the probability density distribution function of pixel number with the gth chromaticity. The average IE (AIE) is computed as follows: where 3 is called the normalised factor. IE(i) denotes the IE of each colour component. Suppose that AIE O and AIE E present the IEs of the original and the enhanced images, respectively. The information distortion rate (ΔAIE) is computed as follows: where the subscripts O and E denote the original and the enhanced images, respectively. The variation range of ΔAIE is from 0 to 1. The information entropy fidelity rate (FIER) is as follows: The variation range of FIER is from 0 to 1. FIER = 0 represents a deep distortion of information.

Computing the constituent fidelity and distortion
We also proposed the method of gradually flattening grey spectrum used to preferably show grey distribution characteristics than a traditional grey histogram. The expression of gradually flattening grey spectrum is as follows [32,33]: where p g and O g denote the grey level of original and target images, respectively. The alphabet m is a positive integer and called the flattening order. Formula (11) possesses a strong Note: IE represents information entropy, CTT represents components of spectra, AB represents average brightness, R -represents red, G -represents green and B -represents blue. performance of mining the lower-layer image information and the visual precision with 1 px. In this method, pixels could be displayed if they were present, not simply classified by their grey levels and it could be used to mine the grey or chromatic information of an image that is actually in existence. However, the traditional grey histogram cannot possess the performance. So, according to the method, we can perform the chromatic level factor (CTT), which represents the complex of the chromatic level of each colour component for an image. More line numbers are associated with good quality of the image. Therefore, we proposed the second fidelity criterion, which states that the constituents of the enhanced image cannot be increased. We defined the constituent distortion rate as how much of the constituent included in the enhanced image does not belong to the original image. The CTT of each chromatic component of an image can be obtained by the number of the spectral lines. The expression is presented as follows [34,35]: where Count is called the counting operator. The averaging constituent (ACT) of an image is defined as follows: Suppose that ACT O and ACT E represent the constituents of the original image and the enhanced images, respectively. The constituent distortion rate (ΔACT) can be computed as follows: The variation range of ΔACT is from 0 to 1. The fidelity rate of the constituent (FCTR) is as follows: The variation range of FCTR is from 0 to 1. FCTR = 0 represents a deep distortion of the constituent.

Computing colour fidelity and distortion
Colour information is significant to visual perception. Evaluating the colour fidelity is an important aspect of measuring the performance of an image enhancement algorithm [36]. According to human visual perception, all colours can be regarded as the three basic colours, red (R), green (G) and blue (B), combined in different proportions [37]. The MSRCR algorithm for three colour channels changes the proportions between channels, namely Therefore, there is an issue regarding how the colour in enhanced images does not necessarily correspond to that in the original image. To judge the colour fidelity of the enhanced image, we proposed the third fidelity criterion in image enhancement. The key to colour fidelity is maintaining the colour ratio. That is, the colour relationship between the enhanced image and the original image can be measured by means of computing the correlation coefficient (CC) between the loop ratios of the average brightness of the three components. The colour fidelity rate represents how much of the colour relation from the original image is preserved in the enhanced image and can be measured by the CC (correlation (COR)) between the average chromaticity of the three colour components of the original and the enhanced image. COR is computed as follows: where AB E i and AB O i denote the average brightness of the three components of the enhanced image and the original image, respectively. COR can be sensitive to colour fidelity. Suppose that COLD denotes the colour distortion rate as follows: The variation range of COLD is from 0 to 1. COLD = 1 means that the colour of the enhanced image is independent of the original image.

Computing the total average distortion (TAD) rate and fidelity rate
From the regression statistics, we can see that the CC of the three distortion indexes of the least-square method with the subjective evaluation is only 0.485 in Table 2. The correlation between the square mean and the subjective evaluation is 0.834. The square average that represents the total FR (TFR) is better consistent with the subjective evaluation results than the method of weight average. Therefore, we shall combine the three fidelity indexes with the square average to construct the distortion score, which is intended to describe the average fidelity for enhanced images. The TAD can be computed as follows.
The TFR is defined as follows: Obviously, the variation range of TFR is from 0 to 1.

Results and analysis
In this section, we present the results of the fidelity and distortion of the images enhanced by MSRCR. To further verify the effectiveness and the rationality of the image fidelity method proposed in this paper, there are two groups of experiments to validate the model: the consistency comparison with subjective evaluation of visual quality on individual distorted types. Then, we compare the fidelity and distortion of MSRCR method with other objective evaluation algorithms.

Performance on individual distorted types
We verified this method using the popular assessment database IQA database (LIVE), each image in these databases has been evaluated by human subjects under controlled conditions and subsequently assigned a quantitative subjective quality score: differential mean opinion score (DMOS). DMOS values are scored by observers, and a smaller DMOS value corresponds to a better image quality. To evaluate whether a metric is statistically consistent with visual perception, predicted metric scores are compared with the subjective ratings using three evaluation criteria suggested by the Video Quality Experts Group [38]:  and the subjective ratings. The estimation TFR is the final fidelity score, which the higher value implies better fidelity, and a smaller DMOS value corresponds to a better image quality. So, the relationship is a negative correlation with the DMOS and TFR. The comparison of the fidelity criteria and the subjective evaluation score is tabulated in Table 3. According to the performances of various types of distortion in Table 3, the fidelity results that we proposed are a negative correlation with the subjective perception, and the evaluation results of all distortion types are consistent with the subjective perception, which indicates good agreement between the fidelity results and the subjective score (DMOS). Thus, the data indicate our method can accurately and automatically predict the fidelity criteria for image quality, particularly in the gblur distortion type.

Fidelity and distortion of MSRCR method with other algorithms
To verify the advantages of our proposed metric, we further apply it to evaluate the image quality returned by the classical image enhancement models. Our purpose, on one hand, is to prove that these methods can significantly improve the quality of the image, it also suffers the problem of sacrificing the fidelity of the image. However, on the other hand, our proposed metric can provide the more objective standard to measure the ability of these models from both perspectives of the image quality and fidelity. Here, we discuss the application of our proposed method on measuring the fidelity of MSRCR in image enhancement processing and the corresponding image quality.

Image enhanced by MSRCR and the subsequent fidelity analysis:
The original image [39] and the enhanced image from the literature [40] are shown in Figs. 4a and b, respectively. The natural image quality evaluator (NIQE) [19] and comprehensive assessment function (CAF) [41] are the widely used indexes for NR IQA. Therefore, we choose CAF and NIQE as a baseline in Table 4. In order to stand out for comparison, we underlined the image quality assessment values in the table. In general, a higher index of CAF means a better performance such that the enhanced image quality is better than that of the original image (10.513 versus 26.928) and a lower index of NIQE means a better image quality (4.6655 versus 3.5447). From these NR indexes, we can get that the quality of Fig. 4b is better. However, the quality index cannot reflect whether there is any distortion or not. From Fig. 4d, the constituents of the enhanced image are increasing compared with the spectra of the original image (Fig. 4c), and many of the constituents at both ends of the spectra are added and are not existent in the spectra of the original image, i.e. additional constituent information may be introduced into the enhanced image. We also computed the fidelity indexes listed in Table 4.
The data in Table 4 show that the image enhanced by MSRCR may have information distortion (ΔAIE = 0.1219), constituent distortion (ΔACT = 0.2967) and colour distortion (COLD = 0.1443). The total average FR (TFR) of the enhanced image is 0.7969. Therefore, the image has been distorted, and the fidelity of the enhanced image compared with the original image is 79.69%.  From Fig. 5, we can see that constituent information distortion is serious, with an obvious difference compared with the baseline. Although the method of MSRCR may improve the image quality, it may not meet the fidelity criteria. Thus, the enhanced image has weak fidelity.

Over-bright image enhanced by MSRCR and the subsequent fidelity analysis:
Over-bright images may be caused by overexposure, and the image quality is not good. For over-bright images, the brightness levels have to be reduced by enhancement methods so that the hidden details can become visible. Here, we used MSRCR and the Zadeh-X method [42], which is another enhancement method, to address over-bright images, and then we analysed the fidelity of the enhanced images. The original image and the images enhanced by MSRCR and Zadeh-X are shown in Figs. 6a-c. The chromatic spectra of the three components of the original and the enhanced images are shown in Figs. 6d-f. For the image enhanced by MSRCR, we can see a spectral shift toward the left-hand sides and that the bandwidths of the spectra become narrower in Fig. 6e. Unfortunately, the image quality of the image   enhanced by MSRCR is not good, even worst than the original image as judged by the NR IQA methods (CAF and NIQE) listed in Table 5, which is in contradiction with the purpose of image enhancement. The result suggests that this enhanced method may not be suitable for enhancing over-bright images. That is, MSRCR can play a role in not only improving image quality, but also degrading the image quality. To test this fidelity criterion, we also chose the enhancement method named Zadeh-X to apply to the image, and we compared it with the MSRCR method. For the image enhanced by the Zadeh-X transformation in Fig. 6c, the image quality indexes are better than that of the original from Table 5. We computed the fidelity values of the enhanced images and list them in Table 5. The data of distortions and fidelities of IEs, constituent information and colour information are listed in Table 5. The data in row 2 demonstrates that the fidelities of IEs and the constituents of the image enhanced by MSRCR are less than those of the corresponding original image. Therefore, the MSRCR method fulfils the first and second fidelity criteria. However, the colour CC, COR, is only 72.8% (colour distortion), which is a very long distance from the original image. Thus, MSRCR may degrade the image quality for this type of image and is not suitable for the enhancement of over-bright images. The data in row 3 of Table 5 demonstrates that the IE and constituent information of the image enhanced by Zadeh-X are less than the corresponding values of the original image, thus, the Zadeh-X method for image enhancement meets the IE fidelity criterion and the constituent fidelity criterion. The CC of the original image and the image enhanced by the Zadeh-X transformation is 99.8%. The Zadeh-X method can meet the fidelity criteria, and its visual effect is better than that of the MSRCR method.

Over-dark image enhancement by MSRCR and the subsequent fidelity analysis:
Here, we discussed an over-dark image and the fidelity of the corresponding image enhanced by MSRCR. The original image is a modified image cited from the literature [43]. The original image contrast is too low; we cannot see anything in it. However, the chromatic spectrum from Fig. 7d may present information in the image, particularly if the spectrum is <20 and is below the human resolution capability, nothing will be distinguished [44]. The images enhanced by MSRCR and Zadeh-X are shown in Figs. 7b and c, respectively. The image quality of the image enhanced by MSRCR is good, and we can see the information hiding in the original image (Fig. 7b), but we cannot distinguish whether the information is the total of the original image. Fig. 7e shows that there is more constituent information than in the original image. Other additional components, which do not belong to the original image, may have been introduced. To evaluate the fidelity of this application of MSRCR, we have also chosen the Zadeh-X method to enhance the original image quality in Fig. 7c and the spectra of Zadeh-X in Fig. 7f to compare with MSRCR; the image quality is better than that of the original image, and the spectral information is less than or equal to that of the original image. We list the data of the original image and the enhanced images in Table 6.
From Table 6, it can be seen from the quality indexes (CAF and NIQE) that the quality of the image enhanced by MSRCR is better than that of Zadeh-X. However, there is serious constituent distortion in the result of MSRCR. That is to say, a quality score may be very good, but it cannot be representative of fidelity. The data in Table 6 show that the enhanced image could be called 'a sheer fabrication out of nothing,' and the MSRCR method for image enhancement has a very strong information distortion effect (ΔAIE = 0.824); however, it has strong colour restoration performance (COR = 1), and there is no colour distortion. The total fidelity value is 0.252; the enhanced image has weak fidelity. That is, the image quality is ensured by bringing out other information at the expense of fidelity in the MSRCR method. A weak relationship is between enhancement and image quality. Therefore, we posit that MSRCR is not suitable for the enhancement of over-dark images. The fidelity of the image enhanced by Zadeh-X is closer to the original one.

Images with good quality enhanced by MSRCR and the subsequent fidelity analysis:
The purpose of image enhancement is to improve the image quality and provide the observer a clearer image, so the processed image should be more effective than the original one. If the image quality is good, how is the performance of this type of image enhanced by MSRCR? The best processing is to retain the original quality. Therefore, we verify the performance of the two enhancement methods by our proposed method.
The original image is shown in Fig. 8a and the image enhanced by MSRCR is shown in Fig. 8b. Conversely, the MSRCR method can cause image quality degradation, where even the quality of the enhanced image is worst than that of the original. As for goodquality images, we can regard them as perfect images. Thus, we choose a reference-IQA approach [45] [structural similarity (SSIM) index] to act as a baseline in Table 7. In general, a higher SSIM  index means a better performance, and we find that the quality of the image enhanced by MSRCR is worst than that of the original image. This method may not have taken into account the degree of image enhancement to the full extent. Therefore, MSRCR may cause image quality degradation for this type of image. For the enhanced image of Zadeh-X in Fig. 8f, the image quality is invariant compared with the original image, and the spectral information is the same as that of the original image. We list the fidelity values of the original image and the enhanced images in Table 7. The MSRCR method may produce serious information distortion (ΔAIE = 0) and constituent distortion (ΔACT = 0). Fig. 8e shows that constituents of the chromatic spectra of the three components of the enhanced image are obviously increased, which may bring out addition information, thus degrading the image quality. The moving patterns of the spectral lines of the three channels are different from each other. According to the traditional concept of image processing, if the image quality is good, the image quality should remain good. In contrast, the Zadeh-X method for image enhancement does not produce any variation of the chromatic spectra, as shown in Fig. 8f, and it retains the image quality, which suggests that the Zadeh-X method for image enhancement possesses the better fidelity and broader adaptability than that of the MSRCR method. As discussed above, enhancement method should take into account the problem of over-enhancement.

Conclusions
Image fidelity depends on the ability to detect differences between images. In this paper, three fidelity criteria and quantitative characterisation methods were proposed. Then, the effectiveness of the approach is tested and the performance indicates that the fidelity criteria have a good correlation with the subjective judgements of image quality. The proposed approach comprehensively considers the effects of IE, constituent information and colour information between the enhanced and the original images. It could reflect the differences between enhanced images and the originals based on the information fidelities. The results showed that the MSRCR method could yield more distortion, especially for over-dark and over-bright images. The proposed method attempts to put forward the computational fidelity of the enhanced image, which provides an additional   There are several extensions to this fidelity technique that might be profitably explored to increase the usefulness and accuracy of the method. First, we should further study the fidelity and distortion issues in the image enhancement process and take into account the degree of fidelity. Second, we will rate the gradation for fidelity or distortion, i.e. gradate the fidelity level for fidelity measurement and decide which fidelity level will be acceptable in the image enhancement process. Finally, a good objective IQA should also reflect the distortion and fidelity of the enhanced image; therefore, we advocate that fidelity issues should also be considered in the quality assessment in the future.