A Novel Distance Matric: Generalized Relative Entropy

Information entropy and its extension, which are important generalization of entropy, have been applied in many research domains today. In this paper, a novel generalized relative entropy is constructed to avoid some defects of traditional relative entropy. We presented the structure of generalized relative entropy after the discussion of defects in relative entropy. Moreover, some properties of the provided generalized relative entropy is presented and proved. The provided generalized relative entropy is proved to have a finite range and is a finite distance metric.


Background
The concept of entropy was proposed by T. Clausius as one of the parameters to reflect the degree of chaos for the object.Later, research found that information was such an abstract concept that is hard to make it clear to obtain its amount.Indeed, it was not until the information entropy was proposed by Shannon that we had a standard measure for the amount of information.Then, some related concepts based on information entropy has been proposed subsequently, such as cross entropy, relative entropy and mutual information, which offered an effective method to solve the complex problems of information processing.Therefore, the study of novel metric based on information entropy was significant in the research domain of information science.
Information entropy was first proposed by Shannon.Assuming an information source I have composed by n different signals Ii, H(I), the information entropy of I, was shown in Eq.1, where p i =      ′     denotes frequency of Ii, E() means mathematical expectation, k > 1 denotes the base of logarithm.When k = 2, the unit of H(I) is bit.
Information entropy was a metric of the chao's degree for an information source.An information source was more complex when its information entropy was larger, and vice versa.The information entropy is an effective internal metric to measure an information source.To extend information entropy to measure the best distribution of an information source, cross entropy was defined in Eq.2 where P was 'real' information source and Q was a 'unreal' distribution information source, pi denotes frequency of components of P and qi denotes frequency of components of Q.
Cross entropy also can act as the reaction of the similarity degree of component's distribution for the two information sources.H(P, Q) = H(P) if and only if all components' distribution were identical.An homologous metric was relative entropy, and was also known as Kullback-Leibler divergence.Its definition was shown in Eq.3, where Eq 3.1 was relative entropy of the discrete random variables and Eq.3.2 was of the continuous random variables.
Relative entropy reflected the difference of the two information sources with different distributions, the larger relative entropy was, the difference of the two information sources was larger, vice versa.However, mutual information, another entropy based metric was also proposed to indicate the relationship of information which contained in two random variables X and Y. Mutual information was defined as relative entropy of p(x,y) and p(x)p(y) in Eq.4,where f(x,y) was joint probability density, fX(x) was marginal probability density of X and fY(y) was marginal probability density of Y.The value of mutual information was non-negative.If and only if X and Y were independent variables with each other, the value of mutual information was equal to zero.
Recently, there are many extensions and applications of information entropy based metric.But all these information entropy based methods have some defects.The most two important defects are: (1) it is not a distance metric; (2) it does not have an upper bound.
So in this paper, Section 2 introduces related works with information entropy based methods; Section 3 provides a novel generalized relative entropy and prove some properties of the provided entropy; Section 4 summarizes the whole content.

Related work
For years, many scholars have studied the applications of various entropy.Earlier, Iwo Białynicki-Birula et al. combined the matching with relative entropy and analyzed the relationship both of them [15].
In 2011, Batina L et al. reviewed mutual information [16].Audenaert K M studied the asymmetry of the relative entropy [17].Maoguo Gong et al. made the best of the BIFT and mutual information to propose a method that can match the object precisely [18].I. Giagkiozis et al. proposed a new method that can took advantage of the knowledge of cross entropy to solve the problem of multi-object programming [19].Tang M and Mao X researched information entropy-based metrics for measuring emergences in artificial societies [20].
In recent years, a great number of scholars have studied entropy based methods in recognition and classification.Soares C and Knobbe A. studied entropy-based discretization methods for ranking data [21].Zhengfu Li et al.
proposed a method to solve the problem of molecular docking used information entropy and ant colony genetic algorithm [22].Chun-Wang Ma et al. used information entropy to analyze the changes of substance in the process of chemical change [23].Kö et al. researched operational meaning of min-and max-entropy [24].
Besides, Müller M P and Pastena M. studied a generalization of majorization based on Shannon entropy [25].
Xiao Zhang et al. proposed a feature selection algorithm for fuzzy rough sets on the basis of information entropy [26].Abolfazl Ebrahimzadeh et al. proposed the concept of logical entropy based on entropy, and applied it to quantum dynamical system [27].Pedro Lopez-Garcia et al. proposed a method to make a prediction for traffic jam in a short period of time combined the genetic algorithm with cross entropy [28].David Sutter et al. studied the monotonicity of cross entropy [29].Opper M provided an estimator for the relative entropy rate of path measures for stochastic differential equations.Tang L et al. studied an EEMD-based multi-scale fuzzy entropy approach for complexity analysis in clean energy markets.

Structure of Generalized Relative Entropy
Nowadays, the relative entropy becomes one of the most important dissimilarity measure between two multidimensional vectors.Let X(x 1 , … x s ) and Y(y 1 , … y s ) be two multidimensional vectors, which are constituted by s components with different counts.The count of i th component is xi in vector X and yi in vector Y.
Therefore, the relative entropy RE(X, Y), which denotes relation from X to Y, is defined in Eq.5, where p x (i) ≝ means the probability of xi in X for each i.Herein, we define p x (i) • log p x (i) p y (i) = 0 when p x (i) = 0 in order to avoid form of equation 0 • log0.In real application, ε > 0 is added in the denominator of log(), which is a very small positive number to avoid form of equation log ∞.
It is admittedly that RE(X, Y) is not a distance metric because usually RE(X, Y) ≠ RE(Y, X) when X ≠ Y.
However, relative entropy does not have finite upper bound, which makes it can not be easily used to measure difference between high dimensional vectors in real application.So in this paper, based on definition of relative entropy, we present a generalized relative entropy d(X, Y) by Eq.6, where s denotes the number of components and k ≥ 1 denotes the control parameter of function d(), r = 0 when X = Y; r = 1 when X ≠ Y.We believe the generalized relative entropy has better properties than relative entropy.Moreover, it is a distance metric.

Properties of Generalized Relative Entropy
Theorem 1 will be presented to prove that the generalized relative entropy d() is a distance metric.However, Lemmas 1-2 and Inferences 1-2 are presented at first.
Then, with consideration of condition that sign "=" appeared, we have Reference 1.

Proof.
We have Eq.8 to prove Lemma 2 based on Eq.9.

Proof.
We have Eq.10 to prove Inference 2 based on Lemma 2.
d(X, Y) = ∑ (p x (i) • log k • p x (i) (k − 1)p x (i) + p y (i) + p y (i) • log k • p y (i) p x (i) + (k − 1)p y (i) ) Inference 2 is proved.■ deduced new uncertain relationship in quantum mechanics based on information entropy[1].A.Uhlmann et al.    applied relative entropy in digital integration, and proved some property of the interpolation theory[2].J. Shore et al. deduced the principle of maximum entropy and minimum cross entropy[3].Andrew M. Fraser et al.analyzed the coordinate of singular factors[4].Steven M.pincus et al. analyzed the complex degree of the system by the entropy[5].Afterwards, Aapo Hyvarinen et al. analyzed independent component and projection pursuit based on entropy[6].In 2000, I.R.Petersen et al. analyzed the optimization problem for the system with constraint of the relative entropy[7].N.kwak et al. classified sample based on mutual information between the input information and the variable category[8].Later, J.P.W.Pluim et al. analyzed the image matching in medicine based on mutual information[9].M. Arif et al. used the entropy to analyze the changes of the center of gravity between the old and young in order to find a method of improving the walking stability for the old[10].Steven J. Phillips et al.analyzed the distribution of species by maximum entropy model[11].V Krishnaveni et al. analyzed the electroencephalogram of human by mutual information[12].Afterwards, Wolf M M et al. researched area laws in quantum systems by used mutual information and correlations[13].Roger A. Baldwin et al. utilized maximum entropy model to find some regular about the selection of habitat of wild animal[14].Sergio Verdu et al.