The Case for Shifting the Rényi Entropy

Abstract

We introduce a variant of the Rényi entropy definition that aligns it with the well-known Hölder mean: in the new formulation, the r-th order Rényi Entropy is the logarithm of the inverse of the r-th order Hölder mean. This brings about new insights into the relationship of the Rényi entropy to quantities close to it, like the information potential and the partition function of statistical mechanics. We also provide expressions that allow us to calculate the Rényi entropies from the Shannon cross-entropy and the escort probabilities. Finally, we discuss why shifting the Rényi entropy is fruitful in some applications.

1. Introduction

The suggestive framework for the description and assessment of information transmission that Shannon proposed and co-developed [1,2,3] soon took hold of the mind of a generation of scientists and overflowed its initial field of application, despite the cautions of the inceptor himself [4]. He had independently motivated and re-discovered the Boltzmann description for the thermodynamic entropy of a system with many micro-states [5]. His build-up of the concept starting from Hartley’s measure of information using the nowadays well-known axiomatic approach created a sub-science—perhaps a science—out of three papers. For information scientists, it is difficult to shatter the intellectual chains of Shannon’s entropy [5,6,7,8,9,10,11].

After Shannon’s introduction of his re-purposing of the Boltzmann entropy to analyze communication, many generalizations of it were proposed, among which Rényi’s [12], Hvarda-Charvat-Tsallis’ [13] and Csiszar’s [14] seem to have found the widest echo. Reviews of information measures with different points of view are [14,15].

In this paper we want to contribute to the characterization and popularization of the Rényi entropy as a proper generalization of the Shannon entropy. Rényi’s suggestion was obtained after noticing some limits to the axiomatic approach [16], later better analyzed by Aczel and Daroczny [17]. His critical realisation was that there are more ways to develop the means of the individual surprisals of a collection of events, whereby he resorted to the Kolmogorov-Nagumo theory of the means [18,19,20]. In fact, Kolmogorov had been present in the history Information Theory from foundational issues [18], to punctual clarification [21], to his own devising of a measure of entropy-complexity. The situation concerning the theory of the means at the time is described in [22].

Rényi was quite aware that entropy is a quantity related to the averages of the information function on a probability distribution: let $X\sim P_X$ be a random variable over a set of outcomes $\mathcal{X}=\{x_i\mid 1\le i\le n\}$ and pmf $P_X$ defined in terms of the non-null values $p_i=P_X(x_i)$. The Rényi entropy for X is defined in terms of that of $P_X$ as $H_{\alpha }(X)=H_{\alpha }(P_X)$ by a case analysis [12]

$$\begin{array}{cc}\hfill \alpha & \ne 1H_{\alpha }(P_X)=\frac{1}{1-\alpha }log\sum _{i=1}^n{p}_i^{\alpha }\hfill \\ \hfill \alpha & =1\underset{\alpha \to 1}{lim}{H}_{\alpha }(P_X)=H(P_X)\hfill \end{array}$$

(1)

where $H(P_X)=-{\sum }_{i=1}^n{p}_{i}log{p}_i$ is the Shannon entropy [1,2,3]. Similarly the associated divergence when $Q\sim Q_X$ is substituted by $P\sim P_X$ on a compatible support is defined in terms of their pmfs $q_i=Q_X(x_i)$ and $p_i=P_X(x_i)$, respectively, as $D_{\alpha }(X\parallel Q)=D_{\alpha }(P_X\parallel Q_X)$ where

$$\begin{array}{cc}\hfill \alpha & \ne 1D_{\alpha }(P_X\parallel Q_X)=\frac{1}{\alpha -1}log\sum _{i=1}^n{p}_i^{\alpha }{q}_i^{1-\alpha }\hfill \\ \hfill \alpha & =1\underset{\alpha \to 1}{lim}{D}_{\alpha }(P_X\parallel Q_X)=D_{KL}\left(P_X\parallel Q_X\right).\hfill \end{array}$$

(2)

and $D_{KL}\left(P_X\parallel Q_X\right)={\sum }_{i=1}^n{p}_{i}log\frac{p_i}{q_i}$ is the Kullback–Leibler divergence [23].

When trying to find the closed form for a generalization of the Shannon entropy that was compatible with all the Faddev axioms but that of linear average, Rényi found that the function $\phi (x)=x^r$ could be used with the Kolmogorov–Nagumo average to obtain such a new form of entropy. Rather arbitrarily, he decided that the constant should be $\alpha =r+1$, thus obtaining (1) and (2), but obscuring the relationship of the entropies of order $\alpha$ and the generalized power means.

We propose to shift the parameter in these definitions back to $r=\alpha -1$ to define the shifted Rényi entropy of order r the value

$$\begin{array}{cc}\hfill {\tilde{H}}_r(P_X)& =-log{M}_r(P_X,P_X)\hfill \end{array}$$

and the shifted Rényi divergence of order r the value

$$\begin{array}{cc}\hfill {\tilde{D}}_r(P_X\parallel Q_X)& =log{M}_r(P_X,\frac{P_X}{Q_X})\hfill \end{array}$$

where $M_r$ is the r-th order weighted generalized power means or Hölder means [24]:

$$\begin{array}{cc}\hfill M_r(\overrightarrow{w},\overrightarrow{x})& ={\left(\sum _{i=1}^n\frac{w_i}{{\sum }_k{w}_k}·x_i^r\right)}^{\frac{1}{r}}.\hfill \end{array}$$

In our opinion, this shifted version may be more fruitful than the original. However, since this could be deemed equally arbitrary, in this paper we argue that this statement of the Rényi entropy greatly clarifies its role vis-a-vis the Hölder means, viz. that most of the properties and special cases of the Rényi entropy arise from similar concerns in the Hölder means. We also provide a brief picture of how the theory surrounding the Rényi entropy would be modified with this change, as well as its relationship to some other magnitudes.

2. Preliminaries

2.1. The Generalized Power Means

Recall that the generalized power or Hölder mean of order r is defined as

$$\begin{array}{c}\hfill M_r(\overrightarrow{w},\overrightarrow{x})={\left(\frac{{\sum }_{i=1}^n{w}_i·x_i^r}{{\sum }_k{w}_k}\right)}^{\frac{1}{r}}={\left(\sum _{i=1}^n\frac{w_i}{{\sum }_k{w}_k}·x_i^r\right)}^{\frac{1}{r}}\end{array}$$

(3)

By formal identification, the generalized power mean is nothing but the weighted f-mean with $f(x)=x^r$ (see Appendix A). In this paper we use the notation where the weighting vector comes first—rather than the opposite, used in [24]—to align it with formulas in information theory, e.g., divergences and cross entropies. Reference [25] provides proof that this functional mean also has the Properties 1–3 of Proposition A1 and Associativity.

The evolution of $M_r(\overrightarrow{w},\overrightarrow{x})$ with r is also called the Hölder path (of an $\overrightarrow{x}$). Important cases of this mean for historical and practical reasons are obtained by giving values to r:

• The (weighted) geometric mean when $r=0$.

  $$\begin{array}{c}\hfill M_0(\overrightarrow{w},\overrightarrow{x})=\underset{r\to 0}{lim}{M}_r(\overrightarrow{w},\overrightarrow{x})={\left({\mathsf{\Pi }}_{i=1}^n{x}_i^{w_i}\right)}^{\frac{1}{{\sum }_k{w}_k}}\end{array}$$

  (4)
• The weighted arithmetic mean when $r=1$.

  $$M_1(\overrightarrow{w},\overrightarrow{x})=\sum _{i=1}^n\frac{w_i}{{\sum }_k{w}_k}·x_i$$
• The weighted harmonic mean for $r=-1$.

  $$M_{-1}(\overrightarrow{w},\overrightarrow{x})={\left(\sum _{i=1}^n\frac{w_i}{{\sum }_k{w}_k}·x_i^{-1}\right)}^{-1}=\frac{{\sum }_k{w}_k}{{\sum }_{i=1}^n{w}_i·\frac{1}{x_i}}$$
• The quadratic mean for $r=2$.

  $$M_2(\overrightarrow{w},\overrightarrow{x})={\left(\sum _{i=1}^n\frac{w_i}{{\sum }_k{w}_k}·x_i^2\right)}^{\frac{1}{2}}$$
• Finally, the max- and min-means appear as the limits:

  $$\begin{array}{cc}\hfill M_{\infty }(\overrightarrow{w},\overrightarrow{x})& =\underset{r\to \infty }{lim}{M}_r(\overrightarrow{w},\overrightarrow{x})=\underset{i=1}{\overset{n}{max}}{x}_i\hfill \\ \hfill M_{-\infty }(\overrightarrow{w},\overrightarrow{x})& =\underset{r\to -\infty }{lim}{M}_r(\overrightarrow{w},\overrightarrow{x})=\underset{i=1}{\overset{n}{min}}{x}_i\hfill \end{array}$$

They all show the following properties:

Proposition 1

(Properties of the weighted power means). Let $\overrightarrow{x},\overrightarrow{w}\in {(0,\infty )}^n$ and $r,s\in (-\infty ,\infty )$. Then, the following formal identities hold, where ${\overrightarrow{x}}^r$ and $\frac{1}{\overrightarrow{x}}$ are to be understood entry-wise,

1. 

(0- and 1-order homogeneity in weights and values) If $k_1,k_2\in {\mathbb{R}}_{\ge 0}$, then $M_r(k_1·\overrightarrow{w},k_2·\overrightarrow{x})=k_1^0·k_2^1·M_r(\overrightarrow{w},\overrightarrow{x}).$

2. 

(Order factorization) If $r\ne 0\ne s$, then $M_{rs}(\overrightarrow{w},\overrightarrow{x})={\left(M_s(\overrightarrow{w},{(\overrightarrow{x})}^r)\right)}^{1/r}.$

3. 

(Reduction to the arithmetic mean) If $r\ne 0$, then $M_r(\overrightarrow{w},\overrightarrow{x})={\left[M_1(\overrightarrow{w},{(\overrightarrow{x})}^r)\right]}^{1/r}$.

4. 

(Reduction to the harmonic mean) If $r\ne 0$, then $M_{-r}(\overrightarrow{w},\overrightarrow{x})={\left[M_{-1}(\overrightarrow{w},{(\overrightarrow{x})}^r)\right]}^{1/r}={\left[M_r(\overrightarrow{w},\frac{1}{\overrightarrow{x}})\right]}^{-1}={\left[M_1(\overrightarrow{w},\frac{1}{{(\overrightarrow{x})}^r})\right]}^{-1/r}$.

5. 

(Monotonicity in r) Furthermore, $\overrightarrow{x}\in {[0,\infty ]}^n$ and $r,s\in [-\infty ,\infty ]$, then

$$\underset{i}{min}{x}_i=M_{-\infty }(\overrightarrow{w},\overrightarrow{x})\le M_r(\overrightarrow{w},\overrightarrow{x})\le M_{\infty }(\overrightarrow{w},\overrightarrow{x})=\underset{i}{max}{x}_i$$

and the mean is a strictly monotonic function of r, that is $r<s$ implies $M_r(\overrightarrow{w},\overrightarrow{x})<M_s(\overrightarrow{w},\overrightarrow{x})$, unless:

• $x_i=k$ is constant, in which case $M_r(\overrightarrow{w},\overrightarrow{x})=M_s(\overrightarrow{w},\overrightarrow{x})=k$.
• $s\le 0$ and some $x_i=0$, in which case $0=M_r(\overrightarrow{w},\overrightarrow{x})\le M_s(\overrightarrow{w},\overrightarrow{x})$.
• $0\le r$ and some $x_i=\infty$, in which case $M_r(\overrightarrow{w},\overrightarrow{x})\le M_s(\overrightarrow{w},\overrightarrow{x})=\infty$.

6. 

(Non-null derivative) Call ${\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x})={\left\{\frac{w_k{x}_k^r}{{\sum }_i{w}_i{x}_i^r}\right\}}_{k=1}^n$. Then

$$\begin{array}{c}\hfill \frac{d}{dr}{M}_r(\overrightarrow{w},\overrightarrow{x})=\frac{1}{r}·M_r(\overrightarrow{w},\overrightarrow{x})ln\frac{M_0({\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x}),\overrightarrow{x})}{M_r(\overrightarrow{w},\overrightarrow{x})}\end{array}$$

(5)

Proof. 

Property 1 follows from the commutativity, associativity and cancellation of sums and products in ${\mathbb{R}}_{\ge 0}$. Property 2 follows from identification in the definition, then Properties 3 and 4 follow from it with $s=1$ and $s=-1$ respectively. Property 5 and the special cases in it are well known and studied extensively in [24]. We will next prove property 6

$$\begin{array}{cc}\frac{d}{dr}{M}_r(\overrightarrow{w},\overrightarrow{x})& =\frac{d}{dr}{e}^{\frac{1}{r}ln\left({\sum }_k\frac{w_k}{{\sum }_i{w}_i}{x}_k^r\right)}=M_r(\overrightarrow{w},\overrightarrow{x})\left(\frac{-1}{r^2}ln\left(\sum _k\frac{w_k}{{\sum }_i{w}_i}{x}_k^r\right)+\frac{1}{r}·\frac{{\sum }_k{w}_k{x}_k^{r}ln{x}_k}{{\sum }_i{w}_i{x}_i^r}\right)\end{array}$$

Note that if we call ${\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x})={\left\{w_k^{\prime }\right\}}_{k=1}^n={\left\{\frac{w_k{x}_k^r}{{\sum }_i{w}_i{x}_i^r}\right\}}_{k=1}^n$, since this is a probability we may rewrite:

$$\begin{array}{c}\hfill \sum _k\frac{w_k{x}_k^r}{{\sum }_i{w}_i{x}_i^r}·ln{x}_k=\sum _k{w}_k^{\prime }ln{x}_k=ln\left(\prod _k{x}_k^{w_k^{\prime }}\right)=ln{M}_0({\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x}),x)\end{array}$$

whence

$$\begin{array}{cc}\hfill \frac{d}{dr}{M}_r(\overrightarrow{w},\overrightarrow{x})& =M_r(\overrightarrow{w},\overrightarrow{x})\left(\frac{1}{r}·ln{M}_0({\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x}),x)-\frac{1}{r}·ln{M}_r(\overrightarrow{w},\overrightarrow{x})\right)\hfill \\ & =\frac{1}{r}·M_r(\overrightarrow{w},\overrightarrow{x})ln\frac{M_0({\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x}),\overrightarrow{x})}{M_r(\overrightarrow{w},\overrightarrow{x})}.\hfill \end{array}$$

 □

Remark 1.

The distribution ${\tilde{q}}_r(\overrightarrow{w},\overrightarrow{x})$ when $\overrightarrow{w}=\overrightarrow{x}$ is extremely important in the theory of generalized entropy functions, where it is called a (shifted) escort distribution (of $\overrightarrow{w}$) [5], and we will prove below that its importance stems, at leasts partially, from this property.

Remark 2.

Notice that in the case where both conditions at the end of Property 1.5 hold—that is for $i\ne j$ we have $x_i=0$ and $x_j=\infty$—then we have for $r\le 0,M_r(\overrightarrow{w},\overrightarrow{x})=0$ and for $0\le r,M_r(\overrightarrow{w},\overrightarrow{x})=\infty$ whence $M_r(\overrightarrow{w},\overrightarrow{x})$ has a discontinuity at $r=0$.

2.2. Rényi’s Entropy

Although the following material is fairly standard, it bears directly into our discussion, hence we introduce it in full.

2.2.1. Probability Spaces, Random Variables and Expectations

Shannon and Rényi set out to find how much information can be gained on average by a single performance of an experiment $\mathsf{\Omega }$ under different suppositions. For that purpose, let $(\mathsf{\Omega },{\mathsf{\Sigma }}_{\mathsf{\Omega }},P)$ be a measure space, with $\mathsf{\Omega }=\{{\omega }_1,\dots ,{\omega }_n\}$ the set of outcomes of a random experiment, ${\mathsf{\Sigma }}_{\mathsf{\Omega }}$ the sigma-algebra of this set and measure $P:\mathsf{\Omega }\to {\mathbb{R}}_{\ge 0},P({\omega }_i)=p_i,1\le k\le n$. We define the support of P, as the set of outcomes with positive probability supp $(P)=\{\omega \in \mathsf{\Omega }\mid P(\omega )>0\}$.

Let $(\mathcal{X},{\mathsf{\Sigma }}_{\mathcal{X}})$ be a measurable space with $\mathcal{X}$ a domain and ${\mathsf{\Sigma }}_{\mathcal{X}}$ its sigma algebra and consider the random variable $X:\mathsf{\Omega }\to \mathcal{X}$, that is, a measurable function so that for each set of $B\in {\mathsf{\Sigma }}_{\mathcal{X}}$ we have $X^{-1}(B)\in {\mathsf{\Sigma }}_{\mathsf{\Omega }}$. Then P induces a measure $P_X$ on $(\mathcal{X},{\mathsf{\Sigma }}_{\mathcal{X}})$ with $\forall x\in {\mathsf{\Sigma }}_{\mathcal{X}},P_X(x)=P(X=x)=P(X^{-1}(x))$, where x is an event in ${\mathsf{\Sigma }}_{\mathcal{X}}$, and $P_X(x)={\sum }_{{\omega }_i\subseteq X^{-1}(x)}P({\omega }_i)$ whereby $(\mathcal{X},{\mathsf{\Sigma }}_{\mathcal{X}},P_X)$ becomes a measure space. We will use mostly $X\sim P_X$ to denote a random variable, instead of its measurable space. The reason for this is that since information measures are defined on distributions, this is the more fundamental notion for us.

Sometimes co-occurring random variables are defined on the same sample space and sometimes on different ones. Hence, we will need another measure space sharing the same measurable space $(\mathsf{\Omega },{\mathsf{\Sigma }}_{\mathsf{\Omega }})$ but different measure, $(\mathsf{\Omega },{\mathsf{\Sigma }}_{\mathsf{\Omega }},Q)$ with $Q({\omega }_i)=q_i$.

Remark 3.

Modernly, discrete distributions are sets or vectors of non-negative numbers adding up to 1, but Rényi developed his theory for “defective distributions”, that is, with ${\sum }_{i}P({\omega }_i)\ne 1$ which are better described as “positive measures”. In fact, we do not need to distinguish whether P is a probability measure in the $(n-1)$-simplex $P\in {\Delta }^{n-1}\iff {\sum }_{i}P({\omega }_i)=1$ or in general a measure $P\in {\mathbb{R}}_{\ge 0}^n$ and nothing precludes using the latter to define entropies—while it provides a bit of generalization this is the road we will take below (see [12,26] on using incomplete distributions with ${\sum }_i{p}_i<1$).

2.2.2. The Approach to Rényi’s Information Functions Based in Postulates

One of the most important applications of the generalized weighted means is to calculate the moments of (non-negative) random variables.

Lemma 1.

Let $X\sim P_X$ be a discrete random variable. Then ther-th moment of Xis:

$$\begin{array}{c}\hfill E_X\left\{X^r\right\}=\sum _i{p}_i{x}_i^r={\left(M_r(P_X,X)\right)}^r\end{array}$$

(6)

This is the concept that Shannon, and afterwards Rényi, used to quantify information by using the distribution as a random variable (Section 3.3).

The postulate approach to characterize Shannon’s information measures can be found in Appendix B. Analogue generalized postulates lead to Rényi’s information functions, but, importantly, he did not consider normalized measures, that is with ${\sum }_k{p}_k=1$.

We follow [27] in stating the Rényi postulates:

• The amount of information provided by a single random event $x_k$ should be a function of its probability $P_X(x_k)=p_k$, not its value $x_k=X({\omega }_k)$, $\mathfrak{I}:[0,1]\to I$ where $I\subseteq \mathbb{R}$ quantifies information.
• This amount of information should be additive on independent events.

  $$\begin{array}{c}\hfill \mathfrak{I}(p,q)=\mathfrak{I}(p)+\mathfrak{I}(q)\end{array}$$

  (7)
• The amount of information of a binary equiprobable decision is one bit.

  $$\begin{array}{c}\hfill \mathfrak{I}(1/2)=1\end{array}$$

  (8)
• If different amounts of information occur with different probabilities the total amount of information $\mathfrak{I}$ is an average of the individual information amounts weighted by the probability of occurrence.

These postulates may lead to the following consequences:

• Postulates 1 and 2 fix Hartley’s function as the single possible amount of information of a basic event

  $$\begin{array}{c}\hfill \mathfrak{I}:[0,1]\to [0,\infty ],p\mapsto \mathfrak{I}(p)=-klogp.\end{array}$$

  (9)
• Postulates 3 fixes the base of the logarithm in Hartley’s formula to 2 by fixing $k=1$. Any other value $k=1$/$logb$ fixes b as the base for the logarithm and changes the unit.
• Postulate 4 defines an average amount of information, or entropy, properly speaking. Its basic formula is a form of the Kolmogorov–Nagumo formula or f-mean (A2) applied to information

  $$\begin{array}{c}\hfill H(P_X,\phi ,\mathfrak{I})={\phi }^{-1}\left({\mathsf{\Sigma }}_{i=1}^n\frac{p_i}{{\sum }_k{p}_k}\phi (\mathfrak{I}(p_i))\right).\end{array}$$

  (10)
  Thus the “entropy” in Information Theory is, by definition, synonym with “aggregate amount of information”, which departs from its physical etymology, despite the numerous analogies between both concepts.
  It has repeatedly been proven that only two forms of the function $\phi$ can actually be used in the Kolmogorov–Nagumo formula that respect the previous postulates [12,26,27]:

  -

  The one generating Shannon’s entropy:

  $$\begin{array}{c}\hfill \phi (h)=ah+b\mathrm{with}a\ne 0,\end{array}$$

  (11)

  -

  That originally used by Rényi himself:

  $$\begin{array}{c}\hfill \phi (h)=2^{(1-\alpha )h},\mathrm{with}\alpha \in [-\infty ,\infty ]\setminus \left\{1\right\}.\end{array}$$

  (12)

Taking the first form (11) and plugging it into (10) leads to Shannon’s measure of information, and taking the second form leads to Rényi’s measure of information (1), so we actually have:

Definition 1

([12,26]). The Rényi entropy of order $\alpha$ for a discrete random variable $X\sim P_X$, is

$$\begin{array}{ccc}\hfill & H_{\alpha }(P_X)=\frac{1}{1-\alpha }log\left(\sum _{i=1}^n\frac{p_i^{\alpha }}{{\sum }_k{p}_k}\right),\alpha \ne 1\hfill & \hfill \underset{\alpha \to 1}{lim}{H}_{\alpha }(P_X)=H(P_X)=-\sum _i\frac{p_i}{{\sum }_k{p}_k}log{p}_i,\end{array}$$

(13)

where the fact that Shannon’s entropy is the Rényi entropy when $\alpha \to 1$ in (1) is found by a continuity argument.

Rényi also used the postulate approach to define the following quantity:

Definition 2

([12,26]). The gain of information or divergence (between distributions) when $Y\sim P_Y$, $P_Y(y_i)=q_i$ is substituted by $X\sim P_X$, $P_X(x_i)=p_i$ being continuous wrt the latter—that is, with supp $Y\subseteq \mathrm{supp}X$—as

$$\begin{array}{ccc}{D}_{\alpha }(P_X\parallel P_Y)=\frac{1}{\alpha -1}log\sum _{i=1}^n{p}_i^{\alpha }{q}_i^{1-\alpha },\alpha \ne 1\hfill & \hfill \underset{\alpha \to 1}{lim}{D}_{\alpha }(P_X\parallel P_Y)& =D_{KL}(P_X\parallel P_Y)\hfill \end{array}$$

and the fact that Kullback–Leibler’s divergence emerges as the limit when $\alpha \to 1$ follows from the same continuity argument as before. Such special cases will not be stated again, as motivated in Section 3.1.

As in the Shannon entropy case, the rest of the quantities arising in Information Theory can be defined in terms of the generalized entropy and its divergence [23,27].

3. Results

3.1. The Shifted Rényi Entropy and Divergence

To leverage the theory of generalized means to our advantage, we start with a correction to Rényi’s entropy definition: The investigation into the form of the transformation function for the Rényi entropy (12) is arbitrary in the parameter $\alpha$ that it chooses. In fact, we may substitute in $r=\alpha -1$ to obtain the pair of formulas:

$$\begin{array}{cccc}\hfill {\phi }^{\prime }(h)& =b^{-rh}\hfill & \hfill {\phi }^{\prime -1}(p)& =\frac{-1}{r}{log}_{b}p\hfill \end{array}$$

(14)

Definition 3.

The shifted Rényi entropy of order $r\ne 0$ for a discrete random variable $X\sim P_X$, is the Kolmogorov–Nagumo ${\phi }^{\prime }$-mean (10) of the information function ${\mathfrak{I}}_{*}(p)=-lnp$ over the probability values.

$$\begin{array}{cccc}\hfill {\tilde{H}}_r\left(P_X\right)& =\frac{-1}{r}{log}_b\left(\sum _i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r\right)\hfill & \hfill \underset{r\to 0}{lim}{\tilde{H}}_r\left(P_X\right)& =H(P_X).\hfill \end{array}$$

(15)

Note that:

• For $r\ne 0$ this is motivated by:

  $$\begin{array}{cc}\hfill {\tilde{H}}_r\left(P_X\right)& =\frac{-1}{r}{log}_b\left(\sum _i\frac{p_i}{{\sum }_k{p}_k}{b}^{r{log}_b{p}_i}\right)=\frac{-1}{r}{log}_b\left(\sum _i\frac{p_i}{{\sum }_k{p}_k}{b}^{{log}_b{p}_i^r}\right)=\frac{-1}{r}{log}_b\left(\sum _i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r\right).\hfill \end{array}$$
• For $r=0$ we can use the linear mean $\phi (h)=ah+b$ with inverse ${\phi }^{-1}(p)=\frac{1}{a}(p-b)$ as per the standard definition, leading to Shannon’s entropy.

Remark 4.

The base of the logarithm is not important as long as it is maintained in ${\phi }^{\prime }(·)$, ${\mathfrak{I}}_{*}(·)$ and their inverses, hence we leave it implicit. For some calculations—e.g., the derivative below—we explicitly provide a particular basis—e.g., ${log}_{e}x=lnx$.

The shifted divergence can be obtained in the same manner—the way that Rényi followed himself [26].

Definition 4.

The shifted Rényi divergence between two distributions $P_X(x_i)=p_i$ and $Q_X(x_i)=q_i$ with compatible support is the following quantity.

$$\begin{array}{cccc}\hfill {\tilde{D}}_r\left(P_X\parallel Q_X\right)& =\frac{1}{r}log\sum _i\frac{p_i}{{\sum }_k{p}_k}{\left(\frac{p_i}{q_i}\right)}^r\hfill & \hfill \underset{r\to 0}{lim}{\tilde{D}}_r\left(P_X\parallel Q_X\right)& =D_{KL}\left(P_X\parallel Q_X\right).\hfill \end{array}$$

(16)

Of course, the values of the Rényi entropy and divergence are not modified by this shifting.

Lemma 2.

The Rényi entropy and the shifted Rényi entropy produce the same value, and similarly for their respective divergences.

Proof. 

if we consider a new parameter $r=\alpha -1$ we have:

$$\begin{array}{cc}\hfill H_{\alpha }(P_X)& =\frac{1}{1-\alpha }log\left(\sum _{i=1}^n\frac{p_i^{\alpha }}{{\sum }_k{p}_k}\right)=\frac{-1}{r}log\left(\sum _{i=1}^n\frac{p_i^{r+1}}{{\sum }_k{p}_k}\right)=-\frac{1}{r}log\left(\sum _{i=1}^n\frac{p_i}{{\sum }_k{p}_k}{p}_i^r\right)={\tilde{H}}_r(P_X).\hfill \end{array}$$

and similarly for the divergence:

$$\begin{array}{cc}\hfill D_{\alpha }(P_X\parallel Q_X)& =\frac{1}{\alpha -1}log\sum _{i=1}^n\frac{p_i^{\alpha }{q}_i^{1-\alpha }}{{\sum }_k{p}_k}=\frac{1}{r}log\sum _{i=1}^n\frac{p_i^{r+1}{q}_i^{-r}}{{\sum }_k{p}_k}=\frac{1}{r}log\sum _{i=1}^n\frac{p_i}{{\sum }_k{p}_k}{\left(\frac{p_i}{q_i}\right)}^r={\tilde{D}}_r\left(P_X\parallel Q_X\right)\hfill \end{array}$$

The Shannon entropy and Kullback–Leibler divergences are clearly the limit cases. □

3.1.1. The Case for Shifting the Rényi Entropy

So what could be the reason for the shifting? First and foremost, it is a re-alignment with the more basic concept of generalized mean.

Proposition 2.

The Shifted Rényi Entropy and Divergence are logarithmic transformations of the generalized power means:

$$\begin{array}{cc}\hfill {\tilde{H}}_r\left(P_X\right)& =log\frac{1}{M_r(P_X,P_X)}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\tilde{D}}_r\left(P_X\parallel Q_X\right)& =log{M}_r(P_X,\frac{P_X}{Q_X})\hfill \end{array}$$

(18)

Proof. 

Simple identification of (15) and (16) in the definition of power mean definitions (3). □

Table 1. Relation between the most usual weighted power means, Rényi entropies and shifted versions of them.

Mean Name	Mean ${\mathit{M}}_{\mathit{r}}(\overrightarrow{\mathit{w}},\overrightarrow{\mathit{x}})$	Shifted Entropy ${\tilde{\mathit{H}}}_{\mathit{r}}({\mathit{P}}_{\mathit{X}})$	Entropy Name	$\mathit{\alpha }$	r
Maximum	${max}_i{x}_i$	${\tilde{H}}_{\infty }=-log{max}_i{p}_i$	min-entropy	∞	∞
Arithmetic	${\sum }_i{w}_i{x}_i$	${\tilde{H}}_1=-log{\sum }_i{p}_i^2$	Rényi’s quadratic	2	1
Geometric	${\mathsf{\Pi }}_i{x}_i^{w_i}$	${\tilde{H}}_0=-{\sum }_i{p}_{i}log{p}_i$	Shannon’s	1	0
Harmonic	${({\sum }_i{w}_i\frac{1}{x_i})}^{-1}$	${\tilde{H}}_{-1}=logn$	Hartley’s	0	$-1$
Minimum	${min}_i{x}_i$	${\tilde{H}}_{-\infty }=-log{min}_i{p}_i$	max-entropy	$-\infty$	$-\infty$

lists the shifting of these entropies and their relation both to the means and to the original Rényi definition in the parameter $\alpha$.

Remark 5.

It is no longer necessary to make the distinction between the case $r\to 0$—Shannon’s—and the rest, since the means are already defined with this caveat. This actually downplays the peculiar features of Shannon’s entropy, arising from the geometric mean when ${\sum }_i{p}_i=1$:

$$\begin{array}{cc}\hfill {\tilde{H}}_0\left(p_x\right)& =log\frac{1}{M_0(P_X,P_X)}=-log\left(\prod _i{p}_i^{p_i}\right)=-\sum _i{p}_{i}log{p}_i\hfill \end{array}$$

However, the prominence of the Shannon entropy will emerge once again in the context of rewriting entropies in terms of each other (Section 3.2).

Since the means are properly defined for all $r\in [-\infty ,\infty ]$, ${\tilde{H}}_r\left(P_X\right)$ is likewise properly defined for all $r\in [-\infty ,\infty ]$—and therefore the non-shifted version with $\alpha =r+1$. This is probably the single strongest argument in favour of the shifting and motivates the following definition.

Definition 5

(The Rényi information spectrum). For fixed $P_X$ we will refer to ${\tilde{H}}_r\left(P_X\right)$ as its Rényi information spectrum over parameter r.

Also, some relationships between magnitudes are clarified in the shifted enunciation with respect to the traditional one, for instance, the relation between the Rényi entropy and divergence.

Lemma 3.

The shifted formulation makes the entropy the self-information with a change of sign:

$$\begin{array}{cc}\hfill {\tilde{H}}_r\left(P_X\right)& ={\tilde{D}}_{-r}\left(P_{XX}\parallel P_X{P}_X\right).\hfill \end{array}$$

(19)

Proof. 

${\tilde{D}}_{-r}\left(P_{XX}\parallel P_X{P}_X\right)={\tilde{D}}_{-r}\left(P_X\parallel P_X{P}_X\right)=\frac{-1}{r}log{\sum }_i{p}_i{\left(\frac{p_i}{p_i{p}_i}\right)}^{-r}=\frac{-1}{r}log{\sum }_i{p}_i{\left(\frac{1}{p_i}\right)}^{-r}={\tilde{H}}_r\left(P_X\right)$. □

Recall that in the common formulation, $H_{\alpha }(P_X)=D_{2-\alpha }(P_X\parallel P_X{P}_X)$ [23].

Another simplification is the fact that the properties of the Rényi entropy and divergence stem from those of the means, inversion and logarithm.

Proposition 3

(Properties of the Rényi spectrum of P_X). Let $r,s\in \mathbb{R}\cup \{\pm \infty \}$, and $P_X,Q_X\in {\Delta }^{n-1}$ where ${\Delta }^{n-1}$ is the simplex over the support supp X, with cardinal $\left|\mathrm{supp}X\right|=n$. Then,

1. 

(Monotonicity) The Rényi entropy is a non-increasing function of the order r.

$$\begin{array}{c}\hfill s\le r\Rightarrow {\tilde{H}}_s\left(P_X\right)\ge {\tilde{H}}_r\left(P_X\right)\end{array}$$

(20)

2. 

(Boundedness) The Rényi spectrum ${\tilde{H}}_r\left(P_X\right)$ is bounded by the limits

$$\begin{array}{c}\hfill {\tilde{H}}_{-\infty }\left(P_X\right)\ge {\tilde{H}}_r\left(P_X\right)\ge {\tilde{H}}_{\infty }\left(P_X\right)\end{array}$$

(21)

3. 

The entropy of the uniform pmf $U_X$ is constant over r.

$$\begin{array}{c}\hfill \forall r\in \mathbb{R}\cup \{\pm \infty \},{\tilde{H}}_r\left(U_X\right)=logn\end{array}$$

(22)

4. 

The Hartley entropy ($r=-1$) is constant over the distribution simplex.

$$\begin{array}{c}\hfill {\tilde{H}}_{-1}\left(P_X\right)=logn\end{array}$$

(23)

5. 

(Divergence from uniformity) The divergence of any distribution $P_X$ from the uniform $U_X$ can be written in terms of the entropies as:

$$\begin{array}{c}\hfill {\tilde{D}}_r\left(P_X\parallel U_X\right)={\tilde{H}}_r\left(U_X\right)-{\tilde{H}}_r\left(P_X\right).\end{array}$$

(24)

6. 

(Derivative of the shifted entropy) The derivative in r of Rényi’s r-th order entropy is

$$\begin{array}{c}\hfill \frac{d}{dr}{\tilde{H}}_r\left(P_X\right)=\frac{-1}{r^2}{\tilde{D}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)=\frac{-1}{r}log\frac{M_0({\tilde{q}}_r(P_X),P_X)}{M_r(P_X,P_X)},\end{array}$$

(25)

where ${\tilde{q}}_r(P_X)={\left\{\frac{p_i{p}_i^r}{{\sum }_k{p}_k{p}_k^r}\right\}}_{i=1}^n$ for $r\in \mathbb{R}\cup \{\pm \infty \}$ are the shifted escort distributions.

7. 

(Relationship with the moments of $P_X$) The shifted Rényi Entropy of order r is the logarithm of the inverse r-th root of the r-th moment of $P_X$.

$$\begin{array}{cc}\hfill {\tilde{H}}_r\left(P_X\right)& =-\frac{1}{r}log{E}_{P_X}\left\{P_X^r\right\}=log\frac{1}{\sqrt[r]{E_{P_X}\left\{P_X^r\right\}}}\hfill \end{array}$$

(26)

Proof. 

Note that properties used in the following are referred to the Proposition they are stated in. Property 1 issues from Property 1.2 and Hartley’s information function being order-inverting or antitone. Since the free parameter r is allowed to take values in $[-\infty ,\infty ]$, Property 2 follows directly from Property 1. With respect to Property 3, we have, from $U_X=1/\left|\mathrm{supp}X\right|=1/n$ and Property A1.3:

$${\tilde{H}}_r(\frac{1}{n})=-log{M}_r(\frac{1}{n},\frac{1}{n})=-log\frac{1}{n}=logn.$$

For Property 4 we have:

$${\tilde{H}}_{-1}(P_X)=-log{(\sum _i{p}_i·p_i^{-1})}^{-1}=-log{(n)}^{-1}=logn$$

While for Property 5,

$$\begin{array}{cc}\hfill {\tilde{D}}_r(P_X\parallel U_X)& =\frac{1}{r}log\left[\sum _i{p}_i{\left(\frac{p_i}{u_i}\right)}^r\right]=\frac{1}{r}log\left[\sum _i{p}_i{\left(\frac{p_i}{1/n}\right)}^r\right]=\frac{1}{r}log\left[n^r\left(\sum _i{p}_i{p}_i^r\right)\right]\hfill \\ & =logn+log{\left(\sum _i{p}_i{p}_i^r\right)}^{1/r}={\tilde{H}}_r(U_X)-{\tilde{H}}_r(P_X).\hfill \end{array}$$

For the third term of Property 6, we have from (17) with natural logarithm, with $P_X$ in the role both of $\overrightarrow{w}$ and $\overrightarrow{x}$

$$\begin{array}{c}\hfill \frac{d}{dr}{\tilde{H}}_r(P_X)=\frac{d}{dr}\left(-ln{M}_r(P_X,P_X)\right)=-\frac{\frac{d}{dr}{M}_r(P_X,P_X)}{M_r(P_X,P_X)},\end{array}$$

whence the property follows directly from (5). For the first identity, though, we have:

$$\begin{array}{cc}\hfill \frac{d{\tilde{H}}_r(P_X)}{dr}& =-\frac{d}{dr}\left[\frac{1}{r}ln\sum _i{p}_i{p}_i^r\right]=-\left[\frac{-1}{r^2}ln\sum _i{p}_i{p}_i^r+\frac{1}{r}\sum _i\frac{p_i{p}_i^r}{{\sum }_i{p}_i{p}_i^r}ln{p}_i\right].\hfill \end{array}$$

If we introduce the abbreviation

$$\begin{array}{cc}\hfill {\tilde{q}}_r(P_X)& ={\tilde{q}}_r(P_X,P_X)={\left\{{\tilde{q}}_r{(P_X)}_i\right\}}_{i=1}^n={\left\{\frac{p_i{p}_i^r}{{\sum }_k{p}_k{p}_k^r}\right\}}_{i=1}^n\hfill \end{array}$$

(27)

noticing that $ln{\sum }_k{p}_k{p}_k^r={\sum }_i{\tilde{q}}_r{(P_X)}_{i}ln({\sum }_k{p}_k{p}_k^r)$, since ${\tilde{q}}_r(P_X)$ is a distribution, and factoring out $-1/r^2$:

$$\begin{array}{cc}\hfill \frac{d{\tilde{H}}_r(P_X)}{dr}& =-\frac{1}{r^2}\left[-\sum _i{\tilde{q}}_r{(P_X)}_{i}ln(\sum _k{p}_k{p}_k^r)+r\left(\sum _i{\tilde{q}}_r{(P_X)}_{i}ln{p}_i\right)\pm \sum _i{\tilde{q}}_r{(P_X)}_{i}ln{p}_i\right]\hfill \\ & =-\frac{1}{r^2}\left[-\sum _i{\tilde{q}}_r{(P_X)}_{i}ln(\sum _k{p}_k{p}_k^r)+(r+1)\sum _i{\tilde{q}}_r{(P_X)}_{i}ln{p}_i-\sum _i{\tilde{q}}_r{(P_X)}_{i}ln{p}_i\right]\hfill \\ & =-\frac{1}{r^2}\left[\sum _i{\tilde{q}}_r{(P_X)}_{i}ln\frac{p_i{p}_i^r}{{\sum }_k{p}_k{p}_k^r}-\sum _i{\tilde{q}}_r{(P_X)}_{i}ln{p}_i\right]=-\frac{1}{r^2}\sum _i{\tilde{q}}_r{(P_X)}_{i}ln\frac{{\tilde{q}}_r{(P_X)}_i}{p_i}\hfill \end{array}$$

and recalling the definition of the shifted divergence we have the result.

For Property 7, in particular, the probability of any event is a function of the random variable $P_X(x_i)=p_i$ whose r-th moment of $P_X$ is

$$\begin{array}{c}\hfill E_X\left\{P_X^r\right\}=\sum _i{p}_i{p}_i^r={\left(M_r(P_X,P_X)\right)}^r\end{array}$$

(28)

The result follows by applying the definition of the shifted entropy in terms of the means. □

Remark 6.

In the preceding proof we have introduced the notion of shifted escort probabilities ${\tilde{q}}_r(P_X)$ acting in the shifted Rényi entropies as the analogues of the escort probabilities in the standard definition (see [5] and Section 2.1). This notion of shifted escort probabilities is the one requested by Property 1.6 by instantiation of variables ${\tilde{q}}_{}(P_X)={\tilde{q}}_{}(P_X,P_X)$. But notice also that ${({\tilde{q}}_r(P_X))}_i=\frac{p_i{p}_i^r}{{\sum }_k{p}_k{p}_k^r}=\frac{p_i^{\alpha }}{{\sum }_k{p}_k^{\alpha }}={(q_{\alpha }(P_X))}_i$ is just the shifting of the traditional escort probabilities [5].

Note that for $P_X\in {\mathbb{R}}_{\ge 0}^n$:

• ${\tilde{q}}_0(P_X)$ is the normalization of $P_X$. In fact, $P_X\in {\Delta }^{n-1}$ if and only if we have ${\tilde{q}}_0(P_X)=P_X$.
• ${\tilde{q}}_{-1}(P_X)(x_i)={\left|\mathrm{supp}{P}_X\right|}^{-1}$ if $x_i\in \mathrm{supp}{P}_X$ and 0 otherwise.
• Furthermore, if $P_X$ has P maxima (M minima), then ${\tilde{q}}_{\infty }(P_X)$ (${\tilde{q}}_{-\infty }(P_X)$) is an everywhere null distribution but at the indices where the maxima (minima) of $P_X$ are situated:

  $$\begin{array}{cccc}\hfill {\tilde{q}}_{\infty }(P_X)(x_i)& =\left\{\begin{array}{cc}\frac{1}{P}\hfill & x_i\in argmax{P}_X\hfill \\ 0\hfill & \mathit{otherwise}\hfill \end{array}\right.\hfill & \hfill {\tilde{q}}_{-\infty }(P_X)(x_i)& =\left\{\begin{array}{cc}\frac{1}{M}\hfill & x_i\in argmin{P}_X\hfill \\ 0\hfill & \mathit{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

Another important point made clear by this relation to the means is the fact that all positive measures have a Rényi spectrum: although so far we conceived the origin of information to be a probability function, nothing precludes applying the same procedure to non-negative, non-normalized quantities with ${\sum }_x{f}_X(x)\ne 1$, e.g., masses, sums, amounts of energy, etc.

It is well-understood that in this situation Rényi’s entropy has to be slightly modified to accept this procedure. The reason for this is Property 1.1 of the means: generalized means are 1-homogeneous in the numbers being averaged, but 0-homogeneous in the weights. In the Rényi spectrum both these roles are fulfilled by the pmf. Again the escort distributions allow us to analyze the measure:

Lemma 4.

Consider a random variable $X\sim M_X$ with non-normalized measure $M_X(x_i)=m_i$ such that ${\sum }_i{m}_i=M\ne 1$. Then the normalized probability measure ${\tilde{q}}_0(M_X)={\{m_i/{\sum }_i{m}_i\}}_{i=1}^n$ provides a Rényi spectrum that is displaced relative to that of the measure as:

$$\begin{array}{c}\hfill {\tilde{H}}_r\left(M_X\right)={\tilde{H}}_r\left({\tilde{q}}_0(M_X)\right)-logM.\end{array}$$

(29)

Proof. 

$$\begin{array}{cc}\hfill {\tilde{H}}_r\left({\tilde{q}}_0(M_X)\right)& =-log{M}_r({\tilde{q}}_0(M_X),{\tilde{q}}_0(M_X))=-\frac{1}{r}log\sum _i\frac{m_i}{M}{\left(\frac{m_i}{M}\right)}^r=logM-\frac{1}{r}log\sum _i\frac{m_i}{M}{m}_i^r\hfill \\ & =logM-log{M}_r(M_X,M_X)=logM+{\tilde{H}}_r\left(M_X\right)\hfill \end{array}$$

 □

Remark 7.

When $M\ge 1,-logM\le 0$ with equality for $M=1$ and that if $M<1$ then $-logM>0$. This last was the original setting Rényi envisioned and catered for in the definitions, but nothing precludes the extension provided by Lemma 4. In this paper, although $P_X$ can be interpreted as a pmf in the formulas, it can also be interpreted as a mass function as in the Lemma above. However, the escort probabilities are always pmfs.

Example 1.

This example uses the UCB admission data from [28]. We analyze the distribution of admissions with count vector $M_X={\left[933585918792584714\right]}^{\top }$ and probabilities ${\tilde{q}}_0(M_X)\approx {\left[0.210.130.200.170.130.16\right]}^{\top }$. The names of the departments are not important, due to the symmetry property. Figure 1a shows the Rényi Spectrum extrapolated from a sample of some orders which include $r\in \{-\infty ,-1,0,1,\infty \}$.

3.1.2. Shifting Other Concepts Related to the Entropies

Other entropy-related concepts may also be shifted. In particular, the cross-entropy has an almost direct translation.

Definition 6.

The shifted Rényi cross-entropy of order $r\in [-\infty ,\infty ]$ between two distributions $P_X(x_i)=p_i$ and $Q_X(x_i)=q_i$ with compatible support is

$$\begin{array}{cc}\hfill {\tilde{X}}_r\left(P_X\parallel Q_X\right)& =log\frac{1}{M_r(P_X,Q_X)}\hfill \end{array}$$

(30)

Note that the case-based definition is redundant: the Shannon cross-entropy appears as ${\tilde{X}}_0\left(P_X\parallel Q_X\right)=-log{M}_0(P_X,Q_X)=-{\sum }_i\frac{p_i}{{\sum }_k{p}_k}log{q}_i$, while for $r\ne 0$ we have ${\tilde{X}}_r\left(P_X\parallel Q_X\right)=-\frac{1}{r}log{\sum }_i{p}_i{\left(\frac{p_i}{q_i}\right)}^r$ by virtue of the definition of the means again.

Perhaps the most fundamental magnitude is the cross-entropy since it is easy to see that:

Lemma 5.

In the shifted formulation both the entropy and the divergence are functions of the cross-entropy:

$$\begin{array}{cccc}\hfill {\tilde{H}}_r\left(P_X\right)& ={\tilde{X}}_r\left(P_X\parallel P_X\right)\hfill & \hfill {\tilde{D}}_r\left(P_X\parallel Q_X\right)& ={\tilde{X}}_{-r}\left(P_X\parallel Q_X/P_X\right)\hfill \end{array}$$

(31)

Proof. 

The first equality is by comparison of definitions, while the second comes from:

$$\begin{array}{cc}\hfill {\tilde{D}}_r\left(P_X\parallel Q_X\right)& =\frac{1}{r}log\sum _i{p}_i{\left(\frac{p_i}{q_i}\right)}^r=-\frac{1}{-r}log\sum _i{p}_i{\left(\frac{q_i}{p_i}\right)}^{-r}={\tilde{X}}_{-r}\left(P_X\parallel Q_X/P_X\right)\hfill \end{array}$$

 □

Note that if we accept the standard criterion in Shannon’s entropy $0\times log\frac{1}{0}=0\times \infty =0$ then the previous expression for the cross-entropy is defined even if $p_i=0$.

3.2. Writing Rényi Entropies in Terms of Each Other

Not every expression valid in the case of Shannon’s entropies can be translated into Rényi entropies: recall from the properties of the Kullback–Leibler divergence its expression in terms of the Shannon entropy and cross-entropy. We have:

$$\begin{array}{cc}\hfill {\tilde{D}}_0\left(P_X\parallel Q_X\right)& =-{\tilde{H}}_0\left(P_X\right)+{\tilde{X}}_0\left(P_X\parallel Q_X\right),\hfill \end{array}$$

(32)

but, in general, ${\tilde{D}}_r\left(P_X\parallel Q_X\right)\ne -{\tilde{H}}_r\left(P_X\right)+{\tilde{X}}_r\left(P_X\parallel Q_X\right)$.

However, the shifting sometimes helps in obtaining “derived expressions”. In particular, the (shifted) escort probabilities are ubiquitous in expressions dealing with Rényi entropies and divergences, and allow us to discover the deep relationships between their values for different r’s.

Lemma 6.

Let $r\in \mathbb{R}\cup \{\pm \infty \}$, $P_X\in {\Delta }^{n-1}$ where ${\Delta }^{n-1}$ is the simplex over the support supp X. Then,

$$\begin{array}{cc}\hfill {\tilde{H}}_r\left(P_X\right)& =\frac{1}{r}{\tilde{D}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)+{\tilde{X}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill {\tilde{H}}_r\left(P_X\right)& =\frac{-1}{r}{\tilde{H}}_0\left({\tilde{q}}_r(P_X)\right)+\frac{r+1}{r}{\tilde{X}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)\hfill \end{array}$$

(34)

Proof. 

First, from the definitions of shifted Rényi entropy and cross-entropy and Property 3.6 we have:

$$\begin{array}{c}\hfill \frac{-1}{r^2}{\tilde{D}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)=\frac{1}{r}\left[{\tilde{H}}_r\left(P_X\right)-{\tilde{X}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)\right]\end{array}$$

Solving for ${\tilde{H}}_r\left(P_X\right)$ obtains the first result. By applying (32) to ${\tilde{q}}_r(P_X)$ and $P_X$ we have:

$$\begin{array}{cc}\hfill {\tilde{D}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)& =-{\tilde{H}}_0\left({\tilde{q}}_r(P_X)\right)+{\tilde{X}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right).\hfill \end{array}$$

(35)

and putting this into (33) obtains the second result.

Another way is to prove it is from the definition of

$$\begin{array}{cc}\hfill {\tilde{H}}_0\left({\tilde{q}}_r(P_X)\right)& =-\sum _i\frac{p_i{p}_i^r}{{\sum }_k{p}_k{p}_k^r}log\frac{p_i{p}_i^r}{{\sum }_k{p}_k{p}_k^r}=\sum _i{\tilde{q}}_r(P_X)log\left(\sum _k{p}_k{p}_k^r\right)-\sum _i{\tilde{q}}_r(P_X)log{p}_i^{r+1}\hfill \\ & =log\left(\sum _k{p}_k{p}_k^r\right)-(r+1)\sum _i{\tilde{q}}_r(P_X)log{p}_i=-r{\tilde{H}}_r\left(P_X\right)+(r+1){\tilde{X}}_0\left({\tilde{q}}_r(P_X)\parallel P_X\right)\hfill \end{array}$$

and reorganize to obtain (34). Again inserting the definition of the Shannon divergence in terms of the cross-entropy (35), into (34) and reorganizing we get (33). □

On other occasions, using the shifted version does not help in simplifying expressions. For instance skew symmetry looks in the standard case as $D_{\alpha }(P_X\parallel Q_X)=\frac{\alpha }{1-\alpha }{D}_{1-\alpha }(Q_X\parallel P_X)$, for any $0<\alpha <1$ ([23], Proposition 2). In the shifted case we have the slightly more general expression for $r\ne 0$:

Lemma 7.

When $Q_X$ is substituted by $P_X$, both probability distributions, on a compatible support, then:

$$\begin{array}{c}\hfill {\tilde{D}}_r\left(P_X\parallel Q_X\right)=-\frac{r+1}{r}·{\tilde{D}}_{-(r+1)}\left(Q_X\parallel P_X\right)\end{array}$$

(36)

Proof. 

By easy rewriting of the divergence ${\tilde{D}}_{-(r+1)}\left(Q_X\parallel P_X\right)$. □

3.3. Quantities Around the Shifted Rényi Entropy

On the one hand, the existence of Hartley’s information function (9) ties up information values to probabilities and vice-versa. On the other, Rényi’s averaging function and its inverse (14) also transform probabilities into information values and vice-versa. In this section we explore the relationship between certain quantities generated by these functions, probabilities and entropies.

3.3.1. The Equivalent Probability Function

Recall that, due to Hartley’s function, from every average measure of information, an equivalent average probability emerges. To see this in a more general light, first define the extension to Hartley’s information function to non-negative numbers ${\mathfrak{I}}_{*}(·):[0,\infty ]\to [-\infty ,\infty ]$ as ${\mathfrak{I}}_{*}(p)=-lnp$. This is one-to-one from $[0,\infty ]$ and total onto $[-\infty ,\infty ]$, with inverse ${\left({\mathfrak{I}}_{*}\right)}^{-1}(h)=e^{-h}$ for $h\in [-\infty ,\infty ]$.

Definition 7.

Let $X\sim P_X$ with Rényi spectrum ${\tilde{H}}_r\left(P_X\right)$. Then the equivalent probability function of ${\tilde{P}}_r\left(P_X\right)$ is the Hartley inverse of ${\tilde{H}}_r\left(P_X\right)$ over all values of $r\in [-\infty ,\infty ]$

$$\begin{array}{c}\hfill {\tilde{P}}_r\left(P_X\right)={\left({\mathfrak{I}}_{*}\right)}^{-1}({\tilde{H}}_r\left(P_X\right))\end{array}$$

(37)

Remark 8.

The equivalent probability function for a fixed probability distribution $P_X$ is a function of parameter r—like the Rényi entropy—whose values are probabilities—in the sense that it produces values in $[0,1]$—but it is not a probability distribution.

Analogously, due to the extended definition of the Hartley information, this mechanism, when operating on a mass measure $M_X$, generates and equivalent mass function ${\tilde{P}}_r\left(M_X\right)$, which is not a mass measure.

Lemma 8.

Let $X\sim P_X$. The equivalent probability function ${\tilde{P}}_r\left(P_X\right)$ is the Hölder path of the probability function $P_X$ (as a set of numbers) using the same probability function as weights.

$$\begin{array}{c}\hfill {\tilde{P}}_r\left(P_X\right)=M_r(P_X,P_X)\end{array}$$

(38)

Proof. 

From the definition, using b as the basis chosen for the logarithm in the information function.

$$\begin{array}{c}\hfill {\tilde{P}}_r\left(P_X\right)={\left({\mathfrak{I}}_{*}\right)}^{-1}({\tilde{H}}_r\left(P_X\right))=b^{-{\tilde{H}}_r\left(P_X\right)}=b^{{log}_b{M}_r(P_X,P_X)}=M_r(P_X,P_X)\end{array}$$

 □

Note that by Remark 8 these means apply, in general, to sets of non-negative numbers and not only to the probabilities in a distribution, given their homogeneity properties. In the light of Lemma 8, the following properties of the equivalent probability function are a corollary of those of the weighted generalized power means of Proposition 1 in Section 2.1.

Corollary 1.

Let $X\sim P_X$ be a random variable with equivalent probability function ${\tilde{P}}_r\left(P_X\right)$. Then:

1. 

For all $r\in [-\infty ,\infty ]$, there holds that

$$\begin{array}{c}\hfill \underset{k}{min}{p}_k={\tilde{P}}_{-\infty }\left(P_X\right)\le {\tilde{P}}_r\left(P_X\right)\le \underset{k}{max}{p}_k={\tilde{P}}_{\infty }\left(P_X\right)\end{array}$$

(39)

2. 

If $P_X\equiv U_X$ the uniform over the same $\mathrm{supp}$, then $\forall k,\forall r\in [-\infty ,\infty ],p_k={\tilde{P}}_r\left(U_X\right)=\frac{1}{|\mathrm{supp}{P}_X|}$.

3. 

if $P_X\equiv {\delta }_X^k$ the Kroneker delta centered on $x_k=X({\omega }_k)$, then ${\tilde{P}}_r\left({\delta }_X^k\right)=u(r)$ where $u(r)$ is the step function.

Proof. 

Claims 1 and 2 issue directly from the properties of the entropies and the inverse to the logarithm. The last claims follows from Remark 2. □

And so, in their turn, the properties of Rényi entropy can be proven from those of the equivalent probability function and Hartley’s generalized information function.

An interesting property might help recovering $P_X$ from the equivalent probability function:

Lemma 9.

Let $X\sim P_X$ be a random variable with equivalent probability function ${\tilde{P}}_r\left(P_X\right)$. Then: for every $p_k$ in $P_X$ there exists an $r_k\in [-\infty ,\infty ]$ such that $p_k={\tilde{P}}_{r_k}\left(P_X\right)$.

Proof. 

This follows from the continuity of the means with respect to its parameters $\overrightarrow{w}$ and $\overrightarrow{x}$. □

So if we could actually find those values $r_k,1\le k\le n$ which return $p_k={\tilde{P}}_r\left(P_X\right)$ we would be able to retrieve $P_X$ by sampling ${\tilde{P}}_{r_k}\left(P_X\right)$ in the appropriate values $P_X={\left\{{\tilde{P}}_{r_k}\left(P_X\right)\right\}}_{k=1}^n$. Since $n\ge 2$ we know that at least two of these values are $r=\pm \infty$ retrieving the value of the highest and lowest probabilities for $k=1$ and $k=n$ when they are sorted by increasing probability value.

Example 2

(Continued). Figure 1b shows the equivalent probability function of the example in the previous section. The dual monotone behaviour with respect to that of the Rényi spectrum is clearly observable. We have also plotted over the axis at $r=0$ the original probabilities of the distribution to set it in the context of the properties in Corollary 1 and Lemma 9.

3.3.2. The Information Potential

In the context of Information Theoretic Learning (ITL) the information potential is an important quantity ([29], Chapter 2).

Definition 8.

Let $X\sim P_X$. Then the information potential ${\tilde{V}}_r\left(P_X\right)$ is

$$\begin{array}{c}\hfill {\tilde{V}}_r\left(P_X\right)=E_{P_X}\left\{P_X^r\right\}=\sum _i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r\end{array}$$

(40)

Note that the original definition of the information potential was presented in terms of parameter $\alpha$ and for distributions with ${\sum }_k{p}_k=1$ in which case $V_{\alpha }(P_X)={\tilde{V}}_r\left(P_X\right)$. Now, recall the conversion function in (14) ${\phi }^{\prime }(h)=b^{-rh}$. The next lemma is immediate using it on (26).

Lemma 10.

Let $X\sim P_X$. The information potential is the ${\phi }^{\prime }$ image of the shifted Rényi entropy

$$\begin{array}{c}\hfill {\tilde{V}}_r\left(P_X\right)={\phi }^{\prime }({\tilde{H}}_r\left(P_X\right))=b^{-r{\tilde{H}}_r\left(P_X\right)}\end{array}$$

(41)

Proof. 

${\tilde{V}}_r\left(P_X\right)=b^{-r{\tilde{H}}_r\left(P_X\right)}=b^{{log}_b({\sum }_i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r)}={\sum }_i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r=E_{P_X}\left\{P_X^r\right\}$ ▯

Incidentally, (28) gives the relation of the information potential and the generalized weighted means.

Remark 9.

The quantity in the right-hand side of (40) is also the normalizing factor or partition function of the moments of the distribution and, as such, appears explicitly in the definition of the escort probabilities (27). Usually other partition functions appear in the estimation of densities based on overt, e.g., maximum entropy [6], or in covert information criteria—e.g., Ising models [5].

3.3.3. Summary

Table 2. Quantities around the shifted Rényi entropy of a discrete distribution $P_X$.

Quantity in Terms of…	Rényi Entropy	Gen. Hölder Mean	Information Potential	Distribution
Rényi entropy	${\tilde{H}}_r\left(P_X\right)$	$-log{M}_r(P_X,P_X)$	$\frac{-1}{r}log{\tilde{V}}_r\left(P_X\right)$	$\frac{-1}{r}log\left({\sum }_i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r\right)$
Gen. Hölder mean	$exp(-{\tilde{H}}_r\left(P_X\right))$	$M_r(P_X,P_X)$	${\left({\tilde{V}}_r\left(P_X\right)\right)}^{\frac{1}{r}}$	${\left({\sum }_i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r\right)}^{\frac{1}{r}}$
Information potential	$exp(-r{\tilde{H}}_r\left(P_X\right))$	$M_r{(P_X,P_X)}^r$	${\tilde{V}}_r\left(P_X\right)=E_{P_X}\left\{P_X^r\right\}$	${\sum }_i\frac{p_i}{{\sum }_k{p}_k}{p}_i^r$

offers a summary of the quantities mentioned above and their relationships, while the domain diagram in Figure 2 summarizes the actions of these functions to obtain the shifted Rényi entropy. A similar diagram is, of course, available for the standard entropy, using $\phi$ with the $\alpha$ parameter.

Note that these quantities have independent motivation: this is historically quite evident in the case of the means [24], and the Rényi information [12] and little bit less so in the case of the information potential which arose in the context of ITL [29], hence motivated by a desire to make Rényi’s entropies more useful. Both quantities are generated from/generate entropy by means of independently motivated functions, Hartley’s transformation (9) and Rényi’s transformation (14), respectively.

Following the original axiomatic approach it would seem we first transform the probabilities into entropies using Hartley’s function and then we use the ${\phi }^{\prime }$ function to work out an average of these using the Kolmogorov–Nagumo formula. But due to the formulas for the information potential and the equivalent probability function we know that this is rather a composition of transformations, than a forward backward moving between entropies and probabilities. It is clear that the Hartley function and Rényi’s choice of averaging function are special for entropies, from the postulate approach to their definition.

3.4. Discussion

A number of decisions taken in the paper might seem arbitrary. In the following, we try to discuss these issues as well as alternatives left for future work.

3.4.1. Other Reparameterization of the Rényi Entropy

Not only the parameter, but also de sign of the parameter is somewhat arbitrary in the form of (12). If we choose $r^{\prime }=1-\alpha$ another generalization evolves that is, in a sense, symmetrical to the shifted Rényi entropy we have presented above, since $r^{\prime }=-r$. This may be better or worse for the general formulas describing entropy, etc., but presents the problem that it no longer aligns with Shannon’s original choice of sign. The $r=0$ order Rényi entropy would actually be Boltzmann’s, negative entropy or negentropy [30] and perhaps more suitable for applications in Thermodynamics [5].

Yet another formulation suggests the use of $\alpha =1/2$, equivalently $r=-1/2$ as the origin of the parameter [31]. From our perspective, this suggests that the origin of the Rényi entropy can be chosen adequately in each application.

3.4.2. Rényi Measures and the Means

The usefulness of the (weighted) means in relation to information-theoretic concerns was already noted and explored in [32]. However, the relationship is not in there explicitly set out in terms of the identity of the Rényi entropies and logarithmic, weighted means of probabilities but rather as a part of establishing bounds for different quantities for discrete channel characterization.

A more direct approach is found in [33] that, inspired by [32], decides to generalize several results from there and other authors concerning the Rényi entropies, divergences and the Rényi centers of a set of distributions. Unlike our proposal, this deep work adheres to the standard definition of Rényi entropies of order $\alpha$ and avoids the issue of negative orders. The focus here is in coding and channel theorems, while ours is a re-definition of the mathematical concept to make similarities with weighted means transparent, yet evident.

3.4.3. Other Magnitudes around the Rényi Entropy

Sometimes the p-norm is used as a magnitude related to the Rényi entropy much as the information potential [29] or directly seeing the relationship with the definition [5].

Definition 9.

For a set of non-negative numbers $\overrightarrow{x}={\left[x_i\right]}_{i=1}^n\in {[0,\infty )}^n$ the p-norm, with $0\le p\le \infty$ is

$$\begin{array}{c}\hfill \parallel \overrightarrow{x}{\parallel }_p={\left(\sum _i{x}_i^p\right)}^{\frac{1}{p}}\end{array}$$

(42)

A more general definition involves both positive and negative components for $\overrightarrow{x}$, as in normed real spaces, but this is not relevant to our purposes for non-negative measures.

The p-norm has the evident problem that it is only defined for positive p whereas (14) proves that negative orders are meaningful and, indeed, interesting. A prior review of results for the negative orders can be found in [23].

We believe this is yet one more advantage of the shifting of the Rényi order: that the relation with the equivalent probability function and the information potential—the moments of the distribution—are properly highlighted.

3.4.4. Redundancy of the Rényi Entropy

Lemma 6 proves that Rényi entropies are very redundant in the sense that given its value for a particular $r_0$ the rest can be written in terms of those entropies with different, but systematically related, r order (see Section 3.4.4).

In particular, Equations (33) and (34) in Lemma 6, and (31) in Lemma 5 allow us to use a good estimator of Shannon’s entropy to estimate the Rényi entropies and related magnitudes for all orders, special or not. Three interesting possibilities for this rewriting are:

• That everything can be written in terms of $r=0$, e.g., in terms of Shannon’s entropy. This is made possible by the existence of estimators for Shannon’s entropy and divergence.
• That everything can be written in terms of a finite $r\ne 0$, e.g., $r=1$. This is possible by means of Properties 1.3 and 1.4 of the generalized power means. The work in [29] is pointing this way (perhaps including also $r=-1$, aka Hartley’s) capitalizing on the fact that Rényi’s entropy for data is well estimated for $r=1$, equivalently $\alpha =2$ ([29], Section 2.6).
• That everything can be written in terms of the extreme values of the entropy, e.g., $r=\pm \infty$. This is suggested by Properties 3.1 and 3.2. Supposing we had a way to estimate either ${\tilde{H}}_{-\infty }\left(P_X\right)$ or ${\tilde{H}}_{\infty }\left(P_X\right)$. Then by a divide-and-conquer type of approach it would be feasible to extract all the probabilities of a distribution out of its Rényi entropy function.

3.4.5. The Algebra of Entropies

Technically, the completed non-negative reals ${\mathbb{R}}_{\ge 0}$, where the means are defined, carry a complete positive semifield structure [34]. This is an algebra similar to a real-valued field but the inverse operation to addition, e.g., subtraction, is missing.

There are some technicalities involving writing the results of the operations of the extremes of the semifields—e.g., multiplication of 0 and ∞—and this makes writing closed expressions for the means with extreme values of $\overrightarrow{w}$ or $\overrightarrow{x}$ complicated. A sample of this is the plethora of conditions on Property 1.5. An extended notation, pioneered by Moreau [35], is however capable of writing a closed expression for the means [36].

Furthermore, taking (minus) logarithms and raising to a real power are isomorphism of semifields, so that the Rényi entropies inhabit a different positive semifield structure [36]. The graph of these isomorphic structures can be seen in Figure 2b. This means that some of the intuitions about operating with entropies are misguided. We believe that failing to give a meaning to the Rényi entropies with negative orders might have been caused by this.

3.4.6. Shifted Rényi Entropies on Continuous Distributions

The treatment we use here may be repeated on continuous measures, but the definitions of Shannon [10,21] and Rényi [26] entropies in such case run into technical difficulties solved, typically, by a process of discretization [27].

Actually we believe that the shifting would also help in this process: a form for the generalized weighted continuous means was long ago established [20] and technically solved by a change of concept and Lebesgue–Stieltjes integration instead of summation ([24], Ch. VI).

Our preliminary analyses show that the relationship with the means given by (17) also holds, and this would mean that the shifting—in aligning the Rényi entropies with the (generalized weighted) continuous means—leverages the theoretical support of the latter to sustain the former.

Definition 10

(Continuous weighted f-mean). Let $\mathsf{\Phi }(\xi )$ be a measure and let f be a monotonic function of ξ with inverse $f^{-1}$. Then a continuous version of (A2) is:

$$\begin{array}{cc}\hfill M_f(\mathsf{\Phi },\xi )& =f^{-1}\left\{\int f(\xi )d\mathsf{\Phi }(\xi )\right\}\hfill \end{array}$$

understood as a Lebesgue–Stieltjes integral.

This definition was already proposed by De Finetti [20] based upon the works of Bonferroni and Kolmogorov and thoroughly developed in ([24], Ch. VI) in connection to the discrete means. With $f(x)=x^r$ the continuous Hölder means $M_r(\mathsf{\Phi },\xi )$ appear. Furthermore De Finetti found ([20], #8) that the form of the f continuous, monotone function f must be

$$\begin{array}{ccc}\hfill f(x)& =a\int \gamma (x)dx+b\hfill & \mathrm{for}\mathrm{arbitrary}a,b(a\ne 0)\hfill \end{array}$$

similar to what Rényi found later for the Shannon entropy.

It is easy to see that an analogue definition of the shifted Rényi entropy but for a continuous probability density $p_X$ with $d{p}_X(x)=p_X(x)dx$ [5,27] is

$$\begin{array}{c}\hfill \tilde{h}(p_X)=\frac{-1}{r}log\int p_X(x)p_X^r(x)dx=-log{M}_r(p_X,p_X)\end{array}$$

(43)

again with the distribution acting as weight and averaged quantity. Compare this to one of the standard forms of the differential Rényi entropy [23]:

$$\begin{array}{c}\hfill h(p_X)=\frac{1}{1-\alpha }ln\int p_X^{\alpha }(x)dx\end{array}$$

The investigation of the properties of (43) is left pending for future work, though.

3.4.7. Pervasiveness of Rényi Entropies

Apart from the evident applications to signal processing and communications [29], physics [5] and cognition [11], the Rényi entropy is a measure of diversity in several disciplines [37]. We believe that, if its applicability comes from the same properties stemming from the means that we have explored in this paper as applied to positive distributions—e.g., of wealth in a population, or energy in a community—, then the expression to be used is (29).

4. Conclusions

In this paper we have advocated for the shifting of the traditional Rényi entropy order from a parameter $\alpha$ to $r=\alpha -1$. The shifting of the Rényi entropy and divergence is motivated by a number of results:

• It aligns them with the power means and explains the apparition of the escort probabilities. Note that the importance of the escort probabilities is justified independently of their link to the means in the shifted version of entropy [5].
• It highlights the Shannon entropy $r=0$ in the role of the “origin” of entropy orders, just as the geometric means is a particular case of the weighted averaged means. This consideration is enhanced by the existence of a formula allowing us to rewrite every other order as a combination of Shannon entropies and cross entropies of escort probabilities of the distribution.
• The shifting of the Rényi entropy aligns it with the moments of the distribution, thus enabling new insights into the moments’ problem.
• It makes the relation between the divergence and the entropy more “symmetrical”.
• It highlights the “information spectrum” quality of the Rényi entropy measure for fixed $P_X$.

The shifting might or might not be justified by applications. If the concept of the means is relevant in the application, we recommend the shifted formulation.