Estimating the variance of Shannon entropy

Leonardo Ricci, Alessio Perinelli, and Michele Castelluzzo

Phys. Rev. E 104, 024220 – Published 25 August 2021

Abstract

The statistical analysis of data stemming from dynamical systems, including, but not limited to, time series, routinely relies on the estimation of information theoretical quantities, most notably Shannon entropy. To this purpose, possibly the most widespread tool is provided by the so-called plug-in estimator, whose statistical properties in terms of bias and variance were investigated since the first decade after the publication of Shannon's seminal works. In the case of an underlying multinomial distribution, while the bias can be evaluated by knowing support and data set size, variance is far more elusive. The aim of the present work is to investigate, in the multinomial case, the statistical properties of an estimator of a parameter that describes the variance of the plug-in estimator of Shannon entropy. We then exactly determine the probability distributions that maximize that parameter. The results presented here allow one to set upper limits to the uncertainty of entropy assessments under the hypothesis of memoryless underlying stochastic processes.

Received 24 April 2021
Accepted 10 August 2021

DOI:https://doi.org/10.1103/PhysRevE.104.024220

Physics Subject Headings (PhySH)

Entropy Mathematical physics

Nonlinear DynamicsStatistical Physics & Thermodynamics

Authors & Affiliations

Leonardo Ricci ^1,2,*, Alessio Perinelli ², and Michele Castelluzzo ¹

¹Department of Physics, University of Trento, 38123 Trento, Italy
²CIMeC, Center for Mind/Brain Sciences, University of Trento, 38068 Rovereto, Italy

^*leonardo.ricci@unitn.it

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 104, Iss. 2 — August 2021

Reuse & Permissions

Access Options

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
Graphical representation of Eq. (20). The red, solid line corresponds to the graph of the function $f (v)$ defined in Eq. (19). The straight lines correspond to the right-hand term of Eq. (20) in the case $M = 7$ and for different values of the parameter $k$ . The blue dots mark the intersection of $f (v)$ with the straight line in the case $k = 1$ , as well as the two abscissae $- {\tilde{v}}_{k, 1}$ , ${\tilde{v}}_{k, 2}$ .
Reuse & Permissions
Figure 2
Maximum variance parameter $Λ_{0, max}$ as a function of the support size $M$ : (red, solid line) exact computation; (red, dashed line) evaluation via Eq. (24) by using the approximated value for $\tilde{v}$ given by Eq. (25); (black, dash-dotted line) approximation described by Eq. (26) and valid for $M ≳ 100$ . Outlier probability $p_{0}$ as a function of the support size $M$ : (blue, solid line) exact computation; (blue, dashed line) evaluation via Eq. (23) by using the approximated value for $\tilde{v}$ given by Eq. (25).
Reuse & Permissions
Figure 3
(a) Color map of the variance parameter $Λ_{0}$ on the probability simplex in the case $M = 3$ . Three points, located at one coordinate being equal to $p_{0} ≅ 0.88$ and the other two equal to $q_{0} ≅ 0.06$ , provide the maximum value of $Λ_{0}$ , namely $Λ_{0, max} ≅ 0.762$ . The curve placed on the plane $π_{s_{0}, s_{1}}$ represents the plot of $Λ_{0}$ on the simplex facet corresponding to $s_{2} = 0$ . (b) Color map of Shannon entropy $H$ on the probability simplex in the case $M = 3$ . The maximum value occurs in the center, where $s_{0} = s_{1} = s_{2} = 1 / 3$ and $Λ_{0}$ vanishes. No additional stationary points are present (which is true at any dimension).
Reuse & Permissions
Figure 4
Plug-in estimator $\hat{Λ_{0}}$ of the variance parameter $Λ_{0}$ as a function of the number of steps $N$ in the case of (a) the “arithmetic progression” distribution and (b) the “maximum variance” distribution. Each red (a) or blue (b) dot and the related error bar corresponds to the sample mean and the sample standard deviation, respectively, of $\hat{Λ_{0}}$ evaluated on a sample $S$ of $10^{4}$ simulated evolutions of $N$ steps. The black solid lines correspond to the expected value of $〈 \hat{Λ_{0}} 〉$ given by Eq. (27). For both panels (a) and (b), the insets show in log-log scale both the same numerical values and theoretical curves referred to the respective asymptotic theoretical values of $Λ_{0}$ [dashed lines: (a), $Λ_{0} ≅ 0.197$ ; (b), $Λ_{0} ≅ 1.246$ ]. Each magenta “ $\times$ ” symbol and the related error bar correspond to the sample mean and the sample standard deviation, respectively, of the coefficient of $N^{- 1}$ in Eq. (40) of Ref. [28] evaluated on the same sample $S$ .
Reuse & Permissions
Figure 5
Variance of the plug-in estimator $\hat{Λ_{0}}$ of the variance parameter $Λ_{0}$ as a function of the number of steps $N$ in the case of (red) the “arithmetic progression” distribution and (blue) the “maximum variance” distribution. Each dot corresponds to the sample variance of $\hat{Λ_{0}}$ evaluated on a sample $S$ of $10^{4}$ simulated evolutions of $N$ steps. The black solid line correspond to the expected value of $s_{\hat{Λ_{0}}}^{2}$ given by Eq. (28). In the case of the “maximum variance” distribution, the expected value of $σ_{\hat{Λ_{0}}}^{2}$ vanishes (see main text). The black dashed line corresponds to a $N^{- 2}$ power law. Finally, the red (blue) inset shows a plot of the “arithmetic progression” (“maximum variance”) distribution.
Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics