Distinct flavors of Zipf's law and its maximum likelihood fitting: Rank-size and size-distribution representations

Álvaro Corral, Isabel Serra, and Ramon Ferrer-i-Cancho

Phys. Rev. E 102, 052113 – Published 10 November 2020

Abstract

In recent years, researchers have realized the difficulties of fitting power-law distributions properly. These difficulties are higher in Zipfian systems, due to the discreteness of the variables and to the existence of two representations for these systems, i.e., two versions depending on the random variable to fit: rank or size. The discreteness implies that a power law in one of the representations is not a power law in the other, and vice versa. We generate synthetic power laws in both representations and apply a state-of-the-art fitting method to each of the two random variables. The method (based on maximum likelihood plus a goodness-of-fit test) does not fit the whole distribution but the tail, understood as the part of a distribution above a cutoff that separates non-power-law behavior from power-law behavior. We find that, no matter which random variable is power-law distributed, using the rank as the random variable is problematic for fitting, in general (although it may work in some limit cases). One of the difficulties comes from recovering the “hidden” true ranks from the empirical ranks. On the contrary, the representation in terms of the distribution of sizes allows one to recover the true exponent (with some small bias when the underlying size distribution is a power law only asymptotically).

Received 11 November 2019
Accepted 18 October 2020

DOI:https://doi.org/10.1103/PhysRevE.102.052113

Physics Subject Headings (PhySH)

Scaling laws of complex systems

Data analysis

Interdisciplinary PhysicsStatistical Physics & Thermodynamics

Authors & Affiliations

Álvaro Corral ^1,2,3,4, Isabel Serra ^1,5, and Ramon Ferrer-i-Cancho⁶

¹Centre de Recerca Matemàtica, Edifici C, Campus Bellaterra, E-08193 Barcelona, Spain
²Departament de Matemàtiques, Facultat de Ciències, Universitat Autònoma de Barcelona, E-08193 Barcelona, Spain
³Barcelona Graduate School of Mathematics, Edifici C, Campus Bellaterra, E-08193 Barcelona, Spain
⁴Complexity Science Hub Vienna, Josefstädter Strasse 39, 1080 Vienna, Austria
⁵Computer Architecture and Operating Systems Group, Barcelona Supercomputing Center (BSC-CNS), E-08034 Barcelona, Spain
⁶Complexity and Quantitative Linguistics Lab, Departament de Ciències de la Computació, Universitat Politècnica de Catalunya, E-08034 Barcelona, Catalonia, Spain

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 102, Iss. 5 — November 2020

Reuse & Permissions

Access Options

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
Estimation of the probability mass function $n (z) / L_{tot}$ of the hidden variable $z$ (hidden rank) associated with type size $n$ for a synthetic sequence with discrete power-law distribution of $z$ , Eq. (14), with exponent $α = 1.2$ and length $L_{tot} = 10^{6}$ , together with the corresponding empirical rank-size relation $n (r) / L_{tot}$ . Both size-one bins and logarithmic bins are shown for $n (z) / L_{tot}$ . The solid line is the original discrete power law from which the tokens are drawn. Notice how the only available quantity in practice, the rank-size relation, deviates from a power law for large ranks (corresponding to intermediate and large values of $z$ ). This deviation (apart from the bias in the ML-estimated exponent) is responsible for the rejection of the power-law hypothesis. (a) Full plot. (b) Restricted plot.
Reuse & Permissions
Figure 2
ML exponent $\hat{α}$ from the rank-size representation in the continuous approximation, as a function of the cutoff in rank $r_{a}$ , for synthetic systems fulfilling Zipf's law for types, Eq. (14), with original value of $α$ equal to 1.2 and different values of $L_{tot}$ . Observe the increasing positive bias from the true value with increasing $r_{a}$ . In all cases, the Kolmogorov-Smirnov test rejects the power-law hypothesis with the ML-estimated exponent. Different values of the simulated $α$ value lead to analogous results. The fitting of a discrete power law yields essentially the same results (not shown).
Reuse & Permissions
Figure 3
(a) Estimated probability mass function $f (n)$ and survivor function $S (n)$ for simulated Zipf's law for types, Eq. (14), taking $α = 1.2$ and $L_{tot} = 10^{6}$ (same data as in Fig. 1), together with the corresponding power-law ML fit for the random variable $n$ , Eq. (9), in the discrete case and also in the continuous approximation (lines). The fit in terms of $S (n)$ , given by Eq. (10), is also shown. We remark that ML estimation does not rely on the estimations of $f (n)$ of $S (n)$ , these are shown here as a visual verification of the goodness of the fits. Expression (8) provides a good fit of $f (n)$ for all $n$ (not shown). (b) Translation of the previous ML fit of the $n$ variable into the rank-size representation, given by the inverted Hurwitz zeta function of Eq. (11). The corresponding values of the upper cutoff $r_{b}$ turn out to be $r_{b} ≃ 5250$ (discrete fit) and $r_{b} ≃ 1350$ (continuous fit).
Reuse & Permissions
Figure 4
ML exponent from the distribution-of-sizes representation $\hat{γ}$ as a function of the cutoff in size $n_{a}$ , for a synthetic system fulfilling Zipf's law for types, Eq. (14), using the (exact) discrete fitting procedure. The original value of $α$ is equal to 1.2 (represented as a straight line for $γ = 1.833$ ) and $L_{tot} = 10^{6}$ (same data as in Fig. 3). The continuous fitting leads to similar results.
Reuse & Permissions
Figure 5
Same as Fig. 3 replacing the simulation of Zipf's law for types by the simulation of Zipf's law for sizes, Eq. (9), with $γ = 1 + 1 / 1.2 ≃ 1.83, n_{a} = 1$ , and $V_{tot} = 133 000$ . (a) Empirical cumulative distribution and probability mass function of sizes, together with discrete and continuous fits. (b) Empirical rank-size representation and fits.
Reuse & Permissions
Figure 6
Direct comparison of the estimated probability mass functions $f (n)$ arising from the simulation of both versions of Zipf's law, Eq. (14) [Fig. 3] and Eq. (9) with $n_{a} = 1$ [Fig. 5]. Notice the bending of $f (n)$ for small $n$ when Zipf's law is simulated for types.
Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics