Numerical and experimental study of the effects of noise on the permutation entropy

We analyze the effects of noise on the permutation entropy of dynamical systems. We take as numerical examples the logistic map and the R\"ossler system. Upon varying the noise strengthfaster, we find a transition from an almost-deterministic regime, where the permutation entropy grows slower than linearly with the pattern dimension, to a noise-dominated regime, where the permutation entropy grows faster than linearly with the pattern dimension. We perform the same analysis on experimental time-series by considering the stochastic spiking output of a semiconductor laser with optical feedback. Because of the experimental conditions, the dynamics is found to be always in the noise-dominated regime. Nevertheless, the analysis allows to detect regularities of the underlying dynamics. By comparing the results of these three different examples, we discuss the possibility of determining from a time series whether the underlying dynamics is dominated by noise or not.


Introduction
Ordinal analysis is a method of time series analysis that consists of computing the probabilities of ordinal patterns, which are defined according to the ordering of D consecutive values in the series [1]. The entropy of these probabilities, referred to as permutation entropy, is a tool to detect possible regularities in the time series. In recent years, ordinal patterns and permutation entropy have been widely used to investigate complex dynamical systems [2,3]. They have been employed in the attempt to distinguish noise from chaos [4][5][6][7], to detect noise-induced order [8], serial correlations [9] and dependencies between two or more time series [10][11][12][13][14][15][16], among many other examples. Applications to experimental time series analysis include classification and discrimination of dynamical states in normal and epileptic EEG [17][18][19][20] and detection of heart rate variability under different physiological and pathological conditions [21][22][23].
Given this growing interest, it is relevant to understand the relation between the permutation entropy and other complexity measures. In particular, a well-established way to characterize the production of information of a dynamical system is the Kolmogorov-Sinai entropy h ks , see e.g. [24,25]. To compute h ks , the time series is discretized by partitioning the phase space into regions and assigning a symbol to each region. Then one computes the probabilities of blocks, which are vectors of D consecutive symbols (more details in the following sections). The entropy of the block probabilities is the block entropy. The Kolmogorov-Sinai entropy is finally obtained as the rate of growth, for D → ∞ and in the limit of a very refined partition, of the block entropy.
Similarly to h ks , one can introduce a permutation entropy rate as the rate of growth for D → ∞ of the permutation entropy. Both the permutation entropy and the Kolmogorov-Sinai entropy measure the "asymptotic" information rate of representations of the time series, the former with ordinal patterns (based in the relative order of consecutive values) and the latter with blocks (based on a partition of the phase space). The permutation entropy rate and h ks are not only conceptually related: for piecewise monotone interval maps on the real line, they were shown to be equal [26]. This result has been later extended to a broad class of dynamics [27,28]. This equivalence is non-trivial considering, for example, that the number of total possible ordinal patterns grows with D as D!, while the number of blocks grows as Q D where Q is the total number of symbols. The two quantities can be equal only thanks to the large number of forbidden ordinal patterns, strongly limiting the growth of the permutation entropy as D is increased.
These mathematical results clarify that, under general hypotheses, per-mutation and block entropies share the same asymptotic behavior. However, due to difficulties in reaching the asymptotic regime, this equivalence can be of little use in many practical cases. For example, it has been noted [26] that the rate of convergence of the permutation entropy to the Kolmogorov-Sinai entropy is extremely slow even for one-dimensional maps, while on the contrary, block entropies converge very quickly, see e.g. [24]. Comparing the two analyses becomes even more problematic for highdimensional and/or noisy dynamics, such as typical experimental time-series. Consider for example the extreme case of a time series dominated by noise, in which all symbols are equally probable and temporal correlations are absent. In this case, the block entropy of length D is equal to D ln(Q), where Q is the total number of symbols, while the permutation entropy with patterns of length D is equal to ln(D!) ∼ D ln D. This means that the block entropy is linear in D, with a slope ln(Q), explicitly dependent on the chosen partition, which diverges only in the limit of a very refined partition, Q → ∞. In contrast, the permutation entropy grows more than linearly, so that their asymptotic slope is infinite. In both cases, the result is an infinite entropy rate. However, to discover it, in the first case one needs to construct a very refined partition. In the second case, one needs to reach large values of D to appreciate that the slope increases logarithmically. Both these tasks can be very difficult when analyzing a finite time series due to statistical limitations.
Our goal is to get a better understanding of how noise influences the permutation entropy. To this aim, we analyze simulated and experimental time series. We mostly focus on permutation entropy as the effect of noise on block entropies is fairly well understood, see e.g. [29,30]. We first analyze time series generated from the logistic map and from Poincaré sections of the three-dimensional Rössler system. We conclude with an experimental example of output intensity data recorded from a semiconductor laser with optical feedback.

Numerical data
We consider two dynamical systems: the one-dimensional logistic map and the three-dimensional Rössler system. In both cases, we study the effect of adding to the dynamical equations a Gaussian white noise, ξ t with ξ t = 0 and temporal correlation ξ t ξ t ′ = δ t,t ′ . We considered also the case of observational noise (not shown), where the dynamics is deterministic but the noise affects the observation, obtaining very similar results.

Logistic map
where x t is the state of the system at iteration t and α is the noise strength.
In order to constrain the variable x t in the interval [0, 1], the values of ξ t that would lead to x t+1 > 1 or x t+1 < 0 are simply discarded and redrawn. Thus, the noise ξ t is temporally uncorrelated, but not purely Gaussian due to this truncation effect. To investigate the variation of the permutation entropy with the noise strength, we computed the permutation entropy, for each value of α, from time series of length N = 12×10 7 . We have also studied other nonlinear one-dimensional maps (Tent, Bernoulli and Quadratic) and obtained very similar results to those of the logistic map (results not shown).

Rössler system
The Rössler equations reaḋ where {X, Y, Z} are the states of the system at time t, α is the noise strength and {a, b, c} are the local parameters set at {0.1, 0.1, 18.0}, respectively.
In order to apply the symbolic methods (ordinal patterns or blocks) we need to discretize the dynamics. Instead of employing temporal sampling [32], we introduce a Poincaré section [33] at X = 0, and analyze the time intervals between consecutive crossings of the Poincaré plane. For each value of α, the permutation entropy is computed from time-series of N = 12 × 10 7 data points.

Experimental data
Experimental data was recorded from the output intensity of a semiconductor laser with optical feedback operating in the low-frequency fluctuations (LFFs) regime. In this regime, the laser intensity displays sudden and apparently random dropouts, followed by gradual recoveries. This spiking dynamics has received considerable attention because the intensity dropouts are induced by stochastic effects and deterministic nonlinearities. The optical feedback introduces a delay which renders the system in principle infinite dimensional. Therefore, the laser in the LFF regime generates complex fluctuations that, because of the stochastic and high-dimensional nature of the underlying dynamics, are suitable to be investigated by means of complexity measures such as the permutation and block entropies.
The experimental setup is the same as in [31] and uses a 650 nm AlGaInP semiconductor laser (SONY SLD1137VS) with optical feedback. The feedback was given through a mirror placed 70 cm apart from the laser cavity, with a round trip of 4.7 ns. The feedback was controlled using a neutral density filter that can adjusts the light intensity injected into the laser. The laser has a solitary threshold current of I th = 28.4 mA. The temperature and current of the laser were stabilized using a combi controller Thorlabs ITC501 with an accuracy of 0.01 C and 0.01 mA, respectively. The current used during the experiment was I = 29.3 mA and the temperature was set at T = 17 C. The neutral density filter was adjusted so that the threshold reduction due to feedback was about 7%. The signal was captured using a photo detector (Thorlabs DET210) connected to a FEMTO HSA-Y-2-40 amplifier and registered with a 1 GHz digital oscilloscope (Agilent Infiniium DSO9104A) with 0.2 ns of sampling. The intensity time series were acquired from the oscilloscope by a LabVIEW program that uses a threshold to detect the times when the intensity drops, and calculates the time intervals between successive threshold crossings (in the following, referred to as interdropout-intervals, IDIs). We recorded in this way time series of more than 10 5 consecutive IDIs. {01, 11, 10}. Then, the permutation entropy [1] or the block entropy [24,25] are simply the entropy of the frequency p i of the different patterns in the time series

Methods of analysis
where M is the number of possible patterns: for permutation entropy, M = D!, while for block entropy, M = Q D .  [26], which for a one-dimensional chaotic map is equal to the Lyapunov exponent λ. In the case of logistic map for a local parameter set at 4 one has λ = ln 2, indicated by the thick black line. As shown in detail in Fig. (2c), we identify three possibilities: In principle, this qualitative feature of the permutation entropy can be applied to experimental time series to assess whether the dynamics is dominated by noise or by the deterministic dynamics.

Results
We remind that this distinction can not be done for the block entropy, as in this case H D − H D−1 is necessarily a decreasing function of D (see e.g. [34,35]). This fundamental difference between permutation entropy and block entropy can be appreciated by comparing the left and right panels of Fig. (2).
For the analysis of the Rossler data, we considered the Poincaré map X = 0, shown in Fig. (3a), and analyzed the sequence of time-intervals between consecutive crossings. Figure ( Before testing the method in experimental data we want to investigate how the choice of the Poincaré section influences the results. We consider a Poincaré section in the plane Z = β, as shown in Fig. (4a), and varying β in the range [0.05 − 26.7], for a fixed value of α = 0. In this case, to discretize the time series, we analyze the time values when the trajectory intersects the Poincaré section and Z grows.    Fig. (3b).
Finally, we analyze experimental data from the laser output intensity, displayed in Fig. (5a). To discretize the data we consider the thresholds indicated with horizontal lines in Fig. (5a), and analyzed the time intervals between consecutive threshold-crossings [7,9,31]. Figure ( For all the thresholds, H D − H D−1 grows monotonically with D. The reason is that the empirical time series is very noisy and the "almost-deterministic" regime is not seen, not even for the highest threshold. Nevertheless, the values of H D − H D−1 lie outside the gray region that indicates values consistent with equally probable ordinal patterns. This reveals that the sequence of intensity dropouts are not completely uncorrelated, and thus, this method can determine regularities also in very noisy data.

Conclusion
We have studied the influence of noise in the permutation entropy of dynamical systems, considering both, simulated data and experimental data. In the simulated data, when increasing the noise strength, a transition between a almost-deterministic regime and a noise-dominated regime was clearly observed. The noise value at which this transition occurs is roughly independent of the size D of the ordinal pattern.
In the almost-deterministic regime, the permutation entropy grows almost linearly or sub-linearly with D. This behavior is qualitatively similar to that of the block entropy. However, to observe a quantitative equivalence it is often needed to analyze extremely long time series, which can be computationally unfeasible even for relatively simple dynamical systems. In the noise-dominated regime, the growth is faster than linear, i.e. the differences H D − H D−1 increase with D. In principle, this fact can be used to determine whether the dynamics is in a noise-dominated or a almost-deterministic regime from an experimental time series where the noise strength can not be externally tuned. However, care must be taken in interpreting the results, as extracting a one-dimensional time series from a purely deterministic high-dimensional time series via a Poincaré map can lead to ordinal patterns which look effectively noisy, as we demonstrated in the example of the Rössler systems. This fact reflects the well-known difficulties of distinguishing deterministic dynamics from noise when dealing with high-dimensional systems [29].