Emerging spectra characterization of catastrophic instabilities in complex systems

Random matrix theory has been widely applied in physics, and even beyond physics. Here, we apply such tools to study catastrophic events, which occur rarely but cause devastating effects. It is important to understand the complexity of the underlying dynamics and signatures of catastrophic events in complex systems, such as the financial market or the environment. We choose the USA S&P-500 and Japanese Nikkei-225 financial markets, as well as the environmental ozone system in the USA. We study the evolution of the cross-correlation matrices and their eigen spectra over different short time-intervals or ‘epochs’. A slight non-linear distortion is applied to the correlation matrix computed for any epoch, leading to the emerging spectrum of eigenvalues, mainly around zero. The statistical properties of the emerging spectrum are intriguing—the smallest eigenvalues and the shape of the emerging spectrum (characterized by the spectral entropy) capture the system instability or criticality. Importantly, the smallest eigenvalue could also signal a precursor to a market catastrophe as well as a ‘market bubble’. We demonstrate in two paradigms the capacity of the emerging spectrum to understand the nature of instability; this is a new and robust feature that can be broadly applied to other physical or complex systems.


Introduction
Critical transitions or sharp changes, which are usually unpredictable, are ubiquitous in complex systems found in nature and society-ranging from natural hazards, such as earthquakes, volcanic eruptions, hurricanes, lightning strikes, extreme weather conditions due to global warming [1][2][3], failure of physical and social structures due to large scale terror strikes [4,5], market crashes and economic slowdowns, electric grid failures, traffic breakdowns, disease and epidemic spreads, etc [6][7][8][9][10][11][12][13]. Such extreme events often reveal underlying dynamical processes and thus provide ground for better scientific understanding of complex systems like stock markets, fractures or earthquakes [14,15], climate, etc. The detailed evolution of complex systems may not have much significance; it may be more relevant to study certain phases of the evolution, like the rare critical events. A forecasting or prediction algorithm may be required for answering important questions related to rare events-predicting the occurrence of seismic events or temperature rise, forecasting crashes or bubbles, etc. Recently, scientists have therefore focused their attention on identifying generic indicators that may detect if a complex system is close to instabilities. Methods from self-organized criticality [16], networks [17], random matrix theory (RMT) [18,19], etc have been used extensively to model and analyze such complex systems [20].
RMT has been applied in various problems [21,22] such as in many-body physics [23], disordered systems and chaos [24], quantum chromodynamics [25], models for interacting fermions [26] and in other fields beyond physics, including financial markets [27][28][29][30][31][32]. Besides traditional methods, the RMT was proposed as a method for noise suppression in financial time series [33][34][35], which essentially proceeded as follows: the correlation matrices were diagonalized by an orthogonal transformation and then the bulk of the spectrum was eliminated; the inverse transformation reconstructed a singular correlation matrix with zero eigenvalues corresponding to the number of eigenvalues retained minus one. The power mapping introduced by Guhr and Kälber [36] was a less radical alternative; this was elaborated further in another paper [37]. The latter paper emphasized that the main advantage of the power mapping was not to omit all the information hidden in the statistics of the bulk. Using certain models, it was later demonstrated by Guhr and Wirtz [38], that the smallest eigenvalues, in particular, carry lot of information, making the highest eigenvalues and their eigenfunctions look like a rather rough approximation. Further applications of the power mapping were shown, e.g., in references [39,40] as well as in reference [41], when focusing on data-sets of short time series in the financial sector. The interest in eigenvalues for sets of N time series of length T, for N T, were mainly focused on the largest one (representing the market mode) or some of the larger ones (representing the market sectors), for the simple reason that there were only a few non-zero eigenvalues left. In reference [42], the authors discussed the use of the power mapping to break the linear dependence among the rows of the corresponding data matrices, leading to a large N − T + 1 dimensional subspace with eigenvalues in the ensuing correlation matrices. The power mapping would lift this degeneracy, even if the power was very close to unity. The resulting spectrum was well separated from the original one and was named as emerging spectrum. For very small deviations from the power unity, analytic results were given for Wishart matrices (white noise data) and numeric exploration of correlated Wishart ensembles were given, as well. Recently, in the context of market states [41], an intermediate view on power mapping for very short time series was adopted, where the significance of the largest eigenvalues were retained but the noise reduction was linked to the determination of optimal clustering choices. Following this idea, some extended studies have been recently done [19,[43][44][45][46].
Pharasi et al [19] had earlier studied the role of power mapping in noise suppression and detection of 'market states' using similarity measures between correlation matrices and corresponding multidimensional scaling maps. Pharasi et al [45] had also briefly reviewed the role of random matrix theory in the studying market dynamics, where they discussed the traditional Marcenko-Pastur distribution in WOE and CWOE, as well as the statistical properties of the emerging spectra arising in WOE and CWOE after application of power mapping.
Our overall aim is the detection of critical instabilities in complex systems using the emerging spectra. It must be mentioned that there is a closely related paper by Rinn et al [47], which explores such stabilities/instabilities in a dynamical way. It may also be mentioned that our approach to the analysis of complex dynamics could potentially be used in many different contexts such as in biological, ecological, and physiological systems [48][49][50][51][52].
In this paper, in the wider framework of RMT, we propose the usage of the shape of the 'emerging spectrum' (characterized by the spectral entropy [53]) as a simple yet generic indicator of critical periods, and the smallest eigenvalue as a signal for precursor to a market catastrophe and a market bubble. Importantly, the indicator can be bench-marked against the Wishart orthogonal ensemble (WOE) and it removes the arbitrariness of threshold, etc. We demonstrate its validity and robustness in two different complex systems.
As paradigm we first choose the stock market, which is a fascinating example of a complex system [54][55][56], where the coherent collective behavior of the economic agents and their repeated nonlinear interactions, often lead to rich structures of correlations and time-dependencies [27][28][29]. The movements in the market prices are often influenced by news or external shocks, which can result in the unforeseen and rapid drop in the prices of a large section of the stock market, labeled as a market crash! On the contrary, the widespread existence of bubbles in financial markets and extreme movements of price return series often result from the unstable relationship between macroeconomic fundamentals of the economy and the asset prices [57]. Since the societal impact of an extreme event like a market crash can be catastrophic [9,58], the understanding of such events [59], the assessment of the associated risks [60], and possible prediction of these events have drawn attention from all quarters: governments, industry participants and academia.
Second, we choose to study environmental pollution [61,62], which in the last few decades has reached alarming levels! One major component is air pollution, where harmful or excessive quantities of substances including gases and particles are introduced into earth's atmosphere [63,64]. According to the intergovernmental panel on climate change, top three most important air pollutants are carbon dioxide (CO 2 ), methane (CH 4 ), and tropospheric ozone (O 3 ) [65,66]. The tropospheric O 3 has been radically worsening the pollution and changing the climatic behavior [64,67]. The ozone pollution in the earth's environment is an example of a complex system-a product of complex interlinked networks of relationships between humans interactions and non-living objects surrounding them, as well as the cycles controlling the flows of chemical elements or compounds that support life and regulate climate. Polluting actions of humans often result in positive feedback loops that are known to destabilize an environmental system and drive it toward an extreme or critical state. We focus on the study of catastrophic instabilities in the environmental ozone pollution vis-à-vis the financial market.

Methodology and results
Our method relies on the time evolution of the cross-correlation matrices for N time-series and the eigen spectra over different time-epochs (of size M), as traditionally analyzed in RMT or in the analysis of adaptive complex systems like financial markets [19,41,45,[68][69][70]. The market returns series are constructed as is the adjusted closure price of stock i on day τ . Then the equal time Pearson correlation coefficients between stocks i and j is defined as where . . . represents the expectation computed over the time-epochs of size M and the day ending on τ , and σ k represents standard deviation of the kth stock evaluated for the same time-epochs. We use C(τ ) to denote the return correlation matrix for the time-epochs ending on day τ .
For this type of analysis, one assumes stationarity of the underlying time series. As this assumption manifestly fails for longer time series, it is often useful to break the long time series of length T, into n time-epochs of size M (such that T/M = n). The assumption of stationarity improves for the shorter time-epochs used. However, if there are N return time series such that N > M, this implies an analysis of highly singular correlation matrices with N − M + 1 zero eigenvalues, which lead to poor statistics. This problem can be avoided by using the non-linear 'power mapping', which was introduced to reduce noise [36,37,42]. It breaks the degeneracy of the zero eigenvalues also named as zero modes [71]. Thus, we apply a small non-linear distortion to the coefficients of the cross-correlation matrix: This breaks the degeneracy of zero-eigenvalues, giving rise to the emerging spectrum near zero [42,71] and the smallest eigenvalues are often negative! We had earlier used the power mapping to study the properties of markets and market states [19,45]. Here, we would like to focus on the evolution of the emerging spectra.   Lorentzian-like, with the smallest eigenvalue λ min = −0.014. Note, the insets of figure 1(c) and (e) elucidate that the emerging spectra are considerably different-slightly deformed semi-circular/Lorentzian-like as the market is normal/critical. Figures 2(a) and (b) show the evolution of the emerging spectra for the USA and JPN markets, respectively. Figures 3(a) and (b) show the statistical time series analyses: (i) market returns r(τ ), (ii) mean market correlation μ(τ ), (iii) smallest eigenvalue of the emerging spectrum λ min , (iv) t-value of the t-test, which tests if lag-1 smallest eigenvalue λ min (τ − 1) has statistically significant effect with mean market correlation μ(τ ), and (v) spectral entropy S ES . The mean of the correlation coefficients and the smallest eigenvalue in the emerging spectra are correlated (∼0.6) to a large extent. Notably, the smallest eigenvalue behaves differently (sharply rising or falling) at the same time when the mean market correlation is very high (crash). It is evident that for the USA, from 2001 onward, the financial market has become more turbulent and in case of JPN, from 1990 onward.

Linear regression model for market correlation on lagged smallest eigenvalue
In order to test the statistical significance of the correlation between last day's (lag-1) smallest eigenvalue λ min (τ − 1) and the current day's mean market correlation μ(τ ), we consider a linear regression model for μ (mean market correlation): where the β's are the coefficients to be estimated, ∼ N(0, σ 2 ), τ = 0, 1, 2, . . . , T is the white noise, and λ min (τ − p)'s are the lag-p smallest eigenvalues. The choice of p is arbitrary to some extent. We tried with values p = 1, 2, . . . , 5. For p = 1 and 2, we observed that almost all t-values are overwhelmingly large, especially because we considered a moving window approach. In the cases of p = 1 and p = 2, due to significant overlap over the windows, the large t-values resulted in too many false positives. On the other hand, when we chose p = 3 and above, the overlap between the windows reduced. As a result, the t-values become moderate as compared to p = 1 and 2 (see figure S6 in the supplementary information: http://stacks.iop.org/NJP/22/063043/mmedia). We also observed that the predictive powers for p = 3, 4 and 5 remained similar. Hence, we chose p = 3 as a parsimonious model choice. (i) market returns r(τ ), (ii) mean market correlation μ(τ ), (iii) smallest eigenvalue of the emerging spectrum λ min , (iv) t-value of the t-test, which tests if lag-1 smallest eigenvalue λ min (τ − 1) has statistically significant effect over mean market correlation μ(τ ), and (v) spectral entropy S ES . The spectral entropy S ES acts as the indicator and reflects the shape of the emerging spectra: high, when the shape resembles the semi-circular Wigner distribution (similar to WOE in red dash) and low, when the shape resembles Lorentzian-like. The S ES is undefined (indicated by pink vertical line) for the epoch that is heavy-tailed Lorentzian-like, which corresponds to a crash (catastrophic instability).
So, the null hypothesis and the alternate hypothesis can be stated mathematically as: The t-value for estimatedβ 1 is calculated as where se is the standard error in statistics. If |t − value| > 2, we can say that the last day's smallest eigenvalue (λ min (τ − 1)) has statistically significant effect over today's mean correlation (μ(τ )). The t-value itself signifies the strength of the signal and shown in figures 3(a) and (b). It seems that for most of the time, the correlation is statistically significant at 2σ levels or higher. The only periods when the λ min (τ − 1) fails to detect with high significance, are the broad periods 1990-91, 2000-02, etc, which act like bubbles/anomalies-the 'Dot-com bubble', 'Housing bubble', etc. Thus, smallest eigenvalue of the emerging spectrum can be effectively used for the characterization of market crashes and as a signal for market bubbles. We run the kernel density estimate to evaluate the normalized probability distribution function of the eigenvalues around zero (emerging spectrum), and compute the probabilities {p i }. We define the entropy of emerging spectrum S ES = − i p i ln p i , and use it to characterize the shape of the this spectrum. The shape of the emerging spectra is quantified by the S ES . When the system changes from normal to critical periods, the shape (distribution) of the emerging spectra changes from distorted Wigner semi-circle to heavy-tailed Lorentzian-like shape; the value of S ES changes from low (normal) to high (critical). We take the mean S ES for the WOE of equivalent size N, as benchmark. The spectral entropy S ES reflects the shape of the emerging spectra: high, when the shape resembles semi-circular Wigner distribution (red dash) and low, when the shape is Lorentzian-like. The S ES is undefined (indicated by pink vertical line) for the epoch that is heavy-tailed Lorentzian, which corresponds to a crash (catastrophic instability). The list of the major crashes is given in table 1. These features could shed light on detection of market anomalies/instabilities and designing appropriate market strategies, etc.  the maximum eigenvalue λ max = 50.015 (not plotted). Inset shows the emerging spectrum using power mapping ( = 0.01), which is heavy-tailed Lorentzian-like, with the smallest eigenvalue λ min = −0.007. Figure 5(a) shows the evolution of the emerging spectra from distorted semi-circle (non-critical period) to Lorenztian-like shape (critical period) with time. Figure 5(b) displays the variation of spectral entropy S ES , which reflects the shape of the emerging spectra: high (e.g., January 2014), when the shape resembles Wigner's semi-circle distribution (red dash) and low (e.g., September 2012), when the shape is Lorentzian-like. However, in ozone system, the mean correlation is never too high ( 0.35) and the Lorentzian-like peaks are low with tails less fat (compared to financial markets in figure 3), indicating the fact that the system never approaches critical state! It is interesting to note that the smallest eigenvalue λ min of the emerging spectra for the environmental ozone neither shows rapid fluctuations nor large changes as compared to market crashes in the financial market. Also, note that the relative changes in the maximum eigenvalue λ max is similar for both the cases, whereas for smallest eigenvalues λ min it is much bigger during the market crash. The spectral entropy S ES reflects the shape of the emerging spectra: high (e.g., January 2014), when the shape resembles semi-circular Wigner distribution (similar to WOE in red dash) and low (e.g., September 2012), when the shape is Lorentzian-like.

Discussions and summary
In summary, our study of the statistical properties of the emerging spectra illustrates for the first time that the shape of the emerging spectrum (captured by the spectral entropy) reflects the instability in a complex system. Also, the smallest eigenvalues of the emerging spectrum also contain significant information about the system's correlation structure. We used the emerging spectral entropy S ES , to quantify the shape of the emerging spectrum. When the system changed from normal to critical periods, the shape (distribution) of the emerging spectra changed from distorted semi-circular to heavy-tailed Lorentzian-like; the value of S ES changed from high (normal) to low (critical). In each case, we took the mean S ES for WOE of equivalent size N, as benchmark. As expected, the spectral entropy S ES never goes above this benchmark. For the financial market, we found that the S ES was undefined (indicated by pink vertical line) for the epoch which corresponds to a crash. For ozone data S ES is always defined. We found that we could retrieve all the major crashes as given in table 1. We also found a few extra instabilities or crashes (the false-positives), for which we could not get any information from internet sources (see table S4 in the supplementary information). In the financial market context, a striking and far reaching result found was that in certain instabilities/crashes the smallest eigenvalue of the emerging spectrum was positively correlated with the largest eigenvalue (and thus with the mean market correlation) rather than just a trivial anti-correlation. We further ran a linear regression model for the mean market cross-correlation μ(τ ) as function of the time-lagged smallest eigenvalue λ min (τ − 1), and found that the two variables have statistically significant correlation except during bubbles/anomalies, implying that an indicator function exists for bubbles which could signal a precursor to market catastrophe. It may be noted that we are interested in the time evolution of the statistical significance (measured by t-value) of the correlation between the variables, μ and λ min (τ − 1), and not the predicted values per se. The proposed regression model can be used for short term forecasting of market instability in a situation like the ongoing COVID-19. We would have to develop a thorough back-testing strategy (or out-of-sample testing strategy) for such forecasting models. However, these are beyond the scope of the present paper.
We also observed a lead-lag effect of the crashes across the globe in terms of the behavior of λ min in various markets (to be reported elsewhere). These features could shed light on detection of market anomalies/instabilities and designing appropriate market strategies, etc. In ozone system, the mean correlation is never too high ( 0.35) (see SI) and the Lorentzian-like peaks are low with less fat tails, which allow the S ES to be well defined (compared to financial markets in figure 3), indicating the fact that the system never approaches critical state. But the emerging spectral entropy does have high value (e.g., January 2014), when the shape resembles Wigner's semi-circle distribution (red dash) and low value (e.g., September 2012), when the shape is Lorentzian-like. So, it can indicate periods which have high variability (less mean correlation) and low variability (less mean correlation), in a computationally less intensive way, as compared to Bayesian spatial interpolation methods using MCMC.
In our proposed method, the only two parameters that can be varied are epoch size M and distortion parameter , which we have studied in details computationally with empirical and surrogate data in references [19,45]. We found that the S ES is not very sensitive to small values of the distortion parameter (see figure S7 in the supplementary information). Therefore, our method to identify and characterize critical periods or instabilities is very simple, computationally cheap, and general (without any arbitrary threshold). Further, we may mention that power mapping reduces the noise and spurious signals quite a bit (for large , as confirmed and reported in [19]). Using different choices of parameters (like window size, shift of windows, distortion parameter) we have observed that the fluctuations still manifest, and so in a sense they are independent of the arbitrariness in the choice of parameters. Using other techniques (see [19,74]) we have confirmed that the financial markets actually have become very volatile (with increasing number of sizable fluctuations), especially after the year 2002 in USA and 1990 in Japan. As mentioned earlier, in the history of financial crashes, not all cases of sizable fluctuations are recorded as market crashes, even though these correlation matrices bear similar signatures of crashes.
In conclusion, we note that the behavior of the spatial correlations of ozone pollutant have interesting and important properties that may help the policy makers on providing risk assessment decisions. On the other hand, it does not display the catastrophic instabilities as seen in financial markets. The new quantities discussed here allow detection and further characterization in simple numerical terms that was not previously available. Thus, these results are certainly of deep significance for the understanding of critical behavior in complex systems and risk management, but beyond that open a new window to the exploration of other complex systems that display catastrophic instabilities.