Taylor’s law for linguistic sequences and random walk models

Taylor’s law describes the fluctuation characteristics underlying a complex system in which the variance of an event within a time span grows by a power law with respect to the mean. Although Taylor’s law has been applied in many natural and social systems, its application for language has been scarce. This article describes a new, natural way to apply Taylor analysis to texts. The method was applied to over 1100 texts across 14 languages and showed how the Taylor exponents of natural language written texts were consistently around 0.58, thus being universal. The exponents were also evaluated for other language-related data, such as speech corpora (0.63 for adult speech, 0.68 for child-directed speech), programming language sources (0.79), and music (0.79). The results show how the Taylor exponent serves to quantify the fundamental structural complexity underlying linguistic time series. To explain the nature of natural language sequences possessing such different degrees of fluctuation, we investigated various mathematical models that could produce a Taylor exponent similar to that of real data. While the majority of previous probabilistic sequential models could not produce a Taylor exponent larger than 0.50, the same as in the independent and identically distributed (i.i.d.) case, random walk sequences on complex networks could produce fluctuation. We show that among various possibilities, random walks on a Barabási-Albert (BA) graph with small mean degree could fulfill the scaling properties of Zipf’s law and the long-range correlation, in addition to having a Taylor’s law exponent larger than 0.5, thus giving a new perspective to reconsider the nature of language.

For words that appear n(=Q) times in a text, there are two extreme possibilities: when the n words are all in the same segment, and on the other hand, when the n words are all in different segments.
When the n words all appear in the same segment, then σ becomes the largest for this case, as follows. Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence.
Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
On the other hand, when the n words all appear in different segments, then σ becomes the smallest, as follows.
n Q Q Q n n Q n n Q n Q n Q

Introduction
Good quantification of the degree of fluctuation underlying human texts has been an issue in the statistical physics domain. Previous studies used analysis methods based on either the long-range correlation (LRC) (Altmann et al 2009, Tanaka-Ishii and Bunde 2016, Lin and Tegmark 2016 or the variance (Ebeling and Pöeschel 1994). All these studies showed that language has power-law behavior due to underlying fluctuation, but none of them could quantify the exact degree or universality underlying language. One difficulty lies in the fact that language is not numerical, so the methods proposed in statistical mechanics, which were based on numerical data, require transformation of a linguistic sequence into a numerical sequence. One method proposed by Lin and Tegmark (2016) used mutual information and applies to text without such transformation, but it could not quantify the LRC underlying language (Takahashi and Tanaka-Ishii 2017). Hence, there is not only the problem of discovering fluctuation but also the question of the analysis method, i.e. whether it can quantify the underlying phenomena. Taylor's law characterizes how the standard deviation of the number of events for a given time and space grows with respect to the mean. Since the pioneering studies of this concept (Smith 1938, Taylor 1961), a substantial number of studies have applied Taylor's law across various domains, including ecology, life science, physics, finance, and human dynamics, as well summarized in (Eisler et al 2007). Despite such diverse application across domains, however, there has been little analysis based on Taylor's law in studying natural language. The only application, to the best of our knowledge, was shown in (Gerlach and Altmann 2014), but they measured the mean and variance of the vocabulary size within a document. This approach essentially differs from the original concept of Taylor analysis, which counts the number of events of every kind and not the Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence.
Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. number of kinds of events. Because of this, the theoretical background of Taylor's law, such as that presented in (Eisler et al 2007), cannot be applied to interpret their results.
In contrast to that previous report, the present paper applies Taylor's law in a novel way by examining the standard deviation of the number of occurrences of every word kind with respect to its mean within a given text length. While the Taylor exponent is analytically proven to be 0.5 for an independent and identically distributed (i.i.d.) process, our results show that the exponent is universally distributed around 0.58 for natural language written texts. Moreover, in verification across various kinds of human linguistic-related sequences, including the speech of infants and adults, programming language code, and music, the exponent could distinguish the kind of data. Previous versions of some of the content reported with this regard appeared originally in the proceedings of the Annual Conference for Computational Linguistics (Kobayashi and Tanaka-Ishii 2018). The content was updated to include results with additional datasets.
To understand the nature of these human linguistic sequences, we reconsidered various mathematical models that can produce a Taylor exponent larger than 0.5. While all the previous sequential models, including Markov and Simon models (Simon 1955), could not produce a Taylor exponent close to that of human language, some random walk models on a complex network graph could. Hence, we present the conditions for such models to fulfill the statistical laws of natural language.

The method
Given a set of elements W (words), let X=X 1 , X 2 , K, X N be a discrete time series of length N, where X i äW for all i=1, 2, K, N, i.e. each X i represents a word. For a given segment length t  D Î (a positive integer), a data sample X is segmented by the length Δt. The number of occurrences of a specific word w k äW is counted for every segment, and the mean μ k and standard deviation σ k across segments are obtained. Doing this for all word gives the distribution of σ with respect to μ, where W | |indicates the number of elements of the set W. Following a previous work (Eisler et al 2007), in this article Taylor's law is defined to hold when μ and σ are correlated by a power law in the following way: Experimentally, the Taylor exponent α is known to take a value within the range of 0.5 1.0   a across a wide variety of domains as reported in (Eisler et al 2007), including finance, meteorology, agriculture, and biology.
Mathematically, it is analytically proven that α=0.5 for an i.i.d process, and thus obviously for a Poisson process. On the other hand, one case with α=1.0 occurs when all segments always contain the same proportions of the elements of W. For example, suppose that W={a, b}. If b always occurs twice as often as a in all segments (e.g., three a and six b in one segment, two a and four b in another, etc.), then both the mean and standard deviation for b are twice those for a, so the exponent is 1.0.
In a real text, this cannot occur for all W, so α<1.0 for natural language text. Nevertheless, for a subset of words in W, this could happen, especially for a regular grammatical sequence. For instance, consider a programming statement: while (i < 1000) do i-. Here, the words while and do always occur once in this type of statement, whereas i always occurs twice. This example shows that the exponent indicates how consistently words depend on each other in W, i.e. how words co-occur systematically in a coherent manner, thus suggesting that the Taylor exponent is partly related to grammaticality.
To measure the Taylor exponent α, the mean and standard deviation were computed for every word kind 2 and then plotted in log-log coordinates. The number of points in this work was thus the number of different word kinds. We fitted the points to a linear function in log-log coordinates by the least-squares method and then estimated the Taylor exponent. The coefficient ĉ and exponent â were then estimated as the following: This fit function could be a problem depending on the distribution of errors between the data points and the regression line. As seen later, the error distribution seems to differ with the kind of data: for a random source the error seems Gaussian, and so the above formula is relevant. On the other hand, for real data the distribution is biased. Changing the fit function according to the data source, however, would cause other essential problems for fair comparison. Here, because most empirical works on Taylor's law used least-squares regression as reported in (Cohen and Xu 2015), this work also uses the above scheme, with the error defined as c,  a (ˆˆ).

Data
Our data consisted of natural language texts and language-related sequences. The details of the data are included in appendix A and can be roughly summarized as follows. The natural language written texts consisted of 1142 single-author long texts in various languages, taken from Project Gutenberg and other sources (long, singleauthor texts), and large-scale newspaper data in three languages (multiple-author texts). We also assembled other types of linguistic-related sequences. The enwiki8 100-MB dump dataset is tagged Wikipedia data. We also assembled speech data from both adults and children: the spoken record of Japan's National Diet (political content), and the 10 longest child-directed speech utterances in CHILDES (Child Language Data Exchange System, a well-studied children's speech archive), in 10 different languages. We also assembled four computer program sources and 12 music performance sources and preprocessed them to exclude annotations. For a reference, we took the text of Moby Dick and generated 10 different shuffled samples. Figure 1 shows a Taylor analysis of the distribution of words in Moby Dick. The horizontal axis indicates the mean frequency of a specific word within a segment of Δt=5620 words, whereas the vertical axis indicates the standard deviation. The choice of Δt is arbitrary but has to be sufficiently smaller than the document length. Among different values of Δt taken from logarithmic bins, we adopted a maximum Δt that could apply to all documents that we used. This resulted in a choice of Δt around 5000 words, specifically Δ t=5620≈10 3.75 .

Results
In the figure, the first graph shows the complete Taylor's law results for Moby Dick. The points were distributed around the regression line in a log-log graph, thus showing the power-law tendency. The exponent was α=0.57, with the standard deviation error of the estimate being ±0.000 9. The tendency of the points seemed to change around μ=1.0, which might seem a crossover, but this is doubtful according to the rationale shown by the second and third graphs in figure 1. These graphs show the regression for only the ranges of μ<1.0 and 1.0  m , respectively. The α value did increase when μ got larger, but very slightly, suggesting that the μ−σ plot did form almost a straight power law.
As explained previously, the Taylor exponent indicates the degree of consistent co-occurrence among words. The value of 0.57 obtained here suggests that the words of natural language texts are not so strongly or consistently coherent with respect to each other. Nevertheless, the value is well above 0.5.
Although the overall global tendency in figure 1 followed a power law, many points deviated significantly from the regression line. The words with the greatest fluctuation were often keywords. For example, among words in Moby Dick with large μ, those with the largest σ included whale, captain, and Ahab, among the book's most important keywords, whereas those with the smallest σ included functional words such as to, that, and all. The values of the standard deviation σ and mean μ for each word kind were plotted for a segment size of Δt=5620. The Taylor exponent obtained by the least-squares method was 0.57. The first graph shows the full picture for the whole range of μ. The second graph shows the fit to the points within the range of μ<1.0, and the third graph shows that within the range of 1.0  m . The second and third graphs indicate that α has the tendency to increase with respect to μ, but very slightly.
The same graphs for other natural language texts (two literary texts in Chinese and French, two newspapers in English and Chinese) are shown in appendix B figure B1. As seen in those graphs, the natural language written texts had almost the same exponent of around 0.58. Analysis of different kinds of data showed how the Taylor exponent differed according to the data source. Another set of graphs is shown in figure B2 for samples from enwiki8 (tagged Wikipedia), the child-directed speech of Thomas (taken from CHILDES), a programming language dataset, and a music source. The distributions appeared different from those of the natural language written texts, and the exponents were significantly larger. This suggests that those datasets contained expressions with fixed forms much more frequently than did the natural language texts. Figure 2 summarizes the overall picture among all data sources. Here, each plot represents one data source (text), and the median and quantiles of the Taylor exponents were calculated for the different kinds of data listed in table A1 of appendix A. The first plot shows the result for the shuffled data, with an exponent of 0.50. We will consider these randomized results more in the next section.
The remaining plots show the results for real data. For all the real data, not a single sample gave an exponent as low as 0.5. The exponents for the texts from Project Gutenberg ranged from 0.53 to 0.68. Appendix C includes a histogram (figure C1) of these texts with respect to the value of α. The number of texts decreased significantly at a value of 0.63, showing that the distribution of the Taylor exponent was rather tight. The kinds of texts in Project Gutenberg at the upper limit of the exponent included structured texts of fixed style, such as lists of histories, and Bibles. As described in more detail in appendix C, a non-parametric test indicated that different languages were not distinguishable by the value of α.
Turning to the last five columns of figure 2, representing enwiki8, speech data (the Diet's spoken record and child-directed speech), programming language, and music data, the Taylor exponents clearly differed from those of the natural language written texts. The enwiki8 dataset showed a higher exponent of 0.63, because of the wiki tags. For the Diet spoken record, another histogram in appendix C (figure C2) shows how tightly the points were located around the mean. Most importantly, the speech data showed higher exponents than did the written texts, with 0.63 for the Diet record and 0.68 for the child-directed speech. Finally, the programming and musical sources produced exponents near 0.79, higher than for any of the natural language data.
Overall, the results show the possibility of applying the Taylor exponent to quantify the structural complexity underlying language data. Grammatical complexity was formalized by Chomsky via his hierarchy (Chomsky 1956), which describes grammar through rewriting rules. The constraints placed on the rules distinguish four levels of grammar: regular, context-free, context-sensitive, and phrase structure. As indicated in (Badii and Politi 1997), however, this does not quantify complexity on a continuous scale. For example, we might want to quantify the complexity of child-directed speech as compared to that of adults, and this could be addressed in only a limited way through the Chomsky hierarchy. Another point is that the hierarchy is sentencebased and does not consider fluctuation in the kinds of words appearing.
The Taylor exponent depended on the two parameters of the experiments, namely, the segment size and data size. First, the exponent differed according to Δt, as shown in figure 3. For each kind of data shown in figure 2, the mean exponent was plotted for various Δt. As reported in (Eisler et al 2007), the exponent is known to grow Figure 2. Box plots of the Taylor exponents for different kinds of data. Each point represents one data sample, and samples from the same kind of data are contained in each box plot. The first plot is for the randomized data, while the remaining plots represent real data, including both the natural language texts and language-related sequences. Each box ranges between the quantiles, with the middle line indicating the median, the whiskers show 1.5 times lengths of the box size, and some extreme values lying beyond. The values indicated above or below the plots are the mean exponents for each data kind.
when the segment size gets larger. The reason is that words occur in a bursty, clustered manner at all Δt length scales: no matter how large the segment size becomes, a segment can include very many or few instances of a given word, leading to larger variance growth. This phenomenon suggests how word co-occurrences in natural language are self-similar across Δt.
The Taylor exponent was initially 0.5 when the segment size was very small. This can be analytically explained as follows (Eisler et al 2007). Consider the case of Δt=1. Let n be the frequency of a particular word in a segment. We then have n 1.0 á ñ  , because the possibility of a specific word appearing in a segment becomes very small. Because n á ñ 2 ≈0, n n n The robustness of figure 2 can be understood by vertically considering figure 3, which includes a dashed vertical line at Δt=5260 on the right side. The α value at this Δt for every document kind gives the mean values shown in figure 2. When Δt was varied, the mean α values would change. Because the plots in figure 3 for different data kinds do not intersect, however, for any Δt, α is able to distinguish the data kind.
As a summary, the importance of the α value is threefold, as follows, although α changes for a Δt: • For any real document, α>0.5 for any Δt>1.
• For a given Δt, α takes highly similar values for documents in a single category.
• α takes different values for documents in different categories.
In contrast, the exponent depended only slightly on the data size. Figure 4 shows the dependency for the two largest datasets used: the New York Times (NYT, 1.5 billion words) and Mainichi (24 years) newspapers. When the data size was increased, the exponent exhibited a slight tendency to decrease. For the NYT, the decrease seemed to have a lower limit, as the figure shows that the exponent stabilized at around 10 7 words.
The reason for this decrease can be explained as follows. The Taylor exponent becomes larger when some words occur in a clustered manner. Making the text size larger increases the number of segments (since Δt was fixed in this experiment). If the number of clusters does not increase as fast as the increase in the number of segments, then the number of clusters per segment becomes smaller, leading to a smaller exponent. In other words, the influence of each consecutive co-occurrence of a particular word decays slightly as the overall text size grows.

Analysis
We are interested in finding what kind of random sequence produces a Taylor exponent larger than 0.5, to better understand the nature of linguistic sequences.
Ways to produce random sequences that model language have been studied especially in the domain of language engineering. These include Markov models, Bayesian models, grammatical models, and neural models. Bayesian models have also been studied substantially in the domain of statistical mechanics and physics, Figure 3. Growth of α with respect to Δt, averaged across datasets within each data kind. Because this analysis required a large amount of computation, for the large datasets (such as newspaper and programming language data), 4 million words were taken from each kind of data and used here. When Δt was small, the Taylor exponent was close to 0.5, as theoretically described in the main text. As Δ t was increased, the value of α grew. The maximum Δ t was 10,000. For the kinds of data investigated here, α grew almost linearly. The results show that, at a given Δt, the Taylor exponent has some capability to distinguish different kinds of text data independently of Δt. The dashed vertical line on the right indicates Δ t=5620, which was the setting used to obtain many of the figures in this article. on the basis of the Simon process (Simon 1955). Previous works such as (Gerlach and Altmann 2013) explain statistical phenomena as extensions of the Simon process.
Our group (Takahashi and Tanaka-Ishii 2018) has also shown how all these models except neural models could not produce a Taylor exponent larger than 0.5. For example, figure 5 shows the Taylor analyses of texts generated by a first-order Markov model trained with the real Moby Dick and by the Simon process. The corresponding rows of table 1 list the details of the data. Both plots were tightly distributed around the regression line with an exponent of 0.5. (Takahashi and Tanaka-Ishii 2018) shows how the exponent cannot be improved by increasing the order of the model or by using advanced techniques.
That work also shows how neural network models can produce a Taylor exponent larger than 0.5. There are limitations, however, in the experimental approach. First, because neural models cannot handle a large vocabulary size, the experiments were all conducted by ignoring the word kind for the least frequent vocabulary.
Hence, although the report shows how the results followed Zipf's law in the region of frequent words, the most important tail part of Zipf's law was not considered. Overall, the final Taylor analysis plots were tightly distributed around the regression line, similarly to figure 5. Second, and above all, there is currently no explanation of why neural models work, and we still do not understand the nature of processes producing a large Taylor exponent.
Apart from neural models, no other language models are known to produce a sequence with a Taylor exponent larger than 0.5. In this article, we report a study of another genre of model: a random walk on a graph, in which successive visits to nodes are considered to form a sequence. Intuitively, a random walk can leave a node and come back to it, thus leading to fluctuation. Previously, (Eisler et al 2007) showed that a kind of random walk model on a Barabási-Albert (BA) graph produces a Taylor exponent larger than 0.5, provided that the random walk meets a specific condition. They assumed multiple random walkers simultaneously, so their approach would not form a single sequence. They excluded the special case of a single walker, and the condition for the random walk to give a Taylor exponent larger than 0.5 is unknown in this case.
In the rest of this paper, we consider using a random walk on a graph to produce a sequence with a Taylor exponent larger than 0.5. To evaluate the exponent's dependence on the text size, parts of each text were taken and the exponents were calculated for those parts, with points taken logarithmically. The window size was Δ t=5620. As the text size grew, the Taylor exponent slightly decreased.

Other scaling properties of language
As will be seen, because of the random walk's nature of going back and forth, a variety of graph settings can produce a Taylor exponent larger than 0.5. Human linguistic sequences, however, exhibit other properties such as the well-known Zipf's law and other power laws. We are interested in random walk models that explain the nature of human linguistic sequences, and we therefore also consider other scaling laws. Precisely, we seek the condition for a random walk on a graph to exhibit not only Taylor's law but also the following two additional scaling laws.
The first is Zipf's law, defined as follows. Let r be the rank of a particular word type and f (r) be its frequency. Zipf's law formulates a power-law relation between the frequency and rank: The second scaling property is the long-range correlation (LRC). A time series is said to be long-range correlated if a similarity function C(s) for two of its subsequences separated by a distance s follows a power law: Table 1. Taylor exponent, Zipf's law, and long-range correlation (LRC) results for real data and various language models. The models include both sequential ones (second block), based on a first-order Markov model (learned from Moby Dick) and the Simon process, and random walk models with various complex network graph structures (third block). The models listed in the second and third blocks are roughly described in the main text, with the details provided in appendix E. The last four columns in the table indicate the judgment of how well the sequence fulfilled each of the scaling laws. For Zipf's law, if the distribution exhibited a power law and the exponent was ξ≈1.0, then the result was judged 'Strong'; if the distribution exhibited a power law with a different exponent from ξ=1.0, then the result was judged 'Weak' (the cutoff was ξ<0.9); otherwise, the judgment was 'NA,' including cases of exponential decay. For the LRC, if the points plotted for the autocorrelation function (C(s) in formula (4), for the intervals described in appendix D) were aligned tightly around the regression line with no negative values before s=50, then the result was judged 'Strong'; if the plot merely presented some power-law tendency (with five values at maximum for our log-bin sampled s<50), then the result was judged 'Weak'; otherwise, the judgment was 'NA.' Finally, the column labeled 'Max scale of μ' indicates the largest μ in the Taylor's law graph as a power of 10. The table shows how strongly all scaling laws were fulfilled by real data (first block). The other models variously fulfilled some of the statistical laws, but only a limited random walk on a Barabási-Albert (BA) graph could fulfill all the laws. The red row indicates a case when all the laws were fulfilled, while the blue row indicates a case when only one law was weakly fulfilled. Both rows are for BA graphs. For these graphs, the max scale presented 3 because the length of walks was 1 million, much longer than Moby Dick (length is 25 thousand words).

Size
Length The most commonly studied similarity function is the autocorrelation function (ACF). As the ACF is applicable only for numerical time series, application of this method to natural language text requires transforming the text into a numerical series. Recent methods do so by considering the intervals of word occurrences (Altmann et al 2009, Tanaka-Ishii andBunde 2016). In this article, we apply the method in (Tanaka-Ishii and Bunde 2016), which is the only available method for quantifying the degree of the power law representing the LRC with respect to a linguistic sequence. The details of the method are described in appendix D. Long-range correlation analysis has a strong relation with Taylor's law, but the two methods capture different aspects underlying a sequence: LRC considers the self-resemblance, whereas fluctuation analysis methods, including Taylor's law, consider the variance of the number of events. Both methods produce a power law when a sequence fluctuates. Because of this difference, however, there are cases when one scaling law is fulfilled but not the other. For example, we saw that the Simon process gives a Taylor exponent of 0.5, but it produces a strong LRC because it increases the number of new words towards the end, leading to a strong similarity almost independently of the distance s (table 1, second row in the second block). As will be seen, the opposite case of a high Taylor exponent but no LRC occurs for random walks on graphs. Real data fulfills both scaling properties (table 1, first block), and we thus seek a random sequence that also fulfills both, in addition to Zipf's law.

Random walks on graph structures
Among various graph structures, in addition to a regular graph as a reference, we chose different structures from complex networks, as listed in the third block of table 1. The leftmost column shows the graph kind, as follows. The 'Moby Dick' graph was a graph obtained from the real data. The regular graphs had degree d for all nodes. The label G n,m indicates a graph with n nodes and m edges, while G n,p indicates a graph with n nodes and edge generation probability p for each sample. The label CWS indicates a small-world graph as defined by the Watts-Strogatz model (Watts and Strogatz 1998), connected with original degree k and edge replacement probability p, while NWS indicates an alternative of the CWS graph as defined by (Newman and Watts 1999). Finally, the label BA indicates a Barabási-Albert graph (Barabasi andReka 1999, Barabási 2016) with mean degree d. Because the BA generative algorithm extends the graph by adding a node with at least one edge, the minimum mean degree d=2.
Every graph was generated with n nodes, and s steps of a random walk were then conducted on the graph. Table 1 lists the results for 1,000,000 walks and 100,000 nodes (second and third columns) except for the Moby Dick graph. All the nodes were uniquely numbered. Every node received a random, unique ID number, and this numbering had no effect on the results for the scaling law analyses. When the walker visited a node, the number was output. This generated a number sequence, which was then subjected to scaling law analysis.
The first column of table 1 also indicates one of two different representative ways of walking, defined as follows.
• Random means that the walker chooses an edge uniformly and randomly, whereas • Preference means that the walker chooses a neighbor node in proportion to the neighbor's degree.
Other parameter values are listed in the fourth column. More detailed definitions and further explanation of every graph in the second and third blocks are included in appendix E.
The remaining columns indicate whether the random walk fulfilled each of the scaling laws. The values of the Taylor exponents are listed (seventh column), and the judgments with respect to Zipf's law, LRC, and Taylor analysis were made using the arbitrary criteria cutoffs detailed in the caption of table 1.
In the third block of the table, the first two rows show how the random walks on the graph acquired from Moby Dick had a Taylor exponent of α ; 0.5, almost like an i.i.d. process. We saw in the second block that the first-order Markov model acquired from Moby Dick behaved similarly. These results indicate that Markov models acquired from human linguistic sequences cannot produce fluctuation.
In contrast, other random walks on different graphs could produce a Taylor exponent larger than 0.50, including even the regular graphs. In contrast, considering Zipf's law and the LRC, only a few of the BA graphs could reproduce a sequence similar to the original Moby Dick. Moreover, comparing the LRC and α columns shows that when α was large, the LRC tended to hold. In the case of the original Moby Dick (first row), the LRC held strongly even though α was not so large, a combination that could be considered to characterize natural language as fulfilling both laws. Appendix F shows the Zipf's law, LRC, and Taylor exponent results for the two examples of random walks on BA graphs that best fulfilled the laws, which are highlighted in red and blue in table 1.
Given these results, we are especially interested in the random walks on BA graphs, because they apparently have a similar tendency to human linguistic sequences with respect to the scaling properties.

Conditions for BA graph to produce scaling properties
Next, the conditions for a BA graph to produce the three scaling properties were studied. The following three parameters are important: the mean degree of all nodes, d; the total number of nodes in the graph, n; and the number of walking steps, s. Note that n corresponds to the vocabulary, whereas s corresponds to the document length, in a sequence.
The most important factor is d. Figure 6 shows the value of α with respect to d for the case of n=s, with n being 1,000,000 or 100,000. The value of α decreased rapidly with respect to d. The random walk by randomly choosing edges gave higher values of α than by using the Preference method. The LRC held strongly only when d was 2 or slightly larger, whereas Zipf's law held only for the Preference case (the two plots at the bottom).
As seen in figure 6, when n equaled s, the graph size made little difference in the results. Therefore, we varied the ratio of s and n and examined the resulting values of α. Figure 7 shows the dependency of α on s/n, with the number of walks increasing to the right. The figure shows the case when d=2.
The results show that as more walks were made, α tended to decrease. This trend was shared with the results shown in figure 4 for how α slightly depended on the natural language text size. Similarly to figure 6, random walks by Preference presented far lower Taylor exponents as compared with those by the Random method. Zipf's law and the LRC were strongly present for all points in the Preference case. On the other hand, for the Random case, the Zipf's law's global exponent was not as steep as −1.0, although the LRC was consistently positive.

Discussion
We have seen that random walks on various graphs fulfill Taylor's law with a larger exponent than 0.5. A random walk on a graph structure fluctuates, because locally connected nodes co-occur through random walks. Because of this, many different experimental settings fulfilled the condition of the Taylor exponent being larger than 0.5, even with regular graphs. Many of these settings did not, however, fulfill the other scaling properties of language. The search for conditions under which a random walk could fulfill Zipf's law, the LRC, and Taylor's law suggest reconsideration of the nature of language. Only a Barabási-Albert (BA) graph with small mean degree d using Preference fulfilled all three scaling laws. Given that most previous language models do not fulfill the three scaling properties (Takahashi and Tanaka-Ishii 2018), our findings provide one possible alternative mathematical model.
Besides these good BA graph results, however, all random walks on a graph structure learned from Moby Dick (first row of the second block and first two rows of the third block of table 1) produced α ; 0.5. This suggests that linguistic sequences cannot be modeled by Markov models, which confirms both previous mathematical results (Lin and Tegmark 2016) and experimental results (Takahashi and Tanaka-Ishii 2018). The main reason is that the mean degree of the Markov models was large (above 10). These Markov graphs, however, form another BA graph, because the degree distribution is known to produce a power law. The closeness of the BA graphs and the Markov graphs generated from a real sequence raises a question of the relation between them.
In the language engineering domain, considering hidden states always contributes to increasing model quality by alleviating the low-frequency problem. For example, neural models, which are the sole model type to date that fulfills the three scaling laws (other than the random walk models discussed here), are also based on hidden states encoding the context. This fact suggests the idea that the nodes of a BA graph might not be direct words but could be some hidden states, and that these hidden states produce words in a probabilistic manner. This would alleviate the problem of a large mean degree (as semantically similar words would aggregate into single nodes).
As mentioned previously, neural models still have the problems of not being able to handle rare events. We consider this study on BA graphs to offer some prospect towards reconsidering the nature of language models. How exactly to reverse-engineer a BA graph language model from a sequence in the form of a language model remains for our future work.

Conclusion
We have shown how Taylor's law holds universally for human linguistic sequences and considered the nature of linguistic sequences via random walk models. With our method, a non-numerical discrete sequence is divided into segments of a given length, and the mean and standard deviation of the frequency of every event (word kind) are measured. Taylor's law is considered to hold when the standard deviation varies with the mean according to a power law, thus giving a value for the Taylor exponent.
Theoretically, an i.i.d. process has a Taylor exponent of 0.5, whereas larger exponents indicate sequences in which words co-occur systematically. Using over 1100 texts across 14 languages, we showed that natural language texts follow Taylor's law, with the exponent distributed around 0.58. This value differed greatly from the exponents for other data sources: enwiki8 (tagged Wikipedia, 0.63), speech data (Diet record, 0.63; child-directed speech, 0.68), programming language data (0.79), and music (0.79). None of the real data exhibited an exponent of 0.5, but it consistently showed a degree of fluctuation distinguishing the data kind.
To explain the statistical mechanics of human linguistic sequences, we sought a mathematical model that could produce a Taylor exponent larger than 0.5, in addition to fulfilling Zipf's law and the long-range correlation (LRC). Motivated by the greater likelihood of fluctuation in a random walk of a graph, we sought the conditions under which a random walk would fulfill Taylor's law. We found that random walks on a graph constructed from a first-order Markov model of real data behaved almost like an i.i.d. process, whereas random walks on a Barabási-Albert (BA) graph with small mean degree could fulfill the scaling laws of natural language. Our findings provide another alternative perspective to consider the nature of human linguistic activities. Table A1 lists all the data used for the article. The data consisted of natural language texts and language-related sequences, listed as different blocks in the table.
The natural language texts consisted of 1142 single-author long texts (first block) extracted from Project Gutenberg and Aozora Bunko across 14 languages. These are the texts above a size threshold (1 megabyte) in the two archives. The second block lists individual samples taken from Project Gutenberg. The third block lists newspaper data taken from the Gigaword corpus, available from the Linguistic Data Consortium in English, Chinese, and other major languages.
Other types of linguistic sequences appear in the fourth block. The enwiki8 100-MB dump dataset consists of tag-annotated text from English Wikipedia. The speech data consisted of the spoken record of Japan's National Diet, and the 10 longest child-directed speech utterances in the CHILDES (Child Language Data Exchange System) database, which was preprocessed by extracting only children's utterances (MacWhinney 2000, Bol 1995, Lieven et al 2009, Rondal 1985, Behrens 2006, Gil and Tadmor 2007, Oshima-Takane et al 1995, Smoczynska 1985, Anđelković et al 2001, Benedet et al 2004, Plunkett and Strömqvist 1992. Four program sources (in Lisp, Haskell, C++, and Python) were crawled from large representative archives, parsed, and stripped of natural language comments. Finally, 12 pieces of musical data (long symphonies and so forth) were transformed from MIDI into text with the software SMF2MML (available at http://shaw.la.coocan.jp/ smf2mml/), with annotations removed. Figure B1 shows typical distributions for natural language written texts, with two single-author texts in Chinese and French, and two multiple-author texts in English and Chinese. The segment size was t 5620 D = words. The points at the upper right represent the most frequent words, whereas those at the lower left represent the least frequent. The exponent α was almost the same even though English, French, and Chinese are different languages using different kinds of script. Figure B2 shows results for other kinds of data, listed in the fourth block of table A1. The segment size was again Δt=5620 words. The exponent became larger as the kind of text differed more from written natural language.

Appendix B. Taylor's law plots for different texts
Appendix C. Histograms of exponents for long texts from project Gutenberg and diet record Figure C1 shows a histogram of the Taylor exponents for the long texts taken from Project Gutenberg. Most of the texts were in English, followed by French, and then other languages, as listed in table A1. The histogram shows how the Taylor exponent ranged mainly between 0.55-0.63, and natural language texts with an exponent larger than 0.63 were rare.
Whether α distinguishes languages is a difficult question. The histogram suggests that Chinese texts exhibited larger values than did texts in Indo-European languages. We conducted a statistical test to evaluate whether this difference was significant as compared to English. Since the numbers of texts were very different, we used the nonparametric statistical test of the Brunner-Munzel method, among various possible methods, to test a null hypothesis of whether α was equal for the two distributions (Brunner and Munzel 2000). The p-value for Chinese was p=1.24×10 −16 , thus rejecting the null hypothesis at the significance level of 0.01. This confirms that α was generally larger for Chinese texts than for English texts. Similarly, the null hypothesis was rejected for Finnish and French, but it was accepted for German. It was also accepted, at the 0.01 significance level, for Japanese texts taken from Aozora Bunko (not shown in the histogram). Because the null hypothesis was accepted for Japanese despite its large linguistic difference from English, we could not conclude whether the Taylor exponent distinguishes languages. Overall, the exponent characterizes written texts in natural language but does not characterize languages. Figure C2 shows a histogram of the Taylor exponents for the spoken record of Japan's Diet (adult speech). The results show how the Taylor exponent ranged tightly around the mean.

Appendix D. Long-range correlation analysis method
A self-contained summary of the analysis scheme used for the long-Range correlation (LRC) is provided here, and a detailed argument for the method is found in (Tanaka-Ishii and Bunde 2016). The method basically uses the autocorrelation function to quantify the LRC. Given a numerical sequence R=r 1 , r 2 , K, r M , of length M, let the mean and standard deviation be μ and σ, respectively. Consider the following autocorrelation function: This is a fundamental function to measure the correlation, the similarity of two subsequences separated by distance s: it calculates the statistical covariance between the original sequence and a subsequence starting from the sth offset element, standardized by the original variance of 2 s . For every s, the value ranges between −1.0 and 1.0, with C(0)=1.0 by definition. For a simple random sequence, such as a random binary sequence, the function gives small values fluctuating around zero for any s, as the sequence has no correlation with itself. The sequence is judged to be long-range correlated when C(s) decays by a power law, as specified in formula (3).
The essential problem lies in the fact that a language sequence is not numerical and thus must be transformed into some numerical sequence. The method of (Tanaka-Ishii and Bunde 2016) transforms a word Figure B1. Examples of Taylor's law for natural language texts. Hong Lou Meng and Les Misérables are Chinese and French literary texts, respectively, appearing in the second block of table A1 as examples of single-author texts. The two newspapers are representative of multiple-author texts, in English and Chinese, respectively. Each point represents a kind of word. The values of σ and μ for each word kind are plotted across texts within segments of size Δt=5620. The Taylor exponents obtained by the least-squares method were all close to the mean value of 0.58. Figure B2. Examples of Taylor's law for the alternative datasets listed in table A1: enwiki8 (tag-annotated Wikipedia), Thomas (longest dataset in CHILDES), Lisp source code, and the music of Bach's St Matthew Passion, all with Δt=5620. These datasets exhibited far larger Taylor exponents than did typical natural language written texts. sequence into a numerical sequence by using intervals of rare words. The following example demonstrates how this is done. Consider the target Romeo in the sequence 'Oh Romeo Romeo wherefore art thou Romeo.' Romeo has a one-word interval between its first and second occurrences, and the third Romeo occurs as the fourth word after the second Romeo. This gives the numerical sequence [1, 4] for this clause and the target word Romeo. The target does not have to be one word but can be any element in a set of words. For example, suppose that the target consisted of two words, the two rarest words in this clause: Romeo, and wherefore. Then, the interval sequence would be [1,1,3], because wherefore occurs right after the second Romeo, and the third Romeo occurs as the third word after wherefore. As rare words occur only in small numbers, consideration of multiple rare words serves to quantify their behavior as an accumulated tendency.
As a summary, the overall procedure is described as follows. Given a numerical sequence of length M, the number of intervals for one Nth of (rare) words is M N ≡M/N−1 3 . For the resulting interval sequence Figure C1. Histogram of Taylor exponents for long texts in Project Gutenberg (1129 texts). The mean value was 0.58. The legend indicates the languages, in frequency order. Each bar shows the number of texts with that value of α. Because of the skew of languages in the original conception of Project Gutenberg, most of the texts are in English, shown in blue, whereas texts in other languages are shown in other colors. The histogram shows how the Taylor exponent ranged fairly tightly around the mean, and natural language texts with an exponent larger than 0.63 were rare. Figure C2. Histogram of Taylor exponents for long texts taken from the Diet record (250 texts). The mean value was 0.63. Each bar shows the number of texts with that value of α. The histogram shows how the Taylor exponent ranged fairly tightly around the mean, within a range of±0.3. R N =r 1 , r 2 , K, r M N (where r i is the interval between the ith and (i+1)-th occurrence of a word in M N words), let the mean and standard deviation be μ N and σ N , respectively. Then the autocorrelation function is calculated for this R N , with M N , μ N , and σ N replacing M, μ, and σ, respectively, in formula (4).
For literary texts, C(s) takes positive values forming a power law (Tanaka-Ishii and Bunde 2016). The blue points in figure D1 represent the actual C s ( ) values in a log-log plot for a sequence of Moby Dick in its entirety. Following (Tanaka-Ishii and Bunde 2016), the values of s were taken up to M N /100 in a logarithmic bin, which is the limit for the resulting C(s) values to remain reliable according to previous fundamental research such as that reported in (Lennartz and Bunde 2009). For s larger than M N /100, the values of the points tended to decrease rapidly. The thick gray line represents the fitted power-law function, which shows that this degree of clustering decayed by a power law with exponent γ=0.317, with a slope error of erb=0.031 7 (the standard deviation of γ) and a fit error (residual) of err=0.001 52 per point. The points were fitted to a linear function in log-log coordinates by the least-squares method. The points were all positive within the chosen range of s.

Appendix E. Details of generative models used in analysis
The analysis included two types of models, namely, sequential and graph models. The second block of table 1 in the main text lists the sequential models, which gave a Taylor exponent of 0.5: • Moby Dick first-order Markov model is a model that generates a word, after a preceding word, with a probability defined as the relative frequency of occurrence in the original Moby Dick. Note that this is also a random walk on a graph, where a node is a word, and an edge is formed between a word and its subsequent word. The edge is directed, and the random walk is made in proportion to the number of times that the two words occur successively.
• Simon uses the Simon process (Simon 1955) to generate a subsequent element, given the past sequence. The process starts with an element sequence. With constant probability a, a new element is generated, whereas with probability 1−a, an element is sampled from the past sequence.
The third block lists the graph models, which often exhibited a Taylor exponent larger than 0.5: • Moby Dick graph models:The first two rows lists graph models constructed from the original Moby Dick. A word forms a node, with an edge indicating that the two words occur successively. Unlike the first-order Markov model mentioned above, the edge is undirected, and the random walk is conducted by either the Preference or Random method (as described in the main text). The resulting Taylor exponents were 0.51 and 0.50, respectively, suggesting that random walks on a Moby Dick graph cannot produce a Taylor exponent as large as that of the original text.
• Regular graphs are those in which all nodes have degree d.
• G n,m is a random graph with n nodes and m edges. Figure D1. Log-log plot of the autocorrelation function C(s) for Moby Dick. The autocorrelation function was applied to intervals, and in the figure, the x-axis indicates the offset s, and the y-axis indicates the value of C(s) for an interval sequence consisting of onesixteenth of all the words. The blue points represent the actual data, the thick gray line is the fitted power law, and the slope γ is shown in the upper right corner, with the residual denoted as err and the error of the slope as erb.
• G n p , is a random graph with n nodes connected with probability p. • CWS is a graphbased on the Watts-Strogatz model, with n nodes that are first connected to k nearest-neighbor nodes; then, with probability p, each edge is reconnected to another node that is connected (Watts and Strogatz 1998).
• NWS is a graph, based on Newman's variation of the WS graph, with n nodes that are first connected to k nearest-neighbor nodes; then, with probability p, a new edge is added for each edge Newman and Watts (1999).
• BA is a Barabási-Albert graph with mean degree d ( Barabasi andReka 1999, Barabási 2016). The graph starts with one node having one edge, and nodes are generated successively by adding them to the preexisting graph by preferential attachment with d/2 edges. Because of this algorithm, the graph's minimal degree is 2. When d is not an integer, edges are probabilistically chosen so that the mean degree becomes the specified value of d. For example, consider the rows with d=2.4 listed in table 1 of the main text. In that case, a node is added to the graph with either one or two edges so that the overall mean becomes d=2.4.
Appendix F. Zipf's law, long-range correlation, and Taylor's law of best models of table A1 Figures F1 and F2 show the rank-frequency distribution, long-range correlation (LRC), and Taylor analysis calculated for the two best random walk sequences discussed in this article. Figure F1 is for a BA model using Preference walk with d=2, which was the best model (listed in red in table 1). The sequence fulfilled all three properties in a reasonable way, although the LRC differed by having much faster decay of C(s). Figure F2 is for a BA model using Preference walk with d=2.4, which was the second best model (listed in blue in table 1). The sequence fulfilled Zipf's law and Taylor's law but was barely long-range correlated, which led to the judgment of 'Weak'.

ORCID iDs
Kumiko Tanaka-Ishii https:/ /orcid.org/0000-0003-1752-3951 Figure F1. Zipf's law, autocorrelation function, and Taylor's law graphs for a BA model using Preference walk with d=2 (listed in red in table 1). A random walk with this setting produced a sequence that fit Zipf's law and had a Taylor exponent larger than 0.5. The sequence was also long-range correlated, with no negative values up to s=50. Figure F2. Zipf's law, autocorrelation function, and Taylor's law graphs for a BA model using Preference walk with d=2.4 (listed in blue in table 1). A random walk with this setting produced a sequence that fit Zipf's law and had a Taylor exponent larger than 0.5. The sequence was barely long-range correlated, however, as some values up to s=50 were negative.