Abstract
Loosely speaking, the Shannon entropy rate is used to gauge a stochastic process’ intrinsic randomness; the statistical complexity gives the cost of predicting the process. We calculate, for the first time, the entropy rate and statistical complexity of stochastic processes generated by finite unifilar hidden semi-Markov models—memoryful, state-dependent versions of renewal processes. Calculating these quantities requires introducing novel mathematical objects (\(\epsilon \)-machines of hidden semi-Markov processes) and new information-theoretic methods to stochastic processes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(379–423), 623–656 (1948)
Crutchfield, J.P., Young, K.: Inferring statistical complexity. Phys. Rev. Lett. 63, 105–108 (1989)
Shalizi, C.R., Crutchfield, J.P.: Computational mechanics: pattern and prediction, structure and simplicity. J. Stat. Phys. 104, 817–879 (2001)
Varn, D.P., Crutchfield, J.P.: Chaotic crystallography: how the physics of information reveals structural order in materials. Curr. Opin. Chem. Eng. 7, 47–56 (2015)
Kelly, D., Dillingham, M., Hudson, A., Wiesner, K.: A new method for inferring hidden Markov models from noisy time sequences. PLoS ONE 7(1), e29703 (2012)
Li, C.-B., Yang, H., Komatsuzaki, T.: Multiscale complex network of protein conformational fluctuations in single-molecule time series. Proc. Natl. Acad. Sci. USA 105, 536–541 (2008)
Li, C.-B., Komatsuzaki, T.: Aggregated Markov model using time series of a single molecule dwell times with a minimum of excessive information. Phys. Rev. Lett. 111, 058301 (2013)
Marzen, S., DeWeese, M.R., Crutchfield, J.P.: Time resolution dependence of information measures for spiking neurons: scaling and universality. Front. Comput. Neurosci. 9, 109 (2015)
Gonzalez, M.C., Hidalgo, C.A., Barabasi, A.-L.: Understanding individual human mobility patterns. Nature 453(7196), 779–782 (2008)
Witt, A., Neiman, A., Kurths, J.: Characterizing the dynamics of stochastic bistable systems by measures of complexity. Phys. Rev. E 55, 5050–5059 (1997)
Clarke, R.W., Freeman, M.P., Watkins, N.W.: Application of computational mechanics to the analysis of natural data: an example in geomagnetism. Phys. Rev. E 67, 016203 (2003)
Dzugutov, M., Aurell, E., Vulpiani, A.: Universal relation between the Kolmogorov-Sinai entropy and the thermodynamical entropy in simple liquids. Phys. Rev. Lett. 81, 1762 (1998)
Gonçalves, W.M., Pinto, R.D., Sartorelli, J.C., de Oliveira, M.J.: Inferring statistical complexity in the dripping faucet experiment. Physica A 257(1–4), 385–389 (1998)
Jay Palmer, A., Fairall, C.W., Brewer, W.A.: Complexity in the atmosphere. IEEE Trans. Geosci. Remote Sens. 38, 2056–2063 (2000)
Cerbus, R.T., Goldburg, W.I.: Information content of turbulence. Phys. Rev. E 88, 053012 (2013)
Kolmogorov, A.N.: Foundations of the Theory of Probability, 2nd edn. Chelsea Publishing Company, New York (1956)
Chaitin, G.: On the length of programs for computing finite binary sequences. J. ACM 13, 145 (1966)
Crutchfield, J.P.: Between order and chaos. Nat. Phys. 8(January), 17–24 (2012)
Akimoto, T., Hasumi, T., Aizawa, Y.: Characterization of intermittency in renewal processes: application to earthquakes. Phys. Rev. E 81, 031133 (2010)
Darmon, D., Sylvester, J., Girvan, M., Rand, W.: Predictability of user behavior in social media: bottom-up versus top-down modeling. arXiv:1306.6111
Yu, S.-Z.: Hidden semi-Markov models. Artif. Intell. 174(2), 215–243 (2010)
Girardin, V.: On the different extensions of the ergodic theorem of information theory. In: Baeza-Yates, R., Glaz, J., Gzyl, H., Husler, J., Palacios, J.L. (eds.) Recent Advances in Applied Probability Theory, pp. 163–179. Springer, New York (2005)
Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley, New York (2006)
Nair, C., Prabhakar, B., Shah, D.: On entropy for mixtures of discrete and continuous variables (2006). arXiv:cs/0607075
Marzen, S., Crutchfield, J.P.: Informational and causal architecture of continuous-time renewal processes. J. Stat. Phys. 168(1), 109–127 (2017)
Brookshear, J.G.: Theory of Computation: Formal Languages, Automata, and Complexity. Benjamin/Cummings, Redwood City (1989)
Victor, J.D.: Binless strategies for estimation of information from neural data. Phys. Rev. E 66(5), 051903 (2002)
Johnson, B.D., Crutchfield, J.P., Ellison, C.J., McTague, C.S.: Enumerating finitary processes. arXiv:1011.0036
Strelioff, C.C., Crutchfield, J.P., Hübler, Alfred: Inferring Markov chains: Bayesian estimation, model comparison, entropy rate, and out-of-class modeling. Phys. Rev. E 76(1), 011106 (2007)
Elliot, T.J., Gu, M.: Occam’s Vorpal Quantum Razor: Memory Reduction when Simulating Continuous-time Stochastic Processes with Quantum Devices (2017). arXiv:1704.04231
Acknowledgements
The authors thank Santa Fe Institute for its hospitality during visits, A. Boyd, C. Hillar, and D. Upper for useful discussions, and T. Elliott for the example of Fig. 2. JPC is an SFI External Faculty member. This material is based upon work supported by, or in part by, the U.S. Army Research Laboratory and the U. S. Army Research Office under Contracts W911NF-13-1-0390 and W911NF-12-1-0288. S.E.M. was funded by a National Science Foundation Graduate Student Research Fellowship, a U.C. Berkeley Chancellor’s Fellowship, and the MIT Physics of Living Systems Fellowship.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Marzen, S.E., Crutchfield, J.P. Structure and Randomness of Continuous-Time, Discrete-Event Processes. J Stat Phys 169, 303–315 (2017). https://doi.org/10.1007/s10955-017-1859-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10955-017-1859-y