Skip to main content
Log in

Predictive models and generative complexity

  • Published:
Journal of Systems Science and Complexity Aims and scope Submit manuscript

Abstract

The causal states of computational mechanics define the minimal sufficient memory for a given discrete stationary stochastic process. Their entropy is an important complexity measure called statistical complexity (or true measure complexity). They induce the ɛ-machine, which is a hidden Markov model (HMM) generating the process. But it is not the minimal one, although generative HMMs also have a natural predictive interpretation. This paper gives a mathematical proof of the idea that the ɛ-machine is the minimal HMM with an additional (partial) determinism condition. Minimal internal state entropy of a generative HMM is in analogy to statistical complexity called generative complexity. This paper also shows that generative complexity depends on the process in a nice way. It is, as a function of the process, lower semi-continuous (w.r.t. weak-* topology), concave, and behaves nice under ergodic decomposition of the process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. P. Crutchfield and K. Young, Inferring statistical complexity, Phys. Rev. Let., 1989, 63: 105–108.

    Article  MathSciNet  Google Scholar 

  2. C. R. Shalizi and J. P. Crutchfield, Computational mechanics: Pattern and prediction, structure and simplicity, Journal of Statistical Physics, 2001, 104: 817–879.

    Article  MathSciNet  MATH  Google Scholar 

  3. N. Ay and J. P. Crutchfield, Reductions of hidden information sources, Journal of Statistical Physics, 2005, 120: 659–684.

    Article  MathSciNet  MATH  Google Scholar 

  4. P. Grassberger, Toward a quantitative theory of self-generated complexity, Int. J. Theor. Phys., 1986, 25: 907–938.

    Article  MathSciNet  MATH  Google Scholar 

  5. D. Zambella and P. Grassberger, Complexity of forecasting in a class of simple models, Complex Systems, 1988, 2: 269–303.

    MathSciNet  MATH  Google Scholar 

  6. H. Jänicke, et al., Multifield visualization using local statistical complexity, IEEE Transactions on Visualization and Computer Graphics, 2007, 13(6): 1384–1391.

    Article  Google Scholar 

  7. R. W. Clarke, M. P. Freeman, and N. W. Watkins, Application of computational mechanics to the analysis of natural data: An example in geomagnetism, Phys. Rev. E, 2003, 67(1): 016203.

    Article  Google Scholar 

  8. J. P. Crutchfield, The calculi of emergence: Computation, dynamics and induction, Physica D, 1994, 75: 11–54.

    Article  MATH  Google Scholar 

  9. W. Löhr and N. Ay, On the generative nature of prediction, Advances in Complex Systems, 2009, 12(2): 169–194.

    Article  MathSciNet  MATH  Google Scholar 

  10. W. Löhr and N. Ay, Non-Sufficient Memories that are Sufficient for Prediction, in Proceedings of Complex’2009, Shanghai, volume 4 part I of LNICST, pages 265–276. Springer, 2009.

  11. Alex Heller, On stochastic processes derived from Markov chains, Annals of Mathematical Statistics, 1965, 36: 1286–1291.

    Article  MathSciNet  MATH  Google Scholar 

  12. Wolfgang Löhr, Models of Discrete-Time Stochastic Processes and Associated Complexity Measures, University of Leipzig, 2010, http://nbn-resolving.de/urn:nbn:de:bsz:15-qucosa-38267.

  13. G. Keller, Equilibrium States in Ergodic Theory, London Mathematical Society, 1998.

  14. L. Dębowski, Ergodic decomposition of excess entropy and conditional mutual information, IPI PAN Reports, nr 993, 2006.

  15. W. Löhr, Properties of the statistical complexity functional and partially deterministic HMMs, Entropy, 2009, 11(3): 385–401.

    Article  MathSciNet  MATH  Google Scholar 

  16. S. Still, J. P. Crutchfield, and C. J. Ellison, Optimal Causal Inference, Informal publication, http://arxiv.org/abs/0708.1580, 2007.

  17. James P. Crutchfield and David P. Feldman, Regularities unseen, randomness observed: Levels of entropy convergence, Chaos, 2003, 13(1): 25–54.

    Article  MathSciNet  MATH  Google Scholar 

  18. W. Bialek, I. Nemenman, and N. Tishby, Predictability, complexity, and learning, Neural Computation, 2001, 13: 2409–2463.

    Article  MATH  Google Scholar 

  19. J. P. Crutchfield and N. H. Packard, Symbolic dynamics of noisy chaos, Physica D, 1983, 7: 201–223.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Additional information

This paper was recommended for publication by Editor Jing HAN.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Löhr, W. Predictive models and generative complexity. J Syst Sci Complex 25, 30–45 (2012). https://doi.org/10.1007/s11424-012-9173-x

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11424-012-9173-x

Key words

Navigation