Skip to main content
SpringerLink
Log in

Generalized Lyapunov Exponents and Aspects of the Theory of Deep Learning

  • Conference paper
  • First Online:
New Trends in Lyapunov Exponents (NTLE 2022)

Part of the book series: CIM Series in Mathematical Sciences ((CIMSMS))

Included in the following conference series:

  • 97 Accesses

Abstract

We discuss certain recent metric space methods and some of the possibilities these methods provide, with special focus on various generalizations of Lyapunov exponents originally appearing in the theory of dynamical systems and differential equations. These generalizations appear for example in topology, group theory, probability theory, operator theory and deep learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akian, Marianne; Gaubert, Stéphane; Walsh, Cormac The max-plus Martin boundary. Doc. Math. 14 (2009), 195–240.

    Google Scholar 

  2. Avelin, B. Karlsson, A. Deep limits and cut-off phenomena for neural networks, J. Mach. Learn. Res. 23 (191) (2022) 29 pp

    Google Scholar 

  3. Barrett, David G.T.; Dherin, Benoit, Implicit Gradient Regularization, ICLR2021

    Google Scholar 

  4. Basteri, A.; Trevisan, D. Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks, arXiv:2203.07379 [cs.LG]

    Google Scholar 

  5. Bellman, Richard, Limit theorems for non-commutative operations. I. Duke Math. J. 21 (1954), 491–500.

    MathSciNet  MATH  Google Scholar 

  6. Benoist, Yves; Quint, Jean-François Random walks on reductive groups. Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics, 62. Springer, Cham, 2016. xi+323 pp.

    Google Scholar 

  7. Blath, Jochen; Hermann, Felix; Slowik, Martin, A branching process model for dormancy and seed banks in randomly fluctuating environments. J. Math. Biol. 83 (2021), no. 2, Paper No. 17, 40 pp.

    Google Scholar 

  8. Blumenthal, Alex A volume-based approach to the multiplicative ergodic theorem on Banach spaces. Discrete Contin. Dyn. Syst. 36 (2016), no. 5, 2377–2403.

    Google Scholar 

  9. Bowen, Lewis; Hayes, Ben; Lin, Yuqing, A multiplicative ergodic theorem for von Neumann algebra valued cocycles, Commun. Math. Phys. (2021), arXiv:2006.13293.

    Google Scholar 

  10. Cantat, Serge; Dujardin, Romain, Random dynamics on real and complex projective surfaces, arXiv:2006.04394

    Google Scholar 

  11. Corach, G.; Porta, H.; Recht, L. Convexity of the geodesic distance on spaces of positive operators. Illinois J. Math. 38 (1994), no. 1, 87–94.

    Article  MathSciNet  MATH  Google Scholar 

  12. Crauel, H.; Doan, T. S.; Siegmund, S. Difference equations with random delay. J. Difference Equ. Appl. 15 (2009), no. 7, 627–647.

    Article  MathSciNet  MATH  Google Scholar 

  13. Crisanti, A.; Paladin, G.; Vulpiani, A. Products of random matrices in statistical physics. With a foreword by Giorgio Parisi. Springer Series in Solid-State Sciences, 104. Springer-Verlag, Berlin, 1993. xiv+166 pp.

    Google Scholar 

  14. Dherin, Benoit; Munn, Michael; Rosca, Mihaela, Barrett, David G.T., Why neural networks find simple solutions: the many regularizers of geometric complexity, NeurIPS (2022)

    Google Scholar 

  15. Dyson,F.J. The dynamics of a disordered linear chain. Phys. Rev. 92, (1953) 1331–1338

    Article  MathSciNet  MATH  Google Scholar 

  16. Ebin, David G. On the space of Riemannian metrics. Bull. Amer. Math. Soc. 74 1968 1001–1003.

    Google Scholar 

  17. Eckmann, J.-P.; Ruelle, D. Ergodic theory of chaos and strange attractors. Rev. Modern Phys. 57 (1985), no. 3, part 1, 617–656.

    Google Scholar 

  18. Erschler, Anna Poisson-Furstenberg boundaries, large-scale geometry and growth of groups. Proceedings of the International Congress of Mathematicians. Volume II, 681–704, Hindustan Book Agency, New Delhi, 2010.

    Google Scholar 

  19. Eynard-Bontemps, Hélène, Navas, Andrés, (Arc-)connectedness for the space of smooth \({\mathbb {Z}}\)ˆd-actions on 1-dimensional manifolds, https://arxiv.org/abs/2103.06940

  20. Fathi, A, Laudenbach, F, and Poénaru, V. Travaux de Thurston sur les surfaces. Astérisque, 66-67. Société Mathématique de France, Paris, 1979. 284 pp.

    Google Scholar 

  21. Furstenberg, Harry Noncommuting random products. Trans. Amer. Math. Soc. 108 1963 377–428.

    Google Scholar 

  22. Glimm, James, A Stone-Weierstrass theorem for C*-algebras, Ann. of Math. 72 (1960), 216–244.

    Article  MathSciNet  MATH  Google Scholar 

  23. González-Tokman, Cecilia; Quas, Anthony A concise proof of the multiplicative ergodic theorem on Banach spaces. J. Mod. Dyn. 9 (2015), 237–255.

    Google Scholar 

  24. González-Tokman, Cecilia Multiplicative ergodic theorems for transfer operators: towards the identification and analysis of coherent structures in non-autonomous dynamical systems. Contributions of Mexican mathematicians abroad in pure and applied mathematics, 31–52, Contemp. Math., 709, Aportaciones Mat., Amer. Math. Soc., Providence, RI, 2018.

    Google Scholar 

  25. Gouëzel, S; Karlsson, A. Subadditive and multiplicative ergodic theorems, J. Eur. Math. Soc. 22 (2020), no. 6, 1893–1915.

    Article  MathSciNet  MATH  Google Scholar 

  26. Guivarc’h, Y. Sur la loi des grands nombres et le rayon spectral d’une marche aléatoire. (French) Conference on Random Walks (Kleebach, 1979) (French), pp. 47–98, 3, Astérisque, 74, Soc. Math. France, Paris, 1980.

    Google Scholar 

  27. Gutiérrez, Armando W.; Walsh, Cormac; Firm non-expansive mappings in weak metric spaces. Arch. Math. (Basel) 119 (2022), no. 4, 389–400.

    Google Scholar 

  28. Gromov, M. Hyperbolic manifolds, groups and actions. Riemann surfaces and related topics: Pro- ceedings of the 1978 Stony Brook Conference (State Univ. New York, Stony Brook, N.Y., 1978), pp. 183–213, Ann. of Math. Stud., 97, Princeton Univ. Press, Princeton, N.J., 1981.

    Google Scholar 

  29. Hanin, Boris; Nica, Mihai Products of many large random matrices and gradients in deep neural networks. Comm. Math. Phys. 376 (2020), no. 1, 287–322.

    Google Scholar 

  30. Hanin, B. Random Neural Networks in the Infinite Width Limit as Gaussian Processes, arXiv:2107.01562 [math.PR]

    Google Scholar 

  31. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Google Scholar 

  32. Horbez, Camille, The horoboundary of outer space, and growth under random automorphisms. Ann. Sci. Éc. Norm. Supér. (4) 49 (2016), no. 5, 1075–1123.

    Google Scholar 

  33. Iasemidis, L.D. Sackellares J.C. REVIEW: Chaos Theory and Epilepsy, The Neuroscientist, (1996) 2 issue 2, page(s): 118–126.

    Google Scholar 

  34. Jakub, C. Nica, M. Depth degeneracy in neural networks: Vanishing angles in fully connected ReLU Networks on Initialization, arXiv preprint arXiv:2302.09712

    Google Scholar 

  35. Kaimanovich, Vadim A.; Masur, Howard The Poisson boundary of the mapping class group. Invent. Math. 125 (1996), no. 2, 221–264.

    Google Scholar 

  36. Kaimanovich, Vadim A. The Poisson formula for groups with hyperbolic properties. Ann. of Math. (2) 152 (2000), no. 3, 659–692.

    Google Scholar 

  37. Kalinin, Boris; Sadovskaya, Victoria Periodic approximation of Lyapunov exponents for Banach cocycles. Ergodic Theory Dynam. Systems 39 (2019), no. 3, 689–706.

    Google Scholar 

  38. Karlsson, Anders, Linear rate of escape and convergence in directions, In: Proceedings of a Workshop at the Schrödinger Institute, Vienna 2001, (Ed. by V.A. Kaimanovich, in collab. with K. Schmidt, W. Woess) de Gruyter, 2004.

    Google Scholar 

  39. Karlsson, Anders, Two extensions of Thurston’s spectral theorem for surface diffeomorphisms. Bull. Lond. Math. Soc. 46 (2014), no. 2, 217–226.

    Article  MathSciNet  MATH  Google Scholar 

  40. Karlsson, Anders, Elements of a metric spectral theory, In: Dynamics, Geometry and Number Theory: the Impact of Margulis on Modern Mathematics The University of Chicago Press, 2022

    Google Scholar 

  41. Karlsson, Anders, From linear to metric functional analysis. Proc. Natl. Acad. Sci. USA 118 (2021), no. 28, Paper No. e2107069118, 5 pp.

    Google Scholar 

  42. Karlsson, Anders, A metric fixed point theorem and some of its applications, preprint 2022.

    Google Scholar 

  43. Karlsson, Anders; Ledrappier, François On laws of large numbers for random walks. Ann. Probab. 34 (2006), no. 5, 1693–1706.

    Google Scholar 

  44. Karlsson, Anders; Ledrappier, François Linear drift and Poisson boundary for random walks. Pure Appl. Math. Q. 3 (2007), no. 4, Special Issue: In honor of Grigory Margulis. Part 1, 1027–1036.

    Google Scholar 

  45. Karlsson, Anders; Ledrappier, François Propriété de Liouville et vitesse de fuite du mouvement brownien. C. R. Math. Acad. Sci. Paris 344 (2007), no. 11, 685–690.

    Google Scholar 

  46. Karlsson, Anders; Margulis, Gregory A., A multiplicative ergodic theorem and nonpositively curved spaces. Comm. Math. Phys. 208 (1999), no. 1, 107–123.

    Google Scholar 

  47. Karlsson, Anders, Monod, Nicolas, Strong law of large numbers with concave moments, unpublished note 2008

    Google Scholar 

  48. Kingman, J. F. C. The ergodic theory of subadditive stochastic processes. J. Roy. Statist. Soc. Ser. B 30 1968 499–510.

    Google Scholar 

  49. Kohlberg, Elon; Neyman, Abraham Asymptotic behavior of nonexpansive mappings in normed linear spaces. Israel J. Math. 38 (1981), no. 4, 269–275.

    Google Scholar 

  50. Lemmens, Bas, Horofunction compactifications of symmetric cones under Finsler distances, arXiv:2111.12468.

    Google Scholar 

  51. Lenzhen, Anna; Rafi, Kasra; Tao, Jing Bounded combinatorics and the Lipschitz metric on Teichmüller space. Geom. Dedicata 159 (2012), 353–371.

    Google Scholar 

  52. Li, Mufan Bill; Nica, Mihai; Roy, Daniel M. The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization, NeurIPS 2022

    Google Scholar 

  53. Lian, Zeng; Lu, Kening Lyapunov exponents and invariant manifolds for random dynamical systems in a Banach space. Mem. Amer. Math. Soc. 206 (2010), no. 967, vi+106 pp.

    Google Scholar 

  54. Lian, Zeng; Young, Lai-Sang Lyapunov exponents, periodic orbits, and horseshoes for semiflows on Hilbert spaces. J. Amer. Math. Soc. 25 (2012), no. 3, 637–665.

    Google Scholar 

  55. Maher, Joseph, Tiozzo, Giulio, Random walks on weakly hyperbolic groups, J. Reine Angew. Math. 742 (2018), 187–239.

    Article  MathSciNet  Google Scholar 

  56. Margulis, G. A. Discrete groups of motions of manifolds of nonpositive curvature. (Russian) Proceedings of the International Congress of Mathematicians (Vancouver, B.C., 1974), Vol. 2, pp. 21–34. Canad. Math. Congress, Montreal, Que., 1975

    Google Scholar 

  57. Martirosyan, D.; Nersesyan, V. Multiplicative ergodic theorem for a non-irreducible random dynamical system. J. Differential Equations 268 (2020), no. 7, 3564–3598.

    Article  MathSciNet  MATH  Google Scholar 

  58. Masai, Hidetoshi, Compactification and distance on Teichmüller space via renormalized volume, arXiv:2108.06059.

    Google Scholar 

  59. McCulloch, W and W. Pitts. A Logical Calculus of Ideas Immanent in Nervous Activity, Bull. Math. Biophys., 5 (1943) 115–133.

    Google Scholar 

  60. Narkhede, M.V.; Bartakke, P.P.; and Sutaone, M.S. A review on weight initialization strategies for neural networks, Artificial intelligence review 55 (2022), no. 1, 291–322.

    Article  Google Scholar 

  61. Neal, R.M. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.

    Google Scholar 

  62. Neusser, Klaus, Time–varying rational expectations models. J. Econom. Dynam. Control 107 (2019), 103731, 17 pp.

    Article  MathSciNet  MATH  Google Scholar 

  63. Oseledec, V. I. A multiplicative ergodic theorem. Characteristic Ljapunov, exponents of dynamical systems. (Russian) Trudy Moskov. Mat. Obšč. 19 1968 179–210.

    Google Scholar 

  64. Papadopoulos, Athanase; Troyanov, Marc Weak Finsler structures and the Funk weak metric. Math. Proc. Cambridge Philos. Soc. 147 (2009), no. 2, 419–437.

    Google Scholar 

  65. Ribeiro, Hugo Sur les espaces à métrique faible. Portugal. Math. 4 (1943), 21–40.

    Google Scholar 

  66. Frank Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain., Psychological review 65 (1958), no. 6, 386.

    Google Scholar 

  67. Ruelle, David Characteristic exponents and invariant manifolds in Hilbert space. Ann. of Math. (2) 115 (1982), no. 2, 243–290.

    Google Scholar 

  68. Sampaio, Luís Miguel, Regularity of the drift for random walks in groups acting on Gromov hyperbolic spaces, https://arxiv.org/abs/2109.13060

  69. Schaumlöffel, Kay-Uwe, Multiplicative ergodic theorems in infinite dimensions, In: Infinite Dimensional Random Dynamical Systems, Lyapunov Exponents, Springer Lecture Notes in Mathematics, volume 1486, (2006) pp 187–195

    MathSciNet  MATH  Google Scholar 

  70. Schmidt,H. Disordered one-dimensional crystals. Phys.Rev.105,425(1957) 425–441

    Google Scholar 

  71. Sejnowski, T.J. The unreasonable effectiveness of deep learning in artificial intelligence, Proc. Nat. Acad. Sci. U.S.A. December 1, 2020 117 (48) 30033–30038.

    Google Scholar 

  72. Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I. and Salakhutdinov, R.. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.

    MathSciNet  MATH  Google Scholar 

  73. Szpankowski, Wojciech Average case analysis of algorithms on sequences. With a foreword by Philippe Flajolet. Wiley-Interscience Series in Discrete Mathematics and Optimization. Wiley-Interscience, New York, 2001. xxii+551 pp.

    Google Scholar 

  74. Thieullen, P. Fibrés dynamiques asymptotiquement compacts. Exposants de Lyapounov. Entropie. Dimension. Ann. Inst. H. Poincaré Anal. Non Linéaire 4 (1987), no. 1, 49–97

    Article  MATH  Google Scholar 

  75. Thurston, William P. On the geometry and dynamics of diffeomorphisms of surfaces. Bull. Amer. Math. Soc. (N.S.) 19 (1988), no. 2, 417–431.

    Google Scholar 

  76. Walsh, Cormac The horoboundary and isometry group of Thurston’s Lipschitz metric. Handbook of Teichmüller theory. Vol. IV, 327–353, IRMA Lect. Math. Theor. Phys., 19, Eur. Math. Soc., Zürich, 2014.

    Google Scholar 

  77. Zheng, Tianyi, Asymptotic behaviors of random walks on countable groups. To appear in the Proceedings of ICM 2022.

    Google Scholar 

  78. Zou, Rui; Cao, Yongluo Livšic theorems for Banach cocycles: existence and regularity. J. Funct. Anal. 280 (2021), no. 5, Paper No. 108889, 37 pp.

    Google Scholar 

Download references

Acknowledgements

This text was written in connection with the conference “New Trends in Lyapunov Exponents” in Lisbon 2022. I thank the organizers and especially Pedro Duarte for the invitation to this very pleasant and stimulating week. I also thank Alex Blumenthal for helpful discussions related to the topics of this paper during this meeting.

The author was supported in part by the Swedish Research Council grant 104651320 and the Swiss NSF grants 200020-200400 and 200021-212864.

Author information

Authors and Affiliations

  1. Section de mathématiques, Université de Genève, Genève, Switzerland

    Anders Karlsson

  2. Mathematics Department, Uppsala University, Uppsala, Sweden

    Anders Karlsson

Authors
  1. Anders Karlsson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anders Karlsson .

Editor information

Editors and Affiliations

  1. Department of Mathematics, CEMAPRE/REM/ISEG, University of Lisbon, Lisbon, Portugal

    João Lopes Dias

  2. Department of Mathematics and CMAFcIO, Faculty of Sciences, University of Lisbon, Lisbon, Portugal

    Pedro Duarte

  3. Department of Mathematics, CEMAPRE/REM/ISEG, University of Lisbon, Lisbon, Portugal

    José Pedro Gaivão

  4. Department of Mathematics, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil

    Silvius Klein

  5. Department of Mathematics, CEMAPRE/REM/ISEG, University of Lisbon, Lisbon, Portugal

    Telmo Peixe

  6. Institute of Mathematics, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    Jaqueline Siqueira

  7. CMAT and Department of Mathematics, University of Minho, Braga, Portugal

    Maria Joana Torres

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Karlsson, A. (2023). Generalized Lyapunov Exponents and Aspects of the Theory of Deep Learning. In: Dias, J.L., et al. New Trends in Lyapunov Exponents. NTLE 2022. CIM Series in Mathematical Sciences. Springer, Cham. https://doi.org/10.1007/978-3-031-41316-2_5

Download citation

Publish with us

Policies and ethics

Search

Navigation