Skip to main content
SpringerLink
Log in

Layered Neural Networks

  • Chapter
Models of Neural Networks I

Part of the book series: Physics of Neural Networks ((NEURAL NETWORKS))

  • 216 Accesses

Synopsis

Some of the recent work done on layered feed-forward networks is reviewed. First we describe exact solutions for the dynamics of such networks, which are expected to respond to an input by going through a sequence of preassigned states on the various layers. The family of networks considered has a variety of interlayer couplings: linear and nonlinear Hebbian, Hebbian with Gaussian synaptic noise and with various kinds of dilution, and the pseudoinverse (projector) matrix of couplings. In all cases our solutions take the form of layer-to-layer recursions for the mean overlap with a (random) key pattern and for the width of the embedding field distribution. Dynamics is governed by the fixed points of these recursions. For all cases nontrivial domains of attraction of the memory states are found. Next we review studies of unsupervised leaming in such networks and the emergence of orientation-selective cells. Finally the main ideas of three supervised leaming procedures, recendy introduced for layered networks, are oudined. All three procedures are based on a search in the space of intemal representations; one is designed for leaming in networks with fixed architecture and has no associated convergence theorem, whereas the other two are guaranteed to converge but may require expansion of the network by an uncontrolled number of hidden units.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. For a recent review of physicists’ contributions see W. Kinzel, Physica Scripta T25, 144 (1989)

    Article  ADS  Google Scholar 

  2. E. Domany, R. Meir and W. Kinzel: Europhys. Lett. 2, 175 (1986)

    Article  ADS  Google Scholar 

  3. R. Meir and E. Domany: Phys. Rev. Lett. 59, 359 (1987)

    Article  ADS  Google Scholar 

  4. R. Meir and E. Domany: Europhys. Lett. 4, 645 (1988)

    Article  ADS  Google Scholar 

  5. R. Meir and E. Domany Phys. Rev. A 37, 608 (1988)

    Article  ADS  Google Scholar 

  6. R. Meir: J. Phys. (Paris) 49, 201 (1988)

    Article  MathSciNet  Google Scholar 

  7. B. Derrida and R. Meir: Phys. Rev. A 38, 3116 (1988)

    Article  MathSciNet  ADS  Google Scholar 

  8. E. Domany, W. Kinzel and R. Meir: J. Phys. A 22, 2081 (1989)

    Article  MathSciNet  ADS  Google Scholar 

  9. R. Linsker: Proc. Nat. Acad. Sci. USA 83, 7508–7512 (1986)

    Article  ADS  Google Scholar 

  10. R. Linsker: Proc. Nat. Acad. Sci. USA 83, 8390–8394 (1986)

    Article  ADS  Google Scholar 

  11. R. Linsker: Proc. Nat. Acad. Sci. USA 83, 8779–8783 (1986)

    Article  ADS  Google Scholar 

  12. R. Linsker, in:Computer Simulation in Brain Science, edited by R. Coltrili (Cambridge University Press, Cambridge 1988)

    Google Scholar 

  13. R. Linsker: IEEE Computer (March 1988) 105–117

    Google Scholar 

  14. 9.12R. Linsker: to be published in the proceedings of the 1988 Denver Conference on Neural Information Processing Systems (Morgan Kauffman)

    Google Scholar 

  15. R. Meir and E. Domany: Phys. Rev. A 37, 2660 (1988)

    Article  ADS  Google Scholar 

  16. T. Grossman, R. Meir, and E. Domany: Complex Systems, 2, 555 (1988)

    MathSciNet  MATH  Google Scholar 

  17. P. Rujan and M. Marchand: Complex Systems 3, 229 (1989)

    MATH  Google Scholar 

  18. M. Mézard and J.P. Nadal: J. Phys. A 22, 2191 (1989)

    Article  MathSciNet  ADS  Google Scholar 

  19. See for example J.E. Hopcroft and R.L. Mattson, Synthesis of Minimal Threshold Logic Networks, IEEE Trans. Electronic Computers, EC-14, 552 (1965)

    Article  Google Scholar 

  20. J.J. Hopfield: Proc. Natl. Acad. USA 79, 2554 (1982)

    Article  MathSciNet  ADS  Google Scholar 

  21. DJ. Amit, H. Gutfreund, and H. Sompolinsky: Ann. Phys. 173, 30 (1987)

    Article  ADS  Google Scholar 

  22. H. Sompolinsky, in: Heidelberg Colloquium on Glassy Dynamics edited by J.L. van Hemmen and I. Morgenstem, Lecture Notes in Physics Vol. 275 (Springer, Berlin, Heidelberg 1987)

    Chapter  Google Scholar 

  23. B. Derrida, E. Gardner, and A. Zippelius: Europhys. Lett. 4, 167 (1987)

    Article  ADS  Google Scholar 

  24. See for example E. Domany, J. Stat. Phys. 51, 743 (1988)

    Article  MathSciNet  ADS  MATH  Google Scholar 

  25. 9.23See H. Ritter, K. Obermayer, K. Schulten, and J. Rubner, this volume, Chap. 8

    Google Scholar 

  26. E. Gardner: J. Phys. A 21, 257 (1988)

    Article  MathSciNet  ADS  Google Scholar 

  27. E. Gardner and B. Derrida, J. Phys. A 21, 271 (1988)

    Article  MathSciNet  ADS  Google Scholar 

  28. W. Krauth and M. Opper: J. Phys. A 22, L519 (1989)

    Article  ADS  Google Scholar 

  29. L Kanter and H. Sompolinsky: Phys. Rev. A 35, 380 (1987)

    Article  ADS  Google Scholar 

  30. A.C.C. Coolen, J.J. Denier van der Gon, and Th.W. Ruijgrok: Proc. nEuro88

    Google Scholar 

  31. A.C.C. Coolen, H J.J. Jonker, and Th.W. Ruijgrok, Utrecht preprint (1989)

    Google Scholar 

  32. M. Opper, J. Kleinz, H. Köhler, and W. Kinzel: J. Phys. A 22, L407 (1989)

    Article  ADS  Google Scholar 

  33. T.B. Kepler and L.F. Abbott: J. Phys. (Paris) 49, 1657 (1988)

    Article  Google Scholar 

  34. H. Homer, D. Bormann, M. Frick, H. Kinzelbach, and A. Schmidt: Z. Phys. B 76,381 (1989)

    Article  ADS  Google Scholar 

  35. I. Kanter: Phys. Rev. A 40, 2611 (1989)

    Article  ADS  Google Scholar 

  36. S. Diedrich and M. Opper: Phys. Rev. Lett. 58, 949 (1987)

    Article  MathSciNet  ADS  Google Scholar 

  37. E. Gardner, N. Stroud, and DJ. Wallace: J. Phys. A 22, 2019 (1989)

    Article  MathSciNet  ADS  Google Scholar 

  38. W. Krauth and M. Mézard: J. Phys. A 21, L745 (1987)

    Article  Google Scholar 

  39. P. Peretto: Neural Networks 1, 309–322 (1988)

    Article  Google Scholar 

  40. J.F. Fontanari and R. Meir: Caltech preprints (1989)

    Google Scholar 

  41. L.F. Abbott and T.B. Kepler: J. Phys. A 22, L711 (1989)

    Article  MathSciNet  ADS  Google Scholar 

  42. FJ. Pineda: Phys. Rev. Lett. 59, 2229 (1987)

    Article  MathSciNet  ADS  Google Scholar 

  43. M. Opper: Europhys. Lett. 8, 389 (1989)

    Article  ADS  Google Scholar 

  44. J.A. Hertz, G.L Thorbergson, and A. Krogh: Physica Scripta T25, 149 (1989)

    Article  ADS  Google Scholar 

  45. W. Kinzel and M. Opper: this volume. Chap.4

    Google Scholar 

  46. For reviews see: J.D. Cowan and D.H. Sharp, Quarterly Reviews of Biophysics, 21 365 (1988)

    Article  Google Scholar 

  47. R.P. Lippmann, IEEE ASSP Magazine, 4, 4 (1987)

    Article  Google Scholar 

  48. T. Kohonen: Self Organization and Associative Memory (Springer, Beriin, Heidelberg 1984)

    Google Scholar 

  49. D.E. Rumelhart and J.L. McClelland: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 2 vols. (MTT Press, Cambridge, Mass. 1986)

    Google Scholar 

  50. M. Minsky and S. Papert:Perceptrons, expanded edition (MIT Press, Cambridge, Mass. 1988)

    MATH  Google Scholar 

  51. D.E. Rumelhart, G.E. Hinton, and R J. Williams: in Parallel Distributed Processing: Explorations in the Microstructure of Cognition edited by D.E. Rumelhart and J.L. McClelland, (MIT Press, Cambridge, Mass. 1986) Vol. 1, p. 318

    Google Scholar 

  52. Y. Le Cun: Proc. Cognitiva, 85, 593 (1985)

    Google Scholar 

  53. PJ. Werbos, Ph.D. thesis. Harvard University (1974)

    Google Scholar 

  54. D.B. Parker, MIT Technical Report TR-47 (1985)

    Google Scholar 

  55. M. Abeles: Local Cortical Circuits (Springer, Berlin, Heidelberg 1982)

    Book  Google Scholar 

  56. H. Sompolinsky: Phys. Rev. A 34, 2571 (1986)

    Article  ADS  Google Scholar 

  57. J.L. van Hemmen and R. Kühn, Phys. Rev. Lett. 57, 913 (1986)

    Article  ADS  Google Scholar 

  58. J.L. van Hemmen, Phys. Rev. A 36, 1959 (1987)

    Article  ADS  Google Scholar 

  59. W. Krauth, M. Mézard, and J.P. Nadal: Complex Systems, 2, 387 (1988)

    MathSciNet  MATH  Google Scholar 

  60. W.A. Litüe: Math. Biosci. 19, 101 (1975)

    Google Scholar 

  61. S. Amari and K. Maginu: Neural Networks, 1, 63 (1988)

    Article  Google Scholar 

  62. See for example W. Feller, An Introduction to Probability Theory and its Applications (Wiley, New York 1966) Vol. 4 p. 256

    MATH  Google Scholar 

  63. 9.54It is very important to realize that the embedding fields Hi and Hj are not independent. Their correlation is M N. This correlation gives rise to the layer-to-layer recursive variation of the width parameter A which in tum, causes the appearance of non trivial domains of attraction

    Google Scholar 

  64. B. Derrida: J. Phys. A 20, L72I (1987)

    Article  Google Scholar 

  65. W. Kinzel: Z. Physik B 60, 205 (1985)

    MathSciNet  ADS  Google Scholar 

  66. W. Krauth, J.P. Nadal, and M. Mézard: J. Phys. A: Math. Gen. 21, 2995 (1988)

    Article  ADS  MATH  Google Scholar 

  67. J. Kleinz: diploma thesis, Justus-Liebig University Glessen (1988)

    Google Scholar 

  68. For a recent review see D. H. Hubel, Los Alamos Science 16, 14 (1988)

    Google Scholar 

  69. D. Kämmen and A. Yuille: Biol. Cybem. 59, 23 (1988)

    Article  Google Scholar 

  70. P. Huben Ann. Statistics 13, 435 (1985)

    Article  Google Scholar 

  71. E. Oja: J. Math. Biol. 15, 267 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  72. R. E. Blahut: Principles and Applications of Information Theory, (Addison-Wesley, 1987)

    Google Scholar 

  73. B. Widrow and R. Winter: Computer 21, 25 (1988)

    Article  Google Scholar 

  74. F. Rosenblatt: Psych. Rev. 62, 386 (1958)

    Article  Google Scholar 

  75. F. RosenblattPrinciples of Neurodynamics (Spartan, New York 1962)

    MATH  Google Scholar 

  76. P.M. Lewis and C.L. Coates: Threshold Logic (Wiley, New York 1967)

    MATH  Google Scholar 

  77. J. Denker, D. Schwartz, B. Wittner, S. Solla, J.J. Hopfield, R. Howard, and L. Jackel: Complex Systems 1, 877–922 (1987)

    MathSciNet  MATH  Google Scholar 

  78. T. Grossman, Complex Systems 3, 407 (1989)

    MathSciNet  Google Scholar 

  79. T. Grossman, in Advances in Neural Information Processing Systems 2, edited by D. Touretzky (Morgan Kaufman, San Mateo 1990) p. 516

    Google Scholar 

  80. R. Rohwer, in Advances in Neural Information Processing Systems 2, edited by D. Touretzky (Morgan Kaufman, San Mateo 1990) p. 538)

    Google Scholar 

  81. A. Krogh, G.I. Thorbergsson, and J.A. Hertz, ibid, p. 773

    Google Scholar 

  82. D. Saad and E. Marom, Complex Systems 4, 107 (1990)

    MathSciNet  MATH  Google Scholar 

  83. D. Nabutovsky, T. Grossman, and E. Domany: Complex Systems 4, 519 (1990)

    MathSciNet  MATH  Google Scholar 

Download references

Authors
  1. Eytan Domany
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ronny Meir
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Electronics, Weizmann Institute of Science, 76100, Rehovot, Israel

    Eytan Domany

  2. Institut für Theoretische Physik, Technische Universität München, D-85747, Garching bei München, Germany

    J. Leo van Hemmen

  3. Department of Physics and Beckman Institute, University of Illinois, Urbana, IL, 61801, USA

    Klaus Schulten

Rights and permissions

Reprints and permissions

Copyright information

© 1995 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Domany, E., Meir, R. (1995). Layered Neural Networks. In: Domany, E., van Hemmen, J.L., Schulten, K. (eds) Models of Neural Networks I. Physics of Neural Networks. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-79814-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-79814-6_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-79816-0

  • Online ISBN: 978-3-642-79814-6

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation