Skip to main content
SpringerLink
Log in

Parallel algorithms/architectures for neural networks

  • Published:
Journal of VLSI signal processing systems for signal, image and video technology Aims and scope Submit manuscript

Abstract

This paper advocates digital VLSI architectures for implementing a wide variety of artificial neural networks (ANNs). A programmable systolic array is proposed, which maximizes the strength of VLSI in terms of intensive and pipelined computing and yet circumvents the limitation on communication. The array is meant to be more general purpose than most other ANN architectures proposed. It may be used for a variety of algorithms in both the retrieving and learning phases of ANNs: e.g., single layer feedback networks, competitive learning networks, and multilayer feed-forward networks. A unified approach to modeling of existing neural networks is proposed. This unified formulation leads to a basic structure for a universal simulation tool and neurocomputer architecture. Fault-tolerance approach and partitioning scheme for large or non-homogeneous networks are also proposed. Finally, the implementations based on commercially available VLSI chips (e.g., Inmos T800) and custom VLSI technology are discussed in great detail.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. Eckmiller and C.V.D. Malsburg.Neural Computers. NATO ASI Series F, Computer and System Science, Springer-Verlag, 1987.

  2. J.D. Cowan and D.H. Sharp. Neural nets. Technical Report, Mathematical Dept., University of Chicago. 1987.

  3. J.A. Anderson,Neurocomputing — Paper Collections. MIT Press, 1988.

  4. W .S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in the nervous activity.Bull. Math. Biophys., 5(115), 1943.

  5. F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain.Psych. Rev., Vol. 65, 1958.

  6. G. Widrow and M.E. Hoff. Adaptive switching circuit.IRE Western Electronic Show and Convention: Convention Record, 96–104, 1960.

  7. K. Steinbush. The learning matrix.Kybernetik (Biological Cybernetics), 36–45, 1961.

  8. J.A. Anderson. A simple neural network generating an interactive memory.Mathematical Biosciences, 14:197–220, 1972.

    Article  MATH  Google Scholar 

  9. D.J. Willshaw.Models of distributed associative memory models. Ph.D. Thesis, University of Edinburgh, 1971.

  10. S.I. Amari, Characteristics of randomly connected threshold-element network systems.Proc. IEEE, 59:35–47, January 1971.

    Article  MathSciNet  Google Scholar 

  11. T. Kohonen. Correlation matrix memories.IEEE Trans. Computers, Vol. C-21, 1972.

  12. K. Fukushima. Cognitron: A self-organizing multilayered neural network.Biological Cybernetics, 20:121–136, 1975.

    Article  Google Scholar 

  13. S. Grossberg. Adaptive pattern classification and universal recoding: Part 1. Parallel development and coding of neural feature detectors.Biological Cybernetics, 23:121–134, 1976.

    Article  MathSciNet  MATH  Google Scholar 

  14. M.A. Cohen and S. Grossberg. Absolute stability of global pattern formation and parallel memory storage by competitive neural networks.IEEE Trans. Systems, Man and Cybernetics, 13(5): 815–825, September 1983.

    Article  MathSciNet  MATH  Google Scholar 

  15. J.J. Hopfield. Neural network and physical systems with emergent collective computational abilities.Proc. Natl’. Acad. Sci. USA, pages 2554–2558, 1982.

  16. J.J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons.Proc. Natl’. Acad. Sci. USA, 81:3088–3092, 1984.

    Article  Google Scholar 

  17. J.J. Hopfield and D.W. Tank. Neural computation of decision in optimization problems.Biological Cybernetics, 52:141–152, 1985.

    MathSciNet  MATH  Google Scholar 

  18. D.E. Rumelhart and D. Zipser. Feature discovery by competitive learning.Cognitive Science, 9:75–112, 1985.

    Article  Google Scholar 

  19. J.L. McClelland and D.E. Rumelhart. Distributed memory and the representation of general and specific information.Journal of Experimental Psychology: General, 114:158–188, 1985.

    Google Scholar 

  20. M. Minsky and S. Papert.Perceptrons: an Introduction to Compuational Geometry. MIT Press, Cambridge, Massachusetts, 1969.

    Google Scholar 

  21. P.J. Werbos. Beyond regression:New tools for prediction and analysis in the behavior science. Ph.D. thesis, Harvard University, Cambridge, 1974.

    Google Scholar 

  22. D. Parker. Learing logic. Technical Report TR-47, Center for Computational Research in Economics and Management Science, MIT, Cambridge, 1985.

    Google Scholar 

  23. D.E. Rumelhart, G.E. Hinton, and R.J. Williams. Learning internal representations by error propagation.Parallel Distributed Processing (PDP): Exploration in the Microstructure of Cognition (Vol. 1), chapter 8, pages 318–362, MIT Press, Cambridge, Massachusetts, 1986.

    Google Scholar 

  24. A. Lapedes and R. Farber. Nonlinear signal processing using neural networks.IEEE, Coif on Neural Information Processing Systems-Natural and Synthetic, Denver, November 1987.

  25. D.J. Burr. Experiments with a connectionist text reader. InProc. IEEE, 1st Intl’ Conf. on Neural Networks, San Diego, pages IV 717-IV 724, 1987.

  26. T.J. Sejnowski. Parallel networks that learn to pronounce English text.Complex Syst., 1:145–168, 1987.

    MATH  Google Scholar 

  27. R. P. Lippmann. An introduction to computing with neural nets.IEEE ASSP Magazine, 4:4–22, 1987.

    Article  Google Scholar 

  28. R.P. Gorman and T.J. Sejnowski, Learned classification of sonar targets using a massively parallel network.IEEE Trans. ASSP 36:1135–1140, July 1988.

    Article  MATH  Google Scholar 

  29. S.Y. Kung and J.N. Hwang. A unified modeling of neural network architectures. InProc. NATO ARW on Sensor-Based Robots, 1989.

  30. D.H. Ackley, G.E. Hinton, and T.J. Sejnowski. A learning algorithm for Boltzmann machines.Cognitive Science, 9:147–169, 1985.

    Article  Google Scholar 

  31. L.R. Rabiner and B.H. Juang. An introduction to hidden Markov models.IEEE ASSP Magazine, 3(1):4–16, January 1986.

    Article  Google Scholar 

  32. D.E. Rumelhart, J. L. McClelland, and the PDP Research Group.Parallel Distributed Processing (PDP): Exploration in the Microstructure of Cognition (Vol. 1). MIT Press, Cambridge, Massachusetts, 1986.

    Google Scholar 

  33. L.E. Baum and G.R. Sell. Growth transformations for functions on manifolds.Pacific Journal of Mathematics, 27(2):211–227, 1968.

    Article  MathSciNet  MATH  Google Scholar 

  34. L.E. Baum and J.A. Eagon. An inequality with applications to statistical estimation for probabilistic function of Markov processes and a model for exology.Amer. Math. Soc. Bull., 73:360–363, May 1967.

    Article  MathSciNet  MATH  Google Scholar 

  35. L.E. Baum, T. Petrie, G. Soules, and N. Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains.Ann. Math Statistic., 41:164–171, 1970.

    Article  MathSciNet  MATH  Google Scholar 

  36. P.F. Stebe. Invariant functions of an iterative process for maximization of a polynomial.Pacific Journal of Mathematics, 43(3):765–783, 1972.

    Article  MathSciNet  MATH  Google Scholar 

  37. J. Lupo,DARPA: Neural Network Study. AFCEA International Press, 1988.

  38. J.N. Hwang and S.Y. Kung. A unifying viewpoint between multilayer perceptrons and hidden Markov models. InIEEE Int’l Symposium on Circuits and Systems, ISCAS’89, pp. 770–773 Portland, May 1989.

  39. R. Linsker. Self-organization in a perceptual network.IEEE Computer Magazine, 21:105–117, March 1988

    Article  Google Scholar 

  40. T. Kohonen.Self-Organization and Associative Memory, Series in Information Science, Vol. 8. Springer-Verlag, New York, 1984.

    Google Scholar 

  41. T. Kohonen. Self-organized formation of topologically correct feature map.Biological Cybernetics, 43:59–69, 1982.

    Article  MathSciNet  MATH  Google Scholar 

  42. D.O. Hebb.The Organization of Behavior Wiley Inc., New York, 1949.

    Google Scholar 

  43. INMOS Ltd.Transputer Development System. Prentice Hall, New Jersey, 1988.

    Google Scholar 

  44. C. Mead and L. Conway.Introduction to VLSI Systems. Addison-Wesley, 1980.

  45. S.Y. Kung.VLSI Array Processors. Prentice Hall Inc., N.J., 1988.

    Google Scholar 

  46. H.T. Kung. Why systolic architectures?IEEE, Computer, 15(1), January 1982.

  47. H.T. Kung and C.E. Leiserson. Systolic arrays (for VLSI). InSparse Matrix Symposium, pages 256–282, SIAM, 1978.

  48. S.Y. Kung, S.C. Lo, S.N. Jean, and J.N. Hwang. Wavefront array processors: from concept to implementation.IEEE Computer Magazine, 18–33, July 1987.

  49. S.Y Kung. On supercomputing with systolic/wavefront array processors.Proc. IEEE, 72(7), July 1984.

  50. S.Y. Kung, K.S. Arun, R.J. Gal-Ezer, and D.V. Bhaskar Rao. Wavefront array processor: language, architecture, and applications.IEEE Transactions on Computers, Special Issue on Parallel and Distributed Computers, C-31(11):1054–1066, Nov 1982.

    Google Scholar 

  51. S.Y. Kung, S. N. Jean, S.C. Lo, and P. S. Lewis. Design methodologies for systolic arrays: mapping algorithms to architectures.Chapter 6, Signal Processing Handbook, pages 145–191, Marcel Dekker Inc., 1988.

  52. J.N. Hwang.Algorithms/Applications/Architectures of Artificial Neural Nets. Ph.D. thesis, Dept. of Electrical Engineering, University of Southern California, December 1988.

  53. K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position.Biological Cybernetics, 36:193–202, April 1980.

    Article  MathSciNet  MATH  Google Scholar 

  54. G.E. Hinton. Connectionist learning procedure. Technical Report CMU-CS-87-115, Carnegie Mellon University, September 1987.

  55. G.A. Carpenter and S. Grossberg. ART2: Self-organization of stable category recognition codes for analog input patterns. InProc. IEEE ICNN’87 San Diego, pages II 727–II 736, 1987.

  56. S.Y. Kung and J.N. Hwang. Parallel architectures for artificial neural nets.IEEE, Int’l Conf on Neural Networks, ICNN’88, San Diego, pages Vol 2: 165–172, July 1988

    Article  Google Scholar 

  57. B. Widrow and R. Winter. Neural nets for adaptive filtering and adaptive pattern recognition.IEEE Computer Magazine, 21:25–39, March 1988.

    Article  Google Scholar 

  58. N.J. Nilsson.Learning Machines. McGraw-Hill Book Company, 1965.

  59. B.H. Juang. On the hidden Markov model and dynamic time warping for speech recognition—A unified view.AT&T Bell Laboratories Technical Journal, 63(7):1213–1243, September 1984.

    Article  MathSciNet  MATH  Google Scholar 

  60. S.E. Levinson, L.R. Rabiner, and M.M. Sondhi. An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition.The Bell System Technical Journal, 62:1035–1074, April 1983.

    Article  MathSciNet  MATH  Google Scholar 

  61. S.Y. Kung and J.N. Hwang. An algebraic projection analysis for optimal hidden units size and learning rate in back-propagation learning.IEEE, Int ’l Conf. on Neural Networks, ICNN’88, San Diego, pages Vol. 1: 363–370, July 1988.

    Article  Google Scholar 

  62. S.Y. Kung and J.N. Hwang. A unified systolic architecture for artificial neural networks. InJournal of Parallel and Distributed Computing, Special Issue on Neural Networks, pp. 358–387, April, 1989.

  63. P. Zafiropulo. Performance evaluation of reliability improvement techniques for single-loop communications systems.IEEE Trans. on Communications, 22(6):742–751, June 1974.

    Article  Google Scholar 

  64. J.Y. Jon and J. A. Abraham. Fault-tolerant matrix arithmetic and signal processing on highly concurrent computing structures.Proc. IEEE, 732–741, May 1986.

  65. E. Manolakos and S.Y. Kung. Transient fault recovery for the wavefront array processors.Proc. IEEE, EUSIPCO’88, France, September 1988.

  66. W. D. Mao and S.Y Kung. Implementation and performance issues of back-propagation neural nets. Technical Report, Electrical Eng. Dept., Princeton University, January 1989.

  67. H.C. Fu, J.N. Hwang, S.Y. Kung, WD. Mao, and J.A. Vlontzos. A universal digital VLSI design for neural networks. Presented inInternational Joint Conference on Neural Networks, Washington D.C., June 1989.

Download references

Author information

Authors and Affiliations

  1. Dept. of Electrical Eng., University of Washington, 98195, Seattle, WA

    J. N. Hwang

  2. Dept. of Electrical Eng., Princeton University, 08544, Princeton, NJ

    S. Y. Kung

Authors
  1. J. N. Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Y. Kung
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hwang, J.N., Kung, S.Y. Parallel algorithms/architectures for neural networks. J VLSI Sign Process Syst Sign Image Video Technol 1, 221–251 (1989). https://doi.org/10.1007/BF02427796

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02427796

Keywords

Search

Navigation