Abstract
In this paper, we propose a new information theoretic competitive learning method. We first construct a learning method in single-layered networks, and then we extend it to supervised multi-layered networks. Competitive unit outputs are computed by the inverse of Euclidean distance between input patterns and connection weights. As distance is smaller, competitive unit outputs are stronger. In realizing competition, neither the winner-take-all algorithm nor the lateral inhibition is used. Instead, the new method is based upon mutual information maximization between input patterns and competitive units. In maximizing mutual information, the entropy of competitive units is increased as much as possible. This means that all competitive units must equally be used in our framework. Thus, no under-utilized neurons or dead neurons are generated. When using multi-layered networks, we can improve noise-tolerance performance by unifying information maximization and minimization. We applied our method with single-layered networks to a simple artificial data problem and an actual road classification problem. In both cases, experimental results confirmed that the new method can produce the final solutions almost independently of initial conditions, and classification performance is significantly improved. Then, we used multi-layered networks, and applied them to a character recognition problem and a political data analysis. In these problem, we could show that noise-tolerance performance was improved by decreasing information content on input patterns to certain points.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Linsker, R.: Self-organization in a perceptual network, Computer 21 (1988), 105–117.
Linsker, R.: How to generate ordered maps by maximizing the mutual information between input and output, Neural Computation 1 (1989), 402–411.
Atick, J. J. and Redlich, A. N.: Toward a theory of early visual processing, Neural Computation 2 (1990), 308–320.
Becker, S.: Mutual information maximization: models of cortical self-organization, Network: Computation in Neural Systems 7 (1996), 7–31.
Becker, S. and Hinton, G. E.: Learning mixture models of spatial coherence, Neural Computation 5 (1993), 267–277.
Kamimura, R. and Nakanishi, S.: Hidden information maximization for feature detection and rule discovery, Network 6 (1995), 577–622.
Kamimura, R. and Nakanishi, S.: Improving generalization performance by information minimization, IEICE Transactions on Information and Systems E78-D(2) (1995), 163–173.
Deco, G., Finnof, W. and Zimmermann, H. G.: Unsupervised mutual information criterion for elimination of overtraining in supervised multiplayer networks, Neural Computation 7 (1995), 86–107.
Akiyama, Y. and Furuya, T.: An extension of the back-propagation learning which performs entropy maximization as well as error minimization, Tech. Rep. NC91-6, IEICE Technical Report, 1991.
Kamimura, R., Kamimura, T. and Shultz, T. R.: Information theoretic competitive learning and linguistic rule acquistion, Transactions of the Japanese Society for Artificial Intelligence 16(2) (2001), 287–298.
Kamimura, R., Kamimura, T. and Uchida, O.: Flexible feature discovery and structural information, Connection Science 13(4) (2001), 323–347.
Kamimura, R., Kamimura, T. and Takeuchi, H.: Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks, Connection Science 14(2) (2002), 137–162.
Kamimura, R.: Progressive feature extraction by greedy network-growing algorithm, to appear in Complex Systems, 2003.
Rumelhart, D. E. and Zipser, D.: Feature discovery by competitive learning, In: D. E. Rumelhart and G. E. H. et al. (eds), Parallel Distributed Processing, Cambridge: MIT Press, vol. 1, pp. 151–193, 1986.
Grossberg, S.: Competitive learning from interactive activation to adaptive resonance, Cognitive Science 11 (1987), 23–63.
DeSieno, D.: Adding a conscience to competitive learning, In: Proceedings of IEEE International Conference on Neural Networks, pp. 117–124, San Diego, IEEE, 1988.
Ahalt, S. C., Krishnamurthy, A. K., Chen, P. and Melton, D. E.: Competitive learning algorithms for vector quantization, Neural Networks 3 (1990), 277–290.
Xu, L.: Rival penalized competitive learning for clustering analysis, RBF net, and curve detection, IEEE Transaction on Neural Networks 4(4) (1993), 636–649.
Luk, A. and Lien, S.: Properties of the generalized lotto-type competitive learning, In: Proceedings of International conference on neural information processing, pp. 1180–1185, San Mateo: CA, Morgan Kaufmann Publishers, 2000.
Hulle, M. M. V.: The formation of topographic maps that maximize the average mutual information of the output responses to noiseless input signals, Neural Computation 9(3) (1997), 595–606.
Gatlin, L. L.: Information Theory and Living Systems, Columbia University Press, 1972.
Hecht-Nielsen, R.: Neurocomputing, Reading, MA: Addison-Wesley, 1990.
McClelland, C. A. and Hoggard, G. D.: Confliict patterns in the interactions among nations, In: R. Hayes (ed.), International Politics and Foreign Policy, p. 714, New York, Free Press, 1969.
Kamimura, R., Kamimura, T. and Shultz, T. R.: Self-organization by information control, In: Proceedings of IASTED International Conference on Applied Informatics, pp. 188–192, Anaheim, CA, ACTA Press, 2001.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Kamimura, R. Information-Theoretic Competitive Learning with Inverse Euclidean Distance Output Units. Neural Processing Letters 18, 163–204 (2003). https://doi.org/10.1023/B:NEPL.0000011136.78760.22
Issue Date:
DOI: https://doi.org/10.1023/B:NEPL.0000011136.78760.22