Abstract
Deep learning is promising approach to extract useful nonlinear representations of data. However, it is usually applied with large training sets, which are not always available in practical tasks. In this paper, we consider stacked autoencoders with logistic regression as the classification layer and study their usefulness for the task of image categorization depending on the size of training sets. Hand-crafted image descriptors are proposed and used for training autoencoders in addition to pixel-level features. New multi-column architecture for autoencoders is also proposed. Conducted experiments showed that useful nonlinear features can be learnt by (stacked) autoencoders only using large training sets, but they can yield positive results due to redundancy reduction also on small training sets. Practically useful results (9.1% error rate for 6 classes) were achieved only using hand-crafted features on the training set containing 4800 images.
Chapter PDF
Similar content being viewed by others
References
He, Y., Kavukcuoglu, K., Wang, Y., Szlam, A., Qi, Y.: Unsupervised Feature Learning by Deep Sparse Coding. arXiv:1312.5783 [cs.LG] (2013)
Le Roux, N., Bengio, Y.: Representational Power of Restricted Boltzmann Machines and Deep Belief Networks. Neural Computation 20(6), 1631–1649 (2008)
Gregor, K., Mnih, A., Wierstra, D.: Deep AutoRegressive Networks. arXiv:1310.8499 [cs.LG] (2013)
Hinton, G.E., Osindero, S., Teh, Y.: A Fast Learning Algorithm for Deep Belief Nets. Neural Computation 18, 1527–1554 (2006)
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy Layer-Wise Training of Deep Networks. In: Advances in Neural Information Processing Systems (NIPS 2006), vol. 19, pp. 153–160 (2007)
Ranzato, M.A., Poultney, C., Chopra, S., LeCun, Y.: Efficient Learning of Sparse Representations with an Energy-Based Model. In: Advances in Neural Information Processing Systems (NIPS 2006), vol. 19 (2007)
Ciresan, D.C., Meier, U., Gambardella, L.M., Schmidhuber, J.: Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition. arXiv:1003.0358 [cs.NE] (2010)
LeCun, Y., Cortes, C.: MNIST handwritten digit database, http://yann.lecun.com/exdb/mnist/
Tenenbaum, J.B., Kemp, C., Griffiths, T.L., Goodman, N.D.: How to Grow a Mind: Statistics, Structure, and Abstraction. Science 331(6022), 1279–1285 (2011)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing Robust Features with Denoising Autoencoders. In: Proc. 25th International Conference on Machine Learning, pp. 1096–1103 (2008)
Cireşan, D., Meier, U., Masci, J., Schmidhuber, J.: Multi-Column Deep Neural Network for Traffic Sign Classification. Neural Networks 32, 333–338 (2012)
Vailaya, A., Jain, A., Zhang, H.J.: On Image Classification: City Images vs. Landscapes. Pattern Recognition 31(12), 1921–1935 (1998)
Haralick, R.M., Shanmugam, K., Dinstein, I.: Textural Features for Image Classification. IEEE Transactions on Systems, Man, and Cybernetics. SMC-3(6), 610–621 (1973)
Deriche, R.: Using Canny’s Criteria to Derive a Recursively Implemented Optimal Edge Detector. International Journal of Computer Vision 1(2), 167–187 (1987)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 IFIP International Federation for Information Processing
About this paper
Cite this paper
Potapov, A., Batishcheva, V., Peterson, M. (2014). Limited Generalization Capabilities of Autoencoders with Logistic Regression on Training Sets of Small Sizes. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H. (eds) Artificial Intelligence Applications and Innovations. AIAI 2014. IFIP Advances in Information and Communication Technology, vol 436. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-44654-6_25
Download citation
DOI: https://doi.org/10.1007/978-3-662-44654-6_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-44653-9
Online ISBN: 978-3-662-44654-6
eBook Packages: Computer ScienceComputer Science (R0)