Skip to main content
SpringerLink
Log in

Comparison of Auto-Encoder Training Algorithms

  • Conference paper
  • First Online:
Human Interaction, Emerging Technologies and Future Systems V (IHIET 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 319))

  • 3188 Accesses

Abstract

Training of deep neural networks is difficult due to vanishing gradients. Therefore, a pre-training procedure based on restricted Boltzmann machines is suggested to resolve this problem. However, new developments in deep learning aim to resolve the problem with vanishing gradients by using rectifier linear units (ReLU). This study compares the performance of a RBM pre-trained auto-encoder with sigmoid activations to the performance of auto-encoder with ReLU activation. The results showed that the ReLU auto-encoder achieved better reconstruction and saved training time, since it doesn't require pre-training .

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kramer, M.A.: Nonlinear principal component analysis using autoassociative neural networks. AIChE J. 37, 233–243 (1991)

    Article  Google Scholar 

  2. Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, a field guide to dynamical recurrent neural networks. IEEE Press (2001)

    Google Scholar 

  3. Hinton, G.E. Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Sci. 313, 504–507 (2006)

    Google Scholar 

  4. Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)

    Article  MathSciNet  Google Scholar 

  5. Haykin, S.S.: Neural Networks and Learning Machines/Simon Haykin. Prentice Hall, New York (2009)

    Google Scholar 

  6. Smolensky, P.: Information Processing in Dynamical Systems: Foundations of Harmony Theory (1986)

    Google Scholar 

  7. Jaitly, N., Hinton, G.: Learning a better representation of speech soundwaves using restricted boltzmann machines. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2011)

    Google Scholar 

  8. Sailor, H.B., Agrawal, D.M., Patil, H.A.: Unsupervised filterbank learning using convolutional restricted boltzmann machine for environmental sound classification. InterSpeech (2017)

    Google Scholar 

  9. Yang, J., Deng, J., Li, S., Hao, Y.: Improved traffic detection with support vector machine based on restricted Boltzmann machine. Soft. Comput. 21(11), 3101–3112 (2015). https://doi.org/10.1007/s00500-015-1994-9

    Article  Google Scholar 

  10. Vrábel, J., Pořı́zka, P., Kaiser, J.: Restricted Boltzmann machine method for dimensionality reduction of large spectroscopic data. Spectrochim. Acta Part B At. Spectrosc. 167, 105849 (2020)

    Google Scholar 

  11. Zhang, Y., Peng, P., Liu, C., Zhang, H.: Anomaly detection for industry product quality inspection based on Gaussian restricted Boltzmann machine. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) (2019)

    Google Scholar 

  12. Tan, C.C., Eswaran, C.: Performance comparison of three types of autoencoder neural networks. In: 2008 Second Asia International Conference on Modelling & Simulation (AMS) (2008)

    Google Scholar 

  13. Nuha, H., Mohandes, M., Liu, B.: Seismic data compression using auto-associative neural network and restricted Boltzmann machine. In: SEG Technical Program Expanded Abstracts 2018, pp. 186–190. Society of Exploration Geophysicists (2018)

    Google Scholar 

  14. Pumsirirat, A., Yan, L.: Credit card fraud detection using deep learning based on auto-encoder and restricted boltzmann machine. Int. J. Adv. Comput. Sci. Appl. 9, 18–25 (2018)

    Google Scholar 

  15. He, D., et al.: Intrusion detection based on stacked autoencoder for connected healthcare systems. IEEE Netw. 33, 64–69 (2019)

    Article  Google Scholar 

  16. Mahmoud, A.M., Alrowais, F., Karamti, H.: A hybrid deep contractive autoencoder and restricted boltzmann machine approach to differentiate representation of female brain disorder. Procedia Comput. Sci. 176, 1033–1042 (2020)

    Article  Google Scholar 

  17. Li, J., Yu, Z.L., Gu, Z., Wu, W., Li, Y., Jin, L.: A hybrid network for ERP detection and analysis based on restricted Boltzmann machine. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 563–572 (2018)

    Article  Google Scholar 

  18. Kuremoto, T., Kimura, S., Kobayashi, K., Obayashi, M.: Time series forecasting using restricted boltzmann machine. In: International Conference on Intelligent Computing (2012)

    Google Scholar 

  19. Qiao, J., Wang, L.: Nonlinear system modeling and application based on restricted Boltzmann machine and improved BP neural network. Appl. Intell. 51(1), 37–50 (2020). https://doi.org/10.1007/s10489-019-01614-1

    Article  Google Scholar 

  20. Ma, M., Sun, C., Chen, X.: Deep coupling autoencoder for fault diagnosis with multimodal sensory data. IEEE Trans. Industr. Inf. 14, 1137–1145 (2018)

    Article  Google Scholar 

  21. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (2011)

    Google Scholar 

  22. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (2010)

    Google Scholar 

  23. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  24. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  25. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2015)

    Google Scholar 

  26. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2481–2495 (2017)

    Article  Google Scholar 

  27. Zhou, Y., Arpit, D., Nwogu, I., Govindaraju, V.: Is joint training better for deep auto-encoders? (2014)

    Google Scholar 

  28. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

Download references

Acknowledgement

The experiments described in this paper became possible thanks to the computing resources and technical support provided by UNITe project: https://unite-bg.eu/.

Author information

Authors and Affiliations

  1. Faculty of Mathematics and Informatics, Sofia University, Sofia, Bulgaria

    Teodor Boyadzhiev, Stela Dimitrova & Simeon Tsvetanov

Authors
  1. Teodor Boyadzhiev
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stela Dimitrova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Simeon Tsvetanov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simeon Tsvetanov .

Editor information

Editors and Affiliations

  1. Institute for Advanced Systems Engineering, University of Central Florida, Orlando, FL, USA

    Tareq Ahram

  2. Campus du Moulin de la Housse, Université de Reims Champagne Ardenne GRESPI, Reims Cedex, France

    Redha Taiar

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Boyadzhiev, T., Dimitrova, S., Tsvetanov, S. (2022). Comparison of Auto-Encoder Training Algorithms. In: Ahram, T., Taiar, R. (eds) Human Interaction, Emerging Technologies and Future Systems V. IHIET 2021. Lecture Notes in Networks and Systems, vol 319. Springer, Cham. https://doi.org/10.1007/978-3-030-85540-6_88

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85540-6_88

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85539-0

  • Online ISBN: 978-3-030-85540-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation