Abstract
Deep learning has been considered a hallmark in a number of applications recently. Among those techniques, the ones based on Restricted Boltzmann Machines have attracted a considerable attention, since they are energy-driven models composed of latent variables that aim at learning the probability distribution of the input data. In a nutshell, the training procedure of such models concerns the minimization of the energy of each training sample in order to increase its probability. Therefore, such optimization process needs to be regularized in order to reach the best trade-off between exploitation and exploration. In this work, we propose an adaptive regularization approach based on temperatures, and we show its advantages considering Deep Belief Networks (DBNs) and Deep Boltzmann Machines (DBMs). The proposed approach is evaluated in the context of binary image reconstruction, thus outperforming temperature-fixed DBNs and DBMs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
The images are originally available in grayscale with resolution of \(28\times 28\), but they were reduced to \(14\times 14\) images.
- 3.
The original training set was reduced to \(2\%\) of its former size, which corresponds to 1, 200 images.
- 4.
- 5.
Since this architecture has been commonly employed in several works in the literature, we opted to employ it in our work either.
- 6.
One sampling iteration was used for all learning algorithms.
References
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Salakhutdinov, R., Hinton, G.E.: An efficient learning procedure for deep Boltzmann machines. Neural Comput. 24(8), 1967–2006 (2012)
Wan, L., Zeiler, M., Zhang, S., LeCun, Y., Fergus, R.: Regularization of neural networks using dropconnect. In: Dasgupta, S., Mcallester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning. JMLR Workshop and Conference Proceedings, ICML 2013, vol. 28, no. 3, pp. 1058–1066 (2013)
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Papa, J.P., Rosa, G.H., Costa, K.A.P., Marana, A.N., Scheirer, W., Cox, D.D.: On the model selection of Bernoulli restricted Boltzmann machines through harmony search. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2015, pp. 1449–1450. ACM, New York (2015)
Papa, J.P., Rosa, G.H., Yang, X.-S.: Quaternion-driven deep belief networks fine-tuning. Applied Soft Computing (2016) (submitted)
Papa, J.P., Rosa, G.H., Marana, A.N., Scheirer, W., Cox, D.D.: Model selection for discriminative restricted Boltzmann machines through meta-heuristic techniques. J. Comput. Sci. 9, 14–18 (2015)
Rosa, G.H., Papa, J.P., Marana, A.N., Scheirer, W., Cox, D.D.: Fine-tuning convolutional neural networks using harmony search. In: Pardo, A., Kittler, J. (eds.) IARP 2015. LNCS, vol. 9423, pp. 683–690. Springer, Cham (2015)
Papa, J.P., Scheirer, W., Cox, D.D.: Fine-tuning deep belief networks using harmony search. Appl. Soft Comput. 46, 875–885 (2016)
Li, G., Deng, L., Xu, Y., Wen, C., Wang, W., Pei, J., Shi, L.: Temperature based restricted Boltzmann machines. Sci. Rep. 6, 1–12 (2016)
Passos, L.A., Papa, J.P.: Temperature-based deep Boltzmann machines, arXiv (2016). http://arxiv.org/abs/1608.07719
Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8), 1771–1800 (2002)
Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1(6), 80–83 (1945)
Tieleman, T.: Training restricted Boltzmann machines using approximations to the likelihood gradient. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1064–1071. ACM, New York (2008)
Acknowledgments
The authors are grateful to FAPESP grants #2014/16250-9 and #2014/12236-1, as well as Capes and CNPq grant #306166/2014-3.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Passos Júnior, L.A., Costa, K.A.P., Papa, J.P. (2017). Deep Boltzmann Machines Using Adaptive Temperatures. In: Felsberg, M., Heyden, A., Krüger, N. (eds) Computer Analysis of Images and Patterns. CAIP 2017. Lecture Notes in Computer Science(), vol 10424. Springer, Cham. https://doi.org/10.1007/978-3-319-64689-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-64689-3_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-64688-6
Online ISBN: 978-3-319-64689-3
eBook Packages: Computer ScienceComputer Science (R0)