Abstract
The article presents the artificial neural networks performance optimization using wavelet transform. The existing approaches of wavelet transform implementation in neural networks imply either transformation before neural network or using “wavenet” architecture, which requires new neural network training approaches. The proposed approach is based on the representation of the neuron as a nonrecursive adaptive filter and wavelet filter application to obtain the low-frequency part of the image. It reduces the image size and filtering interference, which is usually high-frequency. Our wavelet transform model is based on the classical representation of a forward propagation neural network or convolutional layers. It allows designing neural networks with the wavelet transform based on existing libraries and does not require changes in the neural network training algorithm. It was tested on three MNIST-like datasets. As a result of testing, it was found that the speed gain is approximately 50 ± 5% with a slight loss of recognition quality of no more than 4%. For practitioner programmers, the proposed algorithm was tested on real images to distinguish animals and showed similar results as the MNIST-like tests.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
Methods of Computer Image Processing, Soifer, V.A., Ed., Moscow: Fizmatlit, 2003
Forsyth, D.A. and Ponce, J., Computer Vision. A Modern Approach, Upper Saddle River, NJ: Pearson Education, 2003.
Shapiro, L.G. and Stockman, G.C., Computer Vision, Prentice Hall, 2001.
Hastie, R., Tibshirani, R. and Friedman, J., The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, 2009, vol. 2, pp. 1–758.
Ahmed, N. and Rao, K.R., Orthogonal Transforms for Digital Signal Processing, Berlin, 1975.
Ranzato, M.A., Poultney, C., Chopra, S., and LeCun, Y., Efficient learning of sparse representations with an energy-based model, in Advances in Neural Information Processing Systems (NIPS 2006), Platt J., Eds., MIT Press, 2006.
Galushkin, A.I., Tomashevich, D.S., and Tomashevich, N.S., Methods for implementing invariance to affine transformations of two-dimensional images, Inf. Tekhnol., 2001, suppl. no. 1, pp. 1–19.
Phillips, P., Martin, A., Wilson, C., and Przybocki, M., An introduction to evaluation biometric systems, Computer, 2000, vol. 33, pp. 56–63.
Gusev, V.Yu. and Krapivenko, A.V., Method for filtering periodic noise in digital images, Tr. Mosk. Aviats. Inst., Radiotekhn., Elektron., Telekommun. Sist., 2012, no. 50. http://trudymai.ru/published.php?ID=28805.
Kolmogorov, A.N., On the representation of continuous functions of several variables as superpositions of continuous functions of one variable and addition, Dokl. Akad. Nauk SSSR, 1957, vol. 114, no. 5, pз. 953–956.
Arnold, V.I., On the representation of functions of several variables as a superposition of functions of a smaller number of variables, Mat. Prosveshchenie, 1958, no. 3, pp. 41–61.
Gorban, A.N., Generalized approximation theorem and computational capabilities of neural networks, Sib. Zh. Vychisl. Mat., 1998, vol. 1, no. 1, pp. 12–24.
Hecht-Nielsen, R., Neurocomputing, Addison-Wesley, 1989.
Widrow, B., Adaptive sampled-data systems, a statistical theory of adaptation, IRE WESCON Convent. Rec., 1959, vol. 4, pp. 74–85.
Tsypkin, Ya.Z., Information Theory of Identification, Moscow: Nauka, Fizmatlit, 1995.
Kharkevich, A.A., Selected Works in Three Volumes, vol. 3: Information Theory. Image Recognition, Moscow: Nauka, 1973.
McCulloch, W.S. and Pitts, W., A logical calculus of the ideas immanent in nervous activity, Bull. Math. Biophys., 1943, vol. 5, pp. 115–133.
Vershkov, N.A., Kuchukov, V.A., Kuchukova, N.N., and Babenko, M., The wave model of artificial neural network, in Proc. 2020 IEEE Conf. of Russian Young Researchers in Electrical and Electronic Engineering, EIConRus 2020, Moscow, St. Petersburg, 2020, pp. 542–547.
Malykhina, G.F. and Merkusheva, A.V., A method for monitoring the state of a subsystem (object) with incomplete measurement information about the set of parameters that determine its dynamics, Nauch. Priborostroen., 2004, vol. 14, no. 1, pp. 72–84.
Kim, J.S., Cho, Y., and Lim, T.H., Prediction of locations in medical images using orthogonal neural networks, Eur. J. Radiol. Open, 2021, vol. 8, p. 100388.
Jamal, A., Ashour, M., Helmi, R., and Fong, S., A wavelet-neural networks model for time series, in Proc. 11th IEEE Symp. on Computer Applications Industrial Electronics (ISCAIE), Penang, 2021. https://doi.org/10.1109/ISCAIE51753.2021.9431777
Khaustov, P.A., Grigoriev, D.S., and Spitsyn, V.G., Development of an optical character recognition system based on the combined use of a probabilistic neural network and wavelet transform, Izv. Tomsk. Politekh. Univ. Inzh. Georesur., 2013, vol. 323, no. 5, pp. 101–105.
LeCun, Y. and Bengio, Y., Convolutional networks for images, speech, and time-series, in The Handbook of Brain Theory and Neural Networks, Arbib, M.A., Ed., MIT Press, 1995.
Fujieda, S., Takayama, K., and Hachisuka, T., Wavelet Convolutional Neural Networks, 2018. https://arxiv.org/pdf/1805.08620.pdf.
Veitch, D., Wavelet Neural Networks and Their Application in the Study of Dynamical System, Dep. Mathematics Univ. of York, 2005.
Nagornov, O.V., Nikitaev, V.G., Prostokishin, V.M., Tyuflin, S.A., Pronichev, A.N., Bukharova, S.A., Chistov, K.S., Kashafutdinov, R.Z., and Khorkin, V.A., Wavelet Analysis in Examples, Moscow: NRNU MIPT, 2010.
Kerenidis, I., Landman, J. and Mathur, N., Classical and Quantum Algorithms for Orthogonal Neural Networks, 2021. https://arxiv.org/2106.07198v1.
Yubei, J., Chakraborty, Ch., and Yu, S.X., Orthogonal convolutional neural networks, in Proc. 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Seattle, 2020. https://doi.org/10.1109/CVPR42600.2020.01152
Smolentsev, N.K., Fundamentals of Wavelet Theory. Wavelets in Matlab, Moscow: Radio i svyaz’, 2019.
Sikarev, A.A. Lebedev, O.N., Microelectronic Devices for the Formation and Processing of Complex Signals, Moscow: Radio i svyaz’, 1983.
Koehler, G., MNIST Handwritten Digit Recognition in PyTorch. Nextjournal, 2020. https://nextjournal.com/gkoehler/pytorch-mnist. Accessed 28.02.2022.
Qiao, Yu., The MNIST Database of handwritten digits, 2007. http://yann.lecun.com/exdb/mnist/. Accessed 04.03.2022.
GitHub – rois-codh/kmnist: Repository for Kuzushiji-MNIST, Kuzushiji-49, and Kuzushiji-Kanji. https://github.com/rois-codh/kmnist. Accessed 04.03.2022.
A MNIST-like fashion product database. Benchmark. https://github.com/zalandoresearch/fashion-mnist. Accessed 04.03.2022.
Cats and Dogs Dataset. https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip. Accessed 04.03.2022.
PyTorch. https://pytorch.org/get-started/previous-versions/. Accessed 04.03.2022.
ACKNOWLEDGMENTS
This work was supported in part by the Russian Science Foundation, project no. 19-71-10033.
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Vershkov, N., Babenko, M., Tchernykh, A. et al. Optimization of Artificial Neural Networks using Wavelet Transforms. Program Comput Soft 48, 376–384 (2022). https://doi.org/10.1134/S036176882206007X
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S036176882206007X