Abstract
Deep neural networks (DNNs) have achieved impressive success in a variety of classification tasks. However, the presence of noisy labels in training dataset adversely affects the performance of DNNs. Recently, numerous noise-robust loss functions have been proposed to combat the noisy label problem. However, we find that these loss functions are either slow to learn the potential data pattern or not sufficiently robust against noisy labels. Here, we propose an improved categorical cross entropy (ICCE) to deal with this challenge. The ICCE can automatically adjust the weighting scheme based on the predicted probability distribution of DNNs by an exponential item, which makes it gain strong noise robustness and fast learning ability. A theoretical analysis of the ICCE is presented in the context of noisy labels. Experiments on datasets indicate that the ICCE can better improve the performance of DNNs even under high-level noise.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Algan, G., Ulusoy, I.: Image classification with deep learning in the presence of noisy labels: a survey. arXiv preprint arXiv:1912.05170 (2019)
Arpit, D., et al.: A closer look at memorization in deep networks. arXiv: Machine Learning (2017)
Azadi, S., Feng, J., Jegelka, S., Darrell, T.: Auxiliary image regularization for deep CNNs with noisy labels. arXiv: Computer Vision and Pattern Recognition (2015)
Ding, G., Guo, Y., Chen, K., Chu, C., Han, J., Dai, Q.: DECODE: deep confidence network for robust image classification. IEEE Trans. Image Process. 28(8), 3752–3765 (2019)
Frenay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE Trans. Neural Networks 25(5), 845–869 (2014)
Gevaert, C.M., Persello, C., Elberink, S.O., Vosselman, G., Sliuzas, R.: Context-based filtering of noisy labels for automatic basemap updating from UAV data. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 11(8), 2731–2741 (2017)
Ghosh, A., Kumar, H., Sastry, P.: Robust loss functions under label noise for deep neural networks. In: Thirty-First AAAI Conference on Artificial Intelligence, pp. 1919–1925
Jiang, L., Zhou, Z., Leung, T., Li, L., Fei-Fei, L.M.: Regularizing very deep neural networks on corrupted labels. ICML (2018)
Karimi, D., Dou, H., Warfield, S.K., Gholipour, A.: Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med. Image Anal. 65, 101759 (2020)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks, pp. 1097–1105 (2012)
Manwani, N., Sastry, P.: Noise tolerance under risk minimization. IEEE Trans. Cybernet. 43(3), 1146–1151 (2013)
Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., Qu, L.: Making deep neural networks robust to label noise: a loss correction approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944–1952 (2017)
Pelletier, C., Valero, S., Inglada, J., Dedieu, G., Champion, N.: Filtering mislabeled data for improving time series classification. In: 2017 9th International Workshop on the Analysis of Multitemporal Remote Sensing Images (MultiTemp), pp. 1–4. IEEE (2017)
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., Rabinovich, A.: Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596 (2014)
Wang, X., Hua, Y., Kodirov, E., Robertson, N.M.: IMAE for noise-robust learning: mean absolute error does not treat examples equally and gradient magnitude’s variance matters (2019)
Yuan, J.: Learning building extraction in aerial scenes with convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 40(11), 2793–2798 (2018)
Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31, pp. 8778–8788. Curran Associates, Inc. (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, P. et al. (2021). Improved Categorical Cross-Entropy Loss for Training Deep Neural Networks with Noisy Labels. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13022. Springer, Cham. https://doi.org/10.1007/978-3-030-88013-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-88013-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88012-5
Online ISBN: 978-3-030-88013-2
eBook Packages: Computer ScienceComputer Science (R0)