Jinghui Zhang
Abstract
As a new paradigm in training intelligent models, federated learning is widely used to train a global model without requiring local data to be uploaded from end devices. However, there are often mislabeled samples (i.e., noisy samples) in the dataset, which will cause the model update to deviate from the correct direction during the training process, thus reducing the convergence accuracy of the global model. Existing works employ noisy label correction techniques to reduce the impact of noisy samples on model updates by correcting labels; however, such methods necessitate the use of prior knowledge and additional communication costs, which cannot be directly applied to federated learning due to data privacy concerns and limited communication resources. Therefore, this paper proposes a noise-aware local model training method that corrects the noisy labels directly at the end device under the constraints of federated learning. By constructing a label correction model, a joint optimization problem is formally defined for optimizing both the label correction model and the client-side local training model (e.g., classification model). As a solution to this optimization problem, we propose a robustness training algorithm using label correction, along with a cross-validation data sampling algorithm that updates both models simultaneously. It is verified through experiments that the mechanism can effectively improve the model convergence accuracy on noisy datasets in federated learning scenarios.
- [1] . 2019. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning. PMLR, 1062–1070.Google Scholar
- [2] . 2020. Dealing with label quality disparity in federated learning. Federated Learning: Privacy and Incentive (2020), 108–121.Google ScholarCross Ref
- [3] . 2022. Label distribution-based noise correction for multiclass crowdsourcing. International Journal of Intelligent Systems 37, 9 (2022), 5752–5767.Google ScholarDigital Library
- [4] . 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. PMLR, 1126–1135.Google ScholarDigital Library
- [5] . 2013. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems 25, 5 (2013), 845–869.Google ScholarCross Ref
- [6] . 2018. Masking: A new perspective of noisy supervision. Advances in Neural Information Processing Systems 31 (2018).Google Scholar
- [7] . 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems 31 (2018).Google Scholar
- [8] . 2021. Learning cross-modal retrieval with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5403–5413.Google ScholarCross Ref
- [9] . 2018. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. PMLR, 2304–2313.Google Scholar
- [10] . 2022. SAN: Attention-based social aggregation neural networks for recommendation system. International Journal of Intelligent Systems 37, 6 (2022), 3373–3393.Google ScholarDigital Library
- [11] . 2016. The unreasonable effectiveness of noisy data for fine-grained recognition. In European Conference on Computer Vision. Springer, 301–320.Google ScholarCross Ref
- [12] . 2009. Learning multiple layers of features from tiny images. Citeseer (2009).Google Scholar
- [13] . 2021. Optimal feature selection based speech emotion recognition using two-stream deep convolutional neural network. International Journal of Intelligent Systems 36, 9 (2021), 5116–5135.Google ScholarDigital Library
- [14] . 2019. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning. PMLR, 3763–3772.Google Scholar
- [15] . 2018. CleanNet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5447–5456.Google ScholarCross Ref
- [16] . 2020. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning. PMLR, 6226–6236.Google Scholar
- [17] . 2022. Federated social recommendation with graph neural network. ACM Trans. Intell. Syst. Technol. 13, 4, Article
55 (Aug. 2022), 24 pages. Google ScholarDigital Library - [18] . 2019. Artificial intelligence: A survey on evolution, models, applications and future trends. Journal of Management Analytics 6, 1 (2019), 1–29.Google ScholarCross Ref
- [19] . 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. PMLR, 1273–1282.Google Scholar
- [20] . 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 5, 5 (2010), 411–419.Google ScholarCross Ref
- [21] . 2017. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1944–1952.Google ScholarCross Ref
- [22] . 2014. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596 (2014).Google Scholar
- [23] . 2019. Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in Neural Information Processing Systems 32 (2019).Google Scholar
- [24] . 2019. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning. PMLR, 5907–5915.Google Scholar
- [25] . 2020. Learning generalizable and identity-discriminative representations for face anti-spoofing. ACM Trans. Intell. Syst. Technol. 11, 5, Article
60 (Jul. 2020), 19 pages. Google ScholarDigital Library - [26] . 2017. Toward robustness against label noise in training deep discriminative neural networks. Advances in Neural Information Processing Systems 30 (2017).Google Scholar
- [27] . 2021. Tackling instance-dependent label noise via a universal probabilistic model. arXiv preprint arXiv:2101.05467 (2021).Google Scholar
- [28] . 2021. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems 34 (2021).Google Scholar
- [29] . 2020. Robust early-learning: Hindering the memorization of noisy labels. In International Conference on Learning Representations.Google Scholar
- [30] . 2019. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems 32 (2019).Google Scholar
- [31] . 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).Google Scholar
- [32] . 2015. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2691–2699.Google Scholar
- [33] . 2022. Robust federated learning with noisy labels. IEEE Intelligent Systems 37, 2 (2022), 35–43.Google ScholarCross Ref
- [34] . 2022. CLC: A consensus-based label correction approach in federated learning. ACM Transactions on Intelligent Systems and Technology (TIST) 13, 5 (2022), 1–23.Google ScholarDigital Library
- [35] . 2018. Deep learning for environmentally robust speech recognition: An overview of recent developments. ACM Trans. Intell. Syst. Technol. 9, 5, Article
49 (Apr. 2018), 28 pages. Google ScholarDigital Library - [36] . 2021. Meta label correction for noisy label learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’21), Vol. 35. 11053–11061.Google ScholarCross Ref
- [37] . 2021. Learning to disentangle emotion factors for facial expression recognition in the wild. International Journal of Intelligent Systems 36, 6 (2021), 2511–2527.Google ScholarDigital Library
Index Terms
- Noise-aware Local Model Training Mechanism for Federated Learning
Recommendations
Labeling Chaos to Learning Harmony: Federated Learning with Noisy Labels
Federated Learning (FL) is a distributed machine learning paradigm that enables learning models from decentralized private datasets where the labeling effort is entrusted to the clients. While most existing FL approaches assume high-quality labels are ...
Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementFederated learning (FL) is important for privacy-preserving services by training models without collecting raw user data. Most FL algorithms assume all data is annotated, which is impractical due to the high cost of labeling data in real applications. To ...
Meta joint optimization: a holistic framework for noisy-labeled visual recognition
AbstractCollecting large-scale data with clean labels for supervised training is practically challenging. It is easier to collect a dataset with noisy labels, but such noise may degrade the performance of deep neural networks (DNNs). This paper targets at ...
Comments