Abstract
Federated training of large deep neural networks can often be restrictive due to the increasing costs of communicating the updates with increasing model sizes. Various model pruning techniques have been designed in centralized settings to reduce inference times. Combining centralized pruning techniques with federated training seems intuitive for reducing communication costs—by pruning the model parameters right before the communication step. Moreover, such a progressive model pruning approach during training can also reduce training times/costs. To this end, we propose FedSparsify, which performs model pruning during federated training. In our experiments in centralized and federated settings on the brain age prediction task (estimating a person’s age from their brain MRI), we demonstrate that models can be pruned up to 95% sparsity without affecting performance even in challenging federated learning environments with highly heterogeneous data distributions. One surprising benefit of model pruning is improved model privacy. We demonstrate that models with high sparsity are less susceptible to membership inference attacks, a type of privacy attack.
D. Stripelis and U. Gupta—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Communication cost is computed as \(\sum _t^T 2 N_Z^t L \). T represents the total number of federation rounds, \(N_Z^t\) the non-zero model parameters at round t and L the number of participating learners. Factor 2 accounts for the model parameters sent from the controller to the learners and from the learners to the controller within a round.
- 3.
References
Bibikar, S., Vikalo, H., Wang, Z., Chen, X.: Federated dynamic sparse training: computing less, communicating less, yet learning better (2021)
Cole, J.H., Leech, R., Sharp, D.J., Alzheimer’s Disease Neuroimaging Initiative: Prediction of brain age suggests accelerated atrophy after traumatic brain injury. Ann. Neurol. 77(4), 571–581 (2015)
Dayan, I., et al.: Federated learning for predicting clinical outcomes in patients with Covid-19. Nat. Med. 27(10), 1735–1743 (2021)
Ezzati, A., et al.: Predictive value of ATN biomarker profiles in estimating disease progression in Alzheimer’s disease dementia. Alzheimer’s & Dementia 17(11), 1855–1867 (2021)
Farokhi, F., Kaafar, M.A.: Modelling and quantifying membership information leakage in machine learning. arXiv preprint arXiv:2001.10648 (2020)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: International Conference on Learning Representations (2018)
Gupta, U., Lam, P.K., Ver Steeg, G., Thompson, P.M.: Improved brain age estimation with slice-based set networks. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 840–844. IEEE (2021)
Gupta, U., Stripelis, D., Lam, P.K., Thompson, P., Ambite, J.L., Ver Steeg, G.: Membership inference attacks on deep regression models for neuroimaging. In: Medical Imaging with Deep Learning, pp. 228–251. PMLR (2021)
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., Peste, A.: Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res. 22(241), 1–124 (2021)
Jayaraman, B., Wang, L., Evans, D., Gu, Q.: Revisiting membership inference under realistic assumptions. arXiv preprint arXiv:2005.10881 (2020)
Jha, S.K., et al.: An extension of Fano’s inequality for characterizing model susceptibility to membership inference attacks. arXiv preprint arXiv:2009.08097 (2020)
Jiang, Y., et al.: Model pruning enables efficient federated learning on edge devices. IEEE Trans. Neural Netw. Learn. Syst. (2022)
Jónsson, B.A., et al.: Brain age prediction using deep learning uncovers associated sequence variants. Nat. Commun. 10(1), 1–10 (2019)
Kurtz, M., et al.: Inducing and exploiting activation sparsity for fast inference on deep neural networks. In: International Conference on Machine Learning, pp. 5533–5543. PMLR (2020)
Lam, P.K., et al.: Accurate brain age prediction using recurrent slice-based networks. In: 16th International Symposium on Medical Information Processing and Analysis, vol. 11583, p. 1158303. International Society for Optics and Photonics (2020)
Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37(3), 50–60 (2020)
Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: International Conference on Learning Representations (2018)
McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
Miller, K.L., et al.: Multimodal population brain imaging in the UK biobank prospective epidemiological study. Nat. Neurosci. 19(11), 1523–1536 (2016)
Nasr, M., Shokri, R., Houmansadr, A.: Comprehensive privacy analysis of deep learning: passive and active white-box inference attacks against centralized and federated learning. In: IEEE Symposium on Security and Privacy (SP) (2019)
Nasr, M., Shokri, R., Houmansadr, A.: Machine learning with membership privacy using adversarial regularization. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 634–646 (2018)
Peng, H., Gong, W., Beckmann, C.F., Vedaldi, A., Smith, S.M.: Accurate brain age prediction with lightweight deep neural networks. Med. Image Anal. 68, 101871 (2021)
Pustozerova, A., Mayer, R.: Information leaks in federated learning. In: Proceedings of the Network and Distributed System Security Symposium, vol. 10 (2020)
Rieke, N., et al.: The future of digital health with federated learning. NPJ Digit. Med. 3(1), 1–7 (2020)
Ro, J.H., et al.: Scaling language model size in cross-device federated learning. In: ACL Workshop on Federated Learning for Natural Language Processing (2022)
Sheller, M.J., et al.: Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci. Rep. 10(1), 1–12 (2020)
Stripelis, D., Ambite, J.L., Lam, P., Thompson, P.: Scaling neuroscience research using federated learning. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1191–1195. IEEE (2021)
Stripelis, D., Gupta, U., Steeg, G.V., Ambite, J.L.: Federated progressive sparsification (purge, merge, tune)+. arXiv preprint arXiv:2204.12430 (2022)
Stripelis, D., Thompson, P.M., Ambite, J.L.: Semi-synchronous federated learning for energy-efficient training and accelerated convergence in cross-silo settings. ACM Trans. Intell. Syst. Technol. (TIST) (2022)
Truex, S., Liu, L., Gursoy, M.E., Yu, L., Wei, W.: Towards demystifying membership inference attacks. arXiv preprint arXiv:1807.09173 (2018)
Wainberg, M., Merico, D., Delong, A., Frey, B.J.: Deep learning in biomedicine. Nat. Biotechnol. 36(9), 829–838 (2018)
Yang, Q., Liu, Y., Cheng, Y., Kang, Y., Chen, T., Yu, H.: Federated learning. Synthesis Lectures Artif. Intell. Mach. Learn. 13(3), 1–207 (2019)
Zari, O., Xu, C., Neglia, G.: Efficient passive membership inference attack in federated learning. In: NeurIPS PriML Workshop (2021)
Zhu, B., Liu, J.Z., Cauley, S.F., Rosen, B.R., Rosen, M.S.: Image reconstruction by domain-transform manifold learning. Nature 555(7697), 487–492 (2018)
Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Stripelis, D., Gupta, U., Dhinagar, N., Steeg, G.V., Thompson, P.M., Ambite, J.L. (2022). Towards Sparsified Federated Neuroimaging Models via Weight Pruning. In: Albarqouni, S., et al. Distributed, Collaborative, and Federated Learning, and Affordable AI and Healthcare for Resource Diverse Global Health. DeCaF FAIR 2022 2022. Lecture Notes in Computer Science, vol 13573. Springer, Cham. https://doi.org/10.1007/978-3-031-18523-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-18523-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18522-9
Online ISBN: 978-3-031-18523-6
eBook Packages: Computer ScienceComputer Science (R0)
