Philipp Wiesner
Lauritz Thamsen
ABSTRACT
Federated Learning (FL) is an emerging machine learning technique that enables distributed model training across data silos or edge devices without data sharing. Yet, FL inevitably introduces inefficiencies compared to centralized model training, which will further increase the already high energy usage and associated carbon emissions of machine learning in the future. One idea to reduce FL’s carbon footprint is to schedule training jobs based on the availability of renewable excess energy that can occur at certain times and places in the grid. However, in the presence of such volatile and unreliable resources, existing FL schedulers cannot always ensure fast, efficient, and fair training.
We propose FedZero, an FL system that operates exclusively on renewable excess energy and spare capacity of compute infrastructure to effectively reduce a training’s operational carbon emissions to zero. Using energy and load forecasts, FedZero leverages the spatio-temporal availability of excess resources by selecting clients for fast convergence and fair participation. Our evaluation, based on real solar and load traces, shows that FedZero converges significantly faster than existing approaches under the mentioned constraints while consuming less energy. Furthermore, it is robust to forecasting errors and scalable to tens of thousands of clients.
- Ahmed M. Abdelmoniem, Atal Narayan Sahu, Marco Canini, and Suhaib A. Fahmy. 2023. REFL: Resource-Efficient Federated Learning. In EuroSys. ACM. https://doi.org/10.1145/3552326.3567485Google Scholar
Digital Library
- David B. Alencar, Carolina de Mattos Affonso, Roberto C. L. Oliveira, Jorge Laureano Moya Rodríguez, Jandecy Cabral Leite, and Jose Carlos R. Filho. 2017. Different Models for Forecasting Wind Power Generation: Case Study. Energies 10 (2017). https://doi.org/10.3390/en10121976Google ScholarCross Ref
- Amazon. 2022. Amazon’s 2022 Sustainability Report. (2022).Google Scholar
- Amna Arouj and Ahmed M. Abdelmoniem. 2022. Towards Energy-Aware Federated Learning on Battery-Powered Clients. In Workshop on Data Privacy and Federated Learning Technologies for Mobile Edge Network at ACM MobiCom.Google Scholar
- World Bank. 2022. State and Trends of Carbon Pricing 2022. Technical Report. Washington, DC: World Bank.Google Scholar
- Noman Bashir, David Irwin, Prashant Shenoy, and Abel Souza. 2022. Sustainable Computing - Without the Hot Air. In HotCarbon.Google Scholar
- Axel Berg, Mark O’Connor, and Miguel Tairum Cruz. 2021. Keyword Transformer: A Self-Attention Model for Keyword Spotting. In Proc. Interspeech 2021. 4249–4253. https://doi.org/10.21437/Interspeech.2021-1286Google ScholarCross Ref
- Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, and Nicholas D Lane. 2020. Flower: A Friendly Federated Learning Research Framework. arXiv preprint arXiv:2007.14390 (2020).Google Scholar
- Anders Bjørn, Shannon M. Lloyd, Matthew Brander, and H. Damon Matthews. 2022. Renewable energy certificates threaten the integrity of corporate science-based targets. Nature Climate Change 12, 6 (2022).Google Scholar
- Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. 2019. Towards Federated Learning at Scale: System Design. In MLSys. https://proceedings.mlsys.org/paper/2019/file/bd686fd640be98efaae0091fa301e613-Paper.pdfGoogle Scholar
- Jamie M. Bright, Sven Killinger, David Lingfors, and Nicholas A. Engerer. 2018. Improved satellite-derived PV power nowcasting using real-time power data from reference PV systems. Solar Energy 168 (2018). https://doi.org/10.1016/j.solener.2017.10.091Google ScholarCross Ref
- Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2019. LEAF: A Benchmark for Federated Settings. In Workshop on Federated Learning for Data Privacy and Confidentiality at NeurIPS.Google Scholar
- California ISO. 2024. Managing oversupply. http://www.caiso.com/informed/Pages/ManagingOversupply.aspx. accessed Jan. 2024.Google Scholar
- Andrew Chien, Chaojie Zhang, Liuzixuan Lin, and Varsha Rao. 2022. Beyond PUE: Flexible Datacenters Empowering the Cloud to Decarbonize. In HotCarbon.Google Scholar
- Andrew A Chien, Chaojie Zhang, and Hai Duc Nguyen. 2019. Zero-carbon Cloud: Research Challenges for Datacenters as Supply-following Loads. University of Chicago, Tech. Rep. CS-TR-2019-08 (2019).Google Scholar
- Payal Dhar. 2020. The carbon impact of artificial intelligence. Nature Machine Intelligence 2 (2020), 423–425.Google ScholarCross Ref
- Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A. Smith, Nicole DeCario, and Will Buchanan. 2022. Measuring the Carbon Intensity of AI in Cloud Instances. In ACM FAccT. https://doi.org/10.1145/3531146.3533234Google ScholarDigital Library
- Jonatan Enes, Guillaume Fieni, Roberto R. Expósito, Romain Rouvoy, and Juan Touriño. 2020. Power Budgeting of Big Data Applications in Container-based Clusters. In IEEE CLUSTER.Google Scholar
- Gilbert Fridgen, Marc-Fabian Körner, Steffen Walters, and Martin Weibelzahl. 2021. Not All Doom and Gloom: How Energy-Intensive and Temporally Flexible Data Center Applications May Actually Promote Renewable Energy Sources. Business & Information Systems Engineering 63, 3 (2021).Google Scholar
- Google. 2022. 2022 Environmental Report. (2022).Google Scholar
- Başak Güler and Aylin Yener. 2021. A Framework for Sustainable Federated Learning. In 2021 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt). https://doi.org/10.23919/WiOpt52861.2021.9589930Google ScholarCross Ref
- Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. arXiv preprint arXiv:1909.06335 (2019).Google Scholar
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely Connected Convolutional Networks. In CVPR.Google Scholar
- Yae Jee Cho, Jianyu Wang, and Gauri Joshi. 2022. Towards Understanding Biased Client Selection in Federated Learning. In AISTATS.Google Scholar
- Ji Chu Jiang, Burak Kantarci, Sema Oktug, and Tolga Soyata. 2020. Federated Learning in Smart City Sensing: Challenges and Opportunities. Sensors 20, 21 (2020).Google Scholar
- Zhifeng Jiang, Wei Wang, Baochun Li, and Bo Li. 2022. Pisces: Efficient Federated Learning via Guided Asynchronous Training. In ACM Symposium on Cloud Computing (SoCC). https://doi.org/10.1145/3542929.3563463Google ScholarDigital Library
- Lucas Joppa. 2021. Made to measure: Sustainability commitment progress and updates. Microsoft. Retrieved Sept. 2023 from https://blogs.microsoft.com/blog/2021/07/14/made-to-measure-sustainability-commitment-progress-and-updatesGoogle Scholar
- Alexandra I. Khalyasmaa, Stanislav A. Eroshenko, T. Chakravarthy, Venu Gopal Gasi, Sandeep Kumar Yadav Bollu, Raphael Caire, Sai Kumar Reddy Atluri, and Suresh Karrolla. 2019. Prediction of Solar Power Generation Based on Random Forest Regressor Model. In IEEE SIBIRCON. https://doi.org/10.1109/SIBIRCON48586.2019.8958063Google ScholarCross Ref
- Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Technical Report.Google Scholar
- Fan Lai, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. 2021. Oort: Efficient Federated Learning via Guided Participant Selection. In USENIX OSDI. https://www.usenix.org/conference/osdi21/presentation/laiGoogle Scholar
- Chenning Li, Xiao Zeng, Mi Zhang, and Zhichao Cao. 2022. PyramidFL: A Fine-Grained Client Selection Framework for Efficient Federated Learning. In ACM MobiCom. https://doi.org/10.1145/3495243.3517017Google ScholarDigital Library
- Qing’an Li, Chang Cai, Yasunari Kamada, Takao Maeda, Yuto Hiromori, Shuni Zhou, and Jianzhong Xu. 2021. Prediction of power generation of two 30 kW Horizontal Axis Wind Turbines with Gaussian model. Energy 231 (2021). https://doi.org/10.1016/j.energy.2021.121075Google ScholarCross Ref
- Shaohong Li, Xi Wang, Xiao Zhang, Vasileios Kontorinis, Sreekumar Kodakara, David Lo, and Parthasarathy Ranganathan. 2020. Thunderbolt: Throughput-Optimized, Quality-of-Service-Aware Power Capping at Scale. In USENIX OSDI. https://www.usenix.org/conference/osdi20/presentation/li-shaohongGoogle Scholar
- Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. In MLSys.Google Scholar
- Liuzixuan Lin, Victor M. Zavala, and Andrew Chien. 2021. Evaluating Coupling Models for Cloud Datacenters and Power Grids. In ACM e-Energy. https://doi.org/10.1145/3447555.3464868Google ScholarDigital Library
- Longjun Liu, Hongbin Sun, Chao Li, Tao Li, Jingmin Xin, and Nanning Zheng. 2017. Managing Battery Aging for High Energy Availability in Green Datacenters. IEEE Transactions on Parallel and Distributed Systems 28, 12 (2017). https://doi.org/10.1109/TPDS.2017.2712778Google ScholarDigital Library
- H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2016. Communication-Efficient Learning of Deep Networks from Decentralized Data. In AISTATS.Google Scholar
- Microsoft. 2022. 2022 Environmental Sustainability Report. (2022).Google Scholar
- Rakshit Naidu, Harshita Diddee, Ajinkya K Mulay, Aleti Vardhan, Krithika Ramesh, and Ahmed Zamzam. 2021. Towards Quantifying the Carbon Emissions of Differentially Private Machine Learning. In Workshop on Socially Responsible Machine Learning at ICML.Google Scholar
- Anh Nguyen, Tuong Do, Minh Tran, Binh X. Nguyen, Chien Duong, Tu Phan, Erman Tjiputra, and Quang D. Tran. 2022. Deep Federated Learning for Autonomous Driving. In 2022 IEEE Intelligent Vehicles Symposium (IV).Google Scholar
- Jake Oster. 2022. How we count carbon emissions from electricity matters. Amazon. Retrieved Sept. 2023 from https://www.amazon.science/blog/how-we-count-carbon-emissions-from-electricity-mattersGoogle Scholar
- David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. 2022. The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink. Computer 55, 7 (2022). https://doi.org/10.1109/MC.2022.3148714Google ScholarDigital Library
- Xinchi Qiu, Titouan Parcollet, Javier Fernandez-Marques, Pedro Porto Buarque de Gusmao, Daniel J. Beutel, Taner Topal, Akhil Mathur, and Nicholas D. Lane. 2021. A first look into the carbon footprint of federated learning. arXiv preprint arXiv:2102.07627 (2021).Google Scholar
- Ana Radovanovic, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, Saurav Talukdar, Eric Mullen, Kendal Smith, Mariellen Cottman, and Walfredo Cirne. 2022. Carbon-Aware Computing for Datacenters. IEEE Transactions on Power Systems (2022).Google Scholar
- Martin Rapp, Ramin Khalili, Kilian Pfeiffer, and Jörg Henkel. 2022. DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems. In AAAI.Google Scholar
- REN21. 2022. Renewables 2022 Global Status Report. (2022).Google Scholar
- Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletarì, Holger R. Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N. Galtier, Bennett A. Landman, Klaus Maier-Hein, Sébastien Ourselin, Micah Sheller, Ronald M. Summers, Andrew Trask, Daguang Xu, Maximilian Baust, and M. Jorge Cardoso. 2020. The future of digital health with federated learning. npj Digital Medicine 3, 1 (2020).Google Scholar
- René Schwermer, Ruben Mayer, and Hans-Arno Jacobsen. 2023. Energy vs Privacy: Estimating the Ecological Impact of Federated Learning. In ACM e-Energy.Google Scholar
- Jinhyun So, Kevin Hsieh, Behnaz Arzani, Shadi Noghabi, Salman Avestimehr, and Ranveer Chandra. 2022. FedSpace: An Efficient Federated Learning Framework at Satellites and Ground Stations. arXiv preprint arXiv:2202.01267 (2022).Google Scholar
- Behnaz Soltani, Venus Haghighi, Adnan Mahmood, Quan Z. Sheng, and Lina Yao. 2022. A Survey on Participant Selection for Federated Learning in Mobile Networks. In Workshop on Mobility in the Evolving Internet Architecture (MobiArch) at MobiCom.Google ScholarDigital Library
- Abel Souza, Noman Bashir, Jorge Murillo, Walid Hanafy, Qianlin Liang, David Irwin, and Prashant Shenoy. 2023. Ecovisor: A Virtual Energy System for Carbon-Efficient Applications. In ASPLOS.Google Scholar
- Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020. Energy and Policy Considerations for Modern Deep Learning Research. In AAAI.Google Scholar
- Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In ICML.Google Scholar
- Maud Texier. 2021. A timely new approach to certifying clean energy. Google. Retrieved Sept. 2023 from https://cloud.google.com/blog/topics/sustainability/t-eacs-offer-new-approach-to-certifying-clean-energyGoogle Scholar
- Muhammad Tirmazi, Adam Barker, Nan Deng, Md E. Haque, Zhijing Gene Qin, Steven Hand, Mor Harchol-Balter, and John Wilkes. 2020. Borg: The next Generation. In EuroSys.Google ScholarDigital Library
- Cong Wang, Bin Hu, and Hongyi Wu. 2022. Energy Minimization for Federated Asynchronous Learning on Battery-Powered Mobile Devices via Application Co-running. In ICDCS.Google Scholar
- Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In USENIX NSDI.Google Scholar
- Philipp Wiesner, Ilja Behnke, and Odej Kao. 2023. A Testbed for Carbon-Aware Applications and Systems. arxiv:2306.09774 [cs.DC]Google Scholar
- Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Kordian Gontarska, and Lauritz Thamsen. 2021. Let’s Wait Awhile: How Temporal Workload Shifting Can Reduce Carbon Emissions in the Cloud. In ACM Middleware.Google Scholar
- Philipp Wiesner, Dominik Scheinert, Thorsten Wittkopp, Lauritz Thamsen, and Odej Kao. 2022. Cucumber: Renewable-Aware Admission Control for Delay-Tolerant Cloud and Edge Workloads. In International European Conference on Parallel and Distributed Computing (Euro-Par).Google Scholar
- Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim M. Hazelwood. 2022. Sustainable AI: Environmental Implications, Challenges and Opportunities. In MLSys.Google Scholar
- Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. 2019. Federated learning. Morgan & Claypool Publishers.Google Scholar
- Zhaohui Yang, Mingzhe Chen, Walid Saad, Choong Seon Hong, and Mohammad Shikh-Bahaei. 2021. Energy Efficient Federated Learning Over Wireless Communication Networks. IEEE Transactions on Wireless Communications 20, 3 (2021).Google ScholarDigital Library
- Ashkan Yousefpour, Shen Guo, Ashish Shenoy, Sayan Ghosh, Pierre Stock, Kiwan Maeng, Schalk-Willem Krüger, Michael Rabbat, Carole-Jean Wu, and Ilya Mironov. 2023. Green Federated Learning. arxiv:2303.14604 [cs.LG]Google Scholar
- Sai Qian Zhang, Jieyu Lin, and Qi Zhang. 2022. A Multi-Agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learning. In AAAI. https://doi.org/10.1609/aaai.v36i8.20894Google ScholarCross Ref
- Jiajia Zheng, Andrew A. Chien, and Sangwon Suh. 2020. Mitigating Curtailment and Carbon Emissions through Load Migration between Data Centers. Joule 4, 10 (2020).Google Scholar
- Zhi Zhou, Fangming Liu, Yong Xu, Ruolan Zou, Hong Xu, John C.S. Lui, and Hai Jin. 2013. Carbon-Aware Load Balancing for Geo-distributed Cloud Services. In 21st Int. Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS).Google Scholar
- Chen Zhu, Zheng Xu, Mingqing Chen, Jakub Konečný, Andrew Hard, and Tom Goldstein. 2022. Diurnal or Nocturnal? Federated Learning of Multi-branch Networks from Periodically Shifting Distributions. In ICLR.Google Scholar
Index Terms
- FedZero: Leveraging Renewable Excess Energy in Federated Learning
Recommendations
The War of the Efficiencies: Understanding the Tension between Carbon and Energy Optimization
HotCarbon '23: Proceedings of the 2nd Workshop on Sustainable Computer SystemsMajor innovations in computing have been driven by scaling up computing infrastructure, while aggressively optimizing operating costs. The result is a network of worldwide datacenters that consume a large amount of energy, mostly in an energy-...
A first look into the carbon footprint of federated learning
Despite impressive results, deep learning-based technologies also raise severe privacy and environmental concerns induced by the training procedure often conducted in data centers. In response, alternatives to centralized training such as Federated ...
CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency
POMACSCloud platforms are increasing their emphasis on sustainability and reducing their operational carbon footprint. A common approach for reducing carbon emissions is to exploit the temporal flexibility inherent to many cloud workloads by executing them in ...
Comments