Abstract
Deep reinforcement learning (DRL) injects vigorous vitality into congestion control (CC) to efficiently utilize network capacity for Internet communication applications. Existing methods employ a single DRL-based agent to perform CC under Active Queue Management (AQM) or Transmission Control Protocol (TCP) scheme. To enable AQM and TCP to learn to work cooperatively, this paper aims to study CC from a new perspective from the multi-agent system by leveraging multi-agent reinforcement learning (MARL). To this end, we propose a MARL-based Congestion Control framework, MA-CC, which enables senders and routers to gradually learn cross-layer strategies that dynamically adjust congestion window and packet drop rate. We evaluate the proposed scheme in a typical dumbbell-like network model built on the ns-3 simulator. The results show that MA-CC outperforms traditional rule-based and learning-based congestion control algorithms by providing higher throughput while maintaining low transmission latency.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbasloo, S., Yen, C.Y., Chao, H.J.: Classic meets modern: a pragmatic learning-based congestion control for the internet. In: Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, pp. 632–647 (2020)
AlWahab, D.A., Gombos, G., Laki, S.: On a deep Q-Network-based approach for active queue management. In: 2021 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), pp. 371–376. IEEE (2021)
Brakmo, L.S., O’malley, S.W., Peterson, L.L.: TCP Vegas: new techniques for congestion detection and avoidance. In: Proceedings of the Conference on Communications Architectures, Protocols and Applications, pp. 24–35 (1994)
Cardwell, N., Cheng, Y., Gunn, C.S., Yeganeh, S.H., Jacobson, V.: BBR: congestion-based congestion control. Commun. ACM 60(2), 58–66 (2017)
Carlucci, G., De Cicco, L., Holmer, S., Mascolo, S.: Analysis and design of the google congestion control for web real-time communication (WebRTC). In: Proceedings of the 7th International Conference on Multimedia Systems, MMSys 2016, New York, NY, USA. Association for Computing Machinery (2016)
Yawen Chen, Y., et al.: Reinforcement learning meets wireless networks: a layering perspective. IEEE Internet Things J. 8(1), 85–111 (2021)
Dong, M., Li, Q., Zarchy, D., Godfrey, P.B., Schapira, M.: PCC: Re-architecting congestion control for consistent high performance. In: 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pp. 395–408 (2015)
Dong, M., et al.: PCC vivace: online-learning congestion control. In: 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), pp. 343–356 (2018)
Floyd, S., Henderson, T.-T., Gurtov, A.: The NewReno modification to TCP’s fast recovery algorithm. RFC 2582, 05 (1999)
Floyd, S., Jacobson, V.: Random early detection gateways for congestion avoidance. IEEE/ACM Trans. Networking 1(4), 397–413 (1993)
Fu, C.P., Liew, S.C.: TCP Veno: TCP enhancement for transmission over wireless access networks. IEEE J. Sel. Areas Commun. 21(2), 216–228 (2003)
Gawłowicz, P., Zubow, A.: Ns-3 meets OpenAI gym: the playground for machine learning in networking research. In: ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM), p. 11 (2019)
Gettys, J.: Bufferbloat: dark buffers in the internet. IEEE Internet Comput. 15(3), 96 (2011)
Hock, M., Neumeister, F., Zitterbart, M., Bless, R.: TCP LoLa: congestion control for low latencies and high throughput. In: 2017 IEEE 42nd Conference on Local Computer Networks (LCN), pp. 215–218. IEEE (2017)
Jacobson, V.: Modified TCP congestion avoidance algorithm. End2end Interest Mailing List (1990)
Jacobson, V.L.: Congestion avoidance and control. ACM SIGCOMM Comput. Commun. Rev. (1988)
Jiang, H., et al.: When machine learning meets congestion control: a survey and comparison. Comput. Netw. 192, 108033 (2021)
Jin, C., et al.: FAST TCP: from theory to experiments. IEEE Netw. 19(1), 4–11 (2005)
King, R., Baraniuk, R., Riedi, R.: TCP-Africa: an adaptive and fair rapid increase rule for scalable TCP. In: Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies, vol. 3, pp. 1838–1848. IEEE (2005)
Mittal, R., et al.: TIMELY: RTT-based congestion control for the datacenter. ACM SIGCOMM Comput. Commun. Rev. 45(4), 537–550 (2015)
Nichols, K., Jacobson, V.: Controlling queue delay. Commun. ACM 55(7), 42–50 (2012)
Nie, X., et al.: Dynamic TCP initial windows and congestion control schemes through reinforcement learning. IEEE J. Sel. Areas Commun. 37(6), 1231–1247 (2019)
Pan, R., Natarajan, P., Baker, F., White, G.: A lightweight control scheme to address the bufferbloat problem. Technical report, Proportional integral controller enhanced (pie) (2017)
Yuhan, S., Huang, L., Feng, C.: QRED: a q-learning-based active queue management scheme. J. Internet Technol. 19(4), 1169–1178 (2018)
Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085–2087 (2018)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Winstein, K., Balakrishnan, H.: TCP ex machina: computer-generated congestion control. ACM SIGCOMM Comput. Commun. Rev. 43(4), 123–134 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bai, J., Zhang, T., Wang, C., Xie, G. (2023). MA-CC: Cross-Layer Congestion Control via Multi-agent Reinforcement Learning. In: Arai, K. (eds) Intelligent Computing. SAI 2023. Lecture Notes in Networks and Systems, vol 739. Springer, Cham. https://doi.org/10.1007/978-3-031-37963-5_45
Download citation
DOI: https://doi.org/10.1007/978-3-031-37963-5_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-37962-8
Online ISBN: 978-3-031-37963-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)