Abstract
In dense traffic unmanned aerial vehicle (UAV) ad-hoc networks, traffic congestion can cause increased delay and packet loss, which limit the performance of the networks; therefore, a traffic balancing strategy is required to control the traffic. In this study, we propose TQNGPSR, a traffic-aware Q-network enhanced geographic routing protocol based on greedy perimeter stateless routing (GPSR), for UAV ad-hoc networks. The protocol enforces a traffic balancing strategy using the congestion information of neighbors, and evaluates the quality of a wireless link by the Q-network algorithm, which is a reinforcement learning algorithm. Based on the evaluation of each wireless link, the protocol makes routing decisions in multiple available choices to reduce delay and decrease packet loss. We simulate the performance of TQNGPSR and compare it with AODV, OLSR, GPSR, and QNGPSR. Simulation results show that TQNGPSR obtains higher packet delivery ratios and lower end-to-end delays than GPSR and QNGPSR. In high node density scenarios, it also outperforms AODV and OLSR in terms of the packet delivery ratio, end-to-end delay, and throughput.
Similar content being viewed by others
References
Abadi M, Barham P, Chen JM, et al., 2016. TensorFlow: a system for large-scale machine learning. Proc 12th USENIX Conf on Operating Systems Design and Implementation, p.265–283.
Basagni S, Chlamtac I, Syrotiuk VR, et al., 1998. A distance routing effect algorithm for mobility (DREAM). Proc 4th Annual ACM/IEEE Int Conf on Mobile Computing and Networking, p.76–84. https://doi.org/10.1145/288235.288254
Bekmezci I, Sahingoz OK, Temel Ş, 2013. Flying ad-hoc networks (FANETs): a survey. Ad Hoc Netw, 11(3):1254–1270. https://doi.org/10.1016/j.adhoc.2012.12.004
Bolch G, Greiner S, de Meer H, et al., 2006. Queueing Networks and Markov Chains: Modeling and Performance Evaluation with Computer Science Applications (2nd Ed.). John Wiley & Sons, New York, USA.
Boyan JA, Littman ML, 1994. Packet routing in dynamically changing networks: a reinforcement learning approach. Proc 7th Int Conf on Neural Information Processing Systems, p.671–678.
Coutinho N, Matos R, Marques C, et al., 2015. Dynamic dual-reinforcement-learning routing strategies for quality of experience-aware wireless mesh networking. Comput Netw, 88:269–285. https://doi.org/10.1016/j.comnet.2015.06.016
Farahnakian F, Ebrahimi M, Daneshtalab M, et al., 2011. Q-learning based congestion-aware routing algorithm for on-chip network. Proc 2nd Int Conf on Networked Embedded Systems for Enterprise Applications, p.1–7. https://doi.org/10.1109/NESEA.2011.6144949
Jung WS, Yim J, Ko YB, 2017. QGeo: Q-learning-based geographic ad hoc routing protocol for unmanned robotic networks. IEEE Commun Lett, 21(10):2258–2261. https://doi.org/10.1109/LCOMM.2017.2656879
Karp B, Kung HT, 2000. GPSR: greedy perimeter stateless routing for wireless networks. Proc 6th Annual Int Conf on Mobile Computing and Networking, p.243–254. https://doi.org/10.1145/345910.345953
Kenta T, Takeshi M, Shinya K, et al., 2006. Experimental evaluation of an on-demand multipath routing protocol for video transmission in mobile ad hoc networks. J Zhejiang Univ-Sci A, 7(S1):145–150. https://doi.org/10.1631/jzus.2006.AS0145
Ko YB, Vaidya NH, 2000. Location-aided routing (LAR) in mobile ad hoc networks. Wirel Netw, 6(4):307–321. https://doi.org/10.1023/A:1019106118419
Li RL, Li F, Li X, et al., 2014. QGrid: Q-learning based routing protocol for vehicular ad hoc networks. Proc 33rd Int Performance Computing and Communications Conf, p.1–8. https://doi.org/10.1109/PCCC.2014.7017079
Lin SC, Wang P, Luo M, 2016. Control traffic balancing in software defined networks. Comput Netw, 106:260–271. https://doi.org/10.1016/j.comnet.2015.08.004
Lyu N, Song GH, Yang BW, et al., 2019. QNGPSR: a Q-network enhanced geographic ad-hoc routing protocol based on GPSR. Proc 88th Vehicular Technology Conf, p.1–6. https://doi.org/10.1109/VTCFall.2018.8690651
Ma X, Xu Y, Sun GQ, et al., 2013. State-chain sequential feedback reinforcement learning for path planning of autonomous mobile robots. J Zhejiang Univ-Sci C (Comput & Electron), 14(3):167–178. https://doi.org/10.1631/jzus.C1200226
Mnih V, Kavukcuoglu K, Silver D, et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. https://doi.org/10.1038/nature14236
Peng J, Williams RJ, 1996. Incremental multi-step Q-learning. Mach Learn, 22(1-3):283–290. https://doi.org/10.1007/BF00114731
Shi RH, Deng YY, 2008. An improved scheme for reducing the latency of AODV in ad hoc networks. Proc 9th Int Conf for Young Computer Scientists, p.594–598. https://doi.org/10.1109/ICYCS.2008.325
Sutton RS, Barto AG, 1998. Reinforcement Learning: an Introduction. MIT Press, Cambridge, p.1.
Watkins CJCH, Dayan P, 1992. Q-learning. Mach Learn, 8(3-4):279–292. https://doi.org/10.1007/BF00992698
Wu C, Ohzahata S, Kato T, 2012. Routing in VANETs: a fuzzy constraint Q-learning approach. Proc Global Communications Conf, p.195–200. https://doi.org/10.1109/GLOCOM.2012.6503112
Wu C, Ohzahata S, Kato T, 2013. Flexible, portable, and practicable solution for routing in VANETs: a fuzzy constraint Q-learning approach. IEEE Trans Veh Technol, 62(9):4251–4263. https://doi.org/10.1109/TVT.2013.2273945
Xu DH, Chiang M, Rexford J, 2011. Link-state routing with hop-by-hop forwarding can achieve optimal traffic engineering. IEEE/ACM Trans Netw, 19(6):1717–1730. https://doi.org/10.1109/TNET.2011.2134866
Zhan HW, Zhou Y, 2008. Comparison and analysis AODV and OLSR routing protocols in ad hoc network. Proc 4th Int Conf on Wireless Communications, Networking and Mobile Computing, p.1–4. https://doi.org/10.1109/WiCom.2008.578
Zhang JJ, Xi K, Luo M, et al., 2014. Load balancing for multiple traffic matrices using SDN hybrid routing. Proc 15th Int Conf on High Performance Switching and Routing, p.44–49. https://doi.org/10.1109/HPSR.2014.6900880
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the National Natural Science Foundation of China (No. 61501399) and the National Key R&D Program of China (No. 2018AAA0102302)
Contributors
Yi-ning CHEN and Guang-hua SONG designed the research. Yi-ning CHEN, Ni-qi LYU, and Xiao-hong JIANG processed the data. Yi-ning CHEN drafted the manuscript. Guang-hua SONG and Bo-wei YANG helped organize the manuscript. Yi-ning CHEN and Guang-hua SONG revised and finalized the paper.
Compliance with ethics guidelines
Yi-ning CHEN, Ni-qi LYU, Guang-hua SONG, Bo-wei YANG, and Xiao-hong JIANG declare that they have no conflict of interest.
Rights and permissions
About this article
Cite this article
Chen, Yn., Lyu, Nq., Song, Gh. et al. A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks. Front Inform Technol Electron Eng 21, 1308–1320 (2020). https://doi.org/10.1631/FITEE.1900401
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.1900401