Skip to main content
SpringerLink
Log in

Multi-user reinforcement learning based multi-reward for spectrum access in cognitive vehicular networks

  • Published:
Telecommunication Systems Aims and scope Submit manuscript

Abstract

Cognitive Vehicular Networks (CVNs) can improve spectrum utilization by intelligently using idle spectrum, so as to fulfill the needs of communication. The previous researches only considered vehicle-to-vehicle(V2V) links or vehicle-to-infrastructure (V2I) links and ignored the influence of spectrum sensing errors. Therefore, in this paper, V2V links and V2I links are simultaneously discussed in the presence of spectrum sensing errors in the CVNs communication environment that we establish, and a dynamic spectrum access problem aiming at spectrum utilization is framed. In order to solve the above problems, the reinforcement learning method is introduced in this paper. But the impact of two kinds of collisions on the spectrum access rate of cognitive vehicles is neglected in the reinforcement learning method, and the above collisions which exist between cognitive vehicles, between cognitive vehicles and primary vehicles. Hence, different reward functions are designed according to different collision situations, and an improved reinforcement learning method is utilized to improve the success probability of spectrum access. To verify the effectiveness of the improved method, the performance and convergence of the proposed method are significantly better than other methods by comparing with the Myopic method, DQN and traditional DDQN in Python.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Najada, A. L., Mahgoub, H. (2016). I.: Anticipation and alert system of congestion and accidents in vanet using big data analysis for intelligent transportation systems. In: 2016 IEEE symposium series on computational intelligence (SSCI), pp. 1–8. IEEE.

  2. Ullah, A., Yao, X., Shaheen, S., & Ning, H. (2019). Advances in position based routing towards its enabled fog-oriented vanet-a survey. IEEE Transactions on Intelligent Transportation Systems, 21(2), 828–840.

    Article  Google Scholar 

  3. Mutalik, P., Nagaraj, S., Vedavyas, J., Biradar, R. V., Patil, V. G. C. (2016). A comparative study on aodv, dsr and dsdv routing protocols for intelligent transportation system (its) in metro cities for road traffic safety using vanet route traffic analysis (vrta). In: 2016 IEEE international conference on advances in electronics, communication and computer technology (ICAECCT), pp. 383–386. IEEE.

  4. Gao, H., Liu, C., Li, Y., & Yang, X. (2020). V2vr: Reliable hybrid-network-oriented v2v data transmission and routing considering rsus and connectivity probability. IEEE Transactions on Intelligent Transportation Systems, 22(6), 3533–3546.

    Article  Google Scholar 

  5. Villas, L. A., Boukerche, A., Maia, G., Pazzi, R. W., & Loureiro, A. A. (2014). Drive: An efficient and robust data dissemination protocol for highway and urban vehicular ad hoc networks. Computer Networks, 75, 381–394.

    Article  Google Scholar 

  6. Cunha, F., Villas, L., Boukerche, A., Maia, G., Viana, A., Mini, R. A., & Loureiro, A. A. (2016). Data communication in vanets: Protocols, applications and challenges. Ad Hoc Networks, 44, 90–103.

    Article  Google Scholar 

  7. Shaibani, R., Zahary, A. (2018). Survey of context-aware video transmission over vehicular ad-hoc networks (vanets). EAI Endorsed Transactions on Mobile Communications and Applications 4(15).

  8. Wang, R., Xu, Z., Zhao, X., & Hu, J. (2019). V2v-based method for the detection of road traffic congestion. IET Intelligent Transport Systems, 13(5), 880–885.

    Article  Google Scholar 

  9. Priyan, M., & Devi, G. U. (2019). A survey on internet of vehicles: Applications, technologies, challenges and opportunities. International Journal of Advanced Intelligence Paradigms, 12(1–2), 98–119.

    Article  Google Scholar 

  10. Wang, X., Wang, C., Zhang, J., Zhou, M., & Jiang, C. (2016). Improved rule installation for real-time query service in software-defined internet of vehicles. IEEE Transactions on Intelligent Transportation Systems, 18(2), 225–235.

    Article  Google Scholar 

  11. Vasudev, H., Deshpande, V., Das, D., & Das, S. K. (2020). A lightweight mutual authentication protocol for v2v communication in internet of vehicles. IEEE Transactions on Vehicular Technology, 69(6), 6709–6717.

    Article  Google Scholar 

  12. Mitola, J., & Maguire, G. Q. (1999). Cognitive radio: Making software radios more personal. IEEE Personal Communications, 6(4), 13–18.

    Article  Google Scholar 

  13. Mitola, J. (2000). An integrated agent architecture for software defined radio: Dissertation doctor of technology, royal institute of technology, sweden, may 8.

  14. Di Felice, M., Doost-Mohammady, R., Chowdhury, K. R., & Bononi, L. (2012). Smart radios for smart vehicles: Cognitive vehicular networks. IEEE Vehicular Technology Magazine, 7(2), 26–33.

    Article  Google Scholar 

  15. Ahmed, Z., Jamal, H., Khan, S., Mehboob, R., & Ashraf, A. (2009). Cognitive communication device for vehicular networking. IEEE Transactions on Consumer Electronics, 55(2), 371–375.

    Article  Google Scholar 

  16. Di Felice, M., Chowdhury, K.R., Bononi, L. (2011).Cooperative spectrum management in cognitive vehicular ad hoc networks. In: 2011 IEEE vehicular networking conference (VNC), pp. 47–54 IEEE.

  17. Zhang, H., & Guo, C. (2019). Beam alignment-based mmwave spectrum sensing in cognitive vehicular etworks. In: 2019 IEEE global conference on signal and information processing (GlobalSIP), pp. 1–5. IEEE.

  18. Li, M., Zhao, L., & Liang, H. (2017). An smdp-based prioritized channel allocation scheme in cognitive enabled vehicular ad hoc networks. IEEE Transactions on Vehicular Technology, 66(9), 7925–7933.

    Article  Google Scholar 

  19. Cheng, N., Zhang, N., Lu, N., Shen, X., Mark, J. W., & Liu, F. (2013). Opportunistic spectrum access for cr-vanets: A game-theoretic approach. IEEE Transactions on Vehicular Technology, 63(1), 237–251.

    Article  Google Scholar 

  20. Gill, K.S., Heath, K.N., Chuke, S., Haider, A., Gegear, R.J., Ryder, E.F., & Wyglinski, A.M.(2020). Bumblebee-inspired c-v2x dynamic spectrum access testbed using openairinterface. In: 2020 IEEE 91st vehicular technology conference (VTC2020-Spring), pp. 1–5. IEEE.

  21. Yang, C., Fu, Y., Zhang, Y., Xie, S., & Yu, R. (2013). Energy-efficient hybrid spectrum access scheme in cognitive vehicular ad hoc networks. IEEE Communications Letters, 17(2), 329–332.

    Article  Google Scholar 

  22. Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, vol. 30

  23. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (20125) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

  24. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., & Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In: International conference on machine learning, pp. 1928–1937. PMLR.

  25. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., & Ostrovski, G. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533.

    Article  Google Scholar 

  26. Wang, Y., Li, X., Wan, P., & Shao, R. (2021). Intelligent dynamic spectrum access using deep reinforcement learning for vanets. IEEE Sensors Journal, 21(14), 15554–15563.

  27. Choe, C., Ahn, J., Choi, J., Park, D., Kim, M., & Ahn, S. (2020). A robust channel access using cooperative reinforcement learning for congested vehicular networks. IEEE Access, 8, 135540–135557.

    Article  Google Scholar 

  28. Choe, C., Choi, J., Ahn, J., Park, D., & Ahn, S. (2020). Multiple channel access using deep reinforcement learning for congested vehicular networks. In: 2020 IEEE 91st vehicular technology conference (VTC2020-Spring), pp. 1–6. IEEE.

  29. Li, X., Lu, L., Ni, W., Jamalipour, A., Zhang, D., & Du, H. (2022). Federated multi-agent deep reinforcement learning for resource allocation of vehicle-to-vehicle communications. IEEE Transactions on Vehicular Technology, 71(8), 8810–8824.

    Article  Google Scholar 

  30. Sroka, P., & Kliks, A. (2022). Distributed learning for vehicular dynamic spectrum access in autonomous driving. In: 2022 IEEE international conference on pervasive computing and communications workshops and other affiliated events (PerCom Workshops), pp. 605–610. IEEE.

  31. Liu, X., Sun, C., Yau, K. -L. A., & Wu, C. (2022). Joint collaborative big spectrum data sensing and reinforcement learning based dynamic spectrum access for cognitive internet of vehicles. IEEE Transactions on Intelligent Transportation Systems, 1–11.

  32. Meinilä, J., Kyösti, P., Jämsä, T., Hentilä, L. (2009). Winner ii channel models. In: Radio technologies and concepts for IMT-advanced,

  33. Liu, K., Zhao, Q., & Krishnamachari, B. (2010). Dynamic multichannel access with imperfect channel state detection. IEEE Transactions on Signal Processing, 58(5), 2795–2808.

    Article  Google Scholar 

  34. Wang, S., Liu, H., Gomes, P. H., & Krishnamachari, B. (2018). Deep reinforcement learning for dynamic multichannel access in wireless networks. IEEE Transactions on Cognitive Communications and Networking, 4(2), 257–265.

    Article  Google Scholar 

  35. Liang, L., Ye, H., & Li, G. Y. (2019). Spectrum sharing in vehicular networks based on multi-agent reinforcement learning. IEEE Journal on Selected Areas in Communications, 37(10), 2282–2292.

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China under grant no. 61571209 and 61501059, Science and Technology Department of Jilin Provincial, China (Grant No. YDZJ202201ZYTS653),Science and Technology Department of Jilin Provincial, China (Grant No. 20180101336JC).

Author information

Author notes

  1. Quanjun Zhao, Ke Fu, Xiaohui Zhao and Hongliang Sun have contributed equally to this work.

Authors and Affiliations

  1. College of Information and Control Engineering, Jilin Institute of Chemical Technology, Jilin, 132000, China

    Lingling Chen, Quanjun Zhao, Ke Fu & Hongliang Sun

  2. College of Communication Engineering, Jilin University, Changchun, 130012, China

    Lingling Chen

  3. Key Laboratory of Information Science, College of Communication Engineering, Jilin University, Changchun, 130012, China

    Xiaohui Zhao

  4. Department of Communications Engineering, Jilin University, Changchun, 130012, China

    Hongliang Sun

Authors
  1. Lingling Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quanjun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ke Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaohui Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongliang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lingling Chen.

Ethics declarations

Conflict of interest

The authors have not disclosed any competing interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Zhao, Q., Fu, K. et al. Multi-user reinforcement learning based multi-reward for spectrum access in cognitive vehicular networks. Telecommun Syst 83, 51–65 (2023). https://doi.org/10.1007/s11235-023-01004-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11235-023-01004-6

Keywords

Search

Navigation