Skip to main content
SpringerLink
Log in

Hybrid Prediction-Based Approach for Traffic Signal Control Problem

  • Published:
Optical Memory and Neural Networks Aims and scope Submit manuscript

Abstract

Intelligent transportation systems aim at a more efficient solution to the problems of traffic management, traffic analysis and prediction, route guidance, and provide more coordinated and “smarter” use of transport infrastructure in general. In this study, we consider a traffic signal control problem, which is a challenging problem in the transportation area. Traffic signal control aims to optimize traffic flows in road networks, decrease travel and waiting times, and increase the effectiveness of transport infrastructure usage. In this paper, we developed a novel hybrid reinforcement learning-based approach to solve the traffic signal control problem. In the first step of the proposed approach, we predict the number of vehicles that will cross an intersection during a specified time interval using a deep learning approach. This factor is used as one of a component of the system state space. In the second step, we proposed to use a double Q-learning approach to solve the traffic signal control problem using the observed system state space. An experimental study of the proposed algorithm was conducted using an open-source microscopic traffic simulation package SUMO. The effectiveness of the proposed approach is evaluated using both synthetic and real-world traffic scenarios. Experiments demonstrated that the proposed algorithm outperforms other baseline classical and state-of-the-art reinforcement learning-based algorithms in terms of the average waiting time and average travel time criteria.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Similar content being viewed by others

REFERENCES

  1. Silva, B.N., Khan, M., and Han, K., Towards sustainable smart cities: A review of trends, architectures, components, and open challenges in smart cities, Sustainable Cities Soc., 2018, vol. 38, pp. 697–713. https://doi.org/10.1016/j.scs.2018.01.053

    Article  Google Scholar 

  2. Agafonov, A.A. and Yumaganov, A.S., Bus arrival time prediction using recurrent neural network with LSTM architecture, Opt. Mem. Neural Networks, 2019, vol. 28, pp. 222–230. https://doi.org/10.3103/S1060992X19030081

    Article  Google Scholar 

  3. Lim, C., Kim, K.J., and Maglio, P.P., Smart cities with big data: Reference models, challenges, and considerations, Cities, 2018, vol. 82, pp. 86–99. https://doi.org/10.1016/j.cities.2018.04.011

    Article  Google Scholar 

  4. Kandt, J. and Batty, M., Smart cities, big data and urban policy: Towards urban analytics for the long run, Cities, 2021, vol. 109, p. 102992. https://doi.org/10.1016/j.cities.2020.102992

    Article  Google Scholar 

  5. Ismagilova, E., Hughes, L., Dwivedi, Y.K., and Raman, K.R.: Smart cities: Advances in research – An information systems perspective, Int. J. Inf. Manage., 2019, vol. 47, pp. 88–100. https://doi.org/10.1016/j.ijinfomgt.2019.01.004

    Article  Google Scholar 

  6. Schrank, D., Albert, L., Eisele, B., and Lomax, T., Urban Mobility Report, 2021.

  7. Agafonov, A. and Yumaganov, A., Short-term traffic flow forecasting using a distributed spatial-temporal k nearest neighbors model, in Proceedings – 21st IEEE International Conference on Computational Science and Engineering (CSE 2018), 2018, pp. 91–98. https://doi.org/10.1109/CSE.2018.00019

  8. Agafonov, A.A., Short-term traffic data forecasting: A deep learning approach, Opt. Mem. Neural Networks, 2021, vol. 30, no. 1, pp. 1–10. https://doi.org/10.3103/S1060992X21010021

    Article  Google Scholar 

  9. Adart, A., Mouncif, H., and Na¨ımi, M., Vehicular ad-hoc network application for urban traffic management based on markov chains, Int. Arabic. J. Inf. Technol., 2017, vol. 14 (4A Spec. Issue), pp. 624–631.

  10. Connected and Automated Vehicles: market forecast 2020. https://www.gov.uk/government/publications/connected-and-automated-vehicles-market-forecast-2020.

  11. Allsop, R., Estimating the traffic capacity of a signalized road junction, Transp. Res., 1972, vol. 6, no. 3, pp. 245–255. https://doi.org/10.1016/0041-1647(72)90017-2

    Article  Google Scholar 

  12. Webster, F.V., Traffic Signal Settings, H.M. Stationery Office, 195.)

  13. Papageorgiou, M., Kiakaki, C., Dinopoulou, V., and Kotsialos, A., Yibing Wang: Review of road traffic control strategies, Proc. IEEE, 2003, vol. 91, no. 12, pp. 2043–2067. https://doi.org/10.1109/JPROC.2003.819610

    Article  Google Scholar 

  14. Wei, H., Zheng, G., Gayah, V., and Li, Z., A Survey on Traffic Signal Control Methods. arXiv:1904.08117 [cs, stat] (2020). http://arxiv.org/abs/1904.08117, arXiv: 1904.08117.

  15. Qadri, S.S.S.M., G¨ok¸ce, M.A., and Oner, E., State-of-art review of traffic signal control methods: challenges and opportunities, Eur. Transp. Res. Rev., 2020, vol. 12, no. 1, p. 55. https://doi.org/10.1186/s12544-020-00439-1

    Article  Google Scholar 

  16. Guo, Q., Li, L., and (Jeff) Ban, X., Urban traffic signal control with connected and automated vehicles: A survey, Transp. Res., Part C: Emerging Technol., 2019, vol. 101, pp. 313–334. https://doi.org/10.1016/j.trc.2019.01.026

    Article  Google Scholar 

  17. Little, J., Kelson, M., and Gartner, N., MAXBAND: A Program for Setting Signals on Arteries and Triangular Networks, Transp. Res. Rec. J. Transp. Res. Board, 1981, vol. 795, pp. 40–46.

  18. Li, M.T. and Gan, A., Signal timing optimization for oversaturated networks using TRANSYT-7F, Transp. Res. Rec., 1999, vol. 1683, pp. 118–126. https://doi.org/10.3141/1683-15

  19. Varaiya, P., The max-pressure controller for arbitrary networks of signalized intersections, in Advances in Dynamic Network Modeling in Complex Transportation Systems, Ukkusuri, S.V. and Ozbay, K., Eds., Complex Networks and Dynamic Systems, New York: Springer, 2013, pp. 27–66. https://doi.org/10.1007/978-1-4614-6243-92.

  20. Yau, K.L., Qadir, J., Khoo, H., Ling, M., and Komisarczuk, P., A survey on Reinforcement learning models and algorithms for traffic signal control, ACM Comput. Surv., 2017, vol. 50, no. 3. https://doi.org/10.1145/3068287

  21. Greguri’c, M., Vuji’c, M., Alexopoulos, C., and Mileti’c, M., Application of deep reinforcement learning in traffic signal control: An overview and impact of open traffic data, Appl. Sci., 2020, vol. 10, no. 11, p. 4011. https://doi.org/10.3390/app10114011

    Article  Google Scholar 

  22. Palos, P. and Huszak, A., Comparison of Q-learning based traffic light control methods and objective functions, in 2020 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), IEEE, Split, Hvar, Croatia, 2020, pp. 1–6. https://doi.org/10.23919/SoftCOM50211.2020.9238290.

  23. Wei, H., Zheng, G., Yao, H., and Li, Z., IntelliLight: A reinforcement learning approach for intelligent traffic light control, in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, London United Kingdom: ACM, 2018, pp. 2496–2505. https://doi.org/10.1145/3219819.3220096.

  24. Wei, H., Xu, N., Zhang, H., Zheng, G., Zang, X., Chen, C., Zhang, W., Zhu, Y., Xu, K., and Li, Z., CoLight: Learning network-level cooperation for traffic signal control, Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 1913–1922. https://doi.org/10.1145/3357384.3357902, http://arxiv.org/abs/1905.05717. arXiv: 1905.05717.

  25. Chen, C., Wei, H., Xu, N., Zheng, G., Yang, M., Xiong, Y., Xu, K., and Li, Z., Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control, Proceedings of the AAAI Conference on Artificial Intelligence 34(04), 2020, pp. 3414–3421. https://doi.org/10.1609/aaai.v34i04.5744

    Article  Google Scholar 

  26. Liu, Y., Liu, L., and Chen, W.P., Intelligent Traffic Light Control Using Distributed Multi-agent Q Learning. arXiv:1711.10941[cs], 2017. http://arxiv.org/abs/1711.10941. arXiv: 1711.10941.

  27. Li, Z., Yu, H., Zhang, G., Dong, S., and Xu, C.Z., Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning, Transp. Res., Part C: Emerging Technol., 2021, vol. 125, p. 103059. https://doi.org/10.1016/j.trc.2021.103059

    Article  Google Scholar 

  28. Gu, J., Fang, Y., Sheng, Z., and Wen, P., Double deep Q-network with a DualAgent for traffic signal control, Appl. Sci., 2020, vol. 10, no. 5, p. 1622. https://doi.org/10.3390/app10051622

    Article  Google Scholar 

  29. Agafonov, A. and Myasnikov, V., Traffic Signal Control: A Double Q-learning Approach, in Proceedings of the 16th Conference on Computer Science and Intelligence Systems, FedCSIS 2021, 2021, pp. 365–369. https://doi.org/10.15439/2021F109.

  30. Zeng, J., Hu, J., and Zhang, Y., Adaptive traffic signal control with deep recurrent Q-learning, in 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu: IEEE, 2018, pp. 1215–1220. https://doi.org/10.1109/IVS.2018.8500414.

  31. Lopez, P.A., Wiessner, E., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flotterod, Y.P., Hilbrich, R., Lucken, L., Rummel, J., and Wagner, P., Microscopic traffic simulation using SUMO, in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI: IEEE, 2018, pp. 2575–2582. https://doi.org/10.1109/ITSC.2018.8569938.

  32. Hasselt, H., Double Q-learning, Adv. Neural Inform. Process. Syst., 2010, vol. 23.

  33. RESCO, 2021. https://github.com/Pi-Star-Lab/RESCO, original-date: 2021-06-07T17:31:48Z.

  34. TAPASCologne-SUMO Documentation, https://sumo.dlr.de/docs/Data/Scenarios/TAPASCologne.html.

  35. Ault, J., Hanna, J.P., and Sharon, G., Learning an Interpretable Traffic Signal Control Policy. arXiv:1912.11023 [cs, stat], 2020. http://arxiv.org/abs/1912.11023. arXiv: 1912.11023.

Download references

Funding

The work was supported by the Russian Science Foundation, grant no. 21-11-00321, https://rscf.ru/en/project/21-11-00321/.

Author information

Authors and Affiliations

  1. Samara National Research University, 443086, Samara, Russia

    A. A. Agafonov & V. V. Myasnikov

Authors
  1. A. A. Agafonov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. V. Myasnikov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to A. A. Agafonov or V. V. Myasnikov.

Ethics declarations

The authors declare that they have no conflicts of interest.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Agafonov, A.A., Myasnikov, V.V. Hybrid Prediction-Based Approach for Traffic Signal Control Problem. Opt. Mem. Neural Networks 31, 277–287 (2022). https://doi.org/10.3103/S1060992X2203002X

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3103/S1060992X2203002X

Keywords:

Search

Navigation