Skip to main content
SpringerLink
Log in

Multi-agent reinforcement learning based on graph convolutional network for flexible job shop scheduling

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

With the development of Internet of manufacturing things, decentralized scheduling in flexible job shop is arousing great attention. To deal with the challenges confronted by personalized manufacturing, such as high level of flexibility, agility and robustness for dynamic response, we design a centralized-learning decentralized-execution (CLDE) multi-agent reinforcement learning scheduling structure based on Graph Convolutional Network (GCN), namely graph-based multi-agent system (GMAS), to solve the flexible job shop scheduling problem (FJSP). Firstly, according to the product processing network and job shop environment, the probabilistic model of directed acyclic graph for FJSP is constructed. It models the FJSP as the process of topology graph structure predicting, and the scheduling strategy is adjusted by predicting the connection probability among edges. Then, the multi-agent reinforcement learning system consisting of environment module, job agent module, and machine agent module is constructed. The job agents execute scheduling actions by interacting with environment and machine agents in a decentralized way. Meanwhile, the interaction between job agents is extracted as an abstract global action based on GCN. The experimental results demonstrate that GMAS outperforms its rivals on FJSP, especially in complicated situations. Our results thus shed light on a novel direction for FJSP in dynamic and complex scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data that support the findings of this study are openly available in (Flexible Job Shop Problem) at (https://people.idsia.ch//~monaldo/fjsp.html), reference number (Brandimarte, 1993; Hurink et al., 1994).

References

  • Borenstein, D. (2000). A directed acyclic graph representation of routing manufacturing flexibility. European Journal of Operational Research, 127(1), 78–93.

    Article  Google Scholar 

  • Br, S., Turner, D., Mohanty, K., Samsonov, V., & Meisen, T. (2020). Multi agent deep Q-network approach for online job shop scheduling in flexible manufacturing. In ICMSMM 2020: International conference on manufacturing system and multiple machines.

  • Brandimarte, P. (1993). Routing and scheduling in a flexible job shop by tabu search. Annals of Operations Research, 41(3), 157–183.

    Article  Google Scholar 

  • Brucker, P., & Schlie, R. (1990). Job-shop scheduling with multi-purpose machines. Computing (Vienna/New York), 45(4), 369–375.

    Google Scholar 

  • Claus, C. (1998). The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the AAAI-98.

  • Ennigrou, M., & Ghédira, K. (2008). New local diversification techniques for flexible job shop scheduling problem with a multi-agent approach. Autonomous Agents and Multi-Agent Systems, 17(2), 270–287.

    Article  Google Scholar 

  • Hu, J., & Wellman, M. P. (2003). Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research, 4(4), 1039–1069.

    Google Scholar 

  • Hu, L., Liu, Z., Hu, W., Wang, Y., & Wu, F. (2020). Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network. Journal of Manufacturing Systems, 55, 1–14.

    Article  Google Scholar 

  • Hurink, J., Jurisch, B., & Thole, M. (1994). Tabu search for the job-shop scheduling problem with multi-purpose machines. OR Spektrum, 15(4), 205–215.

    Article  Google Scholar 

  • Jiang, J., Dun, C., Huang, T., & Lu, Z. (2018). Graph convolutional reinforcement learning. arXiv. https://arxiv.org/abs/1810.09202v5

  • Jiménez, Y., Palacio, J. C., & Nowé, A. (2020). Multi-agent reinforcement learning tool for job shop scheduling problems. In Optimization and learning, third international conference, OLA 2020. Cádiz, Spain.

  • Li, X., & Gao, L. (2016). An effective hybrid genetic algorithm and tabu search for flexible job shop scheduling problem. International Journal of Production Economics, 174, 93–110.

    Article  Google Scholar 

  • Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the eleventh international conference on machine learning, Morgan Kaufmann (pp. 157–163).

  • Liu, C. L., Chang, C. C., & Tseng, C. J. (2020a). Actor-critic deep reinforcement learning for solving job shop scheduling problems. IEEE Access, 8, 71752–62.

    Article  Google Scholar 

  • Liu, Y., Wang, W., Hu, Y., Hao, J., Chen, X., Gao, Y., & Association for the Advancement of Artificial Intelligence. (2020b). Multi-agent game abstraction via graph attention neural network. In Thirty-fourth AAAI conference on artificial intelligence, the thirty-second innovative applications of artificial intelligence conference and the tenth AAAI symposium on educational advances in artificial intelligence (Vol. 34, pp. 7211–7218).

  • Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in neural information processing system 30 (NIPS 2017) (p. 30).

  • Malysheva, A., Kudenko, D., Shpilman, A., & IEEE. (2019). MAGNet: Multi-agent graph network for deep multi-agent reinforcement learning. In 2019 XVI international symposium problems of redundancy in information and control systems (redundancy) (pp. 171–176).

  • Maoudj, A., Bouzouia, B., Hentout, A., Kouider, A., & Toumi, R. (2019). Distributed multi-agent scheduling and control system for robotic flexible assembly cells. Journal of Intelligent Manufacturing, 30(4), 1629–1644.

    Article  Google Scholar 

  • Meng, F., Yuan, L., & Cohn, T. (2017). Learning how to active learn: A deep reinforcement learning approach. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 595–605). Association for Computational Linguistics.

  • Naderializadeh, N., Hung, F. H., Soleyman, S., & Khosla, D. (2020). Graph convolutional value decomposition in multi-agent reinforcement learning [arXiv]. arXiv. https://arxiv.org/abs/2010.04740

  • Nouiri, M., Bekrar, A., Jemai, A., Niar, S., & Ammari, A. C. (2015). An effective and distributed particle swarm optimization algorithm for flexible job-shop scheduling problem. Journal of Intelligent Manufacturing, 29(3), 603–15.

    Article  Google Scholar 

  • Nouri, H. E., Driss, O. B., & Ghédira, K. (2015). A holonic multiagent model based on a combined genetic algorithm─tabu search for the flexible job shop scheduling problem. In Highlights of practical applications of agents, multi-agent systems, and sustainability—The PAAMS collection. Springer International Publishing.

  • Nouri, H. E., Driss, O. B., & Ghedira, K. (2016). Simultaneous scheduling of machines and transport robots in flexible job shop environment using hybrid metaheuristics based on clustered holonic multiagent model. Computers & Industrial Engineering, 102, 488–501.

    Article  Google Scholar 

  • Park, J., Chun, J., Kim, S. H., Kim, Y., & Park, J. (2021). Learning to schedule job-shop problems: Representation and policy learning using graph neural network and reinforcement learning. International Journal of Production Research, 4, 1–18.

    Google Scholar 

  • Park, I. B., Huh, J., Kim, J., & Park, J. (2019). A reinforcement learning approach to robust scheduling of semiconductor manufacturing facilities. IEEE Transactions on Automation Science and Engineering, 17(3), 1420–31.

    Google Scholar 

  • Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., & Whiteson, S. (2020). Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21, 7234–7284.

    Google Scholar 

  • Silver, D., Lever, G., Heess, N., Degris, T., & Riedmiller, M. (2014). Deterministic policy gradient algorithms. In Proceedings of the 31st international conference on machine learning (p. 32). Beijing, China, JMLR.org.

  • Son, K., Kim, D., Kang, W. J., Hostallero, D., & Yi, Y. (2019). QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning (p. 97).

  • Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., & T. Graepel and Acm. (2018). Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th international conference on autonomous agents ans multiagent systems (AAMAS' 18) (pp. 2085–2087).

  • Sutton, R. S., Mcallester, D., Singh, S., & Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, p. 12.

  • Wang, L., Hu, X., Wang, Y., Xu, S., Ma, S., Yang, K., Liu, Z., & Wang, W. (2021). Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning. Computer Networks, 190, 107969.

    Article  Google Scholar 

  • Wang, Y. F. (2018). Adaptive job shop scheduling strategy based on weighted Q-learning algorithm. Journal of Intelligent Manufacturing, 31(2), 417–432.

    Article  Google Scholar 

  • Xiong, W., & Fu, D. (2018). A new immune multi-agent system for the flexible job shop scheduling problem. Journal of Intelligent Manufacturing, 29, 857–873.

    Article  Google Scholar 

  • Yang, Y., Hao, J., Liao, B., Shao, K., Chen, G., Liu, W., & Tang, H. (2020). Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv. https://arxiv.org/abs/2002.03939

  • Yun, G. K., Lee, S., Son, J., Bae, H., & Chung, B. D. (2020). Multi-agent system and reinforcement learning approach for distributed intelligence in a flexible smart manufacturing system. Journal of Manufacturing Systems, 57, 440–450.

    Article  Google Scholar 

  • Zhang, S., & Wong, T. N. (2017). Flexible job-shop scheduling/rescheduling in dynamic environment: A hybrid MAS/ACO approach. International Journal of Production Research, 55(11), 3173–3196.

    Article  Google Scholar 

  • Zheng, P., Xia, L., Li, C., Li, X., & Liu, B. (2021). Towards self-X cognitive manufacturing network: An industrial knowledge graph-based multi-agent reinforcement learning approach. Journal of Manufacturing Systems, 61, 16–26.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Guangdong Basic and Applied Basic Research Foundation (Grant Nos. 2022A1515010095, 2021A1515010506), the National Natural Science Foundation of China and the Royal Society of Edinburgh (Grant No. 51911530245), and the National Natural Science Foundation of China (Grant Nos. 51905198, 51675186)

Author information

Authors and Affiliations

  1. South China University of Technology, Guangzhou, 510640, Guangdong, China

    Xuan Jing & Xifan Yao

  2. Guangxi University of Science and Technology, Liuzhou, 545006, Guangxi, China

    Min Liu

  3. China University of Geosciences, Wuhan, 430074, China

    Jiajun Zhou

Authors
  1. Xuan Jing
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xifan Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Min Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiajun Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xifan Yao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jing, X., Yao, X., Liu, M. et al. Multi-agent reinforcement learning based on graph convolutional network for flexible job shop scheduling. J Intell Manuf 35, 75–93 (2024). https://doi.org/10.1007/s10845-022-02037-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-022-02037-5

Keywords

Search

Navigation