Skip to main content
SpringerLink
Log in

Double deep Q-network-based self-adaptive scheduling approach for smart shop floor

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In the field of smart manufacturing, the data-driven scheduling approach has become an effective way to solve the smart shop floor scheduling problem with high complexity and dynamics. However, most existing approaches rely too heavily on manual supervision in implementation, resulting in poor adaptability and effectiveness in dynamic production environments. Therefore, this paper proposes a self-adaptive scheduling approach based on double deep Q-network (DDQN), which can reduce manual supervision and realize the autonomy of the whole scheduling process. In the presented approach, first, a self-adaptive scheduling framework, which forms a closed-loop optimization structure for scheduling model evaluation, generation/updating, and application, is designed. Second, the interactive learning mechanism of reinforcement learning is introduced, and the scheduling model is generated through the DDQN algorithm without manual supervision. In addition, dynamic reward function based on simulation is proposed to promote the rationality and accuracy of the reward in reinforcement learning. The effectiveness of the proposed approach is validated on a semiconductor production shop floor, and the experimental results illustrate that the proposed approach can improve the effectiveness of self-adaptive scheduling and significantly reduce the time and labour costs in the dynamic production environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Qu YJ, Ming XG, Liu ZW, Zhang XY, Hou ZT (2019) Smart manufacturing systems: state of the art and future trends. Int J Adv Manuf Technol 103:3751–3768. https://doi.org/10.1007/s00170-019-03754-7

    Article  Google Scholar 

  2. Tao F, Qing QL, Liu A, Kusiak A (2018) Data-driven smart manufacturing. J Manuf Syst 48:157–169. https://doi.org/10.1016/j.jmsy.2018.01.006

    Article  Google Scholar 

  3. Zhang YF, Wang J, Liu SC, Qian C (2017) Game theory based real-time shop floor scheduling strategy and method for cloud manufacturing. Int J Intell Syst 32(4):437–463. https://doi.org/10.1002/int.21868

    Article  Google Scholar 

  4. Zhang GH, Lu XX, Liu X, Zhang LT, Wei SW, Wang WQ (2022) An effective two-stage algorithm based on convolutional neural network for the bi-objective flexible job shop scheduling problem with machine breakdown. Expert Syst Appl 203:117460. https://doi.org/10.1016/j.eswa.2022.117460

    Article  Google Scholar 

  5. Wang JL, Gao PJ, Zhang P, Zhang J, Ip WH (2021) A fuzzy hierarchical reinforcement learning based scheduling method for semiconductor wafer manufacturing systems. J Manuf Syst 61:239–248. https://doi.org/10.1016/j.jmsy.2021.08.008

    Article  Google Scholar 

  6. Fu F, Zhou H (2021) A combined multi-agent system for distributed multi-project scheduling problems. Appl Soft Comput 107:107402. https://doi.org/10.1016/j.asoc.2021.107402

    Article  Google Scholar 

  7. Salido MA, Escamilla J, Giret A, Barber F (2016) A genetic algorithm for energy-efficiency in job-shop scheduling. Int J Adv Manuf Technol 85:1303–1314. https://doi.org/10.1007/s00170-015-7987-0

    Article  Google Scholar 

  8. Caricato P, Grieco A, Nucci F (2008) Simulation and mathematical programming for a multi-objective configuration problem in a hybrid flow shop. In: Winter simulation conference, pp. 1820–1828. https://doi.org/10.1109/WSC.2008.4736271

  9. Gupta AK, Sivakumar AI (2002) Simulation based multi-objective schedule optimization in semiconductor manufacturing. Proc Winter Simul Confer 2002(2):1862–1870. https://doi.org/10.1109/WSC.2002.1166480

    Article  Google Scholar 

  10. Singh A, Mehta NK, Jain PK (2007) Multicriteria dynamic scheduling by swapping of dispatching rules. Int J Adv Manuf Technol 34:988–1007. https://doi.org/10.1007/s00170-006-0674-4

    Article  Google Scholar 

  11. Wang WL, Zhang ZJ, Gao N, Zhao YW (2019) Progress of big data analytics methods based on artificial intelligence technology. Comput Integr Manuf Syst 25(03):529–547. https://doi.org/10.13196/j.cims.2019.03.001

    Article  Google Scholar 

  12. Wan LP, Lan XG, Zhang HB, Zheng NN (2019) A review of deep reinforcement learning theory and application. Pattern Recogn Artif Intell 32(001):67–81. https://doi.org/10.16451/j.cnki.issn1003-6059.201901009

    Article  Google Scholar 

  13. Yang Y, Huang M, Wang ZY, Zhu QB (2020) Robust scheduling based on extreme learning machine for bi-objective flexible job-shop problems with machine breakdowns. Expert Syst Appl 158:113545. https://doi.org/10.1016/j.eswa.2020.113545

    Article  Google Scholar 

  14. Koulinas G, Paraschos P, Koulouriotis D (2020) A decision trees-based knowledge mining approach for controlling a complex production system. Procedia Manuf 51:1439–1445. https://doi.org/10.1016/j.promfg.2020.10.200

    Article  Google Scholar 

  15. Zheng P, Xia LQ, Li CX, Li XY, Liu BF (2021) Towards self-X cognitive manufacturing network: an industrial knowledge graph-based multi-agent reinforcement learning approach. J Manuf Syst 61:16–26. https://doi.org/10.1016/j.jmsy.2021.08.002

    Article  Google Scholar 

  16. Jamwal A, Agrawal R, Sharma M (2022) Deep learning for manufacturing sustainability: models, applications in Industry 4.0 and implications. Int J Inf Manag Data Insights 2(2):100107. https://doi.org/10.1016/j.jjimei.2022.100107

    Article  Google Scholar 

  17. Zhang, J., Gao, L., Qin, W., Lyu, Y. L., and Li, X. Y. (2016). Big-data-driven operational analysis and decision-making methodology in intelligent workshop. Comput Integr Manuf Syst 22(05), 1220–1228. https://doi.org/10.13196/j.cims.2016.05.007.

  18. Wang C, Jiang PY (2019) Deep neural networks based order completion time prediction by using real-time job shop RFID data. J Intell Manuf 30(3):1303–1318. https://doi.org/10.1007/s10845-017-1325-3

    Article  Google Scholar 

  19. Steinbacher LM, Ait-Alla A, Rippel D, Düe T, Freitag M (2022) Modelling framework for reinforcement learning based scheduling applications. IFAC-PapersOnLine 55(10):67–72. https://doi.org/10.1016/j.ifacol.2022.09.369

    Article  Google Scholar 

  20. Wang L, Pan ZX, Wang JJ (2021) A review of reinforcement learning based intelligent optimization for manufacturing scheduling. Complex Syst Model Simul 1(4):257–270. https://doi.org/10.23919/CSMS.2021.0027

    Article  Google Scholar 

  21. Shiue YR, Lee KC, Su CT (2018) Real-time scheduling for a smart factory using a reinforcement learning approach. Comput Ind Eng 125:604–614. https://doi.org/10.1016/j.cie.2018.03.039

    Article  Google Scholar 

  22. Chen XL, Hao XC, Lin HW, Murata T (2010) Rule driven multi objective dynamic scheduling by data envelopment analysis and reinforcement learning. In: ICAL 2010: IEEE international conference on automation and logistics, pp 396–401. https://doi.org/10.1109/ICAL.2010.5585316

  23. Wang LB, Hu X, Wang Y, Xu SJ, Ma SJ, Yang KX et al (2021) Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning. Comput Netw 190(2):107969. https://doi.org/10.1016/j.comnet.2021.107969

    Article  Google Scholar 

  24. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  25. Lin CC, Deng DJ, Chih YL, Chiu HT (2019) Smart manufacturing scheduling with edge computing using multi-class deep Q network. IEEE Trans Ind Informat 15(7):4276–4284. https://doi.org/10.1109/TII.2019.2908210

    Article  Google Scholar 

  26. Waschneck B, Reichstaller A, Belzner L, Altenmuller T, Bauernhansl T, Knapp A, et al. (2018) Deep reinforcement learning for semiconductor production scheduling. In: 29th annual SEMI advanced semiconductor manufacturing conference, pp 301–306. https://doi.org/10.1109/ASMC.2018.8373191

  27. Luo S (2020) Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Appl Soft Comput 91(21):106208. https://doi.org/10.1016/j.asoc.2020.106208

    Article  Google Scholar 

  28. Ma YM, Qiao F, Chen X, Tian K, Wu XH (2015) Dynamic scheduling approach based on SVM for semiconductor production line. Comput Integr Manuf Syst 21(3):7. https://doi.org/10.13196/j.cims.2015.03.018

    Article  Google Scholar 

  29. Hong M, Wang L, Wu LB (2019) Re-understanding of distance from the high point of view: connotation, type and representation. J Neijiang Norm Univer 34(10):20–24. https://doi.org/10.13603/j.cnki.51-1621/z.2019.10.004

    Article  Google Scholar 

  30. Coelho DN, Barreto GA (2019) Approximate linear dependence as a design method for kernel prototype-based classifiers. Adv Intell Syst Comput 976:241–250. https://doi.org/10.1007/978-3-030-19642-4_24

    Article  Google Scholar 

  31. Watkins C, J., Dayan, and Peter. (1992) Q-learning. Mach Learn 8(3–4):279–292. https://doi.org/10.1007/BF00992698

    Article  MATH  Google Scholar 

  32. Zhai JW, Liu Q, Zhang ZZ, Zhong S, Zhu HJ, Zhang P et al (2016) Deep Q-learning with prioritized sampling. Neural Inf Process 9947:13–22. https://doi.org/10.1007/978-3-319-46687-3_2

    Article  Google Scholar 

  33. Hasselt HV, Guez A, Silver D (2016) Deep reinforcement learning with double Q-learning. Thirtieth AAAI Confer Artif Intell 30(1):2094–2100. https://doi.org/10.1609/aaai.v30i1.10295

    Article  Google Scholar 

  34. Xia KS, Sacco C, Kirkpatrick M, Saidy C, Nguyen L, Kircaliali A et al (2020) A digital twin to train deep reinforcement learning agent for smart manufacturing plants: environment, interfaces and intelligence. J Manuf Syst 58:210–230. https://doi.org/10.1016/j.jmsy.2020.06.012

    Article  Google Scholar 

  35. Nian R, Liu JF, Huang B (2020) A review on reinforcement learning: introduction and applications in industrial process control. Comput Chem Eng 139:106886. https://doi.org/10.1016/j.compchemeng.2020.106886

    Article  Google Scholar 

  36. Kempf K (1994) Intel five-machine six step mini-fab description. Intel/ASUReport. http://www.eas.asu.edu/aar/research/in-tel/papers/fabspec.Html

  37. Ma YM, Qiao F, Zhao F, Sutherland JW (2017) Dynamic scheduling of a semiconductor production line based on a composite rule set. Appl Sci 7(10):1052. https://doi.org/10.3390/app7101052

    Article  Google Scholar 

  38. Ma YM, Li SY, Qiao F, Lu XY, Liu J (2022) A data-driven scheduling knowledge management method for smart shop floor. Int J Comput Integr Manuf 35(7):780–793. https://doi.org/10.1080/0951192X.2022.2025622

    Article  Google Scholar 

  39. Fang X, Wang HH, Liu GJ, Tian XJ, Ding GF, Zhang HZ (2022) Industry application of digital twin: from concept to implementation. Int J Adv Manuf Technol 121:4289–4312. https://doi.org/10.1007/s00170-022-09632-z

    Article  Google Scholar 

  40. Zhou BH, Li X, Fung RYK (2015) Dynamic scheduling of photolithography process based on Kohonen neural network. J Intell Manuf 26(1):73–85. https://doi.org/10.1007/s10845-013-0763-9

    Article  Google Scholar 

  41. Serrano-Ruiz JC, Mula J, Poler R (2021) Smart manufacturing scheduling: a literature review. J Manuf Syst 61:265–287. https://doi.org/10.1016/j.jmsy.2021.09.011

    Article  Google Scholar 

Download references

Funding

This research was supported by the National Natural Science Foundation of China (Grant Numbers 62133011, 62273260, 61873191, 61973237).

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Tongji University, Shanghai, 201804, China

    Yumin Ma, Jingwen Cai, Shengyi Li, Juan Liu, Jianmin Xing & Fei Qiao

Authors
  1. Yumin Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingwen Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengyi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianmin Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fei Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Yumin Ma, Shengyi Li, Juan Liu, Jianmin Xing and Fei Qiao. The first draft of the manuscript was written by Jingwen Cai and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Juan Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Ethical approval

This work does not contain any ethical issues or personal information.

Consent to participate

No human or animal was involved in this work; thus, no consent was required.

Consent to publication

All authors have given their permission for publishing this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, Y., Cai, J., Li, S. et al. Double deep Q-network-based self-adaptive scheduling approach for smart shop floor. Neural Comput & Applic 35, 22281–22296 (2023). https://doi.org/10.1007/s00521-023-08877-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08877-3

Keywords

Search

Navigation