{"type":"ScholarlyArticle","title":"On the Study of Curriculum Learning for Inferring Dispatching Policies on the Job Shop Scheduling","authors":[{"name":"Zangir Iklassov","slug":"zangir-iklassov","given":"Zangir","family":"Iklassov","affiliation":"University of Artificial Intelligence"},{"name":"Dmitrii Medvedev","slug":"dmitrii-medvedev","given":"Dmitrii","family":"Medvedev","affiliation":"University of Artificial Intelligence"},{"name":"Ruben Solozabal Ochoa de Retana","slug":"ruben-solozabal-ochoa-de-retana","given":"Ruben","family":"Solozabal Ochoa de Retana","affiliation":"University of Artificial Intelligence"},{"name":"Martin Takac","slug":"martin-takac","given":"Martin","family":"Takac","affiliation":"University of Artificial Intelligence"}],"abstract":"This paper studies the use of Curriculum Learning on Reinforcement Learning (RL) to improve the performance of the dispatching policies learned on the Job-shop Scheduling Problem (JSP). Current works in the literature present a large optimality gap when learning end-to-end solutions on this problem. In this regard, we identify the difficulty for RL to learn directly on large instances as part of the issue and use Curriculum Learning (CL) to mitigate this effect. Particularly, CL sequences the learning process in a curriculum of increasing complexity tasks, which allows learning on large instances that otherwise would be impossible to learn from scratch. In this paper, we present a size-agnostic model that enables us to demonstrate that current curriculum strategies have a major impact on the quality of the solution inferred. In addition, we introduce a novel Reinforced Adaptive Staircase Curriculum Learning (RASCL) strategy, which adjusts the difficulty level during the learning process by revisiting the worst-performing instances. Conducted experiments on Taillard’s and Demirkol’s datasets show that the presented approach significantly improves the current stateof-the-art models on the JSP. It reduces the average optimality gap from 19.35% to 10.46% on Taillard’s instances and from 38.43% to 18.85% on Demirkol’s instances.","doi":"10.24963/ijcai.2023/594","url":"https://ijcai.org/proceedings/2023/594","pdf":"https://ijcai.org/proceedings/2023/0594.pdf","markdown":"https://ijcai.org/proceedings/2023/594.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/594","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":594,"pages":{"first":5350,"last":5358},"keywords":["Planning and Scheduling: PS: Learning in planning and scheduling","Planning and Scheduling: PS: Scheduling"],"videos":[],"references":[{"position":1,"raw":"[Aydin and Öztemel, 2000] M Emin Aydin and Ercan Öztemel. Dynamic job-shop scheduling using reinforce- ment learning agents. Robotics and Autonomous Systems, 33(2-3):169-178, 2000.","title":"Dynamic job-shop scheduling using reinforcement learning agents","authors":"Emin Aydin, Ercan Öztemel","year":"2000","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bello et al., 2016] Irwan Bello, Hieu Pham, Quoc V Le, Mo- hammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016.","title":"Neural combinatorial optimization with reinforcement learning","authors":"Irwan Bello, Hieu Pham, V Quoc, Mohammad Le, Samy Norouzi, Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Demirkol et al., 1998] Ebru Demirkol, Sanjay Mehta, and Reha Uzsoy. Benchmarks for shop scheduling problems. European Journal of Operational Research, 109(1):137- 141, August 1998.","title":"Benchmarks for shop scheduling problems","authors":"Ebru Demirkol, Sanjay Mehta, Reha Uzsoy","year":"1998","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deudon et al., 2018] Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Louis-Martin Rousseau. Learning heuristics for the tsp by policy gradient. In International conference on the integration of constraint programming, artificial intelligence, and operations research, pages 170-181. Springer, 2018.","title":"Learning heuristics for the tsp by policy gradient","authors":"Pierre Michel Deudon, Alexandre Cournut, Yossiri Lacoste, Louis-Martin Adulyasak, Rousseau","year":"2018","venue":"International conference on the integration of constraint programming, artificial intelligence, and operations research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gabel and Riedmiller, 2008] Thomas Gabel and Martin Riedmiller. Adaptive reactive job-shop scheduling with reinforcement learning agents. International Journal of In- formation Technology and Intelligent Computing, 24(4):14- 18, 2008.","title":"Adaptive reactive job-shop scheduling with reinforcement learning agents","authors":"Riedmiller Gabel, Thomas Gabel, Martin Riedmiller","year":"2008","venue":"International Journal of Information Technology and Intelligent Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ingimundardottir and Runarsson, 2018] Helga Ingimundar- dottir and Thomas Philip Runarsson. Discovering dispatch- ing rules from data using imitation learning: A case study for the job-shop problem. Journal of Scheduling, 21(4):413- 428, 2018.","title":"Helga Ingimundardottir and Thomas Philip Runarsson. Discovering dispatching rules from data using imitation learning: A case study for the job-shop problem","authors":null,"year":"2018","venue":"Journal of Scheduling","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Joshi et al., 2019] Chaitanya K Joshi, Thomas Laurent, and Xavier Bresson. An efficient graph convolutional net- work technique for the travelling salesman problem. arXiv preprint arXiv:1906.01227, 2019.","title":"An efficient graph convolutional network technique for the travelling salesman problem","authors":"K Chaitanya, Thomas Joshi, Xavier Laurent, Bresson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kool et al., 2018] Wouter Kool, Herke Van Hoof, and Max Welling. Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475, 2018.","title":"Attention, learn to solve routing problems!","authors":"Wouter Kool, Herke Van Hoof, Max Welling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kwon et al., 2020] Yeong-Dae Kwon, Jinho Choo, By- oungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances in Neural Information Processing Systems, 33:21188-21198, 2020.","title":"Pomo: Policy optimization with multiple optima for reinforcement learning","authors":"Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, Seungjai Min","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2019] Chun-Cheng Lin, Der-Jiunn Deng, Yen- Ling Chih, and Hsin-Ting Chiu. Smart manufacturing scheduling with edge computing using multiclass deep q network. IEEE Transactions on Industrial Informatics, 15(7):4276-4284, 2019.","title":"Smart manufacturing scheduling with edge computing using multiclass deep q network","authors":"Chun-Cheng Lin, Der-Jiunn Deng, Yen-Ling Chih, Hsin-Ting Chiu","year":"2019","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lisicki et al., 2020] Michal Lisicki, Arash Afkanpour, and Graham W Taylor. Evaluating curriculum learning strate- gies in neural combinatorial optimization. arXiv preprint arXiv:2011.06188, 2020.","title":"Evaluating curriculum learning strategies in neural combinatorial optimization","authors":"Michal Lisicki, Arash Afkanpour, Graham W Taylor","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2020] Chien-Liang Liu, Chuan-Chin Chang, and Chun-Jan Tseng. Actor-critic deep reinforcement learning for solving job shop scheduling problems. Ieee Access, 8:71752-71762, 2020.","title":"Actor-critic deep reinforcement learning for solving job shop scheduling problems","authors":"Chien-Liang Liu, Chuan-Chin Chang, Chun-Jan Tseng","year":"2020","venue":"Ieee Access","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahadevan and Theocharous, 1998] Sridhar Mahadevan and Georgios Theocharous. Optimizing production manufacturing using reinforcement learning. In FLAIRS conference, volume 372, page 377, 1998.","title":"Optimizing production manufacturing using reinforcement learning","authors":"Sridhar Mahadevan, Georgios Theocharous","year":"1998","venue":"FLAIRS conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mahadevan et al., 1997] Sridhar Mahadevan, Nicholas Mar- challeck, Tapas K Das, and Abhijit Gosavi. Self-improving factory simulation using continuous-time average-reward reinforcement learning. In Machine Learning Interantional Workshop, pages 202-210, 1997.","title":"Self-improving factory simulation using continuous-time average-reward reinforcement learning","authors":"Nicholas Sridhar Mahadevan, Tapas K Marchalleck, Abhijit Das, Gosavi","year":"1997","venue":"Machine Learning Interantional Workshop","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mao et al., 2019] Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Moham- mad Alizadeh. Learning scheduling algorithms for data processing clusters. In Proceedings of the ACM special interest group on data communication, pages 270-288. 2019.","title":"Learning scheduling algorithms for data processing clusters","authors":"Mao","year":"2019","venue":"Proceedings of the ACM special interest group on data communication","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nazari et al., 2018] Mohammadreza Nazari, Afshin Oroo- jlooy, Lawrence V Snyder, and Martin Takáč. Reinforce- ment learning for solving the vehicle routing problem. In Conference on Neural Information Processing Systems, NeurIPS 2018, 2018.","title":"Reinforcement learning for solving the vehicle routing problem","authors":"Mohammadreza Nazari, Afshin Oroojlooy, Lawrence V Snyder, Martin Takáč","year":"2018","venue":"Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sels et al., 2012] Veronique Sels, Nele Gheysen, and Mario Vanhoucke. A comparison of priority rules for the job shop scheduling problem under different flow time-and tardiness-related objective functions. International Journal of Production Research, 50(15):4255-4270, 2012.","title":"A comparison of priority rules for the job shop scheduling problem under different flow time-and tardiness-related objective functions","authors":"Veronique Sels, Nele Gheysen, Mario Vanhoucke","year":"2012","venue":"International Journal of Production Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2021] Penghao Sun, Zehua Guo, Junchao Wang, Junfei Li, Julong Lan, and Yuxiang Hu. Deepweave: Ac- celerating job completion time with deep reinforcement learning-based coflow scheduling. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3314- 3320, 2021.","title":"Deepweave: Accelerating job completion time with deep reinforcement learning-based coflow scheduling","authors":"Sun","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/458"},{"position":20,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approxima- tion. Advances in neural information processing systems, 12, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David Richard S Sutton, Satinder Mcallester, Yishay Singh, Mansour","year":"1999","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Taillard, 1993] E. Taillard. Benchmarks for basic scheduling problems. European Journal of Operational Research, 64(2):278-285, January 1993.","title":"Benchmarks for basic scheduling problems","authors":"E Taillard","year":"1993","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2015a] Oriol Vinyals, Samy Bengio, and Man- junath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06391, 2015.","title":"Order matters: Sequence to sequence for sets","authors":"Vinyals","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vinyals et al., 2015b] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in neural information processing systems, 28, 2015.","title":"Pointer networks","authors":"Vinyals","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021a] Libing Wang, Xin Hu, Yin Wang, Su- jie Xu, Shijun Ma, Kexin Yang, Zhijun Liu, and Weidong Wang. Dynamic job-shop scheduling in smart manufactur- ing using deep reinforcement learning. Computer Networks, 190:107969, 2021.","title":"Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning","authors":"Wang","year":"2021","venue":"Computer Networks","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021b] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.","title":"A survey on curriculum learning","authors":"Wang","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Waschneck et al., 2018] Bernd Waschneck, André Reich- staller, Lenz Belzner, Thomas Altenmüller, Thomas Bauernhansl, Alexander Knapp, and Andreas Kyek. Opti- mization of global production scheduling with deep rein- forcement learning. Procedia Cirp, 72:1264-1269, 2018.","title":"Optimization of global production scheduling with deep reinforcement learning","authors":"Bernd Waschneck, André Reichstaller, Lenz Belzner, Thomas Altenmüller, Thomas Bauernhansl, Alexander Knapp, Andreas Kyek","year":"2018","venue":"Procedia Cirp","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang and Dietterich, 1995] Wei Zhang and Thomas G Di- etterich. A reinforcement learning approach to job-shop scheduling. In IJCAI, volume 95, pages 1114-1120. Cite- seer, 1995.","title":"A reinforcement learning approach to job-shop scheduling","authors":"Wei Zhang, Thomas G Dietterich","year":"1995","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1995/144"},{"position":31,"raw":"[Zhang et al., 2020a] Cong Zhang, Wen Song, Zhiguang Cao, Jie Zhang, Puay Siew Tan, and Xu Chi. Learning to dis- patch for job shop scheduling via deep reinforcement learn- ing. Advances in Neural Information Processing Systems, 33:1621-1632, 2020.","title":"Learning to dispatch for job shop scheduling via deep reinforcement learning","authors":"Zhang","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2020b] Di Zhang, Dong Dai, Youbiao He, For- rest Sheng Bao, and Bing Xie. Rlscheduler: an automated hpc batch job scheduler using reinforcement learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15. IEEE, 2020.","title":"Rlscheduler: an automated hpc batch job scheduler using reinforcement learning","authors":"Zhang","year":"2020","venue":"SC20: International Conference for High Performance Computing, Networking, Storage and Analysis","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zheng et al., 2019] Shuai Zheng, Chetan Gupta, and Susumu Serita. Manufacturing dispatching using reinforce- ment and transfer learning. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, pages 655-671. Springer, 2019.","title":"Manufacturing dispatching using reinforcement and transfer learning","authors":"Shuai Zheng, Chetan Gupta, Susumu Serita","year":"2019","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}