{"type":"ScholarlyArticle","title":"On Using Admissible Bounds for Learning Forward Search Heuristics","authors":[{"name":"Carlos Núñez-Molina","slug":"carlos-nunez-molina","given":"Carlos","family":"Núñez-Molina","affiliation":"University of Granada , Spain"},{"name":"Masataro Asai","slug":"masataro-asai","given":"Masataro","family":"Asai","affiliation":"MIT-IBM Watson AI Lab , USA"},{"name":"Pablo Mesejo","slug":"pablo-mesejo","given":"Pablo","family":"Mesejo","affiliation":"University of Granada , Spain"},{"name":"Juan Fernandez-Olivares","slug":"juan-fernandez-olivares","given":"Juan","family":"Fernandez-Olivares","affiliation":"University of Granada , Spain"}],"abstract":"In recent years, there has been growing interest in utilizing modern machine learning techniques to learn heuristic functions for forward search algorithms. Despite this, there has been little theoretical understanding of what they should learn, how to train them, and why we do so. This lack of understanding has resulted in the adoption of diverse training targets (suboptimal vs optimal costs vs admissible heuristics) and loss functions (e.g., square vs absolute errors) in the literature. In this work, we focus on how to effectively utilize the information provided by admissible heuristics in heuristic learning. We argue that learning from poly-time admissible heuristics by minimizing mean square errors (MSE) is not the correct approach, since its result is merely a noisy, inadmissible copy of an efficiently computable heuristic. Instead, we propose to model the learned heuristic as a truncated gaussian, where admissible heuristics are used not as training targets but as lower bounds of this distribution. This results in a different loss function from the MSE commonly employed in the literature, which implicitly models the learned heuristic as a gaussian distribution. We conduct experiments where both MSE and our novel loss function are applied to learning a heuristic from optimal plan costs. Results show that our proposed method converges faster during training and yields better heuristics.","doi":"10.24963/ijcai.2024/747","url":"https://ijcai.org/proceedings/2024/747","pdf":"https://ijcai.org/proceedings/2024/0747.pdf","markdown":"https://ijcai.org/proceedings/2024/747.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/747","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Planning and Scheduling","track":"Main Track","number":747,"pages":{"first":6761,"last":6769},"keywords":["Machine Learning: ML: Neuro-symbolic methods","Machine Learning: ML: Knowledge-aided learning","Planning and Scheduling: PS: Learning in planning and scheduling","Search: S: Heuristic search"],"videos":[],"references":[{"position":1,"raw":"[Alkhazraji et al., 2020] Yusra Alkhazraji, Matthias Frorath, Markus Grützner, Malte Helmert, Thomas Liebetraut, Robert Mattmüller, Manuela Ortlieb, Jendrik Seipp, Tobias Springenberg, Philip Stahl, and Jan Wülfing. Pyperplan, 2020.","title":"Pyperplan","authors":"Yusra Alkhazraji, Matthias Frorath, Markus Grützner, Malte Helmert, Thomas Liebetraut, Robert Mattmüller, Manuela Ortlieb, Jendrik Seipp, Tobias Springenberg, Philip Stahl, Jan Wülfing","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arfaee et al., 2011] Shahab Jabbari Arfaee, Sandra Zilles, and Robert C Holte. Learning heuristic functions for large state spaces. Artificial Intelligence, 175(16-17):2075-2098, 2011.","title":"Learning heuristic functions for large state spaces","authors":"Shahab Jabbari Arfaee, Sandra Zilles, Robert C Holte","year":"2011","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res.(JAIR), 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"J. Artif. Intell. Res.(JAIR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Betz and Helmert, 2009] Christoph Betz and Malte Helmert. Planning with h + in theory and practice. In Advances in Artificial Intelligence, volume 5803, pages 9-16. Springer, 2009.","title":"Planning with h + in theory and practice","authors":"Helmert Betz, Christoph Betz, Malte Helmert","year":"2009","venue":"Advances in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bonet and Geffner, 1999] Blai Bonet and Hector Geffner. Planning as heuristic search: New results. In Proc. of Eu- ropean Conference on Planning, pages 360-372. Springer, 1999.","title":"Planning as heuristic search: New results","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"1999","venue":"Proc. of European Conference on Planning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bonet and Geffner, 2001] Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence, 129(1- 2):5-33, 2001.","title":"Planning as heuristic search","authors":"Geffner Bonet, Blai Bonet, Héctor Geffner","year":"2001","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cheng et al., 2021] Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learn- ing. Proc. of the Advances in Neural Information Process- ing Systems (Neurips), 34:13550-13563, 2021.","title":"Heuristic-guided reinforcement learning","authors":"Ching-An Cheng, Andrey Kolobov, Adith Swaminathan","year":"2021","venue":"Proc. of the Advances in Neural Information Processing Systems (Neurips)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2019] Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural Logic Machines. In Proc. of the International Conference on Learning Representations (ICLR), 2019.","title":"Neural Logic Machines","authors":"Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, Denny Zhou","year":"2019","venue":"Proc. of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dowson and Wragg, 1973] D Dowson and A Wragg. Maximum-entropy distributions having prescribed first and second moments (corresp.). IEEE Transactions on Infor- mation Theory, 19(5):689-693, 1973.","title":"Maximum-entropy distributions having prescribed first and second moments (corresp.)","authors":"D Dowson, Wragg","year":"1973","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Eisen et al., 2019] Mark Eisen, Clark Zhang, Luiz FO Cha- mon, Daniel D Lee, and Alejandro Ribeiro. Learning Opti- mal Resource Allocations in Wireless Systems. IEEE Trans- actions on Signal Processing, 67(10):2775-2790, 2019.","title":"Learning Optimal Resource Allocations in Wireless Systems","authors":"Mark Eisen, Clark Zhang, Daniel D Luiz Fo Chamon, Alejandro Lee, Ribeiro","year":"2019","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ernandes and Gori, 2004] Marco Ernandes and Marco Gori. Likely-admissible and sub-symbolic heuristics. In Proc. of European Conference on Artificial Intelligence, volume 16, pages 613-617, 2004.","title":"Likely-admissible and sub-symbolic heuristics","authors":"Gori ; ] Ernandes, Ernandes Marco, Marco Gori","year":"2004","venue":"Proc. of European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ferber et al., 2022] Patrick Ferber, Florian Geißer, Felipe Trevizan, Malte Helmert, and Jörg Hoffmann. Neural network heuristic functions for classical planning: Boot- strapping and comparison to other methods. In Proc. of the International Conference on Automated Planning and Scheduling (ICAPS), volume 32, pages 583-587, 2022.","title":"Neural network heuristic functions for classical planning: Bootstrapping and comparison to other methods","authors":"Patrick Ferber, Florian Geißer, Felipe Trevizan, Malte Helmert, Jörg Hoffmann","year":"2022","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fikes et al., 1972] Richard E Fikes, Peter E Hart, and Nils J Nilsson. Learning and executing generalized robot plans. Artificial Intelligence, 3:251-288, 1972.","title":"Learning and executing generalized robot plans","authors":"Peter E Richard E Fikes, Nils J Hart, Nilsson","year":"1972","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gehring et al., 2022] Clement Gehring, Masataro Asai, Ro- han Chitnis, Tom Silver, Leslie Kaelbling, Shirin Sohrabi, and Michael Katz. Reinforcement learning for classical planning: Viewing heuristics as dense reward generators. In Proc. of the International Conference on Automated Plan- ning and Scheduling (ICAPS), volume 32, pages 588-596, 2022.","title":"Reinforcement learning for classical planning: Viewing heuristics as dense reward generators","authors":"Clement Gehring, Masataro Asai, Rohan Chitnis, Tom Silver, Leslie Kaelbling, Shirin Sohrabi, Michael Katz","year":"2022","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gomoluch et al., 2017] Pawel Gomoluch, Dalal Alrajeh, Alessandra Russo, and Antonio Bucchiarone. Towards learning domain-independent planning heuristics. arXiv preprint arXiv:1707.06895, 2017.","title":"Towards learning domain-independent planning heuristics","authors":"Pawel Gomoluch, Dalal Alrajeh, Alessandra Russo, Antonio Bucchiarone","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hart et al., 1968] Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.","title":"A formal basis for the heuristic determination of minimum cost paths","authors":"Nils J Peter E Hart, Bertram Nilsson, Raphael","year":"1968","venue":"IEEE transactions on Systems Science and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Helmert and Domshlak, 2009] Malte Helmert and Carmel Domshlak. Landmarks, critical paths and abstractions: what's the difference anyway? In Proc. of the Interna- tional Conference on Automated Planning and Scheduling (ICAPS), volume 19, pages 162-169, 2009.","title":"Malte Helmert and Carmel Domshlak. Landmarks, critical paths and abstractions: what's the difference anyway?","authors":"Domshlak Helmert","year":"2009","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Helmert, 2006] Malte Helmert. The fast downward planning system. J. Artif. Intell. Res.(JAIR), 26:191-246, 2006.","title":"The fast downward planning system","authors":"Malte Helmert","year":"2006","venue":"J. Artif. Intell. Res.(JAIR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hoffmann and Nebel, 2001] Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. J. Artif. Intell. Res.(JAIR), 14:253- 302, 2001.","title":"The ff planning system: Fast plan generation through heuristic search","authors":"Nebel Hoffmann, Jörg Hoffmann, Bernhard Nebel","year":"2001","venue":"J. Artif. Intell. Res.(JAIR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jaynes, 1957] Edwin T Jaynes. Information theory and sta- tistical mechanics. Phys. Rev., 106(4):620-630, 1957.","title":"Information theory and statistical mechanics","authors":"T Edwin, Jaynes","year":"1957","venue":"Phys. Rev","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kamran et al., 2021] Danial Kamran, Tizian Engelgeh, Mar- vin Busch, Johannes Fischer, and Christoph Stiller. Mini- mizing safety interference for safe and comfortable auto- mated driving with distributional reinforcement learning. In Proc. of IEEE International Workshop on Intelligent Robots and Systems (IROS), pages 1236-1243. IEEE, 2021.","title":"Minimizing safety interference for safe and comfortable automated driving with distributional reinforcement learning","authors":"Danial Kamran, Tizian Engelgeh, Marvin Busch, Johannes Fischer, Christoph Stiller","year":"2021","venue":"Proc. of IEEE International Workshop on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv preprint arXiv:2010.13187, 2017.","title":"Ilya Loshchilov and Frank Hutter","authors":"Hutter Loshchilov","year":"2017","venue":"Fixing weight decay regularization in adam","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Marom and Rosman, 2020] Ofir Marom and Benjamin Ros- man. Utilising uncertainty for efficient learning of likely- admissible heuristics. In Proc. of the International Con- ference on Automated Planning and Scheduling (ICAPS), volume 30, pages 560-568, 2020.","title":"Utilising uncertainty for efficient learning of likelyadmissible heuristics","authors":"Rosman Marom, Ofir Marom, Benjamin Rosman","year":"2020","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Murray et al., 2023] Andrew Murray, Ashwin Arulselvan, Michael Cashmore, Marc Roper, and Jeremy Frank. A column generation approach to correlated simple temporal networks. In Proc. of the International Conference on Automated Planning and Scheduling (ICAPS), volume 33, pages 295-303, 2023.","title":"A column generation approach to correlated simple temporal networks","authors":"Andrew Murray, Ashwin Arulselvan, Michael Cashmore, Marc Roper, Jeremy Frank","year":"2023","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nazari et al., 2018] Mohammadreza Nazari, Afshin Oroo- jlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. Proc. of the Advances in Neural Information Processing Systems (Neurips), 31:9861-9871, 2018.","title":"Reinforcement learning for solving the vehicle routing problem","authors":"Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, Martin Takác","year":"2018","venue":"Proc. of the Advances in Neural Information Processing Systems (Neurips)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proc. of the International Conference on Machine Learning (ICML), volume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"Proc. of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Richter et al., 2010] Silvia Richter, Jordan Thayer, and Wheeler Ruml. The joy of forgetting: Faster anytime search via restarting. In Proc. of the International Conference on Automated Planning and Scheduling (ICAPS), volume 20, pages 137-144, 2010.","title":"The joy of forgetting: Faster anytime search via restarting","authors":"Silvia Richter, Jordan Thayer, Wheeler Ruml","year":"2010","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Richter et al., 2011] Silvia Richter, Matthias Westphal, and Malte Helmert. Lama 2008 and 2011. In International Planning Competition, pages 117-124, 2011.","title":"Lama 2008 and 2011","authors":"Silvia Richter, Matthias Westphal, Malte Helmert","year":"2011","venue":"International Planning Competition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Russell and Norvig, 2010] Stuart J Russell and Peter Norvig. Artificial intelligence a modern approach. London, 2010.","title":"Artificial intelligence a modern approach","authors":"J Stuart, Peter Russell, Norvig","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Shen et al., 2020] William Shen, Felipe Trevizan, and Sylvie Thiébaux. Learning domain-independent planning heuris- tics with hypergraph networks. In Proc. of the Interna- tional Conference on Automated Planning and Scheduling (ICAPS), volume 30, pages 574-584, 2020.","title":"Learning domain-independent planning heuristics with hypergraph networks","authors":"William Shen, Felipe Trevizan, Sylvie Thiébaux","year":"2020","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Maddi- son, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Pan- neershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Tim- othy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Takahashi et al., 2019] Takeshi Takahashi, He Sun, Dong Tian, and Yebin Wang. Learning heuristic functions for mobile robot path planning using deep neural networks. In Proc. of the International Conference on Automated Planning and Scheduling (ICAPS), volume 29, pages 764- 772, 2019.","title":"Learning heuristic functions for mobile robot path planning using deep neural networks","authors":"Takeshi Takahashi, He Sun, Dong Tian, Yebin Wang","year":"2019","venue":"Proc. of the International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vallati et al., 2015] Mauro Vallati, Lukás Chrpa, Marek Grzes, Thomas Leo McCluskey, Mark Roberts, and Scott Sanner. The 2014 international planning competition: Progress and trends. AI Magazine, 36(3):90-98, 2015.","title":"The 2014 international planning competition: Progress and trends","authors":"Mauro Vallati, Lukás Chrpa, Marek Grzes, Thomas Leo Mccluskey, Mark Roberts, Scott Sanner","year":"2015","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xin et al., 2021] Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. NeuroLKH: Combining Deep Learning Model with Lin-Kernighan-Helsgaun Heuristic for Solving the Traveling Salesman Problem. Proc. of the Advances in Neural Information Processing Systems (Neurips), 34:7472- 7483, 2021.","title":"NeuroLKH: Combining Deep Learning Model with Lin-Kernighan-Helsgaun Heuristic for Solving the Traveling Salesman Problem","authors":"Wen Liang Xin, Zhiguang Song, Jie Cao, Zhang","year":"2021","venue":"Proc. of the Advances in Neural Information Processing Systems (Neurips)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yoon et al., 2008] Sungwook Yoon, Alan Fern, and Robert Givan. Learning control knowledge for forward search plan- ning. J. Mach. Learn. Res.(JMLR), 9(4):683-718, 2008.","title":"Learning control knowledge for forward search planning","authors":"Yoon","year":"2008","venue":"J. Mach. Learn. Res.(JMLR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}