{"type":"ScholarlyArticle","title":"Reinforcement Learning for Cross-Domain Hyper-Heuristics","authors":[{"name":"Florian Mischek","slug":"florian-mischek","given":"Florian","family":"Mischek","affiliation":"Doppler Laboratory for Artificial Intelligence and Optimization for Planning and Scheduling , TU Wien"},{"name":"Nysret Musliu","slug":"nysret-musliu","given":"Nysret","family":"Musliu","affiliation":"Doppler Laboratory for Artificial Intelligence and Optimization for Planning and Scheduling , TU Wien"}],"abstract":"In this paper, we propose a new hyper-heuristic approach that uses reinforcement learning to automatically learn the selection of low-level heuristics across a wide range of problem domains.\r\nWe provide a detailed analysis and evaluation of the algorithm components, including different ways to represent the hyper-heuristic state space and reset strategies to avoid unpromising areas of the solution space.\r\nOur methods have been evaluated using HyFlex, a well-known benchmarking framework for cross-domain hyper-heuristics, and compared with state-of-the-art approaches.\r\nThe experimental evaluation shows that our reinforcement-learning based approach produces results that are competitive with the state-of-the-art, including the top participants of the Cross Domain Hyper-heuristic Search Competition 2011.","doi":"10.24963/ijcai.2022/664","url":"https://ijcai.org/proceedings/2022/664","pdf":"https://ijcai.org/proceedings/2022/0664.pdf","markdown":"https://ijcai.org/proceedings/2022/664.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/664","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Search","track":"Main Track","number":664,"pages":{"first":4793,"last":4799},"keywords":["Machine Learning: Reinforcement Learning","Search: Meta-Reasoning and Meta-Heuristics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986891","length_sec":99,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985034","length_sec":356,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Adriaensen et al., 2015] Steven Adriaensen, Gabriela Ochoa, and Ann Nowé. A benchmark set extension and comparative study for the hyflex framework. In 2015 IEEE Congress on Evolutionary Computation (CEC), pages 784-791, 2015.","title":"A benchmark set extension and comparative study for the hyflex framework","authors":"Steven Adriaensen, Gabriela Ochoa, Ann Nowé","year":"2015","venue":"IEEE Congress on Evolutionary Computation (CEC)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asta and Özcan, 2015] Shahriar Asta and Ender Özcan. A tensor-based selection hyper-heuristic for cross-domain heuristic search. Information Sciences, 299:412-432, 2015.","title":"A tensor-based selection hyper-heuristic for cross-domain heuristic search","authors":"Shahriar Asta, Ender Özcan","year":"2015","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burke et al., 2011] Edmund K. Burke, Michel Gendreau, Matthew Hyde, Graham Kendall, Barry McCollum, Gabriela Ochoa, Andrew J. Parkes, and Sanja Petrovic. The cross-domain heuristic search challenge -an interna- tional research competition. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization, pages 631- 634, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg.","title":"The cross-domain heuristic search challenge -an international research competition","authors":"Edmund K Burke, Michel Gendreau, Matthew Hyde, Graham Kendall, Barry Mccollum, Gabriela Ochoa, Andrew J Parkes, Sanja Petrovic","year":"2011","venue":"Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burke et al., 2019] Edmund K. Burke, Matthew R. Hyde, Graham Kendall, Gabriela Ochoa, Ender Özcan, and John R. Woodward. A Classification of Hyper-Heuristic Approaches: Revisited, pages 453-477. Springer Interna- tional Publishing, Cham, 2019.","title":"A Classification of Hyper-Heuristic Approaches: Revisited","authors":"Edmund K Burke, Matthew R Hyde, Graham Kendall, Gabriela Ochoa, Ender Özcan, John R Woodward","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chuang, 2020] Chung-Yao Chuang. Combining Multiple Heuristics: Studies on Neighborhood-base Heuristics and Sampling-based Heuristics. PhD thesis, Carnegie Mellon University, 2020.","title":"Combining Multiple Heuristics: Studies on Neighborhood-base Heuristics and Sampling-based Heuristics","authors":"Chung-Yao Chuang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Di Gaspero and Urli, 2012] Luca Di Gaspero and Tommaso Urli. Evaluation of a family of reinforcement learning cross-domain optimization heuristics. In Youssef Hamadi and Marc Schoenauer, editors, Learning and Intelligent Optimization, pages 384-389, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.","title":"Evaluation of a family of reinforcement learning cross-domain optimization heuristics","authors":"Luca Di, Gaspero, Tommaso Urli","year":"2012","venue":"Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hsiao et al., 2012] Ping-Che Hsiao, Tsung-Che Chiang, and Li-Chen Fu. A vns-based hyper-heuristic with adap- tive computational budget of local search. In 2012 IEEE Congress on Evolutionary Computation, pages 1-8, 2012.","title":"A vns-based hyper-heuristic with adaptive computational budget of local search","authors":"Hsiao","year":"2012","venue":"IEEE Congress on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Larose, 2011] Mathieu Larose. A hyper-heuristic for the chesc 2011. Report submitted for the CHeSC competition, 2011.","title":"A hyper-heuristic for the chesc","authors":"Mathieu Larose","year":"2011","venue":"Report submitted for the CHeSC competition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Luby et al., 1993] Michael Luby, Alistair Sinclair, and David Zuckerman. Optimal speedup of las vegas algo- rithms. Information Processing Letters, 47(4):173-180, 1993.","title":"Optimal speedup of las vegas algorithms","authors":"Michael Luby, Alistair Sinclair, David Zuckerman","year":"1993","venue":"Information Processing Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Meignan et al., 2010] David Meignan, Abderrafiaa Koukam, and Jean-Charles Créput. Coalition-based metaheuristic: a self-adaptive metaheuristic using rein- forcement learning and mimetism. Journal of Heuristics, 16(6):859-879, 2010.","title":"Coalition-based metaheuristic: a self-adaptive metaheuristic using reinforcement learning and mimetism","authors":"David Meignan, Abderrafiaa Koukam, Jean-Charles Créput","year":"2010","venue":"Journal of Heuristics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mısır et al., 2012] Mustafa Mısır, Katja Verbeeck, Patrick De Causmaecker, and Greet Vanden Berghe. An intelli- gent hyper-heuristic framework for chesc 2011. In Youssef Hamadi and Marc Schoenauer, editors, Learning and In- telligent Optimization, pages 461-466, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.","title":"An intelligent hyper-heuristic framework for chesc","authors":"Mustafa Mısır, Katja Verbeeck, Patrick De Causmaecker, Greet Vanden, Berghe","year":"2011","venue":"Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ochoa et al., 2012a] Gabriela Ochoa, Matthew Hyde, Tim Curtois, Jose A. Vazquez-Rodriguez, James Walker, Michel Gendreau, Graham Kendall, Barry McCollum, An- drew J. Parkes, Sanja Petrovic, and Edmund K. Burke. Hyflex: A benchmark framework for cross-domain heuris- tic search. In Jin-Kao Hao and Martin Middendorf, editors, Evolutionary Computation in Combinatorial Optimiza- tion, pages 136-147, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.","title":"Hyflex: A benchmark framework for cross-domain heuristic search","authors":"Ochoa","year":"2012","venue":"Evolutionary Computation in Combinatorial Optimization","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ochoa et al., 2012b] Gabriela Ochoa, James Walker, Matthew Hyde, and Tim Curtois. Adaptive evolutionary algorithms and extensions to the hyflex hyper-heuristic framework. In International Conference on Parallel Problem Solving from Nature, pages 418-427. Springer, 2012.","title":"Adaptive evolutionary algorithms and extensions to the hyflex hyper-heuristic framework","authors":"Ochoa","year":"2012","venue":"International Conference on Parallel Problem Solving from Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sabar and Kendall, 2015] Nasser R. Sabar and Graham Kendall. Population based monte carlo tree search hyper- heuristic for combinatorial optimization problems. Infor- mation Sciences, 314:225-239, 2015.","title":"Population based monte carlo tree search hyperheuristic for combinatorial optimization problems","authors":"R Nasser, Graham Sabar, Kendall","year":"2015","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}