{"type":"ScholarlyArticle","title":"Levin Tree Search with Context Models","authors":[{"name":"Laurent Orseau","slug":"laurent-orseau","given":"Laurent","family":"Orseau","affiliation":"Alberta Machine Intelligence Institute (Amii) , Canada"},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"Alberta Machine Intelligence Institute (Amii) , Canada"},{"name":"Levi H. S. Lelis","slug":"levi-h-s-lelis","given":"Levi H. S.","family":"Lelis","affiliation":"Department of Computing Science , University of Alberta , Canada and"}],"abstract":"Levin Tree Search (LTS) is a search algorithm that makes use of a policy (a probability distribution over actions) \r\nand comes with a theoretical guarantee on the number of expansions before reaching a goal node, depending on the quality of the policy. \r\nThis guarantee can be used as a loss function, which we call the LTS loss, to optimize neural networks representing the policy (LTS+NN). \r\nIn this work we show that the neural network can be substituted with parameterized context models originating from the online compression literature (LTS+CM). \r\nWe show that the LTS loss is convex under this new model,\r\nwhich allows for using standard convex optimization tools,\r\nand obtain convergence guarantees to the optimal parameters in an online setting for a given set of solution trajectories --- guarantees that cannot be provided for neural networks. \r\nThe new LTS+CM algorithm compares favorably against LTS+NN on several benchmarks: Sokoban (Boxoban), The Witness, and the 24-Sliding Tile puzzle (STP). The difference is particularly large on STP, where LTS+NN fails to solve most of the test instances while LTS+CM solves each test instance in a fraction of a second.\r\nFurthermore, we show that LTS+CM is able to learn a policy that solves the Rubik's cube in only a few hundred expansions, which considerably improves upon previous machine learning techniques.","doi":"10.24963/ijcai.2023/624","url":"https://ijcai.org/proceedings/2023/624","pdf":"https://ijcai.org/proceedings/2023/0624.pdf","markdown":"https://ijcai.org/proceedings/2023/624.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/624","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Search","track":"Main Track","number":624,"pages":{"first":5622,"last":5630},"keywords":["Search: S: Search and machine learning","Search: S: Heuristic search"],"videos":[],"references":[{"position":1,"raw":"[Abel et al., 2020] Zachary Abel, Jeffrey Bosboom, Michael J. Coulombe, Erik D. Demaine, Linus Hamil- ton, Adam Hesterberg, Justin Kopinsky, Jayson Lynch, Mikhail Rudoy, and Clemens Thielen. Who witnesses the witness? finding witnesses in the witness is hard and sometimes impossible. Theor. Comput. Sci., 839:41-102, 2020.","title":"Who witnesses the witness? finding witnesses in the witness is hard and sometimes impossible","authors":"Zachary Abel, Jeffrey Bosboom, Michael J Coulombe, Erik D Demaine, Linus Hamilton, Adam Hesterberg, Justin Kopinsky, Jayson Lynch, Mikhail Rudoy, Clemens Thielen","year":"2020","venue":"Theor. Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agostinelli et al., 2019] Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence, 1, 07 2019.","title":"Solving the rubik's cube with deep reinforcement learning and search","authors":"Agostinelli","year":"2019","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agostinelli et al., 2021] Forest Agostinelli, Alexander Shmakov, Stephen McAleer, Roy Fox, and Pierre Baldi. A* search without expansions: Learning heuristic functions with deep q-networks, 2021. arXiv 2102.04518.","title":"A* search without expansions: Learning heuristic functions with deep q-networks","authors":"Agostinelli","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Allen et al., 2021] Cameron Allen, Michael Katz, Tim Klinger, George Konidaris, Matthew Riemer, and Gerald Tesauro. Efficient black-box planning using macro-actions with focused effects. In Proceedings of the Thirtieth Inter- national Joint Conference on Artificial Intelligence, pages 4024-4031, 2021.","title":"Efficient black-box planning using macro-actions with focused effects","authors":"Cameron Allen, Michael Katz, Tim Klinger, George Konidaris, Matthew Riemer, Gerald Tesauro","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/554"},{"position":5,"raw":"[Boyd and Vandenberghe, 2004] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge Univer- sity Press, Cambridge, England, 2004.","title":"Convex Optimization","authors":"Vandenberghe Boyd","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Budden et al., 2020] David Budden, Adam Marblestone, Eren Sezener, Tor Lattimore, Gregory Wayne, and Joel Veness. Gaussian gated linear networks. In Advances in Neural Information Processing Systems, volume 33, pages 16508-16519.","title":"Gaussian gated linear networks","authors":"David Budden, Adam Marblestone, Eren Sezener, Tor Lattimore, Gregory Wayne, Joel Veness","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games. Cam- bridge University Press, New York, NY, USA, 2006.","title":"Nicolo Cesa-Bianchi and Gabor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Culberson and Schaeffer, 1998] Joseph C. Culberson and Jonathan Schaeffer. Pattern databases. Computational In- telligence, 14(3):318-334, 1998.","title":"Pattern databases","authors":"Joseph C Culberson, Jonathan Schaeffer","year":"1998","venue":"Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Culberson, 1999] Joseph C. Culberson. Sokoban is PSPACE-Complete. In Fun With Algorithms, pages 65- 76, 1999.","title":"Sokoban is PSPACE-Complete","authors":"C Joseph, Culberson","year":"1999","venue":"Fun With Algorithms","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guez et al., 2018] Arthur Guez, Mehdi Mirza, Karol Gre- gor, Rishabh Kabra, Sebastien Racaniere, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, Timothy Lilli- crap, and Victor Valdes. An investigation of model-free planning: boxoban levels. https://github.com/deepmind/ boxoban-levels/, 2018. Accessed: 2023-05-01.","title":"An investigation of model-free planning: boxoban levels","authors":"Guez","year":"2018","venue":null,"doi":null,"url":"https://github.com/deepmind/boxoban-levels/,2018","internal_url":null},{"position":13,"raw":"[Guez et al., 2019] Arthur Guez, Mehdi Mirza, Karol Gre- gor, Rishabh Kabra, Sebastien Racaniere, Theophane We- ber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillicrap. An investigation of model-free planning. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Re- search, pages 2464-2473. PMLR, 2019.","title":"An investigation of model-free planning","authors":"Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, Timothy Lillicrap","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hazan, 2016] Elad Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization, 2(3-4):157-325, 2016.","title":"Introduction to online convex optimization","authors":"Elad Hazan","year":"2016","venue":"Foundations and Trends® in Optimization","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hinton, 2002] Geoffrey E. Hinton. Training Products of Ex- perts by Minimizing Contrastive Divergence. Neural Com- putation, 14(8):1771-1800, 08 2002.","title":"Training Products of Experts by Minimizing Contrastive Divergence","authors":"Geoffrey E Hinton","year":"2002","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jabbari Arfaee et al., 2011] S. Jabbari Arfaee, S. Zilles, and R. C. Holte. Learning heuristic functions for large state spaces. Artificial Intelligence, 175(16-17):2075-2098, 2011.","title":"Learning heuristic functions for large state spaces","authors":"S Jabbari Arfaee, S Zilles, R C Holte","year":"2011","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Joulani et al., 2013] Pooria Joulani, Andras Gyorgy, and Csaba Szepesvari. Online learning under delayed feed- back. In Proceedings of the 30th International Confer- ence on Machine Learning, volume 28(3) of Proceedings of Machine Learning Research, pages 1453-1461. PMLR, 2013.","title":"Online learning under delayed feedback","authors":"Joulani","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Korf, 1997] Richard E. Korf. Finding optimal solutions to rubik's cube using pattern databases. In Proceedings of the Fourteenth National Conference on Artificial Intelli- gence and Ninth Conference on Innovative Applications of Artificial Intelligence, AAAI'97/IAAI'97, page 700-705. AAAI Press, 1997.","title":"Finding optimal solutions to rubik's cube using pattern databases","authors":"Richard E Korf","year":"1997","venue":"Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Conference on Innovative Applications of Artificial Intelligence, AAAI'97/IAAI'97","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mattern, 2013] Christopher Mattern. Linear and geometric mixtures -analysis. Proceedings of the Data Compression Conference, pages 301-310, 02 2013.","title":"Linear and geometric mixtures -analysis","authors":"Christopher Mattern","year":"2013","venue":"Proceedings of the Data Compression Conference","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mattern, 2016] Christopher Mattern. On Statistical Data Compression. PhD thesis, Technische Universität Ilmenau, Fakultät für Informatik und Automatisierung, Feb 2016.","title":"On Statistical Data Compression","authors":"Christopher Mattern","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Matthew, 2005] V Mahoney Matthew. Adaptive weighing of context models for lossless data compression. Florida Institute of Technology CS Dept, Technical Report CS- 2005-16, https://www. cs. fit. edu/Projects/tech reports/cs- 2005-16. pdf, 2005.","title":"Adaptive weighing of context models for lossless data compression","authors":"Matthew Mahoney","year":"2005","venue":null,"doi":null,"url":"https://www.cs.fit.edu/Projects/techreports/cs-2005-16.pdf","internal_url":null},{"position":22,"raw":"[Mittal et al., 2022] Dixant Mittal, Siddharth Aravindan, and Wee Sun Lee. Expose: Combining state-based explo- ration with gradient-based online search, 2022. arXiv 2202.01461.","title":"Expose: Combining state-based exploration with gradient-based online search","authors":"Mittal","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Orseau and Lelis, 2021] Laurent Orseau and Levi H. S. Lelis. Policy-guided heuristic search with guarantees. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 35(14):12382-12390, May 2021.","title":"Policy-guided heuristic search with guarantees","authors":"Laurent Orseau, H S Levi, Lelis","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Orseau et al., 2018] Laurent Orseau, Levi Lelis, Tor Latti- more, and Theophane Weber. Single-agent policy tree search with guarantees. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 31.","title":"Single-agent policy tree search with guarantees","authors":"Laurent Orseau, Levi Lelis, Tor Lattimore, Theophane Weber","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Curran Associates, Inc., 2018.","title":null,"authors":"Curran Associates, Inc","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pearl, 1984] Judea Pearl. Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison- Wesley Longman Publishing Co., Inc., USA, 1984.","title":"Judea Pearl. Heuristics: Intelligent Search Strategies for Computer Problem Solving","authors":"Pearl","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ratner and Warmuth, 1986] Daniel Ratner and Manfred Warmuth. Finding a shortest solution for the nxn exten- sion of the 15-puzzle is intractable. In Proceedings of the Fifth AAAI National Conference on Artificial Intelligence, AAAI'86, page 168-172. AAAI Press, 1986.","title":"Finding a shortest solution for the nxn extension of the 15-puzzle is intractable","authors":"Warmuth Ratner, Daniel Ratner, Manfred Warmuth","year":"1986","venue":"Proceedings of the Fifth AAAI National Conference on Artificial Intelligence, AAAI'86","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rissanen, 1983] J. Rissanen. A universal data compres- sion system. IEEE Transactions on Information Theory, 29(5):656-664, 1983.","title":"A universal data compression system","authors":"J Rissanen","year":"1983","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shalev-Shwartz, 2007] Shai Shalev-Shwartz. Online learn- ing: Theory, algorithms, and applications. PhD thesis, Hebrew University, Jerusalem, 2007.","title":"Online learning: Theory, algorithms, and applications","authors":"Shai Shalev-Shwartz","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Veness et al., 2017] Joel Veness, Tor Lattimore, Avishkar Bhoopchand, Agnieszka Grabska-Barwinska, Christopher Mattern, and Peter Toth. Online learning with gated linear networks, 2017. arXiv 1712.01897.","title":"Online learning with gated linear networks","authors":"Joel Veness, Tor Lattimore, Avishkar Bhoopchand, Agnieszka Grabska-Barwinska, Christopher Mattern, Peter Toth","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Veness et al., 2021] Joel Veness, Tor Lattimore, David Bud- den, Avishkar Bhoopchand, Christopher Mattern, Ag- nieszka Grabska-Barwinska, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, and Marcus Hutter. Gated lin- ear networks. Proceedings of the AAAI Conference on Ar- tificial Intelligence, 35(11):10015-10023, May 2021.","title":"Gated linear networks","authors":"Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Agnieszka Grabska-Barwinska, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, Marcus Hutter","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Willems et al., 1995] Frans M. J. Willems, Yuri M. Shtarkov, and Tjalling J. Tjalkens. The context tree weighting method: Basic properties. IEEE Transactions on Information Theory, 41:653-664, 1995.","title":"The context tree weighting method: Basic properties","authors":"M J Frans, Yuri M Willems, Tjalling J Shtarkov, Tjalkens","year":"1995","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning, pages 928-935, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"Proceedings of the Twentieth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}