{"type":"ScholarlyArticle","title":"Learning and Solving Regular Decision Processes","authors":[{"name":"Eden Abadi","slug":"eden-abadi","given":"Eden","family":"Abadi","affiliation":"Department of Computer Science , Ben Gurion University , Israel"},{"name":"Ronen I. Brafman","slug":"ronen-i-brafman","given":"Ronen I.","family":"Brafman","affiliation":"Department of Computer Science , Ben Gurion University , Israel"}],"abstract":"Regular Decision Processes (RDPs) are a recently introduced model that extends MDPs with non-Markovian dynamics and rewards. The non-Markovian behavior is restricted to depend on regular properties of the history. These can be specified using regular expressions or formulas in linear dynamic logic over finite traces. Fully specified RDPs can be solved by compiling them into an appropriate MDP. Learning RDPs from data is a challenging problem that has yet to be addressed, on which we focus in this paper. Our approach rests on a new representation for RDPs using Mealy Machines that emit a distribution and an expected reward for each state-action pair. Building on this representation, we combine automata learning techniques with history clustering to learn such a Mealy machine and solve it by adapting MCTS to it. We empirically evaluate this approach, demonstrating its feasibility.","doi":"10.24963/ijcai.2020/270","url":"https://ijcai.org/proceedings/2020/270","pdf":"https://ijcai.org/proceedings/2020/0270.pdf","markdown":"https://ijcai.org/proceedings/2020/270.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/270","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":270,"pages":{"first":1948,"last":1954},"keywords":["Planning and Scheduling: Markov Decisions Processes","Knowledge Representation and Reasoning: Action, Change and Causality","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24203/file_24203.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baier et al., 2008] Jorge A. Baier, Christian Fritz, Meghyn Bienvenu, and Sheila McIlraith. Beyond classical plan- ning: Procedural control knowledge and preferences in state-of-the-art planners. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), Nectar Track, pages 1509-1512, Chicago, Illinois, USA, July 13-17 2008.","title":"Beyond classical planning: Procedural control knowledge and preferences in state-of-the-art planners","authors":"Jorge A Baier, Christian Fritz, Meghyn Bienvenu, Sheila Mcilraith","year":"2008","venue":"Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), Nectar Track","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brafman and De Giacomo, 2019] Ronen I. Brafman and Giuseppe De Giacomo. Planning for ltlf /ldlf goals in non-markovian fully observable nondeterministic domains. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 1602-1608, 2019.","title":"Planning for ltlf /ldlf goals in non-markovian fully observable nondeterministic domains","authors":"De Brafman, I Giacomo ; ] Ronen, Giuseppe Brafman, Giacomo De","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019","doi":null,"url":null,"internal_url":"/proceedings/2019/222"},{"position":4,"raw":"[Brafman and Tennenholtz, 2002] Ronen I. Brafman and Moshe Tennenholtz. R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res., 3:213-231, 2002.","title":"R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":5,"raw":"[Camacho et al., 2019] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIlraith. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In Pro- ceedings of the Twenty-Eighth International Joint Confer- ence on Artificial Intelligence (IJCAI), 2019.","title":"LTL and beyond: Formal languages for reward function specification in reinforcement learning","authors":"Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, Sheila A Mcilraith","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2019/840"},{"position":6,"raw":"[De Giacomo and Vardi, 2013] Giuseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear dynamic logic on finite traces. In IJCAI-13, pages 854-860, 2013.","title":"Linear temporal logic and linear dynamic logic on finite traces","authors":"De Giacomo, Giuseppe De Vardi, Moshe Y Giacomo, Vardi","year":"2013","venue":"IJCAI-13","doi":null,"url":null,"internal_url":"/proceedings/2013/132"},{"position":7,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Ma- chine Learning: ECML 2006, 17th European Conference on Machine Learning, Berlin, Germany, September 18-22, 2006, Proceedings, pages 282-293, 2006.","title":"Bandit based monte-carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"Machine Learning: ECML 2006, 17th European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kullback and Leibler, 1951] Solomon Kullback and Richard Leibler. On information and sufficiency. Annals of Mathe- matical Statistics, 22(1):79-86, 1951.","title":"Solomon Kullback and Richard Leibler","authors":"Leibler Kullback","year":"1951","venue":"Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mao et al., 2016] Hua Mao, Yingke Chen, Manfred Jaeger, Thomas D. Nielsen, Kim G. Larsen, and Brian Nielsen. Learning deterministic probabilistic automata from a model checking perspective. Mach. Learn., 105(2):255-299, 2016.","title":"Learning deterministic probabilistic automata from a model checking perspective","authors":"Hua Mao, Yingke Chen, Manfred Jaeger, Thomas D Nielsen, Kim G Larsen, Brian Nielsen","year":"2016","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Puterman, 2005] Martin L. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. Wiley, 2005.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Re- inforcement Learning: An Introduction. MIT Press, Cam- bridge, MA, 1998.","title":"Reinforcement Learning: An Introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Tappler et al., 2019] Martin Tappler, Bernhard K. Aichernig, Giovanni Bacci, Maria Eichlseder, and Kim G. Larsen. L * - based learning of markov decision processes. In Formal Methods -The Next 30 Years -Third World Congress, FM 2019, Porto, Portugal, October 7-11, 2019, Proceedings, pages 651-669, 2019.","title":"L *based learning of markov decision processes","authors":"Martin Tappler, Bernhard K Aichernig, Giovanni Bacci, Maria Eichlseder, Kim G Larsen","year":"2019","venue":"Formal Methods -The Next 30 Years -Third World Congress, FM 2019","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Verwer and Hammerschmidt, 2017] Sicco Verwer and Christian A. Hammerschmidt. flexfringe: A passive automaton learning package. In 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017, Shanghai, China, September 17-22, 2017, pages 638-642, 2017.","title":"flexfringe: A passive automaton learning package","authors":"Hammerschmidt Verwer, Sicco Verwer, Christian A Hammerschmidt","year":"2017","venue":"2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Watkins and Dayan, 1992] Christopher J. C. H. Watkins and Peter Dayan. Technical note q-learning. Machine Learning, 8:279-292, 1992.","title":"Technical note q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}