{"type":"ScholarlyArticle","title":"Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processes","authors":[{"name":"Alessandro Ronca","slug":"alessandro-ronca","given":"Alessandro","family":"Ronca","affiliation":"Sapienza University of Rome , Italy"},{"name":"Gabriel Paludo Licks","slug":"gabriel-paludo-licks","given":"Gabriel","family":"Paludo Licks","affiliation":"Sapienza University of Rome , Italy"},{"name":"Giuseppe De Giacomo","slug":"giuseppe-de-giacomo","given":"Giuseppe","family":"De Giacomo","affiliation":""}],"abstract":"Our work aims at developing reinforcement learning algorithms that do not rely on the Markov assumption. We consider the class of Non-Markov Decision Processes where histories can be abstracted into a finite set of states while preserving the dynamics. We call it a Markov abstraction since it induces a Markov Decision Process over a set of states that encode the non-Markov dynamics. This phenomenon underlies the recently introduced Regular Decision Processes (as well as POMDPs where only a finite number of belief states is reachable). In all such kinds of decision process, an agent that uses a Markov abstraction can rely on the Markov property to achieve optimal behaviour. We show that Markov abstractions can be learned during reinforcement learning. Our approach combines automata learning and classic reinforcement learning. For these two tasks, standard algorithms can be employed. We show that our approach has PAC guarantees when the employed algorithms have PAC guarantees, and we also provide an experimental evaluation.","doi":"10.24963/ijcai.2022/473","url":"https://ijcai.org/proceedings/2022/473","pdf":"https://ijcai.org/proceedings/2022/0473.pdf","markdown":"https://ijcai.org/proceedings/2022/473.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/473","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":473,"pages":{"first":3408,"last":3415},"keywords":["Knowledge Representation and Reasoning: Reasoning about actions","Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis","Planning and Scheduling: Markov Decisions Processes"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987082","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985225","length_sec":705,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abadi and Brafman, 2020] Eden Abadi and Ronen I. Braf- man. Learning and solving regular decision processes. In IJCAI, 2020.","title":"Eden Abadi and Ronen I. Brafman. Learning and solving regular decision processes","authors":"Brafman Abadi","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacchus et al., 1996] Fahiem Bacchus, Craig Boutilier, and Adam J. Grove. Rewarding behaviors. In AAAI, 1996.","title":"Rewarding behaviors","authors":"Bacchus","year":"1996","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balle et al., 2013] Borja Balle, Jorge Castro, and Ricard Gavaldà. Learning probabilistic automata: A study in state distinguishability. Theor. Comp. Sci., 2013.","title":"Learning probabilistic automata: A study in state distinguishability","authors":"Borja Balle, Jorge Castro, Ricard Gavaldà","year":"2013","venue":"Theor. Comp. Sci","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balle et al., 2014] Borja Balle, Jorge Castro, and Ricard Gavaldà. Adaptively learning probabilistic deterministic automata from data streams. Mach. Learn., 2014.","title":"Adaptively learning probabilistic deterministic automata from data streams","authors":"Borja Balle, Jorge Castro, Ricard Gavaldà","year":"2014","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellman, 1957] Richard Bellman. A markovian decision process. J. Math. Mech., 1957.","title":"A markovian decision process","authors":"Richard Bellman","year":"1957","venue":"J. Math. Mech","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brafman and De Giacomo, 2019] Ronen I. Brafman and Giuseppe De Giacomo. Regular decision processes: A model for non-markovian domains. In IJCAI, 2019.","title":"Regular decision processes: A model for non-markovian domains","authors":"De Brafman, I Giacomo ; ] Ronen, Giuseppe Brafman, Giacomo De","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/766"},{"position":7,"raw":"[Brafman and Tennenholtz, 2002] Ronen I. Brafman and Moshe Tennenholtz. R-Max: A general polynomial time algorithm for near-optimal reinforcement learning. JMLR, 2002.","title":"R-Max: A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":8,"raw":"[Brafman et al., 2018] Ronen I. Brafman, Giuseppe De Gia- como, and Fabio Patrizi. LTLf/LDLf non-Markovian re- wards. In AAAI, 2018.","title":"LTLf/LDLf non-Markovian rewards","authors":"Ronen I Brafman, Giuseppe De, Giacomo, Fabio Patrizi","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Clark and Thollard, 2004] Alexander Clark and Franck Thollard. PAC-learnability of probabilistic deterministic finite state automata. J. Mach. Learn. Res., 2004.","title":"PAC-learnability of probabilistic deterministic finite state automata","authors":"Alexander Clark, Franck Thollard","year":"2004","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[De Giacomo et al., 2019] Giuseppe De Giacomo, Luca Ioc- chi, Marco Favorito, and Fabio Patrizi. Foundations for restraining bolts: Reinforcement learning with LTLf/LDLf restraining specifications. In ICAPS, 2019.","title":"Foundations for restraining bolts: Reinforcement learning with LTLf/LDLf restraining specifications","authors":"Giuseppe De, Giacomo, Luca Iocchi, Marco Favorito, Fabio Patrizi","year":"2019","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gaon and Brafman, 2020] Maor Gaon and Ronen I. Braf- man. Reinforcement learning with non-markovian re- wards. In AAAI, 2020.","title":"Reinforcement learning with non-markovian rewards","authors":"Maor Gaon, Ronen I Brafman","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hutter, 2009] Marcus Hutter. Feature reinforcement learn- ing: Part I. Unstructured MDPs. J. Artif. Gen. Intell., 2009.","title":"Feature reinforcement learning: Part I. Unstructured MDPs","authors":"Marcus Hutter","year":"2009","venue":"J. Artif. Gen. Intell","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hutter, 2016] Marcus Hutter. Extreme state aggregation be- yond markov decision processes. Theor. Comp. Sci., 2016.","title":"Extreme state aggregation beyond markov decision processes","authors":"Marcus Hutter","year":"2016","venue":"Theor. Comp. Sci","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Icarte et al., 2018] Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Anthony Valenzano, and Sheila A. McIlraith. Us- ing reward machines for high-level task specification and decomposition in reinforcement learning. In ICML, 2018.","title":"Using reward machines for high-level task specification and decomposition in reinforcement learning","authors":"Rodrigo Toro Icarte, Toryn Q Klassen, Richard Anthony Valenzano, Sheila A Mcilraith","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Icarte et al., 2019] Rodrigo Toro Icarte, Ethan Waldie, To- ryn Q. Klassen, Richard Anthony Valenzano, Margarita P. Castro, and Sheila A. McIlraith. Learning reward ma- chines for partially observable reinforcement learning. In NeurIPS, 2019.","title":"Learning reward machines for partially observable reinforcement learning","authors":"Rodrigo Toro Icarte, Ethan Waldie, Toryn Q Klassen, Richard Anthony Valenzano, Margarita P Castro, Sheila A Mcilraith","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kearns and Singh, 2002] Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time. Mach. Learn., 2002.","title":"Near-optimal reinforcement learning in polynomial time","authors":"J Michael, Kearns, P Satinder, Singh","year":"2002","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kearns and Vazirani, 1994] Michael J. Kearns and Umesh V. Vazirani. An Introduction to Computational Learning Theory. MIT Press, 1994.","title":"An Introduction to Computational Learning Theory","authors":"J Michael, Umesh V Kearns, Vazirani","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lattimore et al., 2013] Tor Lattimore, Marcus Hutter, and Peter Sunehag. The sample-complexity of general rein- forcement learning. In ICML, 2013.","title":"The sample-complexity of general reinforcement learning","authors":"Lattimore","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maillard et al., 2011] Odalric-Ambrym Maillard, Rémi Munos, and Daniil Ryabko. Selecting the state- representation in reinforcement learning. In NeurIPS, 2011.","title":"Selecting the staterepresentation in reinforcement learning","authors":"Maillard","year":"2011","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Majeed and Hutter, 2018] Sultan Javed Majeed and Marcus Hutter. On Q-learning convergence for non-markov deci- sion processes. In IJCAI, 2018.","title":"On Q-learning convergence for non-markov decision processes","authors":"Hutter ; Sultan Javed Majeed, Marcus Majeed, Hutter","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/353"},{"position":22,"raw":"[Nguyen et al., 2013] Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, and Ronald Ortner. Competing with an infinite set of models in reinforcement learning. In AISTATS, 2013.","title":"Competing with an infinite set of models in reinforcement learning","authors":"Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, Ronald Ortner","year":"2013","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Palmer and Goldberg, 2007] Nick Palmer and Paul W. Goldberg. PAC-learnability of probabilistic deterministic finite state automata in terms of variation distance. Theor. Comp. Sci., 2007.","title":"PAC-learnability of probabilistic deterministic finite state automata in terms of variation distance","authors":"Nick Palmer, Paul W Goldberg","year":"2007","venue":"Theor. Comp. Sci","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ron et al., 1998] Dana Ron, Yoram Singer, and Naftali Tishby. On the learnability and usage of acyclic proba- bilistic finite automata. J. Comp. Syst. Sci., 1998.","title":"On the learnability and usage of acyclic probabilistic finite automata","authors":"Dana Ron, Yoram Singer, Naftali Tishby","year":"1998","venue":"J. Comp. Syst. Sci","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ronca and De Giacomo, 2021] Alessandro Ronca and Giuseppe De Giacomo. Efficient PAC reinforcement learning in regular decision processes. In IJCAI, 2021.","title":"Efficient PAC reinforcement learning in regular decision processes","authors":"Alessandro Ronca, Giuseppe De, Giacomo","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/279"},{"position":27,"raw":"[Ronca et al., 2022] Alessandro Ronca, Gabriel Paludo Licks, and Giuseppe De Giacomo. Markov abstractions for PAC reinforcement learning in non-Markov decision processes. CoRR, abs/2205.01053, 2022.","title":"Markov abstractions for PAC reinforcement learning in non-Markov decision processes","authors":"Alessandro Ronca, Gabriel Paludo Licks, Giuseppe De, Giacomo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Adaptive computation and machine learning. MIT Press, 2018.","title":"Reinforcement learning: An introduction. Adaptive computation and machine learning","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Veness et al., 2011] Joel Veness, Kee Siong Ng, Marcus Hutter, William T. B. Uther, and David Silver. A Monte- Carlo AIXI approximation. J. Artif. Intell. Res., 2011.","title":"A Monte-Carlo AIXI approximation","authors":"Joel Veness, Kee Siong Ng, Marcus Hutter, T B William, David Uther, Silver","year":"2011","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2020] Zhe Xu, Ivan Gavran, Yousef Ahmad, Ru- pak Majumdar, Daniel Neider, Ufuk Topcu, and Bo Wu. Joint inference of reward machines and policies for rein- forcement learning. In ICAPS, 2020.","title":"Joint inference of reward machines and policies for reinforcement learning","authors":"Zhe Xu, Ivan Gavran, Yousef Ahmad, Rupak Majumdar, Daniel Neider, Ufuk Topcu, Bo Wu","year":"2020","venue":"ICAPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}