{"type":"ScholarlyArticle","title":"Regular Decision Processes: A Model for Non-Markovian Domains","authors":[{"name":"Ronen I. Brafman","slug":"ronen-i-brafman","given":"Ronen I.","family":"Brafman","affiliation":"Ben-Gurion University , Israel"},{"name":"Giuseppe De Giacomo","slug":"giuseppe-de-giacomo","given":"Giuseppe","family":"De Giacomo","affiliation":"Sapienza Università di Roma , Italy"}],"abstract":"We introduce and study Regular Decision Processes (RDPs), a new, compact, factored model for domains with non-Markovian dynamics and rewards. \r\nIn RDPs, transition and reward functions are specified using formulas in  linear dynamic logic over finite traces, a language with the expressive power of regular expressions. \r\nThis allows specifying complex dependence on the past using intuitive and compact formulas, and provides a model that generalizes  MDPs and k-order MDPs. \r\nRDPs can also approximate POMDPs without having to  postulate the existence of hidden variables, and, in principle, can be learned from observations only.","doi":"10.24963/ijcai.2019/766","url":"https://ijcai.org/proceedings/2019/766","pdf":"https://ijcai.org/proceedings/2019/0766.pdf","markdown":"https://ijcai.org/proceedings/2019/766.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/766","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Planning and Scheduling","track":"Main track","number":766,"pages":{"first":5516,"last":5522},"keywords":["Planning and Scheduling: POMDPs","Planning and Scheduling: Other approaches to planning","Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Albore et al., 2009] Alexandre Albore, Héctor Palacios, and Hector Geffner. A translation-based approach to contingent planning. In IJCAI'09, pages 1623-1628, 2009.","title":"A translation-based approach to contingent planning","authors":"Alexandre Albore, Héctor Palacios, Hector Geffner","year":"2009","venue":"IJCAI'09","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Angluin et al., 2013] Dana Angluin, James Aspnes, Sarah Eisenstat, and Aryeh Kontorovich. On the learnability of shuffle ideals. JMLR, 14:1513-1531, 2013.","title":"On the learnability of shuffle ideals","authors":"Dana Angluin, James Aspnes, Sarah Eisenstat, Aryeh Kontorovich","year":"2013","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bacchus et al., 1996] Fahiem Bacchus, Craig Boutilier, and Adam J. Grove. Rewarding behaviors. In AAAI, 1996.","title":"Rewarding behaviors","authors":"Bacchus","year":"1996","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boutilier et al., 1999] Craig Boutilier, Tom Dean, and Steve Hanks. Decision-theoretic planning: Structural assump- tions and computational leverage. JAIR, 11:1-94, 1999.","title":"Decision-theoretic planning: Structural assumptions and computational leverage","authors":"Craig Boutilier, Tom Dean, Steve Hanks","year":"1999","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brafman and De Giacomo, 2019] Ronen Brafman and Giuseppe De Giacomo. Planning for LTLf/LDLf goals in non-markovian fully observable nondeterministic domains. In IJCAI'19, 2019.","title":"Planning for LTLf/LDLf goals in non-markovian fully observable nondeterministic domains","authors":"De Brafman, Ronen Giacomo, Giuseppe Brafman, Giacomo De","year":"2019","venue":"IJCAI'19","doi":null,"url":null,"internal_url":"/proceedings/2019/222"},{"position":6,"raw":"[Brafman et al., 2018] Ronen Brafman, Giuseppe De Gia- como, and Fabio Patrizi. LTLf/LDLf non-Markovian re- wards. In AAAI, 2018.","title":"LTLf/LDLf non-Markovian rewards","authors":"Ronen Brafman, Giuseppe De, Giacomo, Fabio Patrizi","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Camacho et al., 2017] Alberto Camacho, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. Non-markovian rewards expressed in LTL: guiding search via reward shaping. In SOC, pages 159-160, 2017.","title":"Non-markovian rewards expressed in LTL: guiding search via reward shaping","authors":"Alberto Camacho, Oscar Chen, Scott Sanner, Sheila A Mcilraith","year":"2017","venue":"SOC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cassandra et al., 1994] Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in par- tially observable stochastic domains. In AAAI'94, volume 2, pages 1023-1028, 1994.","title":"Acting optimally in partially observable stochastic domains","authors":"Anthony R Cassandra, Leslie Pack Kaelbling, Michael L Littman","year":"1994","venue":"AAAI'94","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[De Giacomo and Rubin, 2018] Giuseppe De Giacomo and Sasha Rubin. Automata-theoretic foundations of FOND planning for LTLf and LDL f goals. In IJCAI'18, 2018.","title":"Automata-theoretic foundations of FOND planning for LTLf and LDL f goals","authors":"De Giacomo, Rubin, Giuseppe De, Giacomo, Sasha Rubin","year":"2018","venue":"IJCAI'18","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[De Giacomo and Vardi, 2013] Giuseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear dynamic logic on finite traces. In IJCAI'13, 2013.","title":"Linear temporal logic and linear dynamic logic on finite traces","authors":"De Giacomo, Giuseppe De Vardi, Moshe Y Giacomo, Vardi","year":"2013","venue":"IJCAI'13","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ding et al., 2014] Xu Chu Ding, Stephen L. Smith, Calin Belta, and Daniela Rus. Optimal control of markov decision processes with linear temporal logic constraints. IEEE Trans. Automat. Contr., 59(5):1244-1257, 2014.","title":"Optimal control of markov decision processes with linear temporal logic constraints","authors":"Chu Xu, Stephen L Ding, Calin Smith, Daniela Belta, Rus","year":"2014","venue":"IEEE Trans. Automat. Contr","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Favorito, 2018] Marco Favorito. Reinforcement learning for LTLf/LDLf goals: Theory and implementation. Master's thesis, DIAG, Sapienza Univ. Rome, 2018.","title":"Reinforcement learning for LTLf/LDLf goals: Theory and implementation","authors":"Marco Favorito","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fischer and Ladner, 1979] Michael J. Fischer and Richard E. Ladner. Propositional dynamic logic of regular programs. J. Com. Systems and Science, 18, 1979.","title":"Propositional dynamic logic of regular programs","authors":"J Michael, Richard E Fischer, Ladner","year":"1979","venue":"J. Com. Systems and Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fuggitti, 2018] Francesco Fuggitti. LTL and past LTL on finite traces for planning and declarative process mining. Master's thesis, DIAG, Sapienza Univ. Rome, 2018.","title":"LTL and past LTL on finite traces for planning and declarative process mining","authors":"Francesco Fuggitti","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Harel et al., 2000] David Harel, Dexter Kozen, and Jerzy Tiuryn. Dynamic Logic. MIT Press, 2000.","title":"Dynamic Logic","authors":"David Harel, Dexter Kozen, Jerzy Tiuryn","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kwiatkowska et al., 2011] Marta Kwiatkowska, Gethin Nor- man, and David Parker. PRISM 4.0: Verification of prob- abilistic real-time systems. In G. Gopalakrishnan and S. Qadeer, editors, ICGI'11, volume 6806 of LNCS, pages 585-591. Springer, 2011.","title":"PRISM 4.0: Verification of probabilistic real-time systems","authors":"Marta Kwiatkowska, Gethin Norman, David Parker","year":"2011","venue":"ICGI'11","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lambeau et al., 2008] Bernard Lambeau, Christophe Damas, and Pierre Dupont. State-merging DFA induc- tion algorithms with mandatory merge constraints. In Grammatical Inference: Algorithms and Applications, 9th International Colloquium, ICGI 2008, Saint-Malo, France, September 22-24, 2008, Proceedings, pages 139-153, 2008.","title":"State-merging DFA induction algorithms with mandatory merge constraints","authors":"Lambeau","year":"2008","venue":"Grammatical Inference: Algorithms and Applications, 9th International Colloquium, ICGI 2008","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lang et al., 1998] Kevin J. Lang, Barak A. Pearlmutter, and Rodney A. Price. Results of the abbadingo one DFA learn- ing competition and a new evidence-driven state merging algorithm. In ICGI'98, pages 1-12, 1998.","title":"Results of the abbadingo one DFA learning competition and a new evidence-driven state merging algorithm","authors":"Kevin J Lang, Barak A Pearlmutter, Rodney A Price","year":"1998","venue":"ICGI'98","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Littman et al., 1998] Michael L. Littman, Judy Goldsmith, and Martin Mundhenk. The computational complexity of probabilistic planning. Journal of AI Research, 9:1-36, 1998.","title":"The computational complexity of probabilistic planning","authors":"L Michael, Judy Littman, Martin Goldsmith, Mundhenk","year":"1998","venue":"Journal of AI Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Littman et al., 2017] Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Lee Isbell Jr., Min Wen, and James Mac- Glashan. Environment-independent task specifications via GLTL. CoRR, abs/1704.04341, 2017.","title":"Environment-independent task specifications via GLTL","authors":"L Michael, Ufuk Littman, Jie Topcu, Charles Lee Fu, Min Isbell, James Wen, Mac-Glashan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Madani et al., 2003] Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic planning and related stochastic optimization problems. Artif. Intell., 147(1-2):5-34, 2003.","title":"On the undecidability of probabilistic planning and related stochastic optimization problems","authors":"Omid Madani, Steve Hanks, Anne Condon","year":"2003","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos Papadimitriou and John N. Tsitsiklis. The complexity of markov decision processes. Math. Oper. Res., 12(3):441-450, 1987.","title":"The complexity of markov decision processes","authors":"Christos Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pnueli, 1977] Amir Pnueli. The temporal logic of programs. In FOCS, 1977.","title":"The temporal logic of programs","authors":"Amir Pnueli, Pnueli","year":"1977","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Puterman, 2005] Martin L. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. Wiley, 2005.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rudary and Singh, 2003] Matthew R. Rudary and Satinder P. Singh. A nonlinear predictive state representation. In NIPS'03, pages 855-862, 2003.","title":"A nonlinear predictive state representation","authors":"Matthew R Rudary, P Satinder, Singh","year":"2003","venue":"NIPS'03","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Singh et al., 2001] Satinder P. Singh, Michael L. Littman, and Richard S. Sutton. Predictive representations of state. In NIPS'01, pages 1555-1561. MIT Press, December 2001.","title":"Predictive representations of state","authors":"P Satinder, Michael L Singh, Richard S Littman, Sutton","year":"2001","venue":"NIPS'01","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Singh et al., 2003] Satinder P. Singh, Michael L. Littman, Nicholas K. Jong, David Pardoe, and Peter Stone. Learn- ing predictive state representations. In Proceedings of the Twentieth International Conference on Machine Learning, pages 712-719, August 2003.","title":"Learning predictive state representations","authors":"P Satinder, Michael L Singh, Nicholas K Littman, David Jong, Peter Pardoe, Stone","year":"2003","venue":"Proceedings of the Twentieth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tabakov and Vardi, 2005] Deian Tabakov and Moshe Y. Vardi. Experimental evaluation of classical automata con- structions. In LPAR, 2005.","title":"Experimental evaluation of classical automata constructions","authors":"Vardi ; ] Deian Tabakov, Moshe Y Tabakov, Vardi","year":"2005","venue":"LPAR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vardi, 2011] Moshe Y. Vardi. The rise and fall of linear time logic. In GandALF, 2011.","title":"The rise and fall of linear time logic","authors":"Moshe Y Vardi","year":"2011","venue":"GandALF","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2012] Zongzhang Zhang, Michael L. Littman, and Xiaoping Chen. Covering number as a complexity measure for POMDP planning and learning. In AAAI'12, 2012.","title":"Covering number as a complexity measure for POMDP planning and learning","authors":"Zhang","year":"2012","venue":"AAAI'12","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}