{"type":"ScholarlyArticle","title":"Approximate Value Iteration with Temporally Extended Actions (Extended Abstract)","authors":[{"name":"Timothy A. Mann","slug":"timothy-a-mann","given":"Timothy A.","family":"Mann","affiliation":"London , UK"},{"name":"Shie Mannor","slug":"shie-mannor","given":"Shie","family":"Mannor","affiliation":"The Technion , Haifa , Israel"},{"name":"Doina Precup","slug":"doina-precup","given":"Doina","family":"Precup","affiliation":"McGill University , Montreal , Canada"}],"abstract":"The options framework provides a concrete way to implement and reason about temporally extended actions. Existing literature has demonstrated the value of planning with options empirically, but there is a lack of theoretical analysis formalizing when planning with options is more efficient than planning with primitive actions. We provide a general analysis of the convergence rate of a popular Approximate Value Iteration (AVI) algorithm called Fitted Value Iteration (FVI) with options. Our analysis reveals that longer duration options and a pessimistic estimate of the value function both lead to faster convergence. Furthermore, options can improve convergence even when they are suboptimal and sparsely distributed throughout the state space. Next we consider generating useful options for planning based on a subset of landmark states. This suggests a new algorithm, Landmark-based AVI (LAVI), that represents the value function only at landmark states. We analyze OFVI and LAVI using the proposed landmark-based options and compare the two algorithms. Our theoretical and experimental results demonstrate that options can play an important role in AVI by decreasing approximation error and inducing fast convergence.","doi":"10.24963/ijcai.2017/717","url":"https://ijcai.org/proceedings/2017/717","pdf":"https://ijcai.org/proceedings/2017/0717.pdf","markdown":"https://ijcai.org/proceedings/2017/717.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/717","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Journal track","track":"Journal track","number":717,"pages":{"first":5035,"last":5039},"keywords":["Machine Learning: Reinforcement Learning","Constraints and Satisfiability: Dynamic Programming"],"videos":[],"references":[{"position":1,"raw":"[Bertsekas and Tsitsiklis, 1996] Dimitri P. Bertsekas and John Tsitsiklis. Neuro-dynamic programming. Athena Sci- entific, 1996.","title":"Neuro-dynamic programming","authors":"Dimitri P Bertsekas, John Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Littman et al., 1995] Michael L. Littman, Thomas L. Dean, and Leslie Pack Kaelbling. On the complexity of solving Markov decision problems. In Proceedings of the 11 th conference on Uncertainty in artificial intelligence, pages 394-402, 1995.","title":"On the complexity of solving Markov decision problems","authors":"L Michael, Thomas L Littman, Leslie Dean, Kaelbling Pack","year":"1995","venue":"Proceedings of the 11 th conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Mann and Mannor, 2014] Timothy A Mann and Shie Man- nor. Scaling up approximate value iteration with options: Better policies with fewer iterations. In Proceedings of the 31 st International Conference on Machine Learning, 2014.","title":"Scaling up approximate value iteration with options: Better policies with fewer iterations","authors":"Timothy A Mann, Shie Mannor","year":"2014","venue":"Proceedings of the 31 st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Mann et al., 2015] Timothy A. Mann, Shie Mannor, and Doina Precup. Approximate Value Iteration with Tempo- rally Extended Actions. Journal of Artificial Intelligence Research, 53:375-438, 2015.","title":"Approximate Value Iteration with Temporally Extended Actions","authors":"Timothy A Mann, Shie Mannor, Doina Precup","year":"2015","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Mann, 2014] Timothy A. Mann. Cyclic Inventory Manage- ment (CIM). https://code.google.com/p/rddlsim/source/ browse/trunk/files/rddl2/examples/cim.rddl2, 2014. Ac- cessed: 2015-06-29.","title":"Cyclic Inventory Management (CIM)","authors":"Timothy A Mann","year":null,"venue":null,"doi":null,"url":"https://code.google.com/p/rddlsim/source/browse/trunk/files/rddl2/examples/cim.rddl2,2014.Ac-cessed:2015-06-29","internal_url":null},{"position":6,"raw":"[McGovern and Barto, 2001] Amy McGovern and An- drew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In Proceed- ings of the 18 th International Conference on Machine Learning, pages 361 -368, San Fransisco, USA, 2001.","title":"Automatic discovery of subgoals in reinforcement learning using diverse density","authors":"Barto ; G Mcgovern, Barto","year":"2001","venue":"Proceedings of the 18 th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Munos and Szepesvári, 2008] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9:815-857, 2008.","title":"Finite-time bounds for fitted value iteration","authors":"Munos, Szepesvári, Rémi Munos, Csaba Szepesvári","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Puterman, 1994] Martin L Puterman. Markov Decision Pro- cesses -Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.","title":"Markov Decision Processes -Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Scarf, 1959] Herbert Scarf. The optimality of (s,S) policies in the dynamic inventory problem. Technical Report NR- 047-019, Office of Naval Research, April 1959.","title":"The optimality of (s,S) policies in the dynamic inventory problem","authors":"Herbert Scarf","year":"1959","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sethi and Cheng, 1997] Suresh P. Sethi and Feng Cheng. Optimality of (s,S) policies in inventory models with markovian demand. Operations Research, 45(6):931-939, 1997.","title":"Optimality of (s,S) policies in inventory models with markovian demand","authors":"P Suresh, Feng Sethi, Cheng","year":"1997","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Silver and Ciosek, 2012] David Silver and Kamil Ciosek. Compositional planning using optimal option models. In Proceedings of the 29th International Conference on Ma- chine Learning, Edinburgh, 2012.","title":"Compositional planning using optimal option models","authors":"David Silver, Kamil Ciosek","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Simsek and Barto, 2004] Özgür Simsek and Andrew G. Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the 21 st International Conference on Machine Learning, pages 95-102, New York, NY, USA, 2004. ACM.","title":"Using relative novelty to identify useful temporal abstractions in reinforcement learning","authors":"Özgür Simsek, Andrew G Barto","year":"2004","venue":"Proceedings of the 21 st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Stolle and Precup, 2002] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Abstrac- tion, Reformulation, and Approximation, pages 212-223. Springer, 2002.","title":"Learning options in reinforcement learning","authors":"Precup Stolle, Martin Stolle, Doina Precup","year":"2002","venue":"Abstraction, Reformulation, and Approximation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181-211, August 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wolfe and Barto, 2005] Alicia P. Wolfe and Andrew G. Barto. Identifying useful subgoals in reinforcement learn- ing by local graph partitioning. In Proceedings of the 22 nd International Conference on Machine Learning, pages 816-823, 2005.","title":"Identifying useful subgoals in reinforcement learning by local graph partitioning","authors":"Alicia P Wolfe, Andrew G Barto","year":"2005","venue":"Proceedings of the 22 nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yoon et al., 2007] Sung Wook Yoon, Alan Fern, and Robert Givan. FF-Replan: A Baseline for Probabilistic Planning. In Proceedings of the International Conference on Auto- mated Planning and Scheduling, volume 7, pages 352- 359, 2007.","title":"FF-Replan: A Baseline for Probabilistic Planning","authors":"Sung Wook Yoon, Alan Fern, Robert Givan","year":"2007","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}