{"type":"ScholarlyArticle","title":"On Q-learning Convergence for Non-Markov Decision Processes","authors":[{"name":"Sultan Javed Majeed","slug":"sultan-javed-majeed","given":"Sultan Javed","family":"Majeed","affiliation":"http :// www.sultan.pk ,"},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"Research School of Computer Science , Australian National University , Australia; http :// www.sultan.pk ,"}],"abstract":"Temporal-difference (TD) learning is an attractive, computationally efficient framework for model- free reinforcement learning. Q-learning is one of the most widely used TD learning technique that enables an agent to learn the optimal action-value function, i.e. Q-value function. Contrary to its widespread use, Q-learning has only been proven to converge on Markov Decision Processes (MDPs) and Q-uniform abstractions of finite-state MDPs. On the other hand, most real-world problems are inherently non-Markovian: the full true state of the environment is not revealed by recent observations. In this paper, we investigate the behavior of Q-learning when applied to non-MDP and non-ergodic domains which may have infinitely many underlying states. We prove that the convergence guarantee of Q-learning can be extended to a class of such non-MDP problems, in particular, to some non-stationary domains. We show that state-uniformity of the optimal Q-value function is a necessary and sufficient condition for Q-learning to converge even in the case of infinitely many internal states.","doi":"10.24963/ijcai.2018/353","url":"https://ijcai.org/proceedings/2018/353","pdf":"https://ijcai.org/proceedings/2018/0353.pdf","markdown":"https://ijcai.org/proceedings/2018/353.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/353","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":353,"pages":{"first":2546,"last":2552},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Online Learning","Planning and Scheduling: Markov Decisions Processes"],"videos":[],"references":[{"position":1,"raw":"[Baird, 1995] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Pro- ceedings 1995, pages 30-37. Elsevier, 1995.","title":"Residual algorithms: Reinforcement learning with function approximation","authors":"Leemon Baird","year":"1995","venue":"Machine Learning Proceedings 1995","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas and Tsitsiklis, 1995] Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming: an overview, volume 1. Belmont: Athena Scientific, 1995.","title":"Neuro-dynamic programming: an overview","authors":"P Dimitri, John N Bertsekas, Tsitsiklis","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cassandra et al., 1994] Anthony R Cassandra, Leslie Pack Kael- bling, and Michael L Littman. Acting optimaly in partially ob- servable stochastic domains. AAAI, pages 1023-1023, 1994.","title":"Acting optimaly in partially observable stochastic domains","authors":"Leslie Anthony R Cassandra, Pack Kaelbling, Michael L Littman","year":"1994","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cassandra, 1994] Anthony R Cassandra. Optimal Policies for Par- tially Observable Markov Decision Processes. Technical Report August, Report CS-94-14, Brown Univ, 1994.","title":"Optimal Policies for Partially Observable Markov Decision Processes","authors":"Cassandra Anthony","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hutter, 2005] Marcus Hutter. Universal Artificial Intelligence: Se- quential Decisions based on Algorithmic Probability. Springer, Berlin, 2005.","title":"Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability","authors":"Marcus Hutter","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hutter, 2016] Marcus Hutter. Extreme state aggregation beyond Markov decision processes. Theoretical Computer Science, 650:73-91, 2016.","title":"Extreme state aggregation beyond Markov decision processes","authors":"Marcus Hutter","year":"2016","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Leike, 2016] Jan Leike. Nonparametric General Reinforcement Learning. PhD thesis, Australian National University, 2016.","title":"Nonparametric General Reinforcement Learning","authors":"Jan Leike, Leike","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2006] Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. In ISAIM, 2006.","title":"Towards a unified theory of state abstraction for mdps","authors":"Lihong Li, Thomas J Walsh, Michael L Littman","year":"2006","venue":"ISAIM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin and Mitchell, 1992] L.J. Lin and T.M. Mitchell. Memory ap- proaches to reinforcement learning in non-Markovian domains. Artificial Intelligence, 8(7597):28, 1992.","title":"Memory approaches to reinforcement learning in non-Markovian domains","authors":"L J Lin, T M Mitchell","year":"1992","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Littman, 1994] Michael L Littman. Memoryless policies: Theoret- ical limitations and practical results. In From Animals to Animats 3: Proceedings of the third international conference on simula- tion of adaptive behavior, volume 3, page 238. MIT Press, 1994.","title":"Memoryless policies: Theoretical limitations and practical results","authors":"Michael L Littman","year":"1994","venue":"From Animals to Animats 3: Proceedings of the third international conference on simulation of adaptive behavior","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McCallum, 1995] R. Andrew McCallum. Instance-based utile dis- tinctions for reinforcement learning with hidden state. ICML, pages 387-395, 1995.","title":"Instance-based utile distinctions for reinforcement learning with hidden state","authors":"R, Andrew Mccallum","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Murphy, 2000] Kevin P Murphy. A survey of pomdp solution tech- niques. environment, 2:X3, 2000.","title":"A survey of pomdp solution techniques","authors":"Kevin P Murphy","year":"2000","venue":"environment","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pendrith and McGarity, 1998] Mark D Pendrith and Michael J Mc- Garity. An analysis of direct reinforcement learning in non- Markovian domains. ICML, pages 421-429, 1998.","title":"An analysis of direct reinforcement learning in non-Markovian domains","authors":"D Mark, Michael J Mc-Garity Pendrith","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Perkins and Pendrith, 2002] Theodore J Perkins and Mark D Pen- drith. On the existence of fixed points for Q-learning and Sarsa in partially observable domains. ICML, pages 490-497, 2002.","title":"On the existence of fixed points for Q-learning and Sarsa in partially observable domains","authors":"J Theodore, Mark D Perkins, Pendrith","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Singh et al., 1994] S Singh, T Jaakkola, and M Jordan. Learning Without State-Estimation in Partially Observable Markov Deci- sion Processes. ICML, 31(0):37, 1994.","title":"Learning Without State-Estimation in Partially Observable Markov Decision Processes","authors":"T Singh, M Jaakkola, Jordan","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton and Barto, 1984] R S Sutton and A G Barto. Temporal credit assignment in reinforcement learning. Computer Science, page 210, 1984.","title":"Temporal credit assignment in reinforcement learning","authors":"R S Sutton, A G Barto","year":"1984","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton, 1988] Richard S Sutton. Learning to Predict by the Method of Temporal Differences. Machine Learning, 3(1):9-44, 1988.","title":"Learning to Predict by the Method of Temporal Differences","authors":"Richard S Sutton","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Thrun et al., 2005] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Partially Observable Markov Decision Processes. Proba- bilistic Robotics, page 2005, 2005.","title":"Partially Observable Markov Decision Processes","authors":"Sebastian Thrun, Wolfram Burgard, Dieter Fox","year":"2005","venue":"Probabilistic Robotics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tsitsiklis, 1994] John N. Tsitsiklis. Asynchronous Stochastic Ap- proximation and Q-Learning. Machine Learning, 16(3):185-202, 1994.","title":"Asynchronous Stochastic Approximation and Q-Learning","authors":"John N Tsitsiklis","year":"1994","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Watkins and Dayan, 1992] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279-292, 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Whitehead and Lin, 1995] Steven D. Whitehead and Long Ji Lin. Reinforcement learning of non-Markov decision processes. Arti- ficial Intelligence, 73(1-2):271-306, 1995.","title":"Reinforcement learning of non-Markov decision processes","authors":"Steven D Whitehead, Long Ji, Lin","year":"1995","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}