{"type":"ScholarlyArticle","title":"Approximating Optimal Policies for Partially Observable Stochastic Domains","authors":[{"name":"Ronald Parr","slug":"ronald-parr","given":"Ronald","family":"Parr","affiliation":"Computer Science Division University of California Berkeley , CA 94720 , USA"},{"name":"Stuart J. Russell","slug":"stuart-j-russell","given":"Stuart J.","family":"Russell","affiliation":"Computer Science Division University of California Berkeley , CA 94720 , USA"}],"abstract":"The problem of making optimal decisions in uncertain conditions is central to Artificial Intelligence. If the state of the world is known at all times, the world can be modeled as a Markov Decision Process (MDP). MDPs have been studied extensively and many methods are known for determining optimal courses of action, or policies. The more realistic case where state information is only partially observable, Partially Observable Markov Decision Processes (POMDPs), have received much less attention. The best exact algorithms for these problems can be very inefficient in both space and time. We introduce Smooth Partially Observable Value Approximation (SPOVA), a new approximation method that can quickly yield good approximations which can improve over time. This method can be combined with reinforcement learning methods, a combination that was very effective in our test cases.","doi":null,"url":"https://ijcai.org/proceedings/1995/141","pdf":"https://ijcai.org/proceedings/1995/0141.pdf","markdown":"https://ijcai.org/proceedings/1995/141.md","bibtex":"https://ijcai.org/proceedings/1995/bibtex/141","volume":{"slug":"1995","title":"Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1995","url":"https://ijcai.org/proceedings/1995/","year":1995,"location":"Montréal, Québec, Canada","dates":"","editors":"Chris S. Mellish","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Learning","track":"Learning","number":141,"pages":{"first":1088,"last":1094},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Bellman, 1957] Richard Ernest Bellman. Dynamic Pro- gramming. Princeton University Press, Princeton, New Jersey, 1957.","title":"Dynamic Programming","authors":"Richard Ernest, Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cassandra et al., 1994] A. R. Cassandra, L. P. Kaelbling, and M. L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94), pages 1094 LEARNING I 023-1028, Seattle, Washington, August 1994. AAAI Press.","title":"Acting optimally in partially observable stochastic domains","authors":"A R Cassandra, L P Kaelbling, M L Littman","year":"1094","venue":"Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI-94)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chrisman, 1992] Lonnie Chrisman. Reinforcement learning with perceptual aliasing: The perceptual distinctions ap- proach. In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI-92), pages 183-188, San Jose, California, July 1992. AAAI Press. Jaakola et al., in press] Tommi Jaakola, Satinder P. Singh, and Michael I. Jordan. Reinforcement learning algorithm for partially observable Markov decision problems. In Neural Information Processing Systems 7, to appear, in press.","title":"Reinforcement learning with perceptual aliasing: The perceptual distinctions approach","authors":"Lonnie Chrisman, ; Jaakola","year":"1992","venue":"Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI-92)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Lin and Mitchell, 1992] Long-JiLin and Tom M. Mitchell. Memory approaches to reinforcement learning in non- Markovian domains. Technical report, Computer Sci- ence Department, Carnegie-Mellon University, Pittsburgh, Pennsy 1 vania, 1992.","title":"Long-JiLin and Tom M. Mitchell. Memory approaches to reinforcement learning in non-Markovian domains","authors":"Mitchell Lin","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Littman et al., in press]","title":null,"authors":"Littman","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Michael L. Littman, Anthony R. Cassandra, and Leslie P. Kaelbling. Learning policies for partially observable environments: Scaling up. In Proceed- ings of the Twelfth International Conference on Machine Learning, to appear, in press.","title":"Learning policies for partially observable environments: Scaling up","authors":"L Michael, Anthony R Littman, Leslie P Cassandra, Kaelbling","year":null,"venue":"Proceedings of the Twelfth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Littman, 1994] Michael L. Littman. The witness algorithm: Solving partially observable Markov decision processes. Technical report, Computer Science Department, Brown University, Providence, Rhode Island, 1994.","title":"The witness algorithm: Solving partially observable Markov decision processes","authors":"L Michael, Littman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lovejoy, 1991] W. S. Lovejoy. A survey of algorithmic methods for partially observed Markov decision processes. Annals of Operations Research, 28(1--4):47--66, April 1991.","title":"A survey of algorithmic methods for partially observed Markov decision processes","authors":"W S Lovejoy","year":"1991","venue":"Annals of Operations Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Martinetz et al., 1993] Thomas M. Martinetz, Stanislav G. Berkovich, and Klaus J. Schulten. \"neural-gas\" network for vector quantization and its application to time series prediction. IEEE Transactions on Neural Networks, SSC- 4:558-569, 1993.","title":"neural-gas\" network for vector quantization and its application to time series prediction","authors":"M Thomas, Martinetz, G Stanislav, Klaus J Berkovich, Schulten","year":"1993","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[McCallum, 1993] Andrew R. McCallum. Overcoming in- complete perception with utile distinction memory. In Proceedings of the Tenth International Conference on Ma- chine Learning, pages 190-196, Amherst, Massachusetts, July 1993.","title":"Overcoming incomplete perception with utile distinction memory","authors":"Andrew R Mccallum","year":"1993","venue":"Proceedings of the Tenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Morgan Kaufmann.","title":null,"authors":"Morgan Kaufmann","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Russell and Norvig, 1994] Stuart J. Russell and Peter Norvig. Artificial Intelligence: A Modem Approach. Prentice-Hall, Englewood Cliffs, New Jersey, 1994.","title":"Artificial Intelligence: A Modem Approach","authors":"Stuart J Russell, Peter Norvig","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Russell et al., 1994] Stuart J. Russell, John Binder, and Daphne Koller. Adaptive probabilistic networks. Techni- cal Report UCB/CSD-94-824, Computer Science Division, University of California at Berkeley, July 24 1994.","title":"Adaptive probabilistic networks","authors":"Stuart J Russell, John Binder, Daphne Koller","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sondik, 1971] E. J. Sondik. The Optimal Control of Par- tially Observable Markov Decision Processes. PhD thesis, Stanford University, Stanford, California, 1971.","title":"The Optimal Control of Partially Observable Markov Decision Processes","authors":"E J Sondik","year":"1971","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton, 1988] R. S. Sutton. Learning to predict by the meth- ods of temporal differences. Machine Learning, 3:9--44, August 1988.","title":"Learning to predict by the methods of temporal differences","authors":"R S Sutton","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Watkins, 1989] C. J. Watkins. Models of Delayed Reinforce- ment Learning. PhD thesis, Psychology Department, Cam- bridge University, Cambridge, United Kingdom, 1989.","title":"Models of Delayed Reinforcement Learning","authors":"C J Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}