{"type":"ScholarlyArticle","title":"Perception-Aware Point-Based Value Iteration for Partially Observable Markov Decision Processes","authors":[{"name":"Mahsa Ghasemi","slug":"mahsa-ghasemi","given":"Mahsa","family":"Ghasemi","affiliation":"University of Texas at Austin"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"University of Texas at Austin"}],"abstract":"In conventional partially observable Markov decision processes, the observations that the agent receives originate from fixed known distributions. However, in a variety of real-world scenarios, the agent has an active role in its perception by selecting which observations to receive. We avoid combinatorial expansion of the action space from integration of planning and perception decisions, through a greedy strategy for observation selection that minimizes an information-theoretic measure of the state uncertainty. We develop a novel point-based value iteration algorithm that incorporates this greedy strategy to pick perception actions for each sampled belief point in each iteration. As a result, not only the solver requires less belief points to approximate the reachable subspace of the belief simplex, but it also requires less computation per iteration. Further, we prove that the proposed algorithm achieves a near-optimal guarantee on value function with respect to an optimal perception strategy, and demonstrate its performance empirically.","doi":"10.24963/ijcai.2019/329","url":"https://ijcai.org/proceedings/2019/329","pdf":"https://ijcai.org/proceedings/2019/0329.pdf","markdown":"https://ijcai.org/proceedings/2019/329.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/329","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":329,"pages":{"first":2371,"last":2377},"keywords":["Robotics: Sensor Networks","Machine Learning: Reinforcement Learning","Planning and Scheduling: POMDPs","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Araya et al., 2010] Mauricio Araya, Olivier Buffet, Vincent Thomas, and Franc ¸cois Charpillet. A POMDP extension with belief-dependent rewards. In Advances in Neural In- formation Processing Systems, pages 64-72, 2010.","title":"A POMDP extension with belief-dependent rewards","authors":"Mauricio Araya, Olivier Buffet, Vincent Thomas, Franc ¸cois, Charpillet","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Åström, 1965] Karl J Åström. Optimal control of Markov processes with incomplete state information. J. of Mathe- matical Analysis and Applications, 10(1):174-205, 1965.","title":"Optimal control of Markov processes with incomplete state information","authors":"Karl J Åström","year":"1965","venue":"J. of Mathematical Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellman, 1957] Richard Bellman. A Markovian decision process. J. of Mathematics and Mechanics, pages 679- 684, 1957.","title":"A Markovian decision process","authors":"Richard Bellman","year":"1957","venue":"J. of Mathematics and Mechanics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng, 1988] Hsien-Te Cheng. Algorithms for partially ob- servable Markov decision processes. PhD thesis, Univer- sity of British Columbia, 1988.","title":"Algorithms for partially observable Markov decision processes","authors":"Hsien-Te Cheng","year":"1988","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cover and Thomas, 2012] Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012. [Devroye, 1986] Luc Devroye. Sample-based non-uniform random variate generation. In Conf. on Winter Simulation, pages 260-265. ACM, 1986.","title":"Sample-based non-uniform random variate generation","authors":"M Thomas, Joy A Cover, Thomas","year":"1986","venue":"Conf. on Winter Simulation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghasemi and Topcu, 2019] Mahsa Ghasemi and Ufuk Topcu. Perception-aware point-based value iteration for partially observable Markov decision processes. arXiv preprint, 2019.","title":"Perception-aware point-based value iteration for partially observable Markov decision processes","authors":"Topcu ; ] Mahsa Ghasemi, Ufuk Ghasemi, Topcu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hashemi et al., 2018] Abolfazl Hashemi, Mahsa Ghasemi, Haris Vikalo, and Ufuk Topcu. A randomized greedy al- gorithm for near-optimal sensor scheduling in large-scale sensor networks. In American Control Conf., pages 1027- 1032. IEEE, 2018.","title":"A randomized greedy algorithm for near-optimal sensor scheduling in large-scale sensor networks","authors":"Abolfazl Hashemi, Mahsa Ghasemi, Haris Vikalo, Ufuk Topcu","year":"2018","venue":"American Control Conf","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ko et al., 1995] Chun-Wa Ko, Jon Lee, and Maurice Queyranne. An exact algorithm for maximum entropy sampling. Operations Research, 43(4):684-691, 1995.","title":"An exact algorithm for maximum entropy sampling","authors":"Chun-Wa Ko, Jon Lee, Maurice Queyranne","year":"1995","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krause and Golovin, 2014] Andreas Krause and Daniel Golovin. Submodular function maximization. In Tractability: Practical Approaches to Hard Problems, pages 71-104. Cambridge University Press, 2014.","title":"Submodular function maximization","authors":"Golovin Krause, Andreas Krause, Daniel Golovin","year":"2014","venue":"Tractability: Practical Approaches to Hard Problems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kreucher et al., 2005] Chris Kreucher, Keith Kastella, and Alfred O Hero Iii. Sensor management using an active sensing approach. Signal Process., 85(3):607-624, 2005.","title":"Sensor management using an active sensing approach","authors":"Chris Kreucher, Keith Kastella, Alfred O Hero Iii","year":"2005","venue":"Signal Process","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kurniawati et al., 2008] Hanna Kurniawati, David Hsu, and Wee Sun Lee. SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Robotics: Science and systems, 2008.","title":"SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces","authors":"Hanna Kurniawati, David Hsu, Wee Sun, Lee","year":"2008","venue":"Robotics: Science and systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lovejoy, 1991] William S Lovejoy. A survey of algorithmic methods for partially observed Markov decision processes. Annals of Operations Research, 28(1):47-65, 1991.","title":"A survey of algorithmic methods for partially observed Markov decision processes","authors":"S William, Lovejoy","year":"1991","venue":"Annals of Operations Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Natarajan et al., 2015] Prabhu Natarajan, Pradeep K Atrey, and Mohan Kankanhalli. Multi-camera coordination and control in surveillance systems: A survey. ACM Trans. on Multimedia Computing, Communications, and Appli- cations, 11(4):57, 2015.","title":"Multi-camera coordination and control in surveillance systems: A survey","authors":"Prabhu Natarajan, Mohan Pradeep K Atrey, Kankanhalli","year":"2015","venue":"ACM Trans. on Multimedia Computing, Communications, and Applications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nemhauser et al., 1978] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approxima- tions for maximizing submodular set functions-I. Math- ematical Programming, 14(1):265-294, 1978.","title":"An analysis of approximations for maximizing submodular set functions-I","authors":"Laurence A George L Nemhauser, Marshall L Wolsey, Fisher","year":"1978","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos H Papadim- itriou and John N Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):441-450, 1987.","title":"The complexity of Markov decision processes","authors":"Christos H Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pineau et al., 2006] Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun. Anytime point-based approximations for large POMDPs. J. of Artificial Intelligence Research, 27:335-380, 2006.","title":"Anytime point-based approximations for large POMDPs","authors":"Joelle Pineau, Geoffrey Gordon, Sebastian Thrun","year":"2006","venue":"J. of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qian et al., 2017] Chao Qian, Jing-Cheng Shi, Yang Yu, and Ke Tang. On subset selection with general cost con- straints. In Proc. Int. Joint Conf. on Artificial Intelligence, volume 17, pages 2613-2619, 2017.","title":"On subset selection with general cost constraints","authors":"Jing-Cheng Chao Qian, Yang Shi, Ke Yu, Tang","year":"2017","venue":"Proc. Int. Joint Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/364"},{"position":18,"raw":"[Satsangi et al., 2018] Yash Satsangi, Shimon Whiteson, Frans A Oliehoek, and Matthijs TJ Spaan. Exploiting sub- modular value functions for scaling up active perception. Autonomous Robots, 42(2):209-233, 2018.","title":"Exploiting submodular value functions for scaling up active perception","authors":"Yash Satsangi, Shimon Whiteson, Frans A Oliehoek, Matthijs, Spaan","year":"2018","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shamaiah et al., 2010] Manohar Shamaiah, Siddhartha Banerjee, and Haris Vikalo. Greedy sensor selection: Leveraging submodularity. In Proc. IEEE Conf. on Decision and Control, pages 2572-2577. IEEE, 2010.","title":"Greedy sensor selection: Leveraging submodularity","authors":"Manohar Shamaiah, Siddhartha Banerjee, Haris Vikalo","year":"2010","venue":"Proc. IEEE Conf. on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shani et al., 2013] Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point-based POMDP solvers. Au- tonomous Agents and Multi-Agent Systems, 27(1):1-51, 2013.","title":"A survey of point-based POMDP solvers","authors":"Guy Shani, Joelle Pineau, Robert Kaplow","year":"2013","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Smallwood and Sondik, 1973] Richard D Smallwood and Edward J Sondik. The optimal control of partially observ- able Markov processes over a finite horizon. Operations Research, 21(5):1071-1088, 1973.","title":"The optimal control of partially observable Markov processes over a finite horizon","authors":"D Richard, Edward J Smallwood, Sondik","year":"1973","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Smith and Simmons, 2012] Trey Smith and Reid Simmons. Point-based POMDP algorithms: Improved analysis and implementation. arXiv preprint arXiv:1207.1412, 2012.","title":"Trey Smith and Reid Simmons. Point-based POMDP algorithms: Improved analysis and implementation","authors":"Simmons Smith","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sondik, 1978] Edward J Sondik. The optimal control of par- tially observable Markov processes over the infinite hori- zon: Discounted costs. Operations Research, 26(2):282- 304, 1978.","title":"The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs","authors":"J Edward, Sondik","year":"1978","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Spaan and Lima, 2009] Matthijs TJ Spaan and Pedro U Lima. A decision-theoretic approach to dynamic sensor selection in camera networks. In Proc. Int. Conf. on Auto- mated Planning and Scheduling, pages 279--304, 2009.","title":"A decision-theoretic approach to dynamic sensor selection in camera networks","authors":"T J Matthijs, Pedro U Spaan, Lima","year":"2009","venue":"Proc. Int. Conf. on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Spaan and Vlassis, 2005] Matthijs TJ Spaan and Nikos Vlassis. Perseus: Randomized point-based value itera- tion for POMDPs. J. of Artificial Intelligence Research, 24:195-220, 2005.","title":"Perseus: Randomized point-based value iteration for POMDPs","authors":"Vlassis ; ] Spaan, T J Matthijs, Nikos Spaan, Vlassis","year":"2005","venue":"J. of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Spaan et al., 2015] Matthijs TJ Spaan, Tiago S Veiga, and Pedro U Lima. Decision-theoretic planning under uncer- tainty with information rewards for active cooperative per- ception. Autonomous Agents and Multi-Agent Systems, 29(6):1157-1185, 2015.","title":"Decision-theoretic planning under uncertainty with information rewards for active cooperative perception","authors":"Tiago S Matthijs Tj Spaan, Pedro U Veiga, Lima","year":"2015","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Williamson and Shmoys, 2011] David P Williamson and David B Shmoys. The design of approximation algo- rithms. Cambridge University Press, 2011.","title":"The design of approximation algorithms","authors":"P David, David B Williamson, Shmoys","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Zhang, 2001] Nevin Lianwen Zhang and Wei- hong Zhang. Speeding up the convergence of value itera- tion in partially observable Markov decision processes. J. of Artificial Intelligence Research, 14:29-51, 2001.","title":"Speeding up the convergence of value iteration in partially observable Markov decision processes","authors":"Nevin Lianwen, Zhang, Weihong Zhang","year":"2001","venue":"J. of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}