{"type":"ScholarlyArticle","title":"Influence of State-Variable Constraints on Partially Observable Monte Carlo Planning","authors":[{"name":"Alberto Castellini","slug":"alberto-castellini","given":"Alberto","family":"Castellini","affiliation":"Computer Science Department , University of Verona , Italy"},{"name":"Georgios Chalkiadakis","slug":"georgios-chalkiadakis","given":"Georgios","family":"Chalkiadakis","affiliation":"School of Electrical and Computer Engineering , Technical University of Crete , Greece"},{"name":"Alessandro Farinelli","slug":"alessandro-farinelli","given":"Alessandro","family":"Farinelli","affiliation":"Computer Science Department , University of Verona , Italy"}],"abstract":"Online planning methods for partially observable Markov decision processes (POMDPs) have recently gained much interest. In this paper, we propose the introduction of prior knowledge in the form of (probabilistic) relationships among discrete state-variables, for online planning based on the well-known POMCP algorithm. In particular, we propose the use of hard constraint networks and probabilistic Markov random fields to formalize state-variable constraints and we extend the POMCP algorithm to take advantage of these constraints. Results on a case study based on Rocksample show that the usage of this knowledge provides significant improvements to the performance of the algorithm. The extent of this improvement depends on the amount of knowledge encoded in the constraints and reaches the 50% of the average discounted return in the most favorable cases that we analyzed.","doi":"10.24963/ijcai.2019/769","url":"https://ijcai.org/proceedings/2019/769","pdf":"https://ijcai.org/proceedings/2019/0769.pdf","markdown":"https://ijcai.org/proceedings/2019/769.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/769","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Planning and Scheduling","track":"Main track","number":769,"pages":{"first":5540,"last":5546},"keywords":["Planning and Scheduling: Robot Planning","Planning and Scheduling: POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Amato and Oliehoek, 2015] Christopher Amato and Frans A. Oliehoek. Scalable Planning and Learning for Multiagent POMDPs. In Proc. AAAI'15, pages 1995-2002. AAAI Press, 2015.","title":"Scalable Planning and Learning for Multiagent POMDPs","authors":"Christopher Amato, Frans A Oliehoek","year":"2015","venue":"Proc. AAAI'15","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bishop, 2006] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer-Verlag New York, 2006.","title":"Pattern Recognition and Machine Learning","authors":"Christopher M Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boutilier et al., 1999] Craig Boutilier, Thomas Dean, and Steve Hanks. Decision-theoretic Planning: Structural As- sumptions and Computational Leverage. JAIR, 11(1):1- 94, 1999.","title":"Decision-theoretic Planning: Structural Assumptions and Computational Leverage","authors":"Craig Boutilier, Thomas Dean, Steve Hanks","year":"1999","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Browne et al., 2012] Cameron Browne, Edward Powley, Daniel Whitehouse, Simon Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A Survey of Monte Carlo Tree Search Methods. IEEE Trans. Comp. Intell. AI Games, 4(1):1-43, 2012.","title":"A Survey of Monte Carlo Tree Search Methods","authors":"Cameron Browne, Edward Powley, Daniel Whitehouse, Simon Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, Simon Colton","year":"2012","venue":"IEEE Trans. Comp. Intell. AI Games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Castellini et al., 2018] Alberto Castellini, Giovanni Bel- trame, Manuele Bicego, Jason Blum, Matteo Denitto, and Alessandro Farinelli. Unsupervised activity recognition for autonomous water drones. In Proc. SAC 2018, pages 840-842. ACM, 2018.","title":"Unsupervised activity recognition for autonomous water drones","authors":"Alberto Castellini, Giovanni Beltrame, Manuele Bicego, Jason Blum, Matteo Denitto, Alessandro Farinelli","year":"2018","venue":"Proc. SAC 2018","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dechter, 2003] Rina Dechter. Constraint Processing. Mor- gan Kaufmann Pub. Inc., San Francisco, CA, USA, 2003.","title":"Constraint Processing","authors":"Rina Dechter","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duranti, 2015] Pierluigi Duranti. CatOne, Multitask Un- manned Surface Vessel for Hydro-Geological and Environ- ment Surveys, pages 647-652. Springer, 2015.","title":"Multitask Unmanned Surface Vessel for Hydro-Geological and Environment Surveys","authors":"Pierluigi Duranti, Catone","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farinelli et al., 2012] Alessandro Farinelli, Daniele Nardi, Roberta Pigliacampo, Mirco Rossi, and Giuseppe Settem- bre. Cooperative situation assessment in a maritime sce- nario. Int. J. Intelligent Systems, 27(5):477-501, 2012.","title":"Cooperative situation assessment in a maritime scenario","authors":"Alessandro Farinelli, Daniele Nardi, Roberta Pigliacampo, Mirco Rossi","year":"2012","venue":"Int. J. Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ferri et al., 2015] Gabriele Ferri, Alessandro Manzi, Francesco Fornai, Francesco Ciuchi, and Cecilia Laschi. The HydroNet ASV, a Small-Sized Autonomous Cata- maran for Real-Time Monitoring of Water Quality: From Design to Missions at Sea. IEEE J. Oceanic Engineering, 40(3):710-726, 2015.","title":"The HydroNet ASV, a Small-Sized Autonomous Catamaran for Real-Time Monitoring of Water Quality: From Design to Missions at Sea","authors":"Gabriele Ferri, Alessandro Manzi, Francesco Fornai, Francesco Ciuchi, Cecilia Laschi","year":"2015","venue":"IEEE J. Oceanic Engineering","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Griffith and Pradalier, 2015] Shane Griffith and Cedric Pradalier. A Spatially and Temporally Scalable Approach for Long-Term Lakeshore Monitoring. In Field and Service Robotics, volume 113 of Springer Tracts in Advanced Robotics, pages 3-16. Springer, 2015.","title":"A Spatially and Temporally Scalable Approach for Long-Term Lakeshore Monitoring","authors":"Shane Griffith, Cedric Pradalier","year":"2015","venue":"Field and Service Robotics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hamza and Ayanian, 2017] Ameer Hamza and Nora Aya- nian. Forecasting Battery State of Charge for Robot Mis- sions. In Proc. SAC 2017, pages 249-255. ACM, 2017.","title":"Forecasting Battery State of Charge for Robot Missions","authors":"Ayanian Hamza","year":"2017","venue":"Proc. SAC 2017","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hauskrecht, 2000] Milos Hauskrecht. Value-Function Ap- proximations for Partially Observable Markov Decision Processes. JAIR, 13:33-94, 2000.","title":"Milos Hauskrecht. Value-Function Approximations for Partially Observable Markov Decision Processes","authors":"Hauskrecht","year":"2000","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[J. D. Williams, 2005] S. Young J. D. Williams, P. Poupart. Factored partially observable Markov decision processes for dialogue management. In Proc. 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, IJCAI 2005, pages 76-82, 2005.","title":"Factored partially observable Markov decision processes for dialogue management","authors":"S Young, J D Williams, P Poupart","year":"2005","venue":"Proc. 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, IJCAI 2005","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaelbling et al., 1998] Laslie Kaelbling, Michael Littman, and Anthony Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence, 101(1-2):99-134, 1998.","title":"Planning and Acting in Partially Observable Stochastic Domains","authors":"Kaelbling","year":"1998","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit Based Monte-Carlo Planning. In Proc. ECML'06, pages 282-293, Berlin, Heidelberg, 2006. Springer-Verlag.","title":"Bandit Based Monte-Carlo Planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"Proc. ECML'06","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2018] Jongmin Lee, Geon-Hyeong Kim, Pascal Poupart, and Kee-Eung Kim. Monte-Carlo Tree Search for Constrained POMDPs. In NIPS'18, pages 1-17, 2018.","title":"Monte-Carlo Tree Search for Constrained POMDPs","authors":"Jongmin Lee, Geon-Hyeong Kim, Pascal Poupart, Kee-Eung Kim","year":"2018","venue":"NIPS'18","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeSage and Longoria, 2013] J. LeSage and R. Longoria. Characterization of Load Uncertainty in Unstructured Ter- rains and Applications to Battery Remaining Run-time Prediction. J. Field Robotics, 30(3):472-487, 2013.","title":"Characterization of Load Uncertainty in Unstructured Terrains and Applications to Battery Remaining Run-time Prediction","authors":"J Lesage, R Longoria","year":"2013","venue":"J. Field Robotics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McAllester and Singh, 1999] David McAllester and Satin- der Singh. Approximate Planning for Factored POMDPs using Belief State Simplification. In Proc. UAI '99, pages 409-416, 1999.","title":"Approximate Planning for Factored POMDPs using Belief State Simplification","authors":"David Mcallester, Satinder Singh","year":"1999","venue":"Proc. UAI '99","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Müller et al., 2012] Felix Müller, Christian Späth, Thomas Geier, and Susanne Biundo. Exploiting Expert Knowledge in Factored POMDPs. In Proc. ECAI'12, pages 606-611. IOS Press, 2012.","title":"Exploiting Expert Knowledge in Factored POMDPs","authors":"Müller","year":"2012","venue":"Proc. ECAI'12","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Murphy, 2012] Kevin Murphy. Machine Learning: A Prob- abilistic Perspective. The MIT Press, 2012.","title":"Machine Learning: A Probabilistic Perspective","authors":"Kevin Murphy","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos Papadimitriou and John Tsitsiklis. The Complexity of Markov Decision Processes. Math. Oper. Res., 12(3):441-450, August 1987.","title":"The Complexity of Markov Decision Processes","authors":"Christos Papadimitriou, John Tsitsiklis","year":"1987","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ross et al., 2008] Stephane Ross, Joelle Pineau, Sebastien Paquet, and Brahim Chaib-draa. Online Planning Algo- rithms for POMDPs. JAIR, 32:663-704, 2008.","title":"Online Planning Algorithms for POMDPs","authors":"Stephane Ross, Joelle Pineau, Sebastien Paquet, Brahim Chaib-Draa","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Russell and Norvig, 2003] Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson Edu- cation, 2 edition, 2003.","title":"Artificial Intelligence: A Modern Approach","authors":"Stuart Russell, Peter Norvig","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sadrpour et al., 2012] Amir Sadrpour, Jionghua Jin, and Galip Ulsoy. Mission energy prediction for unmanned ground vehicles. In ICRA, pages 2229-2234, 2012.","title":"Mission energy prediction for unmanned ground vehicles","authors":"Amir Sadrpour, Jionghua Jin, Galip Ulsoy","year":"2012","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo Planning in Large POMDPs. In NIPS'10, pages 2164-2172, 2010.","title":"Monte-Carlo Planning in Large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"NIPS'10","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Smith and Simmons, 2004] Trey Smith and Reid Simmons. Heuristic Search Value Iteration for POMDPs. In Proc. UAI '04, pages 520-527, 2004.","title":"Heuristic Search Value Iteration for POMDPs","authors":"Simmons Smith, Trey Smith, Reid Simmons","year":"2004","venue":"Proc. UAI '04","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutton and Barto, 2018] Richard Sutton and Andrew Barto. Reinforcement Learning, An introduction. MIT Press, Cambridge, MA, USA, 2nd edition, 2018.","title":"Reinforcement Learning, An introduction","authors":"Barto Sutton, Richard Sutton, Andrew Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thrun et al., 2005] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, 2005.","title":"Probabilistic Robotics (Intelligent Robotics and Autonomous Agents)","authors":"Sebastian Thrun, Wolfram Burgard, Dieter Fox","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Thrun, 2000] Sebastian Thrun. Monte Carlo POMDPs. In NIPS'00, pages 1064-1070. MIT Press, 2000.","title":"Monte Carlo POMDPs","authors":"Sebastian Thrun","year":"2000","venue":"NIPS'00","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}