{"type":"ScholarlyArticle","title":"Factorized Asymptotic Bayesian Policy Search for POMDPs","authors":[{"name":"Masaaki Imaizumi","slug":"masaaki-imaizumi","given":"Masaaki","family":"Imaizumi","affiliation":"Institute of Statistical Mathematics"},{"name":"Ryohei Fujimaki","slug":"ryohei-fujimaki","given":"Ryohei","family":"Fujimaki","affiliation":"NEC Corporation"}],"abstract":"This paper proposes a novel direct policy search (DPS) method with model selection for partially observed Markov decision processes (POMDPs). DPSs have been standard for learning POMDPs due to their computational efficiency and natural ability to maximize total rewards. An important open challenge for the best use of DPS methods is model selection, i.e., determination of the proper dimensionality of hidden states and complexity of policy functions, to mitigate overfitting in highly-flexible model representations of POMDPs. This paper bridges Bayesian inference and reward maximization and derives marginalized weighted log-likelihood~(MWL) for POMDPs which takes both advantages of Bayesian model selection and DPS. Then we propose factorized asymptotic Bayesian policy search (FABPS) to explore the model and the policy which maximizes MWL by expanding recently-developed factorized asymptotic Bayesian inference. Experimental results show that FABPS outperforms state-of-the-art model selection methods for POMDPs, with respect both to model selection and to expected total rewards.","doi":"10.24963/ijcai.2017/607","url":"https://ijcai.org/proceedings/2017/607","pdf":"https://ijcai.org/proceedings/2017/0607.pdf","markdown":"https://ijcai.org/proceedings/2017/607.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/607","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Planning and Scheduling","track":"Main track","number":607,"pages":{"first":4346,"last":4352},"keywords":["Planning and Scheduling: POMDPs","Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence"],"videos":[],"references":[{"position":1,"raw":"[Abbeel et al., 2010] Pieter Abbeel, Adam Coates, and An- drew Y Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 2010.","title":"Autonomous helicopter aerobatics through apprenticeship learning","authors":"Pieter Abbeel, Adam Coates, Andrew Y Ng","year":"2010","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Åström, 1965] Karl J Åström. Optimal control of markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174-205, 1965.","title":"Optimal control of markov processes with incomplete state information","authors":"Karl J Åström","year":"1965","venue":"Journal of Mathematical Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Busa-Fekete et al., 2014] Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine Learning, 97(3):327-351, 2014.","title":"Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm","authors":"Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, Eyke Hüllermeier","year":"2014","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2009] Chenghui Cai, Xuejun Liao, and Lawrence Carin. Learning to explore and exploit in pomdps. In Advances in Neural Information Processing Systems, pages 198-206, 2009.","title":"Learning to explore and exploit in pomdps","authors":"Cai","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Capitan et al., 2013] Jesus Capitan, Matthijs TJ Spaan, Luis Merino, and Anibal Ollero. Decentralized multi-robot coop- eration with auctioned pomdps. The International Journal of Robotics Research, 32(6):650-671, 2013.","title":"Decentralized multi-robot cooperation with auctioned pomdps","authors":"Capitan","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Debbi et al., 2013] Hichem Debbi, Mustapha Bourahla, and Aimad Debbi. Medical treatment analysis using probabilis- tic model checking. International Journal of Biomedical Engineering and Technology, 12(4):346-359, 2013.","title":"Medical treatment analysis using probabilistic model checking","authors":"Hichem Debbi, Mustapha Bourahla, Aimad Debbi","year":"2013","venue":"International Journal of Biomedical Engineering and Technology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deisenroth and Rasmussen, 2011] Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th Inter- national Conference on Machine Learning, pages 465-472, 2011.","title":"Pilco: A model-based and data-efficient approach to policy search","authors":"Rasmussen Deisenroth, Marc Deisenroth, Carl E Rasmussen","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Doshi-Velez et al., 2015] Finale Doshi-Velez, David Pfau, Frank Wood, and Nicholas Roy. Bayesian nonparametric methods for partially-observable reinforcement learning. IEEE transactions on pattern analysis and machine intelli- gence, 37(2):394-407, 2015.","title":"Bayesian nonparametric methods for partially-observable reinforcement learning","authors":"Doshi-Velez","year":"2015","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Eto et al., 2014] Riki Eto, Ryohei Fujimaki, Satoshi Mori- naga, and Hiroshi Tamano. Fully-automatic bayesian piece- wise sparse linear models. In Proceedings of the 17th Inter- national Conference on Artificial Intelligence and Statistics, pages 238-246, 2014.","title":"Fully-automatic bayesian piecewise sparse linear models","authors":"Riki Eto, Ryohei Fujimaki, Satoshi Morinaga, Hiroshi Tamano","year":"2014","venue":"Proceedings of the 17th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fujimaki and Hayashi, 2012] Ryohei Fujimaki and Kohei Hayashi. Factorized asymptotic bayesian hidden markov models. In Proceedings of the 29th International Confer- ence on Machine Learning, pages 799-806, 2012.","title":"Factorized asymptotic bayesian hidden markov models","authors":"Hayashi Fujimaki, Ryohei Fujimaki, Kohei Hayashi","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fujimaki and Morinaga, 2012] Ryohei Fujimaki and Satoshi Morinaga. Factorized asymptotic bayesian inference for mixture modeling. In International Conference on Artificial Intelligence and Statistics, pages 400-408, 2012.","title":"Factorized asymptotic bayesian inference for mixture modeling","authors":"Morinaga Fujimaki, Ryohei Fujimaki, Satoshi Morinaga","year":"2012","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hayashi and Fujimaki, 2013] Kohei Hayashi and Ryohei Fu- jimaki. Factorized asymptotic bayesian inference for latent feature models. In Advances in Neural Information Pro- cessing Systems, pages 1214-1222, 2013.","title":"Factorized asymptotic bayesian inference for latent feature models","authors":"Fujimaki Hayashi, Kohei Hayashi, Ryohei Fujimaki","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hayashi et al., 2015] Kohei Hayashi, Shinichi Maeda, and Ryohei Fujimaki. Rebuilding factorized information cri- terion: Asymptotically accurate marginal likelihood. In Proceedings of the 32th International Conference on Ma- chine Learning, page 1358-1366, 2015.","title":"Rebuilding factorized information criterion: Asymptotically accurate marginal likelihood","authors":"Kohei Hayashi, Shinichi Maeda, Ryohei Fujimaki","year":"2015","venue":"Proceedings of the 32th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Howard, 1960] Ronald A Howard. Dynamic programming and Markov processes. MIT Press, 1960.","title":"Dynamic programming and Markov processes","authors":"A Ronald, Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Irissappane et al., 2014] Athirai A Irissappane, Frans A Oliehoek, and Jie Zhang. A pomdp based approach to opti- mally select sellers in electronic marketplaces. In Proceed- ings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, pages 1329-1336, 2014.","title":"A pomdp based approach to optimally select sellers in electronic marketplaces","authors":"Irissappane","year":"2014","venue":"Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Konda and Tsitsiklis, 2000] Vijay R Konda and John N Tsit- siklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, pages 1008-1014, 2000.","title":"Actor-critic algorithms","authors":"R Vijay, John N Konda, Tsitsiklis","year":"2000","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Konishi and Kitagawa, 2008] Sadanori Konishi and Gen- shiro Kitagawa. Information criteria and statistical model- ing. Springer Science & Business Media, 2008.","title":"Sadanori Konishi and Genshiro Kitagawa. Information criteria and statistical modeling","authors":"Kitagawa Konishi","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Levine and Koltun, 2013] Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimiza- tion. In Advances in Neural Information Processing Systems, pages 207-215, 2013.","title":"Variational policy search via trajectory optimization","authors":"Koltun Levine, Sergey Levine, Vladlen Koltun","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2015] Chunchen Liu, Lu Feng, Ryohei Fujimaki, and Yusuke Muraoka. Scalable model selection for large- scale factorial relational models. In Proceedings of The 32nd International Conference on Machine Learning, pages 1227-1235, 2015.","title":"Scalable model selection for largescale factorial relational models","authors":"Liu","year":"2015","venue":"Proceedings of The 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peters and Schaal, 2007] Jan Peters and Stefan Schaal. Re- inforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th Inter- national Conference on Machine Learning, pages 745-750. ACM, 2007.","title":"Reinforcement learning by reward-weighted regression for operational space control","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2007","venue":"Proceedings of the 24th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schwarz, 1978] Gideon Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6(2):461-464, 1978.","title":"Estimating the dimension of a model","authors":"Schwarz","year":"1978","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Spaan et al., 2010] Matthijs TJ Spaan, Tiago S Veiga, and Pedro U Lima. Active cooperative perception in network robot systems using pomdps. In 2010 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, pages 4800-4805, 2010.","title":"Active cooperative perception in network robot systems using pomdps","authors":"Tiago S Matthijs Tj Spaan, Pedro U Veiga, Lima","year":"2010","venue":"2010 IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ueno et al., 2012] Tsuyoshi Ueno, Kohei Hayashi, Takashi Washio, and Yoshinobu Kawahara. Weighted likelihood policy search with model selection. In Advances in Neural Information Processing Systems, pages 2357-2365, 2012.","title":"Weighted likelihood policy search with model selection","authors":"Tsuyoshi Ueno, Kohei Hayashi, Takashi Washio, Yoshinobu Kawahara","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Watanabe, 2009] Sumio Watanabe. Algebraic geometry and statistical learning theory, volume 25. Cambridge Univer- sity Press, 2009.","title":"Algebraic geometry and statistical learning theory","authors":"Sumio Watanabe","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Young et al., 2010] Steve Young, Milica Gašić, Simon Keizer, François Mairesse, Jost Schatzmann, Blaise Thom- son, and Kai Yu. The hidden information state model: A practical framework for pomdp-based spoken dialogue man- agement. Computer Speech & Language, 24(2):150-174, 2010.","title":"The hidden information state model: A practical framework for pomdp-based spoken dialogue management","authors":"Steve Young, Milica Gašić, Simon Keizer, François Mairesse, Jost Schatzmann, Blaise Thomson, Kai Yu","year":"2010","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}