{"type":"ScholarlyArticle","title":"DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs","authors":[{"name":"Yunbo Wang","slug":"yunbo-wang","given":"Yunbo","family":"Wang","affiliation":"Tsinghua University"},{"name":"Bo Liu","slug":"bo-liu","given":"Bo","family":"Liu","affiliation":"The University of Texas at Austin"},{"name":"Jiajun Wu","slug":"jiajun-wu","given":"Jiajun","family":"Wu","affiliation":"Stanford University"},{"name":"Yuke Zhu","slug":"yuke-zhu","given":"Yuke","family":"Zhu","affiliation":"The University of Texas at Austin"},{"name":"Simon S. Du","slug":"simon-s-du","given":"Simon S.","family":"Du","affiliation":"Institute for Advanced Study"},{"name":"Li Fei-Fei","slug":"li-fei-fei","given":"Li","family":"Fei-Fei","affiliation":"Stanford University"},{"name":"Joshua B. Tenenbaum","slug":"joshua-b-tenenbaum","given":"Joshua B.","family":"Tenenbaum","affiliation":"Massachusetts Institute of Technology"}],"abstract":"A major difficulty of solving continuous POMDPs is to infer the multi-modal distribution of the unobserved true states and to make the planning algorithm dependent on the perceived uncertainty. We cast POMDP filtering and planning problems as two closely related Sequential Monte Carlo (SMC) processes, one over the real states and the other over the future optimal trajectories, and combine the merits of these two parts in a new model named the DualSMC network. In particular, we first introduce an adversarial particle filter that leverages the adversarial relationship between its internal components. Based on the filtering results, we then propose a planning algorithm that extends the previous SMC planning approach [Piche et al., 2018] to continuous POMDPs with an uncertainty-dependent policy. Crucially, not only can DualSMC handle complex observations such as image input but also it remains highly interpretable. It is shown to be effective in three continuous POMDP domains: the floor positioning domain, the 3D light-dark navigation domain, and a modified Reacher domain.","doi":"10.24963/ijcai.2020/579","url":"https://ijcai.org/proceedings/2020/579","pdf":"https://ijcai.org/proceedings/2020/0579.pdf","markdown":"https://ijcai.org/proceedings/2020/579.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/579","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Planning and Scheduling","track":"Main track","number":579,"pages":{"first":4190,"last":4198},"keywords":["Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: Planning Algorithms","Planning and Scheduling: Planning with Incomplete information"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25225/file_25225.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Beattie et al., 2016] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. DeepMind Lab. arXiv preprint arXiv:1612.03801, 2016.","title":null,"authors":"Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Doucet and Johansen, 2009] Arnaud Doucet and Adam M Johansen. A tutorial on particle filtering and smoothing: Fif- teen years later. Handbook of Nonlinear Filtering, 12(656- 704):3, 2009.","title":"A tutorial on particle filtering and smoothing: Fif-teen years later","authors":"Johansen Doucet, Arnaud Doucet, Adam M Johansen","year":"2009","venue":"Handbook of Nonlinear Filtering","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gordon et al., 1993] Neil J Gordon, David J Salmond, and Adrian FM Smith. Novel approach to nonlinear/non- Gaussian Bayesian state estimation. In IEE Proceedings F (Radar and Signal Processing), pages 107-113, 1993.","title":"Novel approach to nonlinear/non-Gaussian Bayesian state estimation","authors":"David J Neil J Gordon, Adrian Fm Salmond, Smith","year":"1993","venue":"IEE Proceedings F (Radar and Signal Processing)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2015] Shixiang Shane Gu, Zoubin Ghahramani, and Richard E Turner. Neural adaptive sequential Monte Carlo. In NeurIPS, pages 2629-2637, 2015.","title":"Neural adaptive sequential Monte Carlo","authors":"Gu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hafner et al., 2019] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, pages 2555-2565, 2019.","title":"Learning latent dynamics for planning from pixels","authors":"Hafner","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hausknecht and Stone, 2015] Matthew Hausknecht and Pe- ter Stone. Deep recurrent Q-learning for partially observ- able MDPs. In 2015 AAAI Fall Symposium Series, 2015.","title":"Deep recurrent Q-learning for partially observable MDPs","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":"AAAI Fall Symposium Series","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Igl et al., 2018] Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for POMDPs. In ICML, pages 2117- 2126, 2018.","title":"Deep variational reinforcement learning for POMDPs","authors":"Maximilian Igl, Luisa Zintgraf, Anh Tuan, Frank Le, Shimon Wood, Whiteson","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jonschkowski et al., 2018] Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. In RSS, 2018.","title":"Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: End-to-end learning with algorithmic priors","authors":"Jonschkowski","year":"2018","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and act- ing in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack, Kaelbling Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kappen et al., 2012] Hilbert J Kappen, Vicenc ¸Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine Learning, 87(2):159-182, 2012.","title":"Optimal control as a graphical model inference problem","authors":"Hilbert J Kappen, Manfred Vicenc, Opper","year":"2012","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karkus et al., 2017] Peter Karkus, David Hsu, and Wee Sun Lee. QMDP-net: Deep learning for planning under partial observability. In NeurIPS, pages 4694-4704, 2017.","title":"QMDP-net: Deep learning for planning under partial observability","authors":"Peter Karkus, David Hsu, Wee Sun, Lee","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karkus et al., 2018] Peter Karkus, David Hsu, and Wee Sun Lee. Particle filter networks with application to visual localization. In CoRL, 2018.","title":"Particle filter networks with application to visual localization","authors":"Peter Karkus, David Hsu, Wee Sun, Lee","year":"2018","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kempinska and Shawe-Taylor, 2017] Kira Kempinska and John Shawe-Taylor. Adversarial sequential Monte Carlo. In Bayesian Deep Learning (NeurIPS Workshop), 2017.","title":"Kira Kempinska and John Shawe-Taylor","authors":"Shawe-Taylor Kempinska","year":"2017","venue":"Bayesian Deep Learning (NeurIPS Workshop)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kurniawati and Yadav, 2016] Hanna Kurniawati and Vinay Yadav. An online POMDP solver for uncertainty planning in dynamic environment. In Robotics Research, pages 611- 629. 2016.","title":"An online POMDP solver for uncertainty planning in dynamic environment","authors":"Yadav ; ] Kurniawati, Kurniawati Hanna, Vinay Yadav","year":"2016","venue":"Robotics Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Levine and Koltun, 2013] Sergey Levine and Vladlen Koltun. Variational policy search via trajectory optimiza- tion. In NeurIPS, pages 207-215, 2013.","title":"Variational policy search via trajectory optimization","authors":"Koltun Levine, Sergey Levine, Vladlen Koltun","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Levine, 2018] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018.","title":"Reinforcement learning and control as probabilistic inference: Tutorial and review","authors":"Sergey Levine, Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Littman et al., 1995] Michael L Littman, Anthony R Cas- sandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In ICML, 1995.","title":"Learning policies for partially observable environments: Scaling up","authors":"Anthony R Michael L Littman, Leslie Cassandra, Kaelbling Pack","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maddison et al., 2017] Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, An- driy Mnih, Arnaud Doucet, and Yee Teh. Filtering varia- tional objectives. In NeurIPS, pages 6573-6583, 2017.","title":"Filtering variational objectives","authors":"Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, Yee Teh","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Naesseth et al., 2018] Christian A Naesseth, Scott W Linder- man, Rajesh Ranganath, and David M Blei. Variational sequential Monte Carlo. In AISTATS, 2018.","title":"Variational sequential Monte Carlo","authors":"Christian A Naesseth, Rajesh Scott W Linderman, David M Ranganath, Blei","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos H Papadim- itriou and John N Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):441-450, 1987.","title":"The complexity of Markov decision processes","authors":"Christos H Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Piche et al., 2018] Alexandre Piche, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal. Probabilistic plan- ning with sequential Monte Carlo methods. In ICLR, 2018.","title":"Probabilistic planning with sequential Monte Carlo methods","authors":"Alexandre Piche, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, Chris Pal","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Platt Jr et al., 2010] Robert Platt Jr, Russ Tedrake, Leslie Kaelbling, and Tomas Lozano-Perez. Belief space planning assuming maximum likelihood observations. In RSS, 2010.","title":"Belief space planning assuming maximum likelihood observations","authors":"Robert Platt, Russ Tedrake, Leslie Kaelbling, Tomas Lozano-Perez","year":"2010","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ross et al., 2008] Stephane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive POMDPs. In NeurIPS, pages 1225-1232, 2008.","title":"Bayes-adaptive POMDPs","authors":"Stephane Ross, Brahim Chaib-Draa, Joelle Pineau","year":"2008","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Seiler et al., 2015] Konstantin M Seiler, Hanna Kurniawati, and Surya PN Singh. An online and approximate solver for POMDPs with continuous action space. In ICRA, pages 2290-2297, 2015.","title":"An online and approximate solver for POMDPs with continuous action space","authors":"Konstantin M Seiler, Hanna Kurniawati, Surya, Singh","year":"2015","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In NeurIPS, pages 2164-2172, 2010.","title":"Monte-Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Somani et al., 2013] Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. DESPOT: Online POMDP planning with regularization. In NeurIPS, pages 1772-1780, 2013.","title":"DESPOT: Online POMDP planning with regularization","authors":"Adhiraj Somani, Nan Ye, David Hsu, Wee Sun, Lee","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sunberg and Kochenderfer, 2018] Zachary N Sunberg and Mykel J Kochenderfer. Online algorithms for POMDPs with continuous state, action, and observation spaces. In ICAPS, 2018.","title":"Online algorithms for POMDPs with continuous state, action, and observation spaces","authors":"N Zachary, Sunberg, J Mykel, Kochenderfer","year":"2018","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. MuJoCo: A physics engine for model-based control. In IROS, pages 5026-5033, 2012.","title":"MuJoCo: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Todorov, 2008] Emanuel Todorov. General duality between optimal control and estimation. In CDC, pages 4286-4292, 2008.","title":"General duality between optimal control and estimation","authors":"Emanuel Todorov","year":"2008","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Toussaint, 2009] Marc Toussaint. Robot trajectory optimiza- tion using approximate inference. In ICML, pages 1049- 1056, 2009.","title":"Robot trajectory optimization using approximate inference","authors":"Marc Toussaint","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2018] Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learning for POMDPs. arXiv preprint arXiv:1804.06309, 2018.","title":"On improving deep reinforcement learning for POMDPs","authors":"Pengfei Zhu, Xin Li, Pascal Poupart, Guanghui Miao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}