{"type":"ScholarlyArticle","title":"3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations","authors":[{"name":"Zhihua Wang","slug":"zhihua-wang","given":"Zhihua","family":"Wang","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Stefano Rosa","slug":"stefano-rosa","given":"Stefano","family":"Rosa","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Bo Yang","slug":"bo-yang","given":"Bo","family":"Yang","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Sen Wang","slug":"sen-wang","given":"Sen","family":"Wang","affiliation":"School of Engineering and Physical Sciences , Heriot-Watt University"},{"name":"Niki Trigoni","slug":"niki-trigoni","given":"Niki","family":"Trigoni","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Andrew Markham","slug":"andrew-markham","given":"Andrew","family":"Markham","affiliation":"Department of Computer Science , University of Oxford"}],"abstract":"The ability to interact and understand the environment is a fundamental prerequisite for a wide range of applications from robotics to augmented reality. In particular, predicting how deformable objects will react to applied forces in real time is a significant challenge. This is further confounded by the fact that shape information about encountered objects in the real world is often impaired by occlusions, noise and missing regions e.g. a robot manipulating an object will only be able to observe a partial view of the entire solid. In this work we present a framework, 3D-PhysNet, which is able to predict how a three-dimensional solid will deform under an applied force using intuitive physics modelling. In particular, we propose a new method to encode the physical properties of the material and the applied force, enabling generalisation over materials. The key is to combine deep variational autoencoders with adversarial training, conditioned on the applied force and the material properties.We further propose a cascaded architecture that takes a single 2.5D depth view of the object and predicts its deformation. Training data is provided by a physics simulator. The network is fast enough to be used in real-time applications from partial views. Experimental results show the viability and the generalisation properties of the proposed architecture.","doi":"10.24963/ijcai.2018/688","url":"https://ijcai.org/proceedings/2018/688","pdf":"https://ijcai.org/proceedings/2018/0688.pdf","markdown":"https://ijcai.org/proceedings/2018/688.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/688","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Robotics","track":"Main track","number":688,"pages":{"first":4958,"last":4964},"keywords":["Robotics: Learning in Robotics","Robotics: Robotics and Vision","Robotics: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martín Arjovsky, Soumith Chin- tala, and Léon Bottou. Wasserstein gan. CoRR, abs/1701.07875, 2017.","title":"Martín Arjovsky, Soumith Chintala","authors":"Arjovsky","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2017] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Cvae-gan: Fine-grained im- age generation through asymmetric training. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2764-2773, 2017.","title":"Cvae-gan: Fine-grained image generation through asymmetric training","authors":"Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, Gang Hua","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bingning Wang, 2017] Jun Zhao Bingning Wang, Kang Liu. Conditional generative adversarial net- works for commonsense machine comprehension. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 4123-4129, 2017.","title":"Conditional generative adversarial networks for commonsense machine comprehension","authors":"Jun Zhao, Bingning Wang, Kang Liu","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/576"},{"position":4,"raw":"[Brock et al., 2016] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236, 2016.","title":"Generative and discriminative voxel modeling with convolutional neural networks","authors":"Andrew Brock, Theodore Lim, James M Ritchie, Nick Weston","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Byravan and Fox, 2017] Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 173-180. IEEE, 2017.","title":"Se3-nets: Learning rigid body motion using deep neural networks","authors":"Arunkumar Byravan, Dieter Fox","year":"2017","venue":"Robotics and Automation (ICRA), 2017 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Finn et al., 2016] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in Neural Infor- mation Processing Systems, pages 64-72, 2016.","title":"Unsupervised learning for physical interaction through video prediction","authors":"Finn","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frank et al., 2014] Barbara Frank, Cyrill Stachniss, Rüdiger Schmedding, Matthias Teschner, and Wolfram Burgard. Learning object deformation models for robot motion planning. Robotics and Autonomous Systems, 62(8), 2014.","title":"Learning object deformation models for robot motion planning","authors":"Barbara Frank, Cyrill Stachniss, Rüdiger Schmedding, Matthias Teschner, Wolfram Burgard","year":"2014","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Ad- vances in Neural Information Processing Systems, pages 5769-5779, 2017.","title":"Improved training of wasserstein gans","authors":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Aaron C Vincent Dumoulin, Courville","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2017] X. Han, Z. Li, H. Huang, E. Kalogerakis, and Y. Yu. High-resolution shape completion using deep neural networks for global structure and local geometry inference. In IEEE International Conference on Computer Vision (ICCV), October 2017.","title":"High-resolution shape completion using deep neural networks for global structure and local geometry inference","authors":"Han","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Isola et al., 2016] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Larsen et al., 2016] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In Proceedings of the 33rd International Confer- ence on Machine Learning -Volume 48, ICML'16, pages 1558-1566. JMLR.org, 2016.","title":"Autoencoding beyond pixels using a learned similarity metric","authors":"Larsen","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lerer et al., 2016] Adam Lerer, Sam Gross, and Rob Fer- gus. Learning physical intuition of block towers by exam- ple. In Proceedings of the 33rd International Conference on International Conference on Machine Learning -Vol- ume 48, ICML'16, pages 430-438, 2016.","title":"Learning physical intuition of block towers by example","authors":"Adam Lerer, Sam Gross, Rob Fergus","year":"2016","venue":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mottaghi et al., 2016] Roozbeh Mottaghi, Hessam Bagher- inezhad, Mohammad Rastegari, and Ali Farhadi. Newto- nian scene understanding: Unfolding the dynamics of ob- jects in static images. In CVPR, pages 3521-3529, 2016.","title":"Newtonian scene understanding: Unfolding the dynamics of objects in static images","authors":"Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, Ali Farhadi","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Perarnau et al., 2016] Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, and Jose M. Álvarez. Invertible con- ditional gans for image editing. CoRR, abs/1611.06355, 2016.","title":"Invertible conditional gans for image editing","authors":"Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, Jose M Álvarez","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Smith and Meger, 2017] Edward J. Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In 1st Annual Conference on Robot Learn- ing, CoRL 2017, pages 87-96, 2017.","title":"Improved adversarial systems for 3d object generation and reconstruction","authors":"Edward J Smith, David Meger","year":"2017","venue":"1st Annual Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vondrick et al., 2016] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dy- namics. In NIPS, 2016.","title":"Generating videos with scene dynamics","authors":"Carl Vondrick, Hamed Pirsiavash, Antonio Torralba","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2015] Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In Advances in neural information pro- cessing systems, pages 127-135, 2015.","title":"Galileo: Perceiving physical object properties by integrating a physics engine with deep learning","authors":"Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, Josh Tenenbaum","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2017] Jiajun Wu, Erika Lu, Pushmeet Kohli, William T Freeman, and Joshua B Tenenbaum. Learn- ing to see physics via visual de-animation. In Advances in Neural Information Processing Systems, 2017.","title":"Learning to see physics via visual de-animation","authors":"Jiajun Wu, Erika Lu, Pushmeet Kohli, Joshua B William T Freeman, Tenenbaum","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xue et al., 2017] Jia Xue, Hang Zhang, Kristin Dana, and Ko Nishino. Differential angular imaging for material recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.","title":"Differential angular imaging for material recognition","authors":"Jia Xue, Hang Zhang, Kristin Dana, Ko Nishino","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2017] Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, and Niki Trigoni. 3d object reconstruction from a single depth view with adver- sarial learning. In 2017 IEEE International Conference on Computer Vision (ICCV) Workshop, 2017.","title":"3d object reconstruction from a single depth view with adversarial learning","authors":"Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, Niki Trigoni","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV) Workshop","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 2223-2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}