{"type":"ScholarlyArticle","title":"Contact2Grasp: 3D Grasp Synthesis via Hand-Object Contact Constraint","authors":[{"name":"Haoming Li","slug":"haoming-li","given":"Haoming","family":"Li","affiliation":"College of Control Science and Engineering , Zhejiang University , China"},{"name":"Xinzhuo Lin","slug":"xinzhuo-lin","given":"Xinzhuo","family":"Lin","affiliation":"College of Control Science and Engineering , Zhejiang University , China"},{"name":"Yang Zhou","slug":"yang-zhou","given":"Yang","family":"Zhou","affiliation":"OPPO US Research Center"},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"OPPO US Research Center"},{"name":"Yuchi Huo","slug":"yuchi-huo","given":"Yuchi","family":"Huo","affiliation":"State Key Lab of CAD&CG , Zhejiang University , China; Zhejiang Lab , China"},{"name":"Jiming Chen","slug":"jiming-chen","given":"Jiming","family":"Chen","affiliation":"College of Control Science and Engineering , Zhejiang University , China; Key Lab of CS&AUS of Zhejiang Province , China"},{"name":"Qi Ye","slug":"qi-ye","given":"Qi","family":"Ye","affiliation":"College of Control Science and Engineering , Zhejiang University , China; Key Lab of CS&AUS of Zhejiang Province , China"}],"abstract":"3D grasp synthesis generates grasping poses given an input object. Existing works tackle the problem by learning a direct mapping from objects to the distributions of grasping poses. However, because the physical contact is sensitive to small changes in pose, the high-nonlinear mapping between 3D object representation to valid poses is considerably non-smooth, leading to poor generation efficiency and restricted generality. To tackle the challenge, we introduce an intermediate variable for grasp contact areas to constrain the grasp generation; in other words, we factorize the mapping into two sequential stages by assuming that grasping poses are fully constrained given contact maps: 1) we first learn contact map distributions to generate the potential contact maps for grasps; 2) then learn a mapping from the contact maps to the grasping poses. Further, we propose a penetration-aware optimization with the generated contacts as a consistency constraint for grasp refinement. Extensive validations on two public datasets show that our method outperforms state-of-the-art methods regarding grasp generation on various metrics.","doi":"10.24963/ijcai.2023/117","url":"https://ijcai.org/proceedings/2023/117","pdf":"https://ijcai.org/proceedings/2023/0117.pdf","markdown":"https://ijcai.org/proceedings/2023/117.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/117","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":117,"pages":{"first":1053,"last":1061},"keywords":["Computer Vision: CV: 3D computer vision","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Achlioptas et al., 2018] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning rep- resentations and generative models for 3d point clouds. In International Conference on Machine Learning, pages 40- 49, 2018.","title":"Learning representations and generative models for 3d point clouds","authors":"Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, Leonidas Guibas","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In International conference on machine learning, pages 214-223, 2017.","title":"Wasserstein generative adversarial networks","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brahmbhatt et al., 2019a] Samarth Brahmbhatt, Cusuh Ham, Charles C Kemp, and James Hays. Contactdb: An- alyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8709-8719, 2019.","title":"Contactdb: Analyzing and predicting grasp contact via thermal imaging","authors":"Brahmbhatt","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brahmbhatt et al., 2019b] Samarth Brahmbhatt, Ankur Handa, James Hays, and Dieter Fox. Contactgrasp: Functional multi-finger grasp synthesis from contact. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2386-2393, 2019.","title":"Contactgrasp: Functional multi-finger grasp synthesis from contact","authors":"Brahmbhatt","year":"2019","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brahmbhatt et al., 2020] Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object con- tact and hand pose. In European Conference on Computer Vision, pages 361-378, 2020.","title":"Contactpose: A dataset of grasps with object contact and hand pose","authors":"Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, James Hays","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang et al., 2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.","title":"Shapenet: An information-rich 3d model repository","authors":"Thomas Angel X Chang, Leonidas Funkhouser, Pat Guibas, Qixing Hanrahan, Zimo Huang, Silvio Li, Manolis Savarese, Shuran Savva, Hao Song, Su","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Corona et al., 2020] Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5031-5041, 2020.","title":"Ganhand: Predicting human grasp affordances in multi-object scenes","authors":"Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, Grégory Rogez","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2021] Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A bench- mark for visual object affordance understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1778-1787, 2021.","title":"3d affordancenet: A benchmark for visual object affordance understanding","authors":"Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, Kui Jia","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grady et al., 2021] Patrick Grady, Chengcheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1471-1481, 2021.","title":"Contactopt: Optimizing contact to improve grasps","authors":"Patrick Grady, Chengcheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, Charles C Kemp","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hasson et al., 2019] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11807-11816, 2019.","title":"Learning joint reconstruction of hands and manipulated objects","authors":"Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, J Michael, Ivan Black, Cordelia Laptev, Schmid","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2021] Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consis- tency reasoning for human grasps generation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 11107-11116, 2021.","title":"Hand-object contact consistency reasoning for human grasps generation","authors":"Jiang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karunratanakul et al., 2020] Korrawe Karunratanakul, Jin- long Yang, Yan Zhang, Michael J Black, Krikamol Muan- det, and Siyu Tang. Grasping field: Learning implicit rep- resentations for human grasps. In 2020 International Con- ference on 3D Vision, pages 333-344, 2020.","title":"Grasping field: Learning implicit representations for human grasps","authors":"Karunratanakul","year":"2020","venue":"International Conference on 3D Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mahendran et al., 2017] Siddharth Mahendran, Haider Ali, and René Vidal. 3d pose regression using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2174- 2182, 2017.","title":"3d pose regression using convolutional neural networks","authors":"Haider Siddharth Mahendran, René Ali, Vidal","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mandikal and Grauman, 2021] Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In IEEE Interna- tional Conference on Robotics and Automation, pages 6169-6176, 2021.","title":"Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances","authors":"Grauman Mandikal","year":"2021","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mandikal and Grauman, 2022] Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning, pages 651-661, 2022.","title":"Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video","authors":"Grauman Mandikal","year":"2022","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Miller and Allen, 2004] Andrew T Miller and Peter K Allen. Graspit! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, pages 110-122, 2004.","title":"Graspit! a versatile simulator for robotic grasping","authors":"T Andrew, Peter K Miller, Allen","year":"2004","venue":"IEEE Robotics & Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Milletari et al., 2016] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neu- ral networks for volumetric medical image segmentation. In International Conference on 3D Vision, pages 565-571, 2016.","title":"V-net: Fully convolutional neural networks for volumetric medical image segmentation","authors":"Fausto Milletari, Nassir Navab, Seyed-Ahmad Ahmadi","year":"2016","venue":"International Conference on 3D Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mo et al., 2021] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d ob- jects. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 6813-6823, 2021.","title":"Where2act: From pixels to actions for articulated 3d objects","authors":"Mo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qi et al., 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Hao Charles R Qi, Kaichun Su, Leonidas J Mo, Guibas","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Romero et al., 2017] Javier Romero, Dimitris Tzionas, and Michael J Black. Embodied hands: Modeling and cap- turing hands and bodies together. ACM Transactions on Graphics, 2017.","title":"Embodied hands: Modeling and capturing hands and bodies together","authors":"Javier Romero, Dimitris Tzionas, Michael J Black","year":"2017","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Roy and Todorovic, 2016] Anirban Roy and Sinisa Todor- ovic. A multi-scale cnn for affordance segmentation in rgb images. In European conference on computer vision, pages 186-201, 2016.","title":"Anirban Roy and Sinisa Todorovic. A multi-scale cnn for affordance segmentation in rgb images","authors":"Todorovic Roy","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sohn et al., 2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural infor- mation processing systems, 28, 2015.","title":"Learning structured output representation using deep conditional generative models","authors":"Sohn","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Taheri et al., 2020] Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In European conference on computer vision, pages 581-600, 2020.","title":"Grab: A dataset of whole-body human grasping of objects","authors":"Omid Taheri, Nima Ghorbani, Michael J Black, Dimitrios Tzionas","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Taheri et al., 2021] Omid Taheri, Vasileios Choutas, Michael J Black, and Dimitrios Tzionas. Goal: Gener- ating 4d whole-body motion for hand-object grasping. arXiv preprint arXiv:2112.11454, 2021.","title":"Goal: Generating 4d whole-body motion for hand-object grasping","authors":"Omid Taheri, Vasileios Choutas, Michael J Black, Dimitrios Tzionas","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2021] Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. Vat- mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv:2106.14440, 2021.","title":"Vatmart: Learning visual action trajectory proposals for manipulating 3d articulated objects","authors":"Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, Hao Dong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xing et al., 2022] Fangyuan Xing, Shibo He, Victor CM Leung, and Hongxi Yin. Energy efficiency optimization for rate-splitting multiple access-based indoor visible light communication networks. IEEE Journal on Selected Areas in Communications, 40(5):1706-1720, 2022.","title":"Energy efficiency optimization for rate-splitting multiple access-based indoor visible light communication networks","authors":"Fangyuan Xing, Shibo He, C M Victor, Hongxi Leung, Yin","year":"2022","venue":"IEEE Journal on Selected Areas in Communications","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2021] He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura. Manipnet: neural manipulation synthe- sis with a hand-object spatial representation. ACM Trans- actions on Graphics, pages 1-14, 2021.","title":"Manipnet: neural manipulation synthesis with a hand-object spatial representation","authors":"Zhang","year":"2021","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2015] Yixin Zhu, Yibiao Zhao, and Song Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2855-2864, 2015.","title":"Understanding tools: Task-oriented object modeling, learning and recognition","authors":"Yixin Zhu, Yibiao Zhao, Song Chun, Zhu","year":"2015","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}