{"type":"ScholarlyArticle","title":"ABM: Attention before Manipulation","authors":[{"name":"Fan Zhuo","slug":"fan-zhuo","given":"Fan","family":"Zhuo","affiliation":"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ); College of Computer Science and Software Engineering , Shenzhen University , China"},{"name":"Ying He","slug":"ying-he","given":"Ying","family":"He","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , China"},{"name":"Fei Yu","slug":"fei-yu","given":"Fei","family":"Yu","affiliation":"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ); College of Computer Science and Software Engineering , Shenzhen University , China"},{"name":"Pengteng Li","slug":"pengteng-li","given":"Pengteng","family":"Li","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , China"},{"name":"Zheyi Zhao","slug":"zheyi-zhao","given":"Zheyi","family":"Zhao","affiliation":"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ); College of Computer Science and Software Engineering , Shenzhen University , China"},{"name":"Xilong Sun","slug":"xilong-sun","given":"Xilong","family":"Sun","affiliation":"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"}],"abstract":"Vision-language models (VLMs) show promising generalization and zero-shot capabilities, offering a potential solution to the impracticality and cost of enabling robots to comprehend diverse human instructions and scene semantics in the real world. Existing approaches most directly integrate the semantic representations from pre-trained VLMs with policy learning. However, these methods are limited to the labeled data learned, resulting in poor generalization ability to unseen instructions and objects. To address the above limitation, we propose a simple method called \"Attention before Manipulation\" (ABM), which fully leverages the object knowledge encoded in CLIP to extract information about the target object in the image. It constructs an Object Mask Field, serving as a better representation of the target object for the model to separate visual grounding from action prediction and acquire specific manipulation skills effectively. We train ABM for 8 RLBench tasks and 2 real-world tasks via behavior cloning. Extensive experiments show that our method significantly outperforms the baselines in the zero-shot and compositional generalization experiment settings.","doi":"10.24963/ijcai.2024/201","url":"https://ijcai.org/proceedings/2024/201","pdf":"https://ijcai.org/proceedings/2024/0201.pdf","markdown":"https://ijcai.org/proceedings/2024/201.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/201","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":201,"pages":{"first":1816,"last":1824},"keywords":["Robotics: ROB: Robotics and vision","Computer Vision: CV: Embodied vision: Active agents, simulation","Robotics: ROB: Applications","Robotics: ROB: Manipulation"],"videos":[],"references":[{"position":1,"raw":"[Brohan et al., 2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real- world control at scale. arXiv preprint arXiv:2212.06817, 2022.","title":"RT-1: Robotics transformer for realworld control at scale","authors":"Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Gervet et al., 2023] Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3D: 3D Fea- ture Field Transformers for Multi-Task Robotic Manipula- tion. In Proceedings of the Conference on Robot Learning, pages 3949-3965, 2023.","title":"Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation","authors":"Theophile Gervet, Nikolaos Zhou Xian, Katerina Gkanatsios, Fragkiadaki","year":"2023","venue":"Proceedings of the Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goyal et al., 2023] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. RVT: Robotic View Transformer for 3D Object Manipulation. arXiv preprint arXiv:2306.14896, 2023.","title":"RVT: Robotic View Transformer for 3D Object Manipulation","authors":"Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Huang et al., 2023a] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 10608-10615, 2023.","title":"Visual language maps for robot navigation","authors":"Huang","year":"2023","venue":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2023b] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. In- struct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model. arXiv preprint arXiv:2305.11176, 2023.","title":"Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2023c] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.","title":"Voxposer: Composable 3d value maps for robotic manipulation with language models","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[James et al., 2019] Stephen James, Marc Freese, and An- drew J Davison. Pyrep: Bringing v-rep to deep robot learn- ing. arXiv preprint arXiv:1906.11176, 2019.","title":"Pyrep: Bringing v-rep to deep robot learning","authors":"Stephen James, Marc Freese, Andrew J Davison","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[James et al., 2020] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The Robot Learning Benchmark & Learning Environment. IEEE Robotics and Automation Letters, pages 3019-- 3026, 2020.","title":"RLBench: The Robot Learning Benchmark & Learning Environment","authors":"Stephen James, Zicong Ma, David Rovick Arrojo, Andrew J Davison","year":"2020","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jang et al., 2022] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-Z: Zero-shot task gener- alization with robotic imitation learning. In Proceedings of the Conference on Robot Learning, pages 991-1002, 2022.","title":"BC-Z: Zero-shot task generalization with robotic imitation learning","authors":"Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, Chelsea Finn","year":"2022","venue":"Proceedings of the Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2023] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General Robot Manipulation with Multi- modal Prompts. In Proceedings of the Fortieth Interna- tional Conference on Machine Learning, 2023.","title":"VIMA: General Robot Manipulation with Multimodal Prompts","authors":"Jiang","year":"2023","venue":"Proceedings of the Fortieth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Boyi Li, Kilian Q Weinberger, Serge Be- longie, Vladlen Koltun, and Rene Ranftl. Language-driven Semantic Segmentation. In Proceedings of the Interna- tional Conference on Learning Representations, 2022.","title":"Language-driven Semantic Segmentation","authors":"Boyi Li, Q Kilian, Serge Weinberger, Vladlen Belongie, Rene Koltun, Ranftl","year":"2022","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2022] Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. Instruction-following agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.13431, 2022.","title":"Instruction-following agents with jointly pre-trained vision-language models","authors":"Hao Liu, Lisa Lee, Kimin Lee, Pieter Abbeel","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mildenhall et al., 2021] Ben Mildenhall, Pratul P Srini- vasan, Matthew Tancik, Jonathan T Barron, Ravi Ra- mamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.","title":"Nerf: Representing scenes as neural radiance fields for view synthesis","authors":"Ben Mildenhall, P Pratul, Matthew Srinivasan, Jonathan T Tancik, Ravi Barron, Ren Ramamoorthi, Ng","year":"2021","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Proceedings of the Advances in Neural Information Processing Systems, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"Proceedings of the Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Paradis et al., 2021] Samuel Paradis, Minho Hwang, Brijen Thananjeyan, Jeffrey Ichnowski, Daniel Seita, Danyal Fer, Thomas Low, Joseph E Gonzalez, and Ken Goldberg. In- termittent visual servoing: Efficiently learning policies ro- bust to instrument changes for high-precision surgical ma- nipulation. In Proceedings of the IEEE International Con- ference on Robotics and Automation (ICRA), pages 7166- 7173, 2021.","title":"Intermittent visual servoing: Efficiently learning policies robust to instrument changes for high-precision surgical manipulation","authors":"Samuel Paradis, Minho Hwang, Brijen Thananjeyan, Jeffrey Ichnowski, Daniel Seita, Danyal Fer, Thomas Low, Joseph E Gonzalez, Ken Goldberg","year":"2021","venue":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rahmatizadeh et al., 2018] Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 3758-3765, 2018.","title":"Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration","authors":"Rahmatizadeh","year":"2018","venue":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ravi et al., 2020] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501, 2020.","title":"Accelerating 3d deep learning with pytorch3d","authors":"Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, Georgia Gkioxari","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rohmer et al., 2013] Eric Rohmer, Surya PN Singh, and Marc Freese. V-REP: A versatile and scalable robot sim- ulation framework. In Proceedings of the IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems, pages 1321-1326, 2013.","title":"V-REP: A versatile and scalable robot simulation framework","authors":"Eric Rohmer, P N Surya, Marc Singh, Freese","year":"2013","venue":"Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shafiullah et al., 2022] Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663, 2022.","title":"Clip-fields: Weakly supervised semantic fields for robotic memory","authors":"Shafiullah","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2023] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipula- tion. arXiv preprint arXiv:2308.07931, 2023.","title":"Distilled feature fields enable few-shot language-guided manipulation","authors":"William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, Phillip Isola","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shridhar et al., 2022] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the Conference on Robot Learning, pages 894-906, 2022.","title":"Cliport: What and where pathways for robotic manipulation","authors":"Mohit Shridhar, Lucas Manuelli, Dieter Fox","year":"2022","venue":"Proceedings of the Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shridhar et al., 2023] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the Conference on Robot Learning, pages 785-799, 2023.","title":"Perceiver-actor: A multi-task transformer for robotic manipulation","authors":"Mohit Shridhar, Lucas Manuelli, Dieter Fox","year":"2023","venue":"Proceedings of the Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stepputtis et al., 2020] Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13150, 2020.","title":"Language-conditioned imitation learning for robot manipulation tasks","authors":"Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, Heni Ben, Amor","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stone et al., 2023] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Brianna Zitkovich, Fei Xia, Chelsea Finn, et al. Open-world object manipulation us- ing pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023.","title":"Open-world object manipulation using pre-trained vision-language models","authors":"Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Brianna Zitkovich, Fei Xia, Chelsea Finn","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of the Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2023a] Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, and Yang Gao. Programmati- cally Grounded, Compositionally Generalizable Robotic Manipulation. arXiv preprint arXiv:2304.13826, 2023.","title":"Programmatically Grounded, Compositionally Generalizable Robotic Manipulation","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023b] Yixuan Wang, Zhuoran Li, Mingtong Zhang, Katherine Driggs-Campbell, Jiajun Wu, Li Fei-Fei, and Yunzhu Li. Dˆ3 Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation. arXiv preprint arXiv:2309.16118, 2023.","title":"Dˆ3 Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[You et al., 2019] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xi- aodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Train- ing bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.","title":"Large batch optimization for deep learning: Training bert in 76 minutes","authors":"Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeng et al., 2021] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sind- hwani, et al. Transporter networks: Rearranging the vi- sual world for robotic manipulation. In Proceedings of the Conference on Robot Learning, pages 726-747, 2021.","title":"Transporter networks: Rearranging the visual world for robotic manipulation","authors":"Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani","year":"2021","venue":"Proceedings of the Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In Proceedings of the IEEE International Conference on Robotics and Au- tomation (ICRA), pages 5628-5635, 2018.","title":"Deep imitation learning for complex manipulation tasks from virtual reality teleoperation","authors":"Tianhao Zhang, Zoe Mccarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, Pieter Abbeel","year":"2018","venue":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2022] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Proceedings of the European Conference on Computer Vision, pages 696-712, 2022.","title":"Extract free dense labels from clip","authors":"Chong Zhou, Chen Change Loy, Bo Dai","year":"2022","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}