{"type":"ScholarlyArticle","title":"ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spaces","authors":[{"name":"Libing Yang","slug":"libing-yang","given":"Libing","family":"Yang","affiliation":"East China Normal University"},{"name":"Yang Li","slug":"yang-li","given":"Yang","family":"Li","affiliation":"East China Normal University"},{"name":"Long Chen","slug":"long-chen","given":"Long","family":"Chen","affiliation":"The Hong Kong University of Science and Technology"}],"abstract":"Vision-based robotic cloth unfolding has made great progress recently. However, prior works predominantly rely on value learning and have not fully explored policy-based techniques. Recently, the success of reinforcement learning on the large language model has shown that the policy gradient algorithm can enhance policy with huge action space. In this paper, we introduce ClothPPO, a framework that employs a policy gradient algorithm based on actor-critic architecture to enhance a pre-trained model with huge 10^6 action spaces aligned with observation in the task of unfolding clothes. To this end, we redefine the cloth manipulation problem as a partially observable Markov decision process. A supervised pre-training stage is employed to train a baseline model of our policy. In the second stage, the Proximal Policy Optimization (PPO) is utilized to guide the supervised model within the observation-aligned action space. By optimizing and updating the strategy, our proposed method increases the garment's surface area for cloth unfolding under the soft-body manipulation task. Experimental results show that our proposed framework can further improve the unfolding performance of other state-of-the-art methods. Our project is available at https://vpx-ecnu.github.io/ClothPPO-website/.","doi":"10.24963/ijcai.2024/762","url":"https://ijcai.org/proceedings/2024/762","pdf":"https://ijcai.org/proceedings/2024/0762.pdf","markdown":"https://ijcai.org/proceedings/2024/762.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/762","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Robotics","track":"Main Track","number":762,"pages":{"first":6895,"last":6903},"keywords":["Robotics: ROB: Manipulation","Robotics: ROB: Learning in robotics","Robotics: ROB: Perception","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"Shirt Pants Jumpsuit Skirt Dress Flingbot 53.64 / 00.15 / 51.81 65.66 / 01.24 / 51.65 53.15 / 00.07 / 49.62 56.26 / 00.78 / 52.32 66.38 / 01.07 / 51.27 PPO 50.48 / 07.34 / 52.96 52.28 / 01.97 / 52.14 50.24 / -0.66 / 51.61 49.07 / 14.45 / 53.00 52.54 / -18.91 / 53.78 Cloth Funnels 86.91 / 40.98 / 71.10 70.81 / 20.60 / 54.22 73.13 / 21.45 / 62.89 77.99 / 40.82 / 56.09 79.15 / 33.23 / 55.40 Cloth Funnels-P 73.08 / 29.57 / 72.14 71.77 / 22.77 / 55.62 73.76 / 23.03 / 72.68 69.70 / 35.44 / 66.14 76.44 / 30.43 / 68.52 ClothPPO 80.24 / 34.34 / 73.84 80.58 / 30.38 / 72.14 83.04 / 28.10 / 77.17 85.74 / 45.17 / 74.22 86.56 / 40.39 / 73.36","title":"Shirt Pants Jumpsuit Skirt Dress Flingbot","authors":null,"year":null,"venue":"Shirt Pants Jumpsuit Skirt Dress Flingbot","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avigal et al., 2022] Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kröger, and Ken Goldberg. Speedfolding: Learning efficient bimanual folding of garments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-8. IEEE, 2022.","title":"Speedfolding: Learning efficient bimanual folding of garments","authors":"Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kröger, Ken Goldberg","year":"2022","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertiche et al., 2020] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: clothed 3d humans. In Eu- ropean Conference on Computer Vision, pages 344-359. Springer, 2020.","title":"Hugo Bertiche, Meysam Madadi, and Sergio Escalera","authors":"Bertiche","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Canberk et al., 2022] Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Cloth funnels: Canonicalized-alignment for multi-purpose garment manipulation. In International Conference of Robotics and Automation (ICRA), 2022.","title":"Cloth funnels: Canonicalized-alignment for multi-purpose garment manipulation","authors":"Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, Shuran Song","year":"2022","venue":"International Conference of Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen* et al., 2023] Siwei Chen*, Yiqing Xu*, Cunjun Yu*, Linfeng Li, Xiao Ma, Zhongwen Xu, and David Hsu. Daxbench: Benchmarking deformable object manipula- tion with differentiable physics. In ICLR, 2023.","title":"Daxbench: Benchmarking deformable object manipulation with differentiable physics","authors":"Siwei Chen, *, Yiqing Xu, *, Cunjun Yu, *, Linfeng Li, Xiao Ma, Zhongwen Xu, David Hsu","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2024] Jiawei Chen, Lin Chen, Jiang Yang, Tianqi Shi, Lechao Cheng, Zunlei Feng, and Mingli Song. Life regression based patch slimming for vision transform- ers. Neural Networks, page 106340, 2024.","title":"Life regression based patch slimming for vision transformers","authors":"Jiawei Chen, Lin Chen, Jiang Yang, Tianqi Shi, Lechao Cheng, Zunlei Feng, Mingli Song","year":"2024","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Doumanoglou et al., 2016] Andreas Doumanoglou, Jan Stria, Georgia Peleka, Ioannis Mariolis, Vladimir Petrik, Andreas Kargakos, Libor Wagner, Václav Hlaváč, Tae- Kyun Kim, and Sotiris Malassiotis. Folding clothes autonomously: A complete pipeline. IEEE Transactions on Robotics, 32(6):1461-1478, 2016.","title":"Folding clothes autonomously: A complete pipeline","authors":"Andreas Doumanoglou, Jan Stria, Georgia Peleka, Ioannis Mariolis, Vladimir Petrik, Andreas Kargakos, Libor Wagner, Václav Hlaváč, Tae-Kyun Kim, Sotiris Malassiotis","year":"2016","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Franc ¸ois-Lavet et al., 2018] Vincent Franc ¸ois-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, et al. An introduction to deep reinforcement learn- ing. Foundations and Trends® in Machine Learning, 11(3- 4):219-354, 2018.","title":"Vincent Franc ¸ois-Lavet, Peter Henderson, Riashat Islam","authors":"Franc ¸ois-Lavet","year":"2018","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ganapathi et al., 2021] Aditya Ganapathi, Priya Sundare- san, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph E Gonzalez, Nawid Jamali, et al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515-11522. IEEE, 2021.","title":"Learning dense visual correspondences in simulation to smooth and fold real fabrics","authors":"Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph E Gonzalez, Nawid Jamali","year":"2021","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gupta et al., 2021] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learn- ing and evolution. Nature communications, 12(1):5721, 2021.","title":"Embodied intelligence via learning and evolution","authors":"Gupta","year":"2021","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ha and Song, 2021] Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robotic Learning (CoRL), 2021.","title":"Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding","authors":"Huy Ha, Shuran Song","year":"2021","venue":"Conference on Robotic Learning (CoRL)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2023] Kaiwen Hu, Jing Gao, Fangyuan Mao, Xin- hui Song, Lechao Cheng, Zunlei Feng, and Mingli Song. Disassembling convolutional segmentation network. Inter- national Journal of Computer Vision, 131(7):1741-1760, 2023.","title":"Disassembling convolutional segmentation network","authors":"Kaiwen Hu, Jing Gao, Fangyuan Mao, Xinhui Song, Lechao Cheng, Zunlei Feng, Mingli Song","year":"2023","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2023] Zixuan Huang, Xingyu Lin, and David Held. Self-supervised cloth reconstruction via action-conditioned cloth tracking. arXiv preprint arXiv:2302.09502, 2023.","title":"Self-supervised cloth reconstruction via action-conditioned cloth tracking","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018.","title":"Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids","authors":"Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2020] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Confer- ence on Robot Learning, 2020.","title":"Softgym: Benchmarking deep reinforcement learning for deformable object manipulation","authors":"Xingyu Lin, Yufei Wang, Jake Olkin, David Held","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2022] Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256-266. PMLR, 2022.","title":"Learning visible connectivity dynamics for cloth smoothing","authors":"Xingyu Lin, Yufei Wang, Zixuan Huang, David Held","year":"2022","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Matas et al., 2018] Jan Matas, Stephen James, and An- drew J Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734-743. PMLR, 2018.","title":"Sim-to-real reinforcement learning for deformable object manipulation","authors":"Matas","year":"2018","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mo et al., 2022] Kai Mo, Chongkun Xia, Xueqian Wang, Yuhong Deng, Xuehai Gao, and Bin Liang. Foldsformer: Learning sequential multi-step cloth manipulation with space-time attention. IEEE Robotics and Automation Let- ters, 8(2):760-767, 2022.","title":"Foldsformer: Learning sequential multi-step cloth manipulation with space-time attention","authors":"Kai Mo, Chongkun Xia, Xueqian Wang, Yuhong Deng, Xuehai Gao, Bin Liang","year":"2022","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mulero et al., 2023] David Blanco Mulero, Gökhan Alcan, Fares Abu-Dakka, and Ville Kyrki. Qdp: Learning to se- quentially optimise quasi-static and dynamic manipulation primitives for robotic cloth manipulation. In IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems, 2023.","title":"Qdp: Learning to sequentially optimise quasi-static and dynamic manipulation primitives for robotic cloth manipulation","authors":"David Blanco Mulero, Gökhan Alcan, Ville Fares Abu-Dakka, Kyrki","year":"2023","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nair et al., 2017] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE inter- national conference on robotics and automation (ICRA), pages 2146-2153. IEEE, 2017.","title":"Combining self-supervised learning and imitation for vision-based rope manipulation","authors":"Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, Sergey Levine","year":"2017","venue":"IEEE international conference on robotics and automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in Neural Informa- tion Processing Systems, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ramamurthy et al., 2022] Rajkumar Ramamurthy, Prithvi- raj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. 2022.","title":"Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization","authors":"Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Seita et al., 2021] Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Gold- berg, and Andy Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned trans- porter networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4568-4575. IEEE, 2021.","title":"Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks","authors":"Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, Andy Zeng","year":"2021","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sondik, 1971] Edward Jay Sondik. The optimal control of partially observable Markov processes. Stanford Univer- sity, 1971.","title":"The optimal control of partially observable Markov processes","authors":"Edward Jay, Sondik","year":"1971","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Triantafyllou et al., 2016] Dimitra Triantafyllou, Ioannis Mariolis, Andreas Kargakos, Sotiris Malassiotis, and Nikos Aspragathos. A geometric approach to robotic un- folding of garments. Robotics and Autonomous Systems, 75:233-243, 2016.","title":"A geometric approach to robotic unfolding of garments","authors":"Dimitra Triantafyllou, Ioannis Mariolis, Andreas Kargakos, Sotiris Malassiotis, Nikos Aspragathos","year":"2016","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tutz, 1991] Gerhard Tutz. Sequential models in categori- cal regression. Computational Statistics & Data Analysis, 11(3):275-295, 1991.","title":"Sequential models in categorical regression","authors":"Gerhard Tutz, Tutz","year":"1991","venue":"Computational Statistics & Data Analysis","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Weng et al., 2022] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1-6, 2022.","title":"Tianshou: A highly modularized deep reinforcement learning library","authors":"Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, Jun Zhu","year":"2022","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2020a] Jimmy Wu, Xingyuan Sun, Andy Zeng, Shuran Song, Johnny Lee, Szymon Rusinkiewicz, and Thomas Funkhouser. Spatial action maps for mobile ma- nipulation. arXiv preprint arXiv:2004.09141, 2020.","title":"Spatial action maps for mobile manipulation","authors":"Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2020b] Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations, 2020.","title":"Learning to manipulate deformable objects without demonstrations","authors":"Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xiong et al., 2023] Haoyu Xiong, Haoyuan Fu, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, Animesh Garg, and Cewu Lu. Robotube: Learning house- hold manipulation from human videos with simulated twin environments. In Conference on Robot Learning, pages 1- 10. PMLR, 2023.","title":"Robotube: Learning household manipulation from human videos with simulated twin environments","authors":"Haoyu Xiong, Haoyuan Fu, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, Animesh Garg, Cewu Lu","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeng et al., 2020] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sind- hwani, and Johnny Lee. Transporter networks: Rearrang- ing the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.","title":"Transporter networks: Rearranging the visual world for robotic manipulation","authors":"Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, Johnny Lee","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}