{"type":"ScholarlyArticle","title":"PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representations","authors":[{"name":"Tong Sang","slug":"tong-sang","given":"Tong","family":"Sang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Hongyao Tang","slug":"hongyao-tang","given":"Hongyao","family":"Tang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yi Ma","slug":"yi-ma","given":"Yi","family":"Ma","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yan Zheng","slug":"yan-zheng","given":"Yan","family":"Zheng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Zhaopeng Meng","slug":"zhaopeng-meng","given":"Zhaopeng","family":"Meng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Boyan Li","slug":"boyan-li","given":"Boyan","family":"Li","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Zhen Wang","slug":"zhen-wang","given":"Zhen","family":"Wang","affiliation":"School of Artificial Intelligence, Optics and Electronics (iOPEN) and School of Cybersecurity , Northwestern Polytechnical University {2019218044 ,"}],"abstract":"Deep Reinforcement Learning (DRL) has been a promising solution to many complex decision-making problems. Nevertheless, the notorious weakness in generalization among environments prevent widespread application of DRL agents in real-world scenarios. Although advances have been made recently, most prior works assume sufficient online interaction on training environments, which can be costly in practical cases.\r\nTo this end, we focus on an offline-training-online-adaptation setting,\r\nin which the agent first learns from offline experiences collected in environments with different dynamics and then performs online policy adaptation in environments with new dynamics. In this paper, we propose Policy Adaptation with Decoupled Representations (PAnDR) for fast policy adaptation.\r\nIn offline training phase, the environment representation and policy representation are learned through contrastive learning and policy recovery, respectively. The representations are further refined by mutual information optimization to make them more decoupled and complete. With learned representations, a Policy-Dynamics Value Function (PDVF)  network is trained to approximate the values for different combinations of policies and environments from offline experiences. In online adaptation phase, with the environment context inferred from few experiences collected in new environments, the policy is optimized by gradient ascent with respect to the PDVF. Our experiments show that PAnDR outperforms existing algorithms in several representative policy adaptation problems.","doi":"10.24963/ijcai.2022/474","url":"https://ijcai.org/proceedings/2022/474","pdf":"https://ijcai.org/proceedings/2022/0474.pdf","markdown":"https://ijcai.org/proceedings/2022/474.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/474","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":474,"pages":{"first":3416,"last":3422},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986863","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985006","length_sec":341,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alemi et al., 2017] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR, 2017.","title":"Deep variational information bottleneck","authors":"A A Alemi, I Fischer, J V Dillon, K Murphy","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andrychowicz et al., 2020] M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. IJRR, 39(1):3- 20, 2020.","title":"Learning dexterous in-hand manipulation","authors":"M Andrychowicz, B Baker, M Chociej, R Józefowicz, B Mcgrew, J W Pachocki, A Petron, M Plappert, G Powell, A Ray, J Schneider, S Sidor, J Tobin, P Welinder, L Weng, W Zaremba","year":"2020","venue":"IJRR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Badia et al., 2020] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. Daniel Guo, and C. Blun- dell. Agent57: Outperforming the atari human benchmark. In ICML, volume 119, pages 507-517, 2020.","title":"Agent57: Outperforming the atari human benchmark","authors":"A P Badia, B Piot, S Kapturowski, P Sprechmann, A Vitvitskyi, Z Daniel, C Guo, Blundell","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2020] P. Cheng, W. Hao, S. Dai, J. Liu, Z. Gan, and L. Carin. Club: A contrastive log-ratio upper bound of mutual information. In ICML, pages 1779-1788, 2020.","title":"Club: A contrastive log-ratio upper bound of mutual information","authors":"P Cheng, W Hao, S Dai, J Liu, Z Gan, L Carin","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2016] Y. Duan, J. Schulman, X. Chen, P. Bartlett, I. Sutskever, and P. Abbeel. Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611.02779, 2016.","title":"Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning","authors":"Y Duan, J Schulman, X Chen, P Bartlett, I Sutskever, P Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Faccio et al., 2021] F. Faccio, L. Kirsch, and J. Schmidhu- ber. Parameter-based value functions. In ICLR, 2021.","title":"Parameter-based value functions","authors":"F Faccio, L Kirsch, J Schmidhuber","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fakoor et al., 2020] R. Fakoor, P. Chaudhari, S. Soatto, and A. J. Smola. Meta-q-learning. In ICLR, 2020.","title":"Meta-q-learning","authors":"R Fakoor, P Chaudhari, S Soatto, A J Smola","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finn et al., 2017] C. Finn, P. Abbeel, and S. Levine. Model- agnostic meta-learning for fast adaptation of deep net- works. In ICML, 2017.","title":"Modelagnostic meta-learning for fast adaptation of deep networks","authors":"C Finn, P Abbeel, S Levine","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu et al., 2021] H. Fu, H. Tang, J. Hao, C. Chen, X. Feng, D. Li, and W. Liu. Towards effective context for meta- reinforcement learning: an approach based on contrastive learning. In AAAI, pages 7457-7465, 2021.","title":"Towards effective context for metareinforcement learning: an approach based on contrastive learning","authors":"H Fu, H Tang, J Hao, C Chen, X Feng, D Li, W Liu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grover et al., 2018] A. Grover, M. Al-Shedivat, J. K. Gupta, Y. Burda, and H. Edwards. Learning policy representations in multiagent systems. In ICML, volume 80, pages 1797- 1806, 2018.","title":"Learning policy representations in multiagent systems","authors":"A Grover, M Al-Shedivat, J K Gupta, Y Burda, H Edwards","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2020] Z. Guo, B. Pires, B. Piot, J. Grill, F. Altché, R. Munos, and M. Azar. Bootstrap latent- predictive representations for multitask reinforcement learning. In ICML, volume 119, pages 3875-3886, 2020.","title":"Bootstrap latentpredictive representations for multitask reinforcement learning","authors":"Z Guo, B Pires, B Piot, J Grill, F Altché, R Munos, M Azar","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Harb et al., 2020] J. Harb, T. Schauland D. Precup, and P. Bacon. Policy evaluation networks. CoRR, abs/2002.11833, 2020.","title":"Policy evaluation networks","authors":"J Harb, T Schauland, D Precup, P Bacon","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2020] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Gir- shick. Momentum contrast for unsupervised visual repre- sentation learning. In CVPR, pages 9726-9735, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"K He, H Fan, Y Wu, S Xie, R B Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kirk et al., 2021] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel. A survey of generalisation in deep rein- forcement learning. CoRR, abs/2111.09794, 2021.","title":"A survey of generalisation in deep reinforcement learning","authors":"R Kirk, A Zhang, E Grefenstette, T Rocktäschel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Laskin et al., 2020] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for rein- forcement learning. In ICML, pages 5639-5650, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"M Laskin, A Srinivas, P Abbeel","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2020] K. Lee, Y. Seo, S. Lee, H. Lee, and J. Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In ICML, pages 5757-5766, 2020.","title":"Context-aware dynamics model for generalization in model-based reinforcement learning","authors":"K Lee, Y Seo, S Lee, H Lee, J Shin","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2021] Y. Ma, X.n Hao, J. Hao, J. Lu, X. Liu, X. Tong, M. Yuan, Z. Li, J. Tang, and Z. Meng. A hier- archical reinforcement learning based optimization frame- work for large-scale dynamic pickup and delivery prob- lems. In NeurIPS, pages 23609-23620, 2021.","title":"A hierarchical reinforcement learning based optimization framework for large-scale dynamic pickup and delivery problems","authors":"Y Ma, X Hao, J Hao, J Lu, X Liu, X Tong, M Yuan, Z Li, J Tang, Z Meng","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Notin et al., 2021] P. Notin, J. M. Hernández-Lobato, and Y. Gal. Improving black-box optimization in VAE latent space using decoder uncertainty. CoRR, abs/2107.00096, 2021.","title":"Improving black-box optimization in VAE latent space using decoder uncertainty","authors":"P Notin, J M Hernández-Lobato, Y Gal","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Packer et al., 2018] C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song. Assessing generalization in deep reinforcement learning. CoRR, abs/1810.12282, 2018.","title":"Assessing generalization in deep reinforcement learning","authors":"C Packer, K Gao, J Kos, P Krähenbühl, V Koltun, D Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2018] X. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, pages 1-8, 2018.","title":"Sim-to-real transfer of robotic control with dynamics randomization","authors":"Peng","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Poole et al., 2019] B. Poole, S. Ozair, A. Oord, A. Alemi, and G. Tucker. On variational bounds of mutual informa- tion. In ICML, volume 97, pages 5171-5180, 2019.","title":"On variational bounds of mutual information","authors":"B Poole, S Ozair, A Oord, A Alemi, G Tucker","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raileanu et al., 2020] R. Raileanu, M. Goldstein, A. Szlam, and R. Fergus. Fast adaptation to new environments via policy-dynamics value functions. In ICML, pages 7920- 7931, 2020.","title":"Fast adaptation to new environments via policy-dynamics value functions","authors":"R Raileanu, M Goldstein, A Szlam, R Fergus","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rakelly et al., 2019] K. Rakelly, A. Zhou, C.Finn, S. Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In ICML, volume 97, pages 5331-5340, 2019.","title":"Efficient off-policy metareinforcement learning via probabilistic context variables","authors":"K Rakelly, A Zhou, C Finn, S Levine, D Quillen","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Re- inforcement learning -an introduction. Adaptive compu- tation and machine learning. MIT Press, 1998.","title":"Reinforcement learning -an introduction. Adaptive computation and machine learning","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tang et al., 2020] H. Tang, Z. Meng, J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, and L. Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. CoRR, abs/2010.09536, 2020.","title":"What about inputting policy in value function: Policy representation and policy-extended value function approximator","authors":"H Tang, Z Meng, J Hao, C Chen, D Graves, D Li, C Yu, H Mao, W Liu, Y Yang, L Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yarats et al., 2021] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2021.","title":"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels","authors":"D Yarats, I Kostrikov, R Fergus","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2019] Y. Zheng, C. Fan, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng, Y. Liu, R. Shen, and Y. Chen. Wuji: Automatic online combat game testing using evolution- ary deep reinforcement learning. In ASE, pages 772-784, 2019.","title":"Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning","authors":"Y Zheng, C Fan, X Xie, T Su, L Ma, J Hao, Z Meng, Y Liu, R Shen, Y Chen","year":"2019","venue":"ASE","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2021] Y. Zheng, Y. Liu, X. Xie, Y. Liu, L. Ma, J. Hao, and Y. Liu. Automatic web testing using curiosity- driven reinforcement learning. In ICSE, pages 423-435, 2021.","title":"Automatic web testing using curiositydriven reinforcement learning","authors":"Y Zheng, Y Liu, X Xie, Y Liu, L Ma, J Hao, Y Liu","year":"2021","venue":"ICSE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}