{"type":"ScholarlyArticle","title":"Learning Embeddings for Sequential Tasks Using Population of Agents","authors":[{"name":"Mridul Mahajan","slug":"mridul-mahajan","given":"Mridul","family":"Mahajan","affiliation":"Max Planck Institute for Software Systems"},{"name":"Georgios Tzannetos","slug":"georgios-tzannetos","given":"Georgios","family":"Tzannetos","affiliation":"Max Planck Institute for Software Systems"},{"name":"Goran Radanovic","slug":"goran-radanovic","given":"Goran","family":"Radanovic","affiliation":"Max Planck Institute for Software Systems"},{"name":"Adish Singla","slug":"adish-singla","given":"Adish","family":"Singla","affiliation":"Max Planck Institute for Software Systems"}],"abstract":"We present an information-theoretic framework to learn fixed-dimensional embeddings for tasks in reinforcement learning. We leverage the idea that two tasks are similar if observing an agent's performance on one task reduces our uncertainty about its performance on the other. This intuition is captured by our information-theoretic criterion which uses a diverse agent population as an approximation for the space of agents to measure similarity between tasks in sequential decision-making settings. In addition to qualitative assessment, we empirically demonstrate the effectiveness of our techniques based on task embeddings by quantitative comparisons against strong baselines on two application scenarios: predicting an agent's performance on a new task by observing its performance on a small quiz of tasks, and selecting tasks with desired characteristics from a given set of options.","doi":"10.24963/ijcai.2024/523","url":"https://ijcai.org/proceedings/2024/523","pdf":"https://ijcai.org/proceedings/2024/0523.pdf","markdown":"https://ijcai.org/proceedings/2024/523.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/523","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":523,"pages":{"first":4733,"last":4741},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Representation learning","Machine Learning: ML: Multi-task and transfer learning"],"videos":[],"references":[{"position":1,"raw":"[Achille et al., 2019] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2Vec: Task Embedding for Meta-Learning. In ICCV, 2019.","title":"Task2Vec: Task Embedding for Meta-Learning","authors":"Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, C Charless, Stefano Fowlkes, Pietro Soatto, Perona","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arnekvist et al., 2018] Isac Arnekvist, Danica Kragic, and Johannes Andreas Stork. VPE: Variational Policy Embed- ding for Transfer Reinforcement Learning. In ICRA, 2018.","title":"VPE: Variational Policy Embedding for Transfer Reinforcement Learning","authors":"Isac Arnekvist, Danica Kragic, Johannes Andreas Stork","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Athar et al., 2020] Ali Athar, Sabarinath Mahadevan, Aljoša Ošep, Laura Leal-Taixé, and Bastian Leibe. STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos. In ECCV, 2020.","title":"STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos","authors":"Ali Athar, Sabarinath Mahadevan, Aljoša Ošep, Laura Leal-Taixé, Bastian Leibe","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bain and Sammut, 1995] Michael Bain and Claude Sammut. A Framework for Behavioural Cloning. In Machine Intelli- gence, Intelligent Agents, 1995.","title":"A Framework for Behavioural Cloning","authors":"Michael Bain, Claude Sammut","year":"1995","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bing et al., 2023] Z. Bing, D. Lerch, K. Huang, and A. Knoll. Meta-Reinforcement Learning in Non-Stationary and Dy- namic Environments. In IEEE TPAMI, 2023.","title":"Meta-Reinforcement Learning in Non-Stationary and Dynamic Environments","authors":"Z Bing, D Lerch, K Huang, A Knoll","year":"2023","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. CoRR, abs/1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bunel et al., 2018] Rudy Bunel, Matthew J. Hausknecht, Ja- cob Devlin, Rishabh Singh, and Pushmeet Kohli. Lever- aging Grammar and Reinforcement Learning for Neural Program Synthesis. In ICLR, 2018.","title":"Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis","authors":"Rudy Bunel, Matthew J Hausknecht, Jacob Devlin, Rishabh Singh, Pushmeet Kohli","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cer et al., 2018] Daniel Matthew Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Universal Sentence Encoder for English. In EMNLP, 2018.","title":"Universal Sentence Encoder for English","authors":"Daniel Matthew Cer, Yinfei Yang, Sheng Yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St, Noah John, Mario Constant, Steve Guajardo-Cespedes, Chris Yuan, Brian Tar, Ray Strope, Kurzweil","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devidze et al., 2021] Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. Expli- cable Reward Design for Reinforcement Learning Agents. In NeurIPS, 2021.","title":"Explicable Reward Design for Reinforcement Learning Agents","authors":"Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, Adish Singla","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Eimer et al., 2021] Theresa Eimer, André Biedenkapp, Frank Hutter, and Marius Lindauer. Self-Paced Context Evaluation for Contextual Reinforcement Learning. In ICML, 2021.","title":"Self-Paced Context Evaluation for Contextual Reinforcement Learning","authors":"Theresa Eimer, André Biedenkapp, Frank Hutter, Marius Lindauer","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fu et al., 2020] Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. To- wards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning. In AAAI, 2020.","title":"Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning","authors":"Fu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Furuta et al., 2021] Hiroki Furuta, Tatsuya Matsushima, Tadashi Kozuno, Yutaka Matsuo, Sergey Levine, Ofir Nachum, and Shixiang Shane Gu. Policy Information Ca- pacity: Information-Theoretic Measure for Task Complex- ity in Deep Reinforcement Learning. In ICML, 2021.","title":"Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning","authors":"Furuta","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ghosh et al., 2022] Ahana Ghosh, Sebastian Tschiatschek, Sam Devlin, and Adish Singla. Adaptive Scaffolding in Block-based programming via Synthesizing New Tasks as Pop quizzes. In AIED, 2022.","title":"Adaptive Scaffolding in Block-based programming via Synthesizing New Tasks as Pop quizzes","authors":"Ahana Ghosh, Sebastian Tschiatschek, Sam Devlin, Adish Singla","year":"2022","venue":"AIED","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gupta et al., 2018] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- Reinforcement Learning of Structured Exploration Strate- gies. In NIPS, 2018.","title":"Meta-Reinforcement Learning of Structured Exploration Strategies","authors":"Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hallak et al., 2015] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov Decision Processes. CoRR, abs/1502.02259, 2015.","title":"Contextual Markov Decision Processes","authors":"Assaf Hallak, Dotan Di Castro, Shie Mannor","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He-Yueya and Singla, 2021] Joy He-Yueya and Adish Ku- mar Singla. Quizzing Policy Using Reinforcement Learn- ing for Inferring the Student Knowledge State. In EDM, 2021.","title":"Joy He-Yueya and Adish Kumar Singla. Quizzing Policy Using Reinforcement Learning for Inferring the Student Knowledge State","authors":"He-Yueya, Singla","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jaderberg and et al., 2019] Max Jaderberg and et al. Human- level Performance in 3D Multiplayer Games with Population-based Reinforcement Learning. In Science, 2019.","title":"Max Jaderberg and et al. Humanlevel Performance in 3D Multiplayer Games with Population-based Reinforcement Learning","authors":"Jaderberg","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Klink et al., 2020a] P. Klink, C. D'Eramo, J. Peters, and J. Pajarinen. Self-Paced Deep Reinforcement Learning. In NeurIPS, 2020.","title":"Self-Paced Deep Reinforcement Learning","authors":"Klink","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Klink et al., 2020b] Pascal Klink, Carlo D' Eramo, Jan R Peters, and Joni Pajarinen. Self-Paced Deep Reinforcement Learning. In NeurIPS, 2020.","title":"Self-Paced Deep Reinforcement Learning","authors":"Klink","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lan et al., 2019] Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta Reinforcement Learning with Task Embedding and Shared Policy. In IJCAI, 2019.","title":"Meta Reinforcement Learning with Task Embedding and Shared Policy","authors":"Lin Lan, Zhenguo Li, Xiaohong Guan, Pinghui Wang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/387"},{"position":21,"raw":"[Li et al., 2021] Lanqing Li, Rui Yang, and Dijun Luo. FO- CAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization. In ICLR, 2021.","title":"FO-CAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization","authors":"Lanqing Li, Rui Yang, Dijun Luo","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Luce, 1959] R. Duncan Luce. Individual Choice Behavior: A Theoretical analysis. Wiley, 1959.","title":"Individual Choice Behavior: A Theoretical analysis","authors":"R, Duncan Luce","year":"1959","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mahajan et al., 2023] Mridul Mahajan, Georgios Tzannetos, Goran Radanovic, and Adish Singla. Learning Embeddings for Sequential Tasks Using Population of Agents. CoRR, abs/2306.03311, 2023.","title":"Learning Embeddings for Sequential Tasks Using Population of Agents","authors":"Georgios Mridul Mahajan, Goran Tzannetos, Adish Radanovic, Singla","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Repre- sentations of Words and Phrases and their Compositionality. In NIPS, 2013.","title":"Distributed Representations of Words and Phrases and their Compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Parker-Holder et al., 2020] Jack Parker-Holder, Aldo Pac- chiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective Diversity in Population Based Reinforcement Learning. In NeurIPS, 2020.","title":"Effective Diversity in Population Based Reinforcement Learning","authors":"Parker-Holder","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Peng et al., 2020] Xingchao Peng, Yichen Li, and Kate Saenko. Domain2Vec: Domain Embedding for Unsuper- vised Domain Adaptation. In ECCV, 2020.","title":"Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation","authors":"Peng","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global Vectors for Word Representation. In EMNLP, 2014.","title":"GloVe: Global Vectors for Word Representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qin et al., 2022] Rongjun Qin, F. Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-Agent Policy Transfer via Task Rela- tionship Modeling. CoRR, abs/2203.04482, 2022.","title":"Multi-Agent Policy Transfer via Task Relationship Modeling","authors":"F Rongjun Qin, Tonghan Chen, Lei Wang, Xiaoran Yuan, Zongzhang Wu, Chongjie Zhang, Yang Zhang, Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rakelly et al., 2019] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient Off- Policy Meta-Reinforcement Learning via Probabilistic Con- text Variables. In ICML, 2019.","title":"Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables","authors":"Rakelly","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Schäfer et al., 2022] Lukas Schäfer, Filippos Christianos, Amos J. Storkey, and Stefano V. Albrecht. Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Re- inforcement Learning. CoRR, abs/2207.02249, 2022.","title":"Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Reinforcement Learning","authors":"Lukas Schäfer, Filippos Christianos, Amos J Storkey, Stefano V Albrecht","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sodhani et al., 2021a] Shagun Sodhani, Ludovic Denoyer, Pierre-Alexandre Kamienny, and Olivier Delalleau. MTEnv -Environment interface for mulit-task reinforcement learn- ing. GitHub, 2021.","title":"MTEnv -Environment interface for mulit-task reinforcement learning","authors":"Sodhani","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sodhani et al., 2021b] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-Task Reinforcement Learning with Context-based Representations. In ICML, 2021.","title":"Multi-Task Reinforcement Learning with Context-based Representations","authors":"Sodhani","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sun et al., 2014] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Representation from Predicting 10,000 Classes. In CVPR, 2014.","title":"Deep Learning Face Representation from Predicting 10,000 Classes","authors":"Yi Sun, Xiaogang Wang, Xiaoou Tang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to Compare: Relation Network for Few-Shot Learning. In CVPR, 2018.","title":"Learning to Compare: Relation Network for Few-Shot Learning","authors":"Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, H S Philip, Timothy M Torr, Hospedales","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Ap- proximation. In NIPS, 1999.","title":"Policy Gradient Methods for Reinforcement Learning with Function Approximation","authors":"David Richard S Sutton, Satinder Mcallester, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tylkin et al., 2021] Paul Tylkin, Goran Radanovic, and David C. Parkes. Learning Robust Helpful Behaviors in Two-Player Cooperative Atari Environments. In AAMAS, 2021.","title":"Learning Robust Helpful Behaviors in Two-Player Cooperative Atari Environments","authors":"Paul Tylkin, Goran Radanovic, David C Parkes","year":"2021","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tzannetos et al., 2023] Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal Curriculum for Reinforcement Learning Agents. In TMLR, 2023.","title":"Proximal Curriculum for Reinforcement Learning Agents","authors":"Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, Adish Singla","year":"2023","venue":"TMLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[van der Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. In JMLR, 2008.","title":"Visualizing Data using t-SNE","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Vinyals and et al., 2019] Oriol Vinyals and et al. Grandmas- ter Level in StarCraft II Using Multi-agent Reinforcement Learning. In Nature, 2019.","title":"Oriol Vinyals and et al. Grandmaster Level in StarCraft II Using Multi-agent Reinforcement Learning","authors":"Vinyals","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Vuorio et al., 2019] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal Model-Agnostic Meta- Learning via Task-Aware Modulation. In NeurIPS, 2019.","title":"Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation","authors":"Risto Vuorio, Shao-Hua, Hexiang Sun, Joseph J Hu, Lim","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Walke et al., 2022] Homer Walke, Jonathan Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't Start From Scratch: Leveraging Prior Data to Auto- mate Robotic Reinforcement Learning. In CoRL, 2022. [Yengera et al., 2021]","title":"Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning","authors":"Walke","year":"2021","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"Gaurav Yengera, Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Curricu- lum Design for Teaching via Demonstrations: Theory and Applications. In NeurIPS, 2021.","title":"Curriculum Design for Teaching via Demonstrations: Theory and Applications","authors":"Gaurav Yengera, Rati Devidze, Parameswaran Kamalaruban, Adish Singla","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yoo et al., 2022] Minjong Yoo, Sangwoo Cho, and Honguk Woo. Skills Regularized Task Decomposition for Multi- task Offline Reinforcement Learning. In NeurIPS, 2022.","title":"Skills Regularized Task Decomposition for Multitask Offline Reinforcement Learning","authors":"Yoo","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2018] Yu Zhang, Ying Wei, and Qiang Yang. Learning to Multitask. In NeurIPS, 2018.","title":"Learning to Multitask","authors":"Yu Zhang, Ying Wei, Qiang Yang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhang et al., 2021] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn- ing Invariant Representations for Reinforcement Learning without Reconstruction. In ICLR, 2021.","title":"Learning Invariant Representations for Reinforcement Learning without Reconstruction","authors":"Amy Zhang, Thomas Rowan, Roberto Mcallister, Yarin Calandra, Sergey Gal, Levine","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}