{"type":"ScholarlyArticle","title":"On Adaptivity and Safety in Sequential Decision Making","authors":[{"name":"Sapana Chaudhary","slug":"sapana-chaudhary","given":"Sapana","family":"Chaudhary","affiliation":"Texas A&M University , College Station , TX , USA"}],"abstract":"Sequential decision making is an important field in machine learning, encompassing techniques such as online optimization, structured bandits, and reinforcement learning, which have numerous applications such as recommendation systems, online advertising, conversational agents, and robot learning. However, two key challenges face real-world sequential decision making: the need for adaptable models and the need for safety during both learning and execution. Adaptability refers to the ability of a model to quickly adapt to new and diverse environments, which is especially challenging in environments where feedback is sparse. To address this challenge, we propose using meta reinforcement learning with sub-optimal demonstration data. Safety is also critical in real-world sequential decision making. A model that adheres to safety requirements can avoid dangerous outcomes and ensure the safety of humans and other agents in the environment. We propose an approach based on online convex optimization that ensures safety at every time step. Addressing these challenges can lead to the development of more robust, safe, and adaptable AI systems that can perform a wide range of tasks and operate in a variety of environments.","doi":"10.24963/ijcai.2023/813","url":"https://ijcai.org/proceedings/2023/813","pdf":"https://ijcai.org/proceedings/2023/0813.pdf","markdown":"https://ijcai.org/proceedings/2023/813.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/813","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":813,"pages":{"first":7077,"last":7078},"keywords":["Machine Learning: ML: Optimization","Machine Learning: ML: Reinforcement learning","AI Ethics, Trust, Fairness: ETF: Safety and robustness"],"videos":[],"references":[{"position":1,"raw":"[Chang et al., 2023] Ting-Jui Chang, Sapana Chaudhary, Dileep Kalathil, and Shahin Shahrampour. Dynamic regret analysis of safe distributed online optimization for convex and non-convex problems. arXiv preprint arXiv:2302.12320, 2023.","title":"Dynamic regret analysis of safe distributed online optimization for convex and non-convex problems","authors":"Ting-Jui Chang, Sapana Chaudhary, Dileep Kalathil, Shahin Shahrampour","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chaudhary and Kalathil, 2022] Sapana Chaudhary and Dileep Kalathil. Safe online convex optimization with unknown linear safety constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6175-6182, 2022.","title":"Safe online convex optimization with unknown linear safety constraints","authors":"Kalathil Chaudhary, Sapana Chaudhary, Dileep Kalathil","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhary and Ravindran, 2022] Sapana Chaudhary and Balaraman Ravindran. Smooth imitation learning via smooth costs and smooth policies. In 5th Joint Inter- national Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), pages 63-71, 2022.","title":"Smooth imitation learning via smooth costs and smooth policies","authors":"Ravindran Chaudhary, Sapana Chaudhary, Balaraman Ravindran","year":"2022","venue":"5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dorfman et al., 2020] Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02598, 2020.","title":"Offline meta learning of exploration","authors":"Ron Dorfman, Idan Shenfeld, Aviv Tamar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Rakelly et al., 2019] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcement learning via probabilistic con- text variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.","title":"Efficient offpolicy meta-reinforcement learning via probabilistic context variables","authors":"Rakelly","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Rengarajan et al., 2022] Desik Rengarajan, Sapana Chaud- hary, Jaewon Kim, Dileep Kalathil, and Srinivas Shakkot- tai. Enhanced meta reinforcement learning using demon- strations in sparse reward environments. arXiv preprint arXiv:2209.13048, 2022.","title":"Enhanced meta reinforcement learning using demonstrations in sparse reward environments","authors":"Rengarajan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Zintgraf et al., 2019] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hof- mann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019.","title":"Varibad: A very good method for bayes-adaptive deep rl via meta-learning","authors":"Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}