{"type":"ScholarlyArticle","title":"Online Learning in Changing Environments","authors":[{"name":"Lijun Zhang","slug":"lijun-zhang","given":"Lijun","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"The usual goal of online learning is to minimize the regret, which measures the performance of  online learner against a fixed comparator. However, it is not suitable for changing environments in which the best decision may change over time.  To address this limitation, new performance measures, including dynamic regret and adaptive regret have been proposed to guide the design of online algorithms. In dynamic regret, the  learner is compared with a sequence of comparators, and in adaptive regret, the learner is required to minimize the regret over every interval. In this paper, we will review the recent developments in this area, and highlight our contributions. Specifically, we have proposed novel algorithms to minimize the dynamic regret and adaptive regret, and investigated the relationship between them.","doi":"10.24963/ijcai.2020/731","url":"https://ijcai.org/proceedings/2020/731","pdf":"https://ijcai.org/proceedings/2020/0731.pdf","markdown":"https://ijcai.org/proceedings/2020/731.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/731","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Early Career","track":"Early Career","number":731,"pages":{"first":5178,"last":5182},"keywords":["Machine Learning: Big data; Scalability","Machine Learning: Online Learning","Machine Learning: Time-series;Data Streams"],"videos":[],"references":[{"position":1,"raw":"[Besbes et al., 2015] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations Research, 63(5):1227-1244, 2015.","title":"Non-stationary stochastic optimization","authors":"Omar Besbes, Yonatan Gur, Assaf Zeevi","year":"2015","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games. Cam- bridge University Press, 2006.","title":"Nicolò Cesa-Bianchi and Gábor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Daniely et al., 2015] Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. In Pro- ceedings of the 32nd International Conference on Machine Learning, pages 1405-1411, 2015.","title":"Strongly adaptive online learning","authors":"Amit Daniely, Alon Gonen, Shai Shalev-Shwartz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Freund et al., 1997] Yoav Freund, Robert E. Schapire, Yo- ram Singer, and Manfred K. Warmuth. Using and combi- ning predictors that specialize. In Proceedings of the 29h Annual ACM Symposium on Theory of Computing, pages 334-343, 1997.","title":"Using and combining predictors that specialize","authors":"Yoav Freund, Robert E Schapire, Yoram Singer, Manfred K Warmuth","year":"1997","venue":"Proceedings of the 29h Annual ACM Symposium on Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[György et al., 2012] András György, Tamás Linder, and Gábor Lugosi. Efficient tracking of large classes of experts. IEEE Transactions on Information Theory, 58(11):6709-6725, 2012.","title":"Efficient tracking of large classes of experts","authors":"András György, Tamás Linder, Gábor Lugosi","year":"2012","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hall and Willett, 2013] Eric C. Hall and Rebecca M. Wil- lett. Dynamical models and tracking regret in online con- vex programming. In Proceedings of the 30th Internati- onal Conference on Machine Learning, pages 579-587, 2013.","title":"Dynamical models and tracking regret in online convex programming","authors":"Eric C Hall, Rebecca M Willett","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hazan and Seshadhri, 2007] Elad Hazan and C. Seshadhri. Adaptive algorithms for online decision problems. Elec- tronic Colloquium on Computational Complexity, 88, 2007.","title":"Adaptive algorithms for online decision problems","authors":"Elad Hazan, C Seshadhri","year":"2007","venue":"Electronic Colloquium on Computational Complexity","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hazan and Seshadhri, 2009] Elad Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In Proceedings of the 26th Annual International Confe- rence on Machine Learning, pages 393-400, 2009.","title":"Efficient learning algorithms for changing environments","authors":"Elad Hazan, C Seshadhri","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hazan et al., 2007] Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex op- timization. Machine Learning, 69(2-3):169-192, 2007.","title":"Logarithmic regret algorithms for online convex optimization","authors":"Elad Hazan, Amit Agarwal, Satyen Kale","year":"2007","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hazan, 2016] Elad Hazan. Introduction to online convex optimization. Foundations and Trends in Optimization, 2(3-4):157-325, 2016.","title":"Introduction to online convex optimization","authors":"Elad Hazan","year":"2016","venue":"Foundations and Trends in Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jadbabaie et al., 2015] Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, and Karthik Sridharan. Online opti- mization: Competing with dynamic comparators. In Pro- ceedings of the 18th International Conference on Artificial Intelligence and Statistics, 2015.","title":"Online optimization: Competing with dynamic comparators","authors":"Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, Karthik Sridharan","year":"2015","venue":"Proceedings of the 18th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jun et al., 2017a] Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Improved strongly adaptive online learning using coin betting. In Proceedings of the 20th International Conference on Artificial Intelli- gence and Statistics, pages 943-951, 2017.","title":"Improved strongly adaptive online learning using coin betting","authors":"Jun","year":"2017","venue":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jun et al., 2017b] Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Online learning for changing environments using coin betting. Electronic Journal of Statistics, 11(2):5282-5310, 2017.","title":"Online learning for changing environments using coin betting","authors":"Jun","year":"2017","venue":"Electronic Journal of Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Littlestone and Warmuth, 1994] Nick Littlestone and Man- fred K. Warmuth. The weighted majority algorithm. In- formation and Computation, 108(2):212-261, 1994.","title":"Nick Littlestone and Manfred K. Warmuth. The weighted majority algorithm","authors":"Warmuth Littlestone","year":"1994","venue":"Information and Computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mokhtari et al., 2016] Aryan Mokhtari, Shahin Shahram- pour, Ali Jadbabaie, and Alejandro Ribeiro. Online opti- mization in dynamic environments: Improved regret rates for strongly convex problems. In Proceedings of the 55th IEEE Conference on Decision and Control, pages 7195- 7201, 2016.","title":"Online optimization in dynamic environments: Improved regret rates for strongly convex problems","authors":"Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, Alejandro Ribeiro","year":"2016","venue":"Proceedings of the 55th IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shalev-Shwartz, 2011] Shai Shalev-Shwartz. Online lear- ning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107-194, 2011.","title":"Online learning and online convex optimization","authors":"Shai Shalev-Shwartz","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[van Erven and Koolen, 2016] Tim van Erven and Wouter M Koolen. MetaGrad: Multiple learning rates in online lear- ning. In Advances in Neural Information Processing Sys- tems 29, pages 3666-3674, 2016.","title":"MetaGrad: Multiple learning rates in online learning","authors":"Van Erven, ; ] Tim Koolen, Van Erven, Wouter M Koolen","year":"2016","venue":"Advances in Neural Information Processing Systems 29","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Guanghui Wang, Dakuan Zhao, and Li- jun Zhang. Minimizing adaptive regret with one gradient per iteration. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2762- 2768, 2018.","title":"Minimizing adaptive regret with one gradient per iteration","authors":"Guanghui Wang, Dakuan Zhao, Lijun Zhang","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/383"},{"position":19,"raw":"[Yang et al., 2016] Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Op- timal dynamic regret of online learning with true and noisy gradient. In Proceedings of the 33rd International Confe- rence on Machine Learning, pages 449-457, 2016.","title":"Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient","authors":"Tianbao Yang, Lijun Zhang, Rong Jin, Jinfeng Yi","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2017] Lijun Zhang, Tianbao Yang, Jinfeng Yi, Rong Jin, and Zhi-Hua Zhou. Improved dynamic regret for non-degenerate functions. In Advances in Neural In- formation Processing Systems 30, pages 732-741, 2017.","title":"Improved dynamic regret for non-degenerate functions","authors":"Lijun Zhang, Tianbao Yang, Jinfeng Yi, Rong Jin, Zhi-Hua Zhou","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2018a] Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31, pages 1323-1333, 2018.","title":"Adaptive online learning in dynamic environments","authors":"Zhang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2018b] Lijun Zhang, Tianbao Yang, Rong Jin, and Zhi-Hua Zhou. Dynamic regret of strongly adaptive methods. In Proceedings of the 35th International Confe- rence on Machine Learning, 2018.","title":"Dynamic regret of strongly adaptive methods","authors":"Zhang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019a] Lijun Zhang, Tie-Yan Liu, and Zhi- Hua Zhou. Adaptive regret of convex and smooth functi- ons. In Proceedings of the 36th International Conference on Machine Learning, pages 7414-7423, 2019.","title":"Adaptive regret of convex and smooth functions","authors":"Zhang","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2019b] Lijun Zhang, Guanghui Wang, Wei- Wei Tu, and Zhi-Hua Zhou. Dual adaptivity: A univer- sal algorithm for minimizing the adaptive regret of convex functions. ArXiv e-prints, arXiv:1906.10851, 2019.","title":"Dual adaptivity: A universal algorithm for minimizing the adaptive regret of convex functions","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2020] Lijun Zhang, Shiyin Lu, and Tianbao Yang. Minimizing dynamic regret and adaptive regret si- multaneously. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2020.","title":"Minimizing dynamic regret and adaptive regret simultaneously","authors":"Lijun Zhang, Shiyin Lu, Tianbao Yang","year":"2020","venue":"Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Ma- chine Learning, pages 928-936, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"Proceedings of the 20th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}