{"type":"ScholarlyArticle","title":"Mitigating Disparity while Maximizing Reward: Tight Anytime Guarantee for Improving Bandits","authors":[{"name":"Vishakha Patil","slug":"vishakha-patil","given":"Vishakha","family":"Patil","affiliation":"Indian Institute of Science , Bangalore"},{"name":"Vineet Nair","slug":"vineet-nair","given":"Vineet","family":"Nair","affiliation":"Arithmic Labs"},{"name":"Ganesh Ghalme","slug":"ganesh-ghalme","given":"Ganesh","family":"Ghalme","affiliation":"Indian Institute of Technology , Hyderabad"},{"name":"Arindam Khan","slug":"arindam-khan","given":"Arindam","family":"Khan","affiliation":"Indian Institute of Science , Bangalore"}],"abstract":"We study the Improving Multi-Armed Bandit problem, where the reward obtained from an arm increases with the number of pulls it receives. This model provides an elegant abstraction for many real-world problems in domains such as education and employment, where decisions about the distribution of opportunities can affect the future capabilities of communities and the disparity between them. A decision-maker in such settings must consider the impact of her decisions on future rewards in addition to the standard objective of maximizing her cumulative reward at any time. We study the tension between two seemingly conflicting objectives in the horizon-unaware setting: a) maximizing the cumulative reward at any time and b) ensuring that arms with better long-term rewards get sufficient pulls even if they initially have low rewards. We show that, surprisingly, the two objectives are aligned with each other. Our main contribution is an anytime algorithm for the IMAB problem that achieves the best possible cumulative reward while ensuring that the arms reach their true potential given sufficient time. Our algorithm mitigates the initial disparity due to lack of opportunity and continues pulling an arm until it stops improving. We prove the optimality of our algorithm by showing that a) any algorithm for the IMAB problem, no matter how utilitarian, must suffer Omega(T) policy regret and Omega(k) competitive ratio with respect to the optimal offline policy, and b) the competitive ratio of our algorithm is O(k).","doi":"10.24963/ijcai.2023/456","url":"https://ijcai.org/proceedings/2023/456","pdf":"https://ijcai.org/proceedings/2023/0456.pdf","markdown":"https://ijcai.org/proceedings/2023/456.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/456","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":456,"pages":{"first":4100,"last":4108},"keywords":["AI Ethics, Trust, Fairness: ETF: Fairness and diversity","Machine Learning: ML: Online learning","Uncertainty in AI: UAI: Sequential decision making"],"videos":[],"references":[{"position":1,"raw":"[Andrew et al., 2013] Lachlan Andrew, Siddharth Barman, Katrina Ligett, Minghong Lin, Adam Meyerson, Alan Roytman, and Adam Wierman. A tale of two metrics: Simultaneous bounds on competitiveness and regret. In Conference on Learning Theory, pages 741-763. PMLR, 2013.","title":"A tale of two metrics: Simultaneous bounds on competitiveness and regret","authors":"Lachlan Andrew, Siddharth Barman, Katrina Ligett, Minghong Lin, Adam Meyerson, Alan Roytman, Adam Wierman","year":"2013","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2012] Raman Arora, Ofer Dekel, and Ambuj Tewari. Online bandit learning against an adaptive ad- versary: from regret to policy regret. arXiv preprint arXiv:1206.6400, 2012.","title":"Online bandit learning against an adaptive adversary: from regret to policy regret","authors":"Raman Arora, Ofer Dekel, Ambuj Tewari","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002a] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002b] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Auer","year":"2002","venue":"SIAM journal on computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barocas et al., 2019] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning. http: //www.fairmlbook.org, 2019. Accessed: 2023-05-29.","title":"Fairness and machine learning","authors":"Solon Barocas, Moritz Hardt, Arvind Narayanan","year":"2019","venue":null,"doi":null,"url":"http://www.fairmlbook.org","internal_url":null},{"position":6,"raw":"[Besbes et al., 2014] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non- stationary rewards. Advances in neural information pro- cessing systems, 27, 2014.","title":"Stochastic multi-armed-bandit problem with nonstationary rewards","authors":"Omar Besbes, Yonatan Gur, Assaf Zeevi","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Borodin and El-Yaniv, 2005] Allan Borodin and Ran El- Yaniv. Online computation and competitive analysis. cam- bridge university press, 2005.","title":"Online computation and competitive analysis","authors":"El-Yaniv ; Borodin, Allan Borodin, Ran El-Yaniv","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Buchbinder et al., 2012] Niv Buchbinder, Shahar Chen, Joshep Seffi Naor, and Ohad Shamir. Unified algorithms for online learning and competitive analysis. In Confer- ence on Learning Theory, pages 5-1. JMLR Workshop and Conference Proceedings, 2012.","title":"Unified algorithms for online learning and competitive analysis","authors":"Buchbinder","year":"2012","venue":"JMLR Workshop and Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daniely and Mansour, 2019] Amit Daniely and Yishay Mansour. Competitive ratio vs regret minimization: achieving the best of both worlds. In Algorithmic Learning Theory, pages 333-368. PMLR, 2019.","title":"Competitive ratio vs regret minimization: achieving the best of both worlds","authors":"Amit Daniely, Yishay Mansour","year":"2019","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dwork et al., 2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214- 226, 2012.","title":"Fairness through awareness","authors":"Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, Richard Zemel","year":"2012","venue":"Proceedings of the 3rd innovations in theoretical computer science conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garivier and Moulines, 2011] Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switch- ing bandit problems. In International Conference on Algo- rithmic Learning Theory, pages 174-188. Springer, 2011.","title":"On upper-confidence bound policies for switching bandit problems","authors":"Moulines Garivier, Aurélien Garivier, Eric Moulines","year":"2011","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ghalme et al., 2022] Ganesh Ghalme, Vineet Nair, Vishakha Patil, and Yilun Zhou. Long-term resource allocation fairness in average markov decision process (amdp) environment. In Proceedings of the 21st Interna- tional Conference on Autonomous Agents and Multiagent Systems, pages 525-533, 2022.","title":"Long-term resource allocation fairness in average markov decision process (amdp) environment","authors":"Ganesh Ghalme, Vineet Nair, Vishakha Patil, Yilun Zhou","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hardt et al., 2016] Moritz Hardt, Eric Price, and Nati Sre- bro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.","title":"Equality of opportunity in supervised learning","authors":"Moritz Hardt, Eric Price, Nati Srebro","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Heidari et al., 2016] Hoda Heidari, Michael J Kearns, and Aaron Roth. Tight policy regret bounds for improving and decaying bandits. In IJCAI, pages 1562-1570, 2016.","title":"Tight policy regret bounds for improving and decaying bandits","authors":"Hoda Heidari, Michael J Kearns, Aaron Roth","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hossain et al., 2021] Safwan Hossain, Evi Micha, and Nis- arg Shah. Fair algorithms for multi-agent multi-armed bandits. Advances in Neural Information Processing Sys- tems, 34, 2021.","title":"Fair algorithms for multi-agent multi-armed bandits","authors":"Safwan Hossain, Evi Micha, Nisarg Shah","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Immorlica et al., 2019] Nicole Immorlica, Karthik Abinav Sankararaman, Robert Schapire, and Aleksandrs Slivkins. Adversarial bandits with knapsacks. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 202-219. IEEE, 2019.","title":"Adversarial bandits with knapsacks","authors":"Nicole Immorlica, Karthik Abinav Sankararaman, Robert Schapire, Aleksandrs Slivkins","year":"2019","venue":"IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[J. Baker, 2019] Dominique J. Baker. Why might states ban affirmative action? https://www.brookings.edu/blog/ brown-center-chalkboard/2019/04/12/why-might-states- ban-affirmative-action/, 2019. Accessed: 2023-05-29.","title":"Why might states ban affirmative action?","authors":"Dominique J Baker","year":"2023","venue":null,"doi":null,"url":"https://www.brookings.edu/blog/brown-center-chalkboard/2019/04/12/why-might-states-ban-affirmative-action/,2019","internal_url":null},{"position":18,"raw":"[Jabbari et al., 2017] Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in reinforcement learning. In International con- ference on machine learning, pages 1617-1626. PMLR, 2017.","title":"Fairness in reinforcement learning","authors":"Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Aaron Roth","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Joseph et al., 2016] Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in learn- ing: Classic and contextual bandits. arXiv preprint arXiv:1605.07139, 2016.","title":"Fairness in learning: Classic and contextual bandits","authors":"Matthew Joseph, Michael Kearns, Jamie Morgenstern, Aaron Roth","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jovanovic and Nyarko, 1995] Boyan Jovanovic and Yaw Nyarko. A bayesian learning model fitted to a variety of empirical learning curves. Brookings Papers on Economic Activity. Microeconomics, 1995:247-305, 1995.","title":"A bayesian learning model fitted to a variety of empirical learning curves","authors":"Boyan Jovanovic, Yaw Nyarko","year":"1995","venue":"Brookings Papers on Economic Activity. Microeconomics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kesselheim and Singla, 2020] Thomas Kesselheim and Sahil Singla. Online learning with vector costs and bandits with knapsacks. In Conference on Learning Theory, pages 2286-2305. PMLR, 2020.","title":"Online learning with vector costs and bandits with knapsacks","authors":"Singla Kesselheim, Thomas Kesselheim, Sahil Singla","year":"2020","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kleinberg et al., 2017] Jon M. Kleinberg, Sendhil Mul- lainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. In Christos H. Papadim- itriou, editor, 8th Innovations in Theoretical Computer Sci- ence Conference, ITCS 2017, January 9-11, 2017, Berke- ley, CA, USA, volume 67 of LIPIcs, pages 43:1-43:23. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2017.","title":"Inherent trade-offs in the fair determination of risk scores","authors":"Jon M Kleinberg, Sendhil Mullainathan, Manish Raghavan","year":"2017","venue":"8th Innovations in Theoretical Computer Science Conference, ITCS 2017","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Levine et al., 2017] Nir Levine, Koby Crammer, and Shie Mannor. Rotting bandits. Advances in neural information processing systems, 30, 2017.","title":"Rotting bandits","authors":"Nir Levine, Koby Crammer, Shie Mannor","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2019] Fengjiao Li, Jia Liu, and Bo Ji. Combinato- rial sleeping bandits with fairness constraints. IEEE Trans- actions on Network Science and Engineering, 7(3):1799- 1813, 2019.","title":"Combinatorial sleeping bandits with fairness constraints","authors":"Fengjiao Li, Jia Liu, Bo Ji","year":"2019","venue":"IEEE Transactions on Network Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lindner et al., 2021] David Lindner, Hoda Heidari, and An- dreas Krause. Addressing the long-term impact of ml de- cisions via policy regret. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 537-544. International Joint Conferences on Artificial Intelligence, 2021.","title":"Addressing the long-term impact of ml decisions via policy regret","authors":"David Lindner, Hoda Heidari, Andreas Krause","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21","doi":null,"url":null,"internal_url":"/proceedings/2021/75"},{"position":26,"raw":"[Metelli et al., 2022] Alberto Maria Metelli, Francesco Trovo, Matteo Pirola, and Marcello Restelli. Stochastic rising bandits. In International Conference on Machine Learning, pages 15421-15457. PMLR, 2022.","title":"Stochastic rising bandits","authors":"Alberto Maria Metelli, Francesco Trovo, Matteo Pirola, Marcello Restelli","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Patil et al., 2021] Vishakha Patil, Ganesh Ghalme, Vineet Nair, and Y. Narahari. Achieving fairness in the stochas- tic multi-armed bandit problem. J. Mach. Learn. Res., 22:174:1-174:31, 2021.","title":"Achieving fairness in the stochastic multi-armed bandit problem","authors":"Vishakha Patil, Ganesh Ghalme, Vineet Nair, Y Narahari","year":"2021","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the se- quential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sahoo, 2009] Niranjan Sahoo. Reservation policy and its implementation across domains in india. Academic Foun- dation in association with Observer Research Foundation, 2009.","title":"Reservation policy and its implementation across domains in india","authors":"Niranjan Sahoo","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Son and Sethi, 2006] Lisa K Son and Rajiv Sethi. Metacog- nitive control and optimal learning. Cognitive Science, 30(4):759-774, 2006.","title":"Metacognitive control and optimal learning","authors":"K Lisa, Rajiv Son, Sethi","year":"2006","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Staff, 2018] GlobalIsAsian Staff. Meritocracy in singapore: Solution or problem? https://lkyspp.nus.edu.sg/gia/article/ meritocracy-in-singapore-solution-or-problem, 2018. Ac- cessed: 2023-05-29.","title":"GlobalIsAsian Staff. Meritocracy in singapore: Solution or problem?","authors":null,"year":"2018","venue":null,"doi":null,"url":"-cessed:2023-05-29","internal_url":null},{"position":32,"raw":"[Tekin and Liu, 2012] Cem Tekin and Mingyan Liu. Online learning of rested and restless bandits. IEEE Transactions on Information Theory, 58(8):5588-5611, 2012.","title":"Online learning of rested and restless bandits","authors":"Liu Tekin, Cem Tekin, Mingyan Liu","year":"2012","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[V. Reeves and Halikias, 2017] Richard V. Reeves and Dimitrios Halikias. Race gaps in sat scores high- light inequality and hinder upward mobility. https: //www.brookings.edu/research/race-gaps-in-sat-scores- highlight-inequality-and-hinder-upward-mobility/, 2017. Accessed: 2023-05-29.","title":"Race gaps in sat scores highlight inequality and hinder upward mobility","authors":"Richard V Reeves, Dimitrios Halikias","year":"2023","venue":null,"doi":null,"url":"https://www.brookings.edu/research/race-gaps-in-sat-scores-highlight-inequality-and-hinder-upward-mobility/,2017","internal_url":null},{"position":34,"raw":"[Wang et al., 2021] Lequn Wang, Yiwei Bai, Wen Sun, and Thorsten Joachims. Fairness of exposure in stochastic bandits. In Marina Meila and Tong Zhang, editors, Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10686-10696. PMLR, 18-24 Jul 2021.","title":"Fairness of exposure in stochastic bandits","authors":"Wang","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}